If you’re inclined to cancel because you feel terrible every time you hang out with this person or these people, that’s very good information to have! In that case, don’t just consider canceling; consider whether this relationship is actually serving you and worth your time in the first place.
