Friendship is a reciprocal arrangement. You are not morally obliged to support someone who is making the world a worse place. Quite the opposite. You should choose people who want things to be better, not worse. It’s a good thing, not a selfish thing, to choose people who are good for you. It’s appropriate and praiseworthy to associate with people whose lives would be improved if they saw your life improve.
