Yelling at someone you love, or saying something cutting or mean out of hurt or anger, that’s worth an apology. Texting while driving and hitting a biker—that’s worth an apology. Saying no to an invitation or an offer? I don’t see anything wrong with that. You are simply stating your preference.