Don’t deny your responsibility. Attempts to redirect your meaning (“You’re misinterpreting what I said!” “I was only kidding!” “Can’t you take a joke?”) will sound like you’re shifting the blame on to him and will stimulate more defensiveness. Be accountable for your behavior.