Instead of taking him on directly, you hurt something he values, something close to home. You make him understand that every time he bothers you he can expect damage, even if on a smaller scale. The only way to make you stop attacking him in your irritating fashion is for him to stop attacking you. You are like a wasp on his skin: most people leave wasps alone.