You aren’t alone. Every single pan/bi/poly person I know has experienced exactly what you’re going through. People seem to think they have a right to define other people’s sexualities for them.
Your feelings are valid, your sexuality is valid, and your friend needs to learn to respect you, or else they aren’t a very good friend.