You can’t build a personal brand if you try to be all things for all people. When you’re afraid of alienating a group of subscribers, or a particular target audience, then you can never attract your ideal audience. The more you alienate people, the more you attract your own kind of people too.
