Doing magic in front of someone who doesn’t believe in it is loads harder. Worse, if their disbelief trumps either your certainty or your mana, and the spell doesn’t come off, you’ll probably have trouble the next time you try and cast it, whether the unbeliever’s still there or not. Do that a few more times and you’ll stop being able to do magic at all.
