Join the Dark Side, Our Trains Run On Time; or It's Bonfire Night, Let's Talk Extremism

I'm going to hazard a guess that if Voldemort appeared in your home town with a signup sheet, you probably wouldn't reach for your pen. In fact, I'd bet the majority of you would hear his first few agenda items and slam the door in his noseless face. 
If that sounds like you, you're clearly not the average citizen of Fantasyland. Typical Dark Lords seem able to gather an army in no time at all-- and not just foot soldiers who got bullied into the job, but genuine supporters. Never mind that there's nothing currently wrong with their lives or their country, they're going to help the Dark Lord burn it down anyway. Because evil. Or because insanity (which I find to be a huge authorial cop-out). As far as motives go, I find this unrealistic and a bit jarring. 
In real life, fascism doesn't appear in a vacuum, fueled by the sheer force of evil. People who are reasonably content with their lives, or see an outlet for addressing their problems, are unlikely to want to trash the status quo. Even a quick perusal of the rise of historical dictators and fascist movements around the world shows such movements build momentum based on deep-seated social discontent. In a world where the economy is unraveling, or disease is spreading wildly, or the environment is changing, and people feel they've lost control, the easy answer is to point to an easy target and direct everyone's frustration into enraged action. 
Essentially, such an extremist movement needs two main 'talking points' to gain any kind of traction. First, there needs to be an enemy, who are the purported source of the problem. This talking point is all about rage (frustration with what is currently broken) and fear (that the enemy will make the problem worse). Second, there needs to be a game plan for making things better. For maximum effect, this isn't just about exterminating the enemy. It's a grand vision for the future, for meeting peoples' needs and wants.

In the context of speculative fiction and worldbuilding, that means you the author need to establish two key features of your setting:
1) A systemic inequality or source of public outrage, social problems, etc. This should be a complex problem, otherwise the extremists have a point. The agitators will either be at the receiving end of the injustice and feel they have identified a singular cause to be eliminated, or perceive themselves to be the recipients of injustice.
2) Some cool temptation with which to lure in new recruits. In Susan Cooper's The Dark is Rising series, the Dark offers instant gratification; other temptations include elevated social status in the promised new order, cool forbidden powers, or riches stolen from the enemy. Whatever this is, it has to have strong appeal.

If you make your extremist movement part of your setting, rather than plopping it in willy-nilly, it will serve as a much more formidable force for your protagonists to defeat.