Maybe Nate Silver Was Wrong

By James Kwak

I think Nate Silver does a good job aggregating polls to make meaningful quantitative predictions about upcoming elections. But as he said himself shortly before the election, if the polls he relies on are systematically biased, then his forecasts are going to be off.* Many people have noted that Silver (and other quantitative poll aggregators like Sam Wang) correctly predicted an Obama victory and the outcomes in most if not all states.

But the fact remains that Obama did modestly better than the polls, and hence the poll aggregators, expected (not to mention than the Romney campaign expected). We shouldn’t read too much into this, as even where Obama significantly overperformed—like in Iowa, where Silver forecast a 3.2 percentage point victory and the actual came in at 5.7 points—the results were within the confidence intervals. But it’s also possible that the polls really were systematically biased, only they were biased against Obama—not against Romney, as conservative pundits were claiming in the last days.

Why would that be? One possibility is turnout. Many polls incorporate a likely voter model, which weights the sample to try to approximate the expected composition of the electorate. Gallup’s problem, I believe I read somewhere, was that they expected the electorate to be whiter than it turned out to be. In retrospect, we have anecdotal evidence that the electorate was younger and less white than many people (such as Paul Ryan) expected. And one common explanation is that this was due to the strength of the Obama campaign’s get-out-the-vote operation.

One piece of evidence for this theory is that Obama’s performance relative to expectations was especially good in the swing states, where you would expect him to have devoted most of his GOTV efforts.** Of the nine major swing states (in order of competitiveness, according to Silver, Florida, North Carolina, Virginia, Colorado, Iowa, New Hampshire, Ohio, Nevada, and Wisconsin), Obama beat Silver’s poll-based forecast in seven; on average, including the two states where he underperformed, he beat the polls by 1.1 percentage points. Of the other forty-one states and the District of Columbia, by contrast, Obama overperformed in only twenty-three, or just over half, and on average he beat the polls by only 0.4 percentage points.

Now, this is not something that Nate Silver was supposed to predict. Just before the election, his forecast is based almost entirely on the polls. And he freezes his model several months before the election, precisely because he doesn’t want it to be influenced by subjective judgments.

But this is exactly the kind of thing that journalists (and their subspecies known as pundits) are supposed to predict. That Obama would have the best turnout operation ever is not something that Nate Silver could predict in January. But all those people who don’t believe in polls, who think that old-fashioned beat reporting and gut instinct are the way to predict elections, could have done the work to figure out that Obama had the best turnout operation ever. Based on that research, the pundits and the political experts could then have said, “I expect Obama will do better than the polls, because the current generation of likely voter models does not take into account the strength of Obama’s turnout operation.” And that would have added value to what Silver was doing.

In other words, this was an election where old-fashioned reporting and punditry could have provided some insight into the outcome. But they didn’t, because the pundits were too busy spinning false stories about momentum (which were provably false, since momentum does show up in polls) instead of looking for relevant facts.

In theory, poll aggregation should not be the last word in election forecasting; there should be a place for political expertise. But with the “experts” we’ve got now, it is the last word.

* Does anyone know why all posts from the first six days of November have vanished from Silver’s blog?

** You would also expect him to have devoted most of his other efforts in those states, but those activities, such as TV advertising, should have shown up in the polls before Election Day.