In Defense of Nate Silver-Updated

Some people seem to be sharpening the knives to go after Nate Silver if Obama loses. I’d like to register an early dissent.

There are valid criticisms of black box statistical modeling. On the one hand, we’re asked to view the results credibly without knowing what the special sauce used to bake them is. You could mount that criticism of Silver as much as you could of any other modeler. Or pollster, for that matter. So at the end of the day you have to either make your peace with the black boxes, or write them off as value-less.

I happen to find some value in them. They aren’t predictive–but I’d argue they’re not really meant to be. They’re simply informative–just more data points from which we cobble together our understanding of a system (an election) which is so multi-variate that, as Scott Fitzgerald once wrote about Hollywood, is so complex that no more than a handful of men can keep the entire equation in their heads.

What’s more, Silver’s a very agile writer. Like Michael Lewis he has a gift for explaining complicated numerical concepts. (I would not agree with the charge that Silver often makes simple mathematical concepts sound grandiose and complex.) And finally, Silver hedges. Always and everywhere. Some people might take this to be weasely on his part, but it strikes me as just the opposite: It’s humility. Silver is in the numbers business, but he understands that the numbers don’t tell us everything. So you’ll never hear him say, “X has happened so Y must happen.” Just the opposite, actually. Silver understands the limits of his own models. He acknowledges those limits nearly every time he writes. I think this ought to be applauded.

If Romney wins should that discredit Silver’s models? Only so far as anybody ever used them as oracular constructs instead of analytical tools.

One final word: People seem to think that it would reflect badly on Silver if Romney were to win while Silver’s model shows only a 25 percent chance of victory. But isn’t 25 percent kind of a lot? If I told you there was a 1-in-4 chance of you getting hit by a bus tomorrow, would you think that 25 percent seemed like a big number or a little number? Or, to put it another way, a .250 hitter gets on base once a game, so you’d never look at him in any given at bat and think there was no chance he’d get a hit.

Ultimately I’d suggest that the real test for Nate Silver is the same as it is for any analyst, on any subject. Not “did he predict an outcome correctly” (or “did he predict the outcome I prefer”) but “does his work add value to our understanding of the subject.”

Speaking only for myself, the answer to that question is an unqualified yes.

Update: On Twitter, Jim Henley (@UOJim) pointed to a meditation he wrote on probability. It’s written from the perspective of a D&D nerd with cancer and it’s very much worth reading on its own, apart from its tangential bearing on our larger discussion here.

Also, Galley Friend A.W. offers the following dissent:

I like Silver quite a lot.  But I have one major quibble: His numerical specificity occludes the enormous subjectivity inherent in his weighing and discounting of various polls.  (See http://fivethirtyeight.blogs.nytimes.com/methodology/)

He’s selling a false certainty.  It’s the Washington equivalent of Wall Street’s now-infamous “value at risk” (VaR) models at the center of recent Wall Street meltdowns.  (E.g., http://www.futuresmag.com/2010/12/01/var-the-number-that-killed-us)

In end, my problem with Silver’s presentation is the same as Naked Capitalism’s indictment of VaR:

“But VaR is a particularly troubling example, more so because it is sufficiently, dangerously simple minded enough that regulators and managers a step or two removed from markets have become overly attached to its deceptive simplicity.”  (http://www.nakedcapitalism.com/2012/05/jp-morgan-loss-bomb-confirms-that-its-time-to-kill-var.html)

Obviously, in scrutinizing and combining polls, such relative judgments are unavoidable.  But Silver ought to be more transparent and up-front in presenting how those subjective judgments affect his bottom-line numbers.