The Signal and the Noise: Why So Many Predictions Fail-but Some Don't

by Nate Silver

The story the data tells us is often the one we’d like to hear, and we usually make sure that it has a happy ending.

The numbers have no way of speaking for themselves. We speak for them. We imbue them with meaning.

We are wired to detect patterns and respond to opportunities and threats without much hesitation.

The problem, Poggio says, is that these evolutionary instincts sometimes lead us to see patterns when there are none there.

We think we want information when we really want knowledge.

Human beings have an extraordinary capacity to ignore risks that threaten their livelihood, as though this will make them go away.

I wouldn’t question anyone who says the normal laws of probability don’t apply when it comes to the Red Sox or the Chicago Cubs.

Ultimately, the right attitude is that you should make the best forecast possible today—regardless of what you said last week, last month, or last year.

“When the facts change, I change my mind,” the economist John Maynard Keynes famously said. “What do you do, sir?”

Whether information comes in a quantitative or qualitative flavor is not as important as how you use it.

But statheads can have their biases too. One of the most pernicious ones is to assume that if something cannot easily be quantified, it does not matter.

Many times, in fact, it is possible to translate qualitative information into quantitative information.*

Good innovators typically think very big and they think very small.

New ideas are sometimes found in the most granular details of a problem where few others bother to look. And they are sometimes found when you are doing your most abstract and philosophical thinking, considering why the world is the way that it is and whether there might be an alternative to the dominant paradigm.

What could go wrong? Chaos theory. You may have heard the expression: the flap of a butterfly’s wings in Brazil can set off a tornado in Texas. It comes from the title of a paper19 delivered in 1972 by MIT’s Edward Lorenz, who began his career as a meteorologist.

What is it, exactly, that humans can do better than computers that can crunch numbers at seventy-seven teraFLOPS? They can see.

The forecasters know the flaws in the computer models.

The unique resource that these forecasters were contributing was their eyesight.

Distort a series of letters just slightly—as with the CAPTCHA technology that is often used in spam or password protection—and very “smart” computers get very confused. They are too literal-minded, unable to recognize the pattern once its subjected to even the slightest degree of manipulation. Humans by contrast, out of pure evolutionary necessity, have very powerful visual cortexes. They rapidly parse through any distortions in the data in order to identify abstract qualities like pattern and organization—qualities that happen to be very important in different types of weather systems.

The best forecasters, Hoke explained, need to think visually and abstractly while at the same time being able to sort through the abundance of information the computer provides them with.

“With four parameters I can fit an elephant,” the mathematician John von Neumann once said of this problem.59 “And with five I can make him wiggle his trunk.”

Overfitting represents a double whammy: it makes our model look better on paper but perform worse in the real world.

Michael Babyak, who has written extensively on this problem,60 puts the dilemma this way: “In science, we seek to balance curiosity with skepticism.”

An oft-told joke: a statistician drowned crossing a river that was only three feet deep on average.

During recessions, the economy can fall into a vicious cycle: businesses won’t hire until they see more consumer demand, but consumer demand is low because businesses aren’t hiring and consumers can’t afford their products.

If you’re looking for an economic forecast, the best place to turn is the average or aggregate prediction rather than that of any one economist.

Things like Google search traffic patterns, for instance, can serve as leading indicators for economic data series like unemployment.

The philosophy of this book is that prediction is as much a means as an end.

As another mathematician said, “The best model of a cat is a cat.”

Language, for instance, is a type of model, an approximation that we use to communicate with one another. All languages contain words that have no direct cognate in other languages, even though they are both trying to explain the same universe.

Finding patterns is easy in any kind of data-rich environment; that’s what mediocre gamblers do. The key is in determining whether the patterns represent noise or signal.

Usually, however, we focus on the newest or most immediately available information, and the bigger picture gets lost.

Sometimes, the new evidence is so powerful that it overwhelms everything else, and we can go from assigning a near-zero probability of something to a near-certainty of it almost instantly.

Humans and computers apply different heuristics when they play chess. When they play against each other, the game usually comes down to who can find his opponent’s blind spots first.

Humans, instead, are more capable of focusing on the most important elements and seeing the strategic whole, which sometimes adds up to more than the sum of its parts.

My general advice, in the broader context of forecasting, is to lean heavily toward the “bug” interpretation when your model produces an unexpected or hard-to-explain result. It is too easy to mistake noise for a signal.

Computers have their blind spots as well, but they can avoid these failures of the imagination by at least considering all possible moves.

We have to view technology as what it always has been—a tool for the betterment of the human condition. We should neither worship at the altar of technology nor be frightened by it. Nobody has yet designed, and perhaps no one ever will, a computer that thinks like a human being.49 But computers are themselves a reflection of human progress and human ingenuity: it is not really “artificial” intelligence if a human designed the artifice.