The Signal and the Noise: Why So Many Predictions Fail-but Some Don't

by Nate Silver

Barack Obama stock.75 Eventually the anomalies were corrected, but it took some time—often four to six hours—before prices fully rebounded to their previous values. Many traders were convinced that the rogue trader knew something they didn’t—perhaps he had inside information about some impending scandal? This is herding. And there’s evidence

Age risks: we share so much information that our independence is reduced. Instead, we seek out others who think just like us and brag about how many “friends” and “followers” we have. In the market, prices may occasionally follow the lead of the worst investors.

However, these tracks happen to run along the same road, as though some city decided to hold a Formula 1 race but by some bureaucratic oversight forgot to close one lane to commuter traffic.

Sometimes, like during the financial crisis, there is a big accident, and regular investors get run over.

This sort of duality, what the physicist Didier Sornette calls “the fight between order and disorder,”99 is common in complex systems, which are those governed by the ...

well described by a few simple laws

And yet they are essentially unpredictable from day to day.

they periodically undergo violent and highly nonlinear* phase changes from orderly to chaotic and back again.

In science, dubious forecasts are more likely to be exposed—and the truth is more likely to prevail. In politics, a domain in which the truth enjoys no privileged status, it’s anybody’s guess.

You should work to reduce your biases, but to say you have none is a sign that you have many. To state your beliefs up front—to say “Here’s where I’m coming from”12—is a way to operate in good faith and to recognize that you perceive reality through a subjective filter.

Most of the time, we do not appreciate how noisy the data is, and so our bias is to place too much weight on the newest data point.

It’s often the outliers that make the news.

But we can have the opposite bias when we become too personally or professionally invested in a problem, failing to change our minds when the facts do.

The more often you are willing to test your ideas, the sooner you can begin to avoid these problems and learn from your mistakes.

Prediction is difficult for us for the same reason that it is so important: it is where objective and subjective reality intersect.

Distinguishing the signal from the noise requires both scientific knowledge and self-knowledge:

Unpredictable” has been on the rise again in recent years.

How good we think we are at prediction and how good we really are may even be inversely correlated.

An unpredicted disaster, even if it has no direct effect on us, shakes our confidence that we are in control of our fate.