The Signal and the Noise: Why So Many Predictions Fail-but Some Don't

by Nate Silver

You can get lost in the narrative.

The Boston Red Sox failed to make the playoffs in 2011 despite having a 99.7 percent chance of doing so at one point27—although I wouldn’t question anyone who says the normal laws of probability don’t apply when it comes to the Red Sox or the Chicago Cubs.

In baseball, even the best hitters fail a majority of the time, and every player will enter a slump at certain points during the season. The ability to cope with this failure requires a short memory and a certain sense of humor.

“I like to see a hitter, when he flails at a pitch, when he takes a big swing and to the fans it looks ridiculous, I like to look down and see a smile on his face. And then the next time—bam—four hundred feet!”

This is the essence of Beane’s philosophy: collect as much information as possible, but then be as rigorous and disciplined as possible when analyzing it.

“There are a lot of things I wrote in the eighties that weren’t right,” he told me. “The big change was my having children. I know it’s a cliché, but once you have children you start to understand that everyone is somebody’s baby. It is an insiders-outsiders thing. You grow up and these people are characters on TV or video games or baseball cards—you don’t really think about the fact that these guys are humans and doing the best they can.”

You’ve given me an overly specific solution to a general problem. This is overfitting, and it leads to worse predictions.

But the overfit model scores those extra points in essence by cheating—by fitting noise rather than signal. It actually does a much worse job of explaining the real world.58

“With four parameters I can fit an elephant,” the mathematician John von Neumann once said of this problem.59 “And with five I can make him wiggle his trunk.”

Earthquakes may be an inherently complex process. The theory of complexity that the late physicist Per Bak and others developed is different from chaos theory, although the two are often lumped together. Instead, the theory suggests that very simple things can behave in strange and mysterious ways when they interact with one another.

The most pessimistic interpretation, advanced by economists including Jeffrey Sachs of Columbia University, is that the pattern reflects profound structural problems in the American economy: among them, increasing competition from other countries, an imbalance between the service and manufacturing sectors, an aging population, a declining middle class, and a rising national debt. Under this theory, we have entered a new and unhealthy normal, and the problems may get worse unless fundamental changes are made.

a butterfly flapping its wings in Brazil can theoretically cause a tornado in Texas. But in loosely the same way, a tsunami in Japan or a longshoreman’s strike in Long Beach can affect whether someone in Texas finds a job.

Hatzius refers to this chain of cause and effect as a “story.” It is a story about the economy—and although it might be a data-driven story, it is one grounded in the real world.

“When I wrote The Population Bomb I thought our interests in sex and children were so strong that it would be hard to change family size,” Paul Ehrlich told me in a brief interview. “We found out that if you treat women decently and give them job opportunities, the fertility rate goes down.”

“What we find again and again and again is that the more a particular condition is on people’s minds and the more it’s a current topic of discussion, the closer the reporting gets to 100 percent.”

The most effective flu prediction might be one that fails to come to fruition because it motivates people toward more healthful choices.

As the statistician George E. P. Box wrote, “All models are wrong, but some models are useful.”

Successful gamblers, instead, think of the future as speckles of probability, flickering upward and downward like a stock market ticker to every new jolt of information. When their estimates of these probabilities diverge by a sufficient margin from the odds on offer, they may place a bet.

So Voulgaris is not just looking for patterns. Finding patterns is easy in any kind of data-rich environment; that’s what mediocre gamblers do. The key is in determining whether the patterns represent noise or signal.

In the essay, Bayes considered the age-old theological question of how there could be suffering and evil in the world if God was truly benevolent. Bayes’s answer, in essence, was that we should not mistake our human imperfections for imperfections on the part of God, whose designs for the universe we might not fully understand. “Strange therefore . . . because he only sees the lowest part of this scale, [he] should from hence infer a defeat of happiness in the whole,” Bayes wrote in response to another theologian.

FIGURE 8-7: SCIENTIFIC METHOD Step in Scientific Method63 Sports Betting Example Observe a phenomenon Cavaliers games are frequently going over the game total. Develop a hypothesis to explain the phenomenon Cavaliers games are going over because Ricky Davis is playing for a new contract and trying to score as many points as possible. Formulate a prediction from the hypothesis Davis’s incentives won’t change until the end of the season. Therefore: (i) he’ll continue to play at a fast pace, and, (ii) future Cavaliers games will continue to be high-scoring as a result. Test the prediction Place your bet.

A heuristic approach to problem solving consists of employing rules of thumb when a deterministic solution to a problem is beyond our practical capacities.

Campbell had to ask the age-old programmer’s question: was the new move a feature of the program—a eureka moment that indicated it was growing yet more skilled? Or was it a bug?

“In the early stages of debugging Deep Blue, when it would make a move that was unusual, I would say, ‘Oh, there’s something wrong,’” Campbell told me. “We’d dig in and look at the code and eventually figure out what the problem was. But that happened less and less as time went on. As it continued to make these unusual moves, we’d look in and see that it had figured out something that is difficult for humans to see.”

Google performs extensive testing on search and its other products. “We ran six thousand experiments on search last year and probably another six thousand or so on the ad monetization side,” he said. “So Google is doing on a rough order of ten thousand experiments a year.” Some of these experiments are highly visible—occasionally involving rolling out a whole new product line. But most are barely noticeable: moving the placement of a logo by a few pixels, or slightly permuting the background color on an advertisement, and then seeing what effect that has on click-throughs or monetization. Many of the experiments are applied to as few as 0.5 percent of Google’s users, depending on how promising the idea seems to be.

If you do detect a pattern, particularly an obvious-seeming one, the odds are that other investors will have found it as well, and the signal will begin to cancel out or even reverse itself.

A common experiment in economics classrooms, usually employed when the professor needs some extra lunch money, is to hold an auction wherein students submit bids on the number of pennies in a jar.77 The student with the highest bid pays the professor and wins the pennies (or an equivalent amount in paper money if he doesn’t like loose change). Almost invariably, the winning student will find that he has paid too much. Although some of the students’ bids are too low and some are about right, it’s the student who most overestimates the value of the coins in the jar who is obligated to pay for them; the worst forecaster takes the “prize.” This is known as the “winner’s curse.”

One economist sees a $100 bill sitting on the street and reaches to grab it. “Don’t bother,” the other economist says. “If it were a real $100 bill, someone would already have picked it up.”

Avoiding buying during a bubble, or selling during a panic, requires deliberate and conscious effort. You need to have the presence of mind to ignore it. Otherwise you will make the same mistakes that everyone else is making.

The cognitive shortcuts that our mind takes—our heuristics—are what get investors into trouble. The idea that something going up will continue to go up couldn’t be any more instinctive. It just happens to be completely wrong when it comes to the stock market.

My view on trading markets (and toward free-market capitalism more generally) is the same as Winston Churchill’s attitude toward democracy.100 I think it’s the worst economic system ever invented—except for all the other ones.

I downloaded data from eighty-four countries for which estimates of both obesity rates and daily caloric consumption are publicly available.5 Looked at in this way, the relationship seems surprisingly tenuous. The daily consumption in South Korea, which has a fairly meat-heavy diet, is about 3,070 calories per person per day, slightly above the world average. However, the obesity rate there is only about 3 percent. The Pacific island nation of Nauru, by contrast, consumes about as many calories as South Korea per day,6 but the obesity rate there is 79 percent. If you plot the eighty-four countries on a graph (figure 12-1) there seems to be only limited evidence of a connection between obesity and calorie consumption; it would not qualify as “statistically significant” by standard tests.*

Healthy skepticism needs to proceed from this basis. It needs to weigh the strength of new evidence against the overall strength of the theory, rather than rummaging through fact and theory alike for argumentative and ideological convenience, as is the cynical practice when debates become partisan and politicized.

the Maunder Minimum, a period of about seventy years during the late seventeenth and early eighteenth centuries when there was very little sunspot activity, may have triggered cooler temperatures in Europe and North America.

Meanwhile, our government spends hundreds of billions toward economic stimulus programs, or initiates wars in the Middle East, under the pretense of what are probably far more speculative forecasts than are pertinent in climate science.

“The thing is, many people are going around talking as if they looked at the data. I guarantee that nobody ever has,” Schmidt told me after New York’s October 2011 snowstorm, which various media outlets portrayed as evidence either for or against global warming.

Attributing every weather anomaly to manmade climate change—other than the higher temperatures the global warming phenomenon is named for—is a high-stakes gamble, rooted more in politics than in science. There is little consensus about the ways that climate change might manifest itself other than through temperature increases and probably rising sea levels. Obviously, charging that every snowfall is evidence against the theory is just as ridiculous.

When you’re a Jet, you’re a Jet all the way.

It is much easier after the event to sort the relevant from the irrelevant signals. After the event, of course, a signal is always crystal clear; we can now see what disaster it was signaling, since the disaster has occurred. But before the event it is obscure and pregnant with conflicting meanings. It comes to the observer embedded in an atmosphere of “noise,” i.e., in the company of all sorts of information that is useless and irrelevant for predicting the particular disaster.

When a possibility is unfamiliar to us, we do not even think about it. Instead we develop a sort of mind-blindness to it. In medicine this is called anosognosia:20 part of the physiology of the condition prevents a patient from recognizing that they have the condition.

An unknown unknown is when we haven’t really thought to ask the question in the first place.

“You can reasonably predict behavior if people would prefer not to die,” Rumsfeld told me. “But if people are just as happy dying, or feel that it’s a privilege or that it achieves their goal, then they’re going to behave in a very different way.”

terrorists are not purely seeking to maximize their body count; instead, they want to maximize the amount of fear they inflict on a population so as to alter its behavior in some way. Death and destruction are just a means toward that end. “You may kill people to achieve that,” Rumsfeld told me. “But that is not its purpose.”

From the Iranian Revolution through September 10, 2001, there were more than 4,000 attempted or successful terror attacks in NATO countries. But more than half the death toll had been caused by just seven of them. The three largest attacks—the Air India disaster, the Lockerbie bombing, and Oklahoma City—had accounted for more than 40 percent of the fatalities all by themselves.

The conundrum of why terrorists don’t target shopping centers would seem ridiculous to someone in Israel, where it happens all the time.

This is perhaps the easiest Bayesian principle to apply: make a lot of forecasts. You may not want to stake your company or your livelihood on them, especially at first.* But it’s the only way to get better.

Specifically, I split the sample data into even- and odd-numbered months; if a player is truly skilled, he should be winning in the even-numbered months as well as the odd-numbered ones. I then applied a regression analysis to predict a player’s win rate (measured as big blinds won per one hundred hands) from one half of the sample to the other; the results of the regression are taken to be tantamount to the player’s long term success rate. The variables in the regression were a player’s win rate, multiplied by the natural logarithm of the number of hands who he played, along with a variable indicating how many hands he played of the ones he was dealt. Players who were too tight or too loose were considerably less likely to replicate their success from one period to the next, holding other factors equal—in fact, this was often a better predictor of a player’s out-of-sample win rate than the past win rate itself, unless he had played a very large number of hands.

When I was working on this book and came across a thorny problem that I couldn’t quite resolve, I found it much more productive to walk around, subjecting my brain to random inputs, than to stare at my computer screen or sit in a coffee shop. One of the advantages of living in New York is that it offers 24/7 access to the spontaneous behavior of eight million human beings who might jog your mind or your memory.

The title of this chapter is inspired by a line from the poem “The Road to Wisdom,” by the Danish mathematician Piet Hein: “to err and err and err again, but less and less and less.”

As the Harvard professor H. L. “Skip” Gates says, “Conspiracy theories are an irresistible labor-saving device in the face of complexity.”