The Signal and the Noise: Why So Many Predictions Fail-but Some Don't

by Nate Silver

But these subtleties were soon lost to the popular imagination. Machine had triumphed over man! It was like when HAL 9000 took over the spaceship. Like the moment when, exactly thirteen seconds into “Love Will Tear Us Apart,” the synthesizer overpowers the guitar riff, leaving rock and roll in its dust.43 Except it wasn’t true. Kasparov had been the victim of a large amount of human frailty—and a tiny software bug.

In theory, programming a computer to play chess is easy: if you let a chess program’s search algorithms run for an indefinite amount of time, then all positions can be solved by brute force. “There is a well-understood algorithm to solve chess,” Campbell told me. “I could probably write the program in half a day that could solve the game if you just let it run long enough.” In practice, however, “it takes the lifetime of the universe to do that,” he lamented.

We should probably not describe the computer as “creative” for finding the moves; instead, it did so more through the brute force of its calculation speed. But it also had another advantage: it did not let its hang-ups about the right way to play chess get in the way of identifying the right move in those particular circumstances. For a human player, this would have required the creativity and confidence to see beyond the conventional thinking. People marveled at Fischer’s skill because he was so young, but perhaps it was for exactly that reason that he found the moves: he had the full breadth of his imagination at his disposal. The blind spots in our thinking are usually of our own making and they can grow worse as we age. Computers have their blind spots as well, but they can avoid these failures of the imagination by at least considering all possible moves.

Campbell somewhat mischievously referred to an incident that had occurred toward the end of the first game in their 1997 match with Kasparov. “A bug occurred in the game and it may have made Kasparov misunderstand the capabilities of Deep Blue,” Campbell told me. “He didn’t come up with the theory that the move that it played was a bug.” The bug had arisen on the forty-fourth move of their first game against Kasparov; unable to select a move, the program had defaulted to a last-resort fail-safe in which it picked a play completely at random. The bug had been inconsequential, coming late in the game in a position that had already been lost; Campbell and team repaired it the next day. “We had seen it once before, in a test game played earlier in 1997, and thought that it was fixed,” he told me. “Unfortunately there was one case that we had missed.” In fact, the bug was anything but unfortunate for Deep Blue: it was likely what allowed the computer to beat Kasparov. In the popular recounting of Kasparov’s match against Deep Blue, it was the second game in which his problems originated—when he had made the almost unprecedented error of forfeiting a position that he could probably have drawn. But what had inspired Kasparov to commit this mistake? His anxiety over Deep Blue’s forty-fourth move in the first game—the move in which the computer had moved its rook for no apparent purpose. Kasparov had concluded that the counterintuitive play must be a sign of superior intelligence. ...

Computers are very, very fast at making calculations. Moreover, they can be counted on to calculate faithfully—without getting tired or emotional or changing their mode of analysis in midstream. But this does not mean that computers produce perfect forecasts, or even necessarily good ones. The acronym GIGO (“garbage in, garbage out”) sums up this problem. If you give a computer bad data, or devise a foolish set of instructions for it to analyze, it won’t spin straw into gold. Meanwhile, computers are not very good at tasks that require creativity and imagination, like devising strategies or developing theories about the way the world works.

ESPN broadcasts presented a highly sanitized version of what reality actually looks like at the poker table. For one thing, out of the necessity of compressing more than forty hours of play involving more than eight hundred players into six hours of broadcasts, they showed only a small fraction of the hands as they were actually played. What’s more, because of the ingenious invention of the “hole cam”—pinhole-size cameras installed around the edge of the table beside each player—the cards of not just Moneymaker but those of each of his opponents were revealed to the home audience as the hand was being played out, giving the audience the feeling of being clairvoyant. Poker is a pretty easy game if you know what cards your opponent holds. Moneymaker was cast as the protagonist who could do no wrong. Hands that a sober analysis might have concluded he’d played poorly were invariably praised by the announcers—rash bluffs became gutsy ones, premature folds became perceptive ones. Moneymaker was not some slightly-above-average schmoe getting the cards of his life*1 but a poker savant who was cunning enough to have developed into a world-class player almost overnight. The viewer was led to believe that poker is easy to learn, easy to profit from, and incredibly action-packed—none of which is true.

As Arthur Conan Doyle once said, “Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.” This is sound logic, but we have a lot of trouble distinguishing the impossible from the highly improbable and sometimes get in trouble when we try to make too fine a distinction.

There is a learning curve that applies to poker and to most other tasks that involve some type of prediction. The key thing about a learning curve is that it really is a curve: the progress we make at performing the task is not linear.

The name for the curve comes from the well-known business maxim called the Pareto principle or 80-20 rule (as in: 80 percent of your profits come from 20 percent of your customers16). As I apply it here, it posits that getting a few basic things right can go a long way. In poker, for instance, simply learning to fold your worst hands, bet your best ones, and make some effort to consider what your opponent holds will substantially mitigate your losses. If you are willing to do this, then perhaps 80 percent of the time you will be making the same decision as one of the best poker players like Dwan—even if you have spent only 20 percent as much time studying the game.

The first 20 percent often begins with having the right data, the right technology, and the right incentives. You need to have some information—more of it rather than less, ideally—and you need to make sure that it is quality-controlled. You need to have some familiarity with the tools of your trade—having top-shelf technology is nice, but it’s more important that you know how to use what you have. You need to care about accuracy—about getting at the objective truth—rather than about making the most pleasing or convenient prediction, or the one that might get you on television. Then you might progress to a few intermediate steps, developing some rules of thumb (heuristics) that are grounded in experience and common sense and some systematic process to make a forecast rather than doing so on an ad hoc basis. These things aren’t exactly easy—many people get them wrong. But they aren’t hard either, and by doing them you may be able to make predictions 80 percent as reliable as those of the world’s foremost expert.

However, when a field is highly competitive, it is only through this painstaking effort around the margin that you can make any money. There is a “water level” established by the competition and your profit will be like the tip of an iceberg: a small sliver of competitive advantage floating just above the surface, but concealing a vast bulwark of effort that went in to support it.

I’ve been fortunate enough to take advantage of fields where the water level was set pretty low, and getting the basics right counted for a lot. Baseball, in the pre-Moneyball era, used to be one of these. Billy Beane got an awful lot of mileage by recognizing a few simple things, like the fact that on-base percentage is a better measure of a player’s offensive performance than his batting average. Nowadays pretty much everyone realizes that. In politics, I’d expect that I’d have a small edge at best if there were a dozen clones of FiveThirtyEight. But often I’m effectively “competing” against political pundits, like those on The McLaughlin Group, who aren’t really even trying to make accurate predictions.

It is often possible to make a profit by being pretty good at prediction in fields where the competition succumbs to poor incentives, bad habits, or blind adherence to tradition—or because you have better data or technology than they do. It is much harder to be very good in fields where everyone else is getting the basics right—and you may be fooling yourself if you think you have much of an edge.

Here you see the statistical echo of the 80/20 rule: there’s a much larger difference between the very worst players and the average ones than between the average ones and the best. The better players are doing just a few things differently from one another, while those at the lower end of the curve are getting even the basics wrong, diverging wildly from optimal strategy.

It is emphatically the case, however, that if you can’t spot one or two bad players in the game, you probably shouldn’t be playing in it. In poker, the line between success and failure is very thin and the presence of a single fish can make the difference.

Poker abides by a “trickle up” theory of wealth: the bottom 10 percent of players are losing money quickly enough to support a relatively large middle class of break-even players. But what happens when the fish—the sucker—busts out, as someone losing money at this rate is bound to do? Several of the marginally winning players turn into marginally losing ones (figure 10-8b). In fact, we now estimate that only the very best player at the table is still making money over the long run, and then less than he did before.

What’s more, the subtraction of the fish from the table can have a cascading effect on the other players. The one who was formerly the next-to-worst player is now the sucker, and will be losing money at an even faster rate than before. So he may bust out too, in turn making the remaining players’ task yet more challenging. The entire equilibrium of the poker ecosystem can be thrown out of balance.

Much more commonly, the answer is that there is not just one fishy player who loses money in perpetuity but a steady stream of them who take their turn in the barrel, losing a few hundred or a few thousand dollars and then quitting.

Many of the professional players, reliant on the game for income, had soldiered on and kept playing, but most of the amateurs withdrew their funds or went broke. The fragile ecology of the poker economy was turned upside down—without those weak players to prop the game up, the water level had risen, and some of the sharks turned into suckers.

I had hit a wall, playing uncreative and uninspired poker. When I did play, I combined the most dangerous trait of the professional player—the sense that I was entitled to win money—with the bad habits of the amateur, playing late into the evening, sometimes after having been out with friends.

Luck and skill are often portrayed as polar opposites. But the relationship is a little more complicated than that.

In figure 10-9, I’ve plotted the batting averages achieved by American League players in April 2011 on one axis, and the batting averages for the same players in May 2011 on the other one.28 There seems to be no correlation between the two. (A player named Brendan Ryan, for instance, hit .184 in April but .384 in May.) And yet, we know from looking at statistics over the longer term—what baseball players do over whole seasons or over the course of their careers—that hitting ability differs substantially from player to player.

I’ve modeled the potential profits and losses for a player with the statistics I just described. The bands in the chart show the plausible range of wins and losses for the player, enough to cover 95 percent of all possible cases. After he plays 60,000 hands—about as many as he’d get in if he played forty hours a week in a casino every week for a full year—the player could plausibly have made $275,000 or have lost $35,000. In essence, this player could go to work every day for a year and still lose money. This is why it is sometimes said that poker is a hard way to make an easy living.

The Bayesian method described in the book The Mathematics of Poker, for instance, would suggest that a player who had made $30,000 in his first 10,000 hands at a $100/$200 limit hold ’em game was nevertheless more likely than not to be a long-term loser.

“There is no other game that I know of where humans are so smug, and think that they just play like wizards, and then play so badly,” he told me. “Basically it’s because they don’t know anything, and they think they must be God-like, and the truth is that they aren’t. If computer programs feed on human hubris, then in poker they will eat like kings.” This quality, of course, is not unique to poker.

More broadly, overconfidence is a huge problem in any field in which prediction is involved.

Nor are poker players very much like roulette players; they are probably much more like investors, in fact. According to one study of online poker players, 52 percent have at least a bachelor’s degree34—about twice the rate in the U.S. population as a whole, and four times the rate among those who purchase lottery tickets.35 Most poker players are smart enough to know that some players really do make money over the long term—and this is what can get them in trouble.

Angelo, like most poker players, had his ups and downs—not just in his results but also in the quality of his play. When he was playing his best, he was very good. But he wasn’t always playing his best—very often, he was on tilt. “I was a great tilter,” Angelo reflected in his book, Elements of Poker, referring to a state of overaggressive play brought on by a loss of perspective.37 “I knew all the different kinds. I could do steaming tilt, simmering tilt, too loose tilt, too tight tilt, too aggressive tilt, too passive tilt, playing too high tilt, playing too long tilt, playing too tired tilt, entitlement tilt, annoyed tilt, injustice tilt, frustration tilt, sloppy tilt, revenge tilt, underfunded tilt, overfunded tilt, shame tilt, distracted tilt, scared tilt, envy tilt, this-is-the-worst-pizza-I’ve-ever-had tilt, I-just-got-showed-a-bluff tilt, and of course, the classics: I-gotta-get-even tilt, and I-only-have-so-much-time-to-lose-this-money tilt, also known as demolition tilt.”

As we have seen, it is considerably easier to lose money at poker when you play badly than to make money when you are playing well.

It is quite plausible for someone who plays at a world-class level 90 percent of the time to lose money from the game overall if he tilts the other 10 percent of the time.

The fundamental reason that poker players tilt is that this balance is so often out of whack: over the short term, and often over the medium term, a player’s results aren’t very highly correlated with his skill.

The investor who calls the stock market bottom is heralded as a genius, even if he had some buggy statistical model that just happened to get it right. The general manager who builds a team that wins the World Series is assumed to be better than his peers, even if, when you examine his track record, the team succeeded despite the moves he made rather than because of them. And this is certainly the case when it comes to poker. Chris Moneymaker wouldn’t have been much of a story if the marketing pitch were “Here’s some slob gambler who caught a bunch of lucky cards.” Sometimes we take consideration of luck too far in the other direction, when we excuse predictions that really were bad by claiming they were unlucky. The credit-rating agencies used a version of this excuse when their incompetence helped to usher in the financial collapse. But as a default, just as we perceive more signal than there really is when we make predictions, we also tend to attribute more skill than is warranted to successful predictions when we assess them later.

In the 1950s, the average share of common stock in an American company was held for about six years before being traded—consistent with the idea that stocks are a long-term investment. By the 2000s, the velocity of trading had increased roughly twelvefold. Instead of being held for six years, the same share of stock was traded after just six months.

Economics 101 teaches that trading is rational only when it makes both parties better off. A baseball team with two good shortstops but no pitching trades one of them to a team with plenty of good arms but a shortstop who’s batting .190. Or an investor who is getting ready to retire cashes out her stocks and trades them to another investor who is just getting his feet wet in the market. But very little of the trading that occurs on Wall Street today conforms to this view. Most of it reflects true differences of opinion—contrasting predictions—about the future returns of a stock.* Never before in human history have so many predictions been made so quickly and for such high stakes.

And yet, a central premise of this book is that we must accept the fallibility of our judgment if we want to come to more accurate predictions. To the extent that markets are reflections of our collective judgment, they are fallible too. In fact, a market that makes perfect predictions is a logical impossibility.

“Past performance is not indicative of future results” appears in mutual-fund brochures for a reason.

Efficient-market hypothesis is sometimes mistaken for an excuse for the excesses of Wall Street; whatever else those guys are doing, it seems to assert, at least they’re behaving rationally. A few proponents of the efficient-market hypothesis might interpret it in that way. But as the theory was originally drafted, it really makes just the opposite case: the stock market is fundamentally and profoundly unpredictable. When something is truly unpredictable, nobody from your hairdresser to the investment banker making $2 million per year is able to beat it consistently.

At one point during the dot-com boom, the market value of technology companies accounted for about 35 percent of the value of all stocks in the United States,41 implying they would soon come to represent more than a third of private-sector profits. What’s interesting is that the technology itself has in some ways exceeded our expectations. Can you imagine what an investor in 2000 would have done if you had shown her an iPad? And told her that, within ten years, she could use it to browse the Internet on an airplane flying 35,000 feet over Missouri and make a Skype call* to her family in Hong Kong? She would have bid Apple stock up to infinity. Nevertheless, ten years later, in 2010, technology companies accounted for only about 7 percent of economic activity.42 For every Apple, there were dozens of companies like Pets.com that went broke. Investors were behaving as though every company would be a winner, that they wouldn’t have to outcompete each other, leading to an utterly unrealistic assumption about the potential profits available to the industry as a whole.

A common experiment in economics classrooms, usually employed when the professor needs some extra lunch money, is to hold an auction wherein students submit bids on the number of pennies in a jar.77 The student with the highest bid pays the professor and wins the pennies (or an equivalent amount in paper money if he doesn’t like loose change). Almost invariably, the winning student will find that he has paid too much. Although some of the students’ bids are too low and some are about right, it’s the student who most overestimates the value of the coins in the jar who is obligated to pay for them; the worst forecaster takes the “prize.” This is known as the “winner’s curse.” The stock market has some of the same properties. Every now and then, the trader most willing to buy a stock will be the one who really does have some unique or proprietary insight about a company. But most traders are about average, and they’re using mostly the same models with most of the same data. If they decide a stock is substantially undervalued when their peers disagree, most of the time it will be because they’ve placed too much confidence in their forecasting abilities and mistaken the noise in their model for a signal.

There is a kind of symbiosis between the irrational traders and the skilled ones—just as, in a poker game, good players need some fish at the table to make the game profitable to play in. In the financial literature, these irrational traders are known as “noise traders.” As the economist Fisher Black wrote in a 1986 essay simply called “Noise”: Noise makes trading in financial markets possible, and thus allows us to observe prices for financial assets. [But] noise also causes markets to be somewhat inefficient. . . . Most generally, noise makes it very difficult to test either practical or academic theories about the way that financial or economic markets work. We are forced to act largely in the dark.

But, if you think a market is efficient—efficient enough that you can’t really beat it for a profit—then it would be irrational for you to place any trades. In fact, efficient-market hypothesis is intrinsically somewhat self-defeating. If all investors believed the theory—that they can’t make any money from trading since the stock market is unbeatable—there would be no one left to make trades and therefore no market at all.

You should not rush out and become an options trader. As the legendary investor Benjamin Graham advises, a little bit of knowledge can be a dangerous thing in the stock market.92 After all, any investor can do as well as the average investor with almost no effort. All he needs to do is buy an index fund that tracks the average of the S&P 500.93 In so doing he will come extremely close to replicating the average portfolio of every other trader, from Harvard MBAs to noise traders to George Soros’s hedge fund manager.

“Everybody thinks they have this supersmart mutual fund manager,” Henry Blodget told me. “He went to Harvard and has been doing it for twenty-five years. How can he not be smart enough to beat the market? The answer is: Because there are nine million of him and they all have a fifty-million-dollar budget and computers that are collocated in the New York Stock Exchange. How could you possibly beat that?”

Suppose that you had invested $10,000 in the S&P 500 in 1970, planning to cash it out forty years later upon your retirement in 2009. There were plenty of ups and downs during this period. But if you stuck with your investment through thick and thin, you would have made a profit of $63,000 when you retired, adjusted for inflation and not counting the original principal.95 If instead you had “played it safe” by pulling your money out of the market every time it had fallen more than 25 percent from its previous peak, waiting until the market rebounded to 90 percent of its previous high before reinvesting, you would have just $18,000 in profit—a meager return of 2.6 percent per year.96 Many investors, unfortunately, behave in exactly this fashion. Worse yet, they tend to make their initial investments at times when the market is overvalued, in which case they may struggle to make a positive return of any kind over the long term.

Avoiding buying during a bubble, or selling during a panic, requires deliberate and conscious effort. You need to have the presence of mind to ignore it. Otherwise you will make the same mistakes that everyone else is making.

The cognitive shortcuts that our mind takes—our heuristics—are what get investors into trouble. The idea that something going up will continue to go up couldn’t be any more instinctive. It just happens to be completely wrong when it comes to the stock market. Our instincts related to herding may be an even more fundamental problem. Oftentimes, it will absolutely be right to do what everyone else is doing, or at least to pay some attention to it. If you travel to a strange city and need to pick a restaurant for dinner, you probably want to select the one that has more customers, other things being equal. But occasionally it will backfire: you wind up at the tourist trap. Likewise, when we were making forecasts in Bayesland, it usually behooved us to pay some attention to our neighbors and adjust our beliefs accordingly, rather than adopt the stubborn and often implausible notion that we knew better than everyone else. I pay quite a bit of attention to what the consensus view is—what a market like Intrade is saying—when I make a forecast. It is never an absolute constraint. But the further I move away from that consensus, the stronger my evidence has to be before I come to the view that I have things right and everyone else has it wrong.

Complex systems like these can at once seem very predictable and very unpredictable. Earthquakes are very well described by a few simple laws (we have a very good idea of the long-run frequency of a magnitude 6.5 earthquake in Los Angeles). And yet they are essentially unpredictable from day to day. Another characteristic of these systems is that they periodically undergo violent and highly nonlinear* phase changes from orderly to chaotic and back again. For Sornette and others who take highly mathematical views of the market, the presence of periodic bubbles seems more or less inevitable, an intrinsic property of the system. I am partial toward this perspective. My view on trading markets (and toward free-market capitalism more generally) is the same as Winston Churchill’s attitude toward democracy.100 I think it’s the worst economic system ever invented—except for all the other ones. Markets do a good job most of the time, but I don’t think we’ll ever be rid of bubbles.

What’s clear, however, is that we’ll never detect a bubble if we start from the presumption that markets are infallible and the price is always right. Markets cover up some of our warts and balance out some of our flaws. And they certainly aren’t easy to outpredict. But sometimes the price is wrong.

“It’s critical to have a diversity of models,” I was told by Kerry Emanuel, an MIT meteorologist who is one of the world’s foremost theorists about hurricanes. “You do not want to put all your eggs in one basket.” One of the reasons this is so critical, Emanuel told me, is that in addition to the different assumptions these models employ, they also contain different bugs. “That’s something nobody likes to talk about,” he said. “Different models have different coding errors. You cannot assume that a model with millions and millions of lines of code, literally millions of instructions, that there isn’t a mistake in there.” If you’re used to thinking about the global warming debate as series of arguments between “skeptics” and “believers,” you might presume that this argument emanates from a scientist on the skeptical side of the aisle. In fact, although Emanuel has described himself as conservative and Republican40—which is brave enough at MIT—he would probably not think of himself as a global warming skeptic. Instead, he is a member in good standing of the scientific establishment, having been elected to the National Academy of Sciences.

A survey of climate scientists conducted in 200842 found that almost all (94 percent) were agreed that climate change is occurring now, and 84 percent were persuaded that it was the result of human activity. But there was much less agreement about the accuracy of climate computer models.