The Signal and the Noise: The Art and Science of Prediction

by Nate Silver

Greed and fear are volatile quantities, however, and the balance can get out of whack. When there is an excess of greed in the system, there is a bubble. When there is an excess of fear, there is a panic.

One of the pervasive risks that we face in the information age, as I wrote in the introduction, is that even if the amount of knowledge in the world is increasing, the gap between what we know and what we think we know may be widening.

Quite a lot of evidence suggests that aggregate or group forecasts are more accurate than individual ones, often somewhere between 15 and 20 percent more accurate depending on the discipline.

You will need to learn how to express—and quantify—the uncertainty in your predictions. You will need to update your forecast as facts and circumstances change. You will need to recognize that there is wisdom in seeing the world from a different viewpoint. The more you are willing to do these things, the more capable you will be of evaluating a wide variety of information without abusing it.

A good baseball projection system must accomplish three basic tasks: Account for the context of a player’s statistics Separate out skill from luck Understand how a player’s performance evolves as he ages—what is known as the aging curve

The key to making a good forecast, as we observed in chapter 2, is not in limiting yourself to quantitative information. Rather, it’s having a good process for weighing the information appropriately. This is the essence of Beane’s philosophy: collect as much information as possible, but then be as rigorous and disciplined as possible when analyzing it.

In the most competitive industries, like sports, the best forecasters must constantly innovate. It’s easy to adopt a goal of “exploit market inefficiencies.” But that doesn’t really give you a plan for how to find them and then determine whether they represent fresh dawns or false leads. It’s hard to have an idea that nobody else has thought of. It’s even harder to have a good idea—and when you do, it will soon be duplicated.

Good innovators typically think very big and they think very small.

So meteorologists have been after something else. Instead of a statistical model, they wanted a living and breathing one that simulated the physical processes that govern the weather.

This logic is a little circular. TV weathermen say they aren’t bothering to make accurate forecasts because they figure the public won’t believe them anyway. But the public shouldn’t believe them, because the forecasts aren’t accurate.

Earthquakes may be an inherently complex process. The theory of complexity that the late physicist Per Bak and others developed is different from chaos theory, although the two are often lumped together. Instead, the theory suggests that very simple things can behave in strange and mysterious ways when they interact with one another.

As Hatzius sees it, economic forecasters face three fundamental challenges. First, it is very hard to determine cause and effect from economic statistics alone. Second, the economy is always changing, so explanations of economic behavior that hold in one business cycle may not apply to future ones. And third, as bad as their forecasts have been, the data that economists have to work with isn’t much good either.

Hatzius noted, for instance, that the unemployment rate is usually taken to be a lagging indicator. And sometimes it is. After a recession, businesses may not hire new employees until they are confident about the prospects for recovery, and it can take a long time to get all the unemployed back to work again. But the unemployment rate can also be a leading indicator for consumer demand, since unemployed people don’t have much ability to purchase new goods and services.

As the statistician George E. P. Box wrote, “All models are wrong, but some models are useful.”90 What he meant by that is that all models are simplifications of the universe, as they must necessarily be. As another mathematician said, “The best model of a cat is a cat.”91 Everything else is leaving out some sort of detail. How pertinent that detail might be will depend on exactly what problem we’re trying to solve and on how precise an answer we require.

In weather, much of the problem is that our knowledge of the initial conditions is incomplete. Even though we have a very good idea of the rules by which the weather system behaves, we have incomplete information about the position of all the molecules that form clouds and rainstorms and hurricanes. Hence, the best we can do is to make probabilistic forecasts.

What’s more, the subtraction of the fish from the table can have a cascading effect on the other players. The one who was formerly the next-to-worst player is now the sucker, and will be losing money at an even faster rate than before. So he may bust out too, in turn making the remaining players’ task yet more challenging. The entire equilibrium of the poker ecosystem can be thrown out of balance.

My conclusion at the time was that the composition of the player pool had changed dramatically. Many of the professional players, reliant on the game for income, had soldiered on and kept playing, but most of the amateurs withdrew their funds or went broke. The fragile ecology of the poker economy was turned upside down—without those weak players to prop the game up, the water level had risen, and some of the sharks turned into suckers.26

When we play poker, we control our decision-making process but not how the cards come down. If you correctly detect an opponent’s bluff, but he gets a lucky card and wins the hand anyway, you should be pleased rather than angry, because you played the hand as well as you could. The irony is that by being less focused on your results, you may achieve better ones.

One conceit of economics is that markets as a whole can perform fairly rationally, even if many of the participants within them are irrational. But irrational behavior in the markets may result precisely because individuals are responding rationally according to their incentives. So long as most traders are judged on the basis of short-term performance, bubbles involving large deviations of stock prices from their long-term values are possible—and perhaps even inevitable.

In the market, prices may occasionally follow the lead of the worst investors. They are the ones making most of the trades.

Avoiding buying during a bubble, or selling during a panic, requires deliberate and conscious effort. You need to have the presence of mind to ignore it. Otherwise you will make the same mistakes that everyone else is making.

There might be a terrific opportunity to short a bubble or long a panic once every fifteen or twenty years when one comes along in your asset class. But it’s very hard to make a steady career out of that, doing nothing for years at a time.

This sort of duality, what the physicist Didier Sornette calls “the fight between order and disorder,”99 is common in complex systems, which are those governed by the interaction of many separate individual parts. Complex systems like these can at once seem very predictable and very unpredictable.

Empirical studies of consensus-driven predictions have found mixed results, in contrast to a process wherein individual members of a group submit independent forecasts and those are averaged or aggregated together, which can almost always be counted on to improve predictive accuracy.38

There is an alternative, however, when you have some knowledge of the structure behind the system. This second type of model essentially creates a simulation of the physical mechanics of some portion of the universe. It takes much more work to build than a purely statistical method and requires a more solid understanding of the root causes of the phenomenon. But it is potentially more accurate.

An admonition like “The more complex you make the model the worse the forecast gets” is equivalent to saying “Never add too much salt to the recipe.” How much complexity—how much salt—did you begin with? If you want to get good at forecasting, you’ll need to immerse yourself in the craft and trust your own taste buds.

The level of industrial activity is fairly constant, but CO2 circulates quickly into the atmosphere and remains there for a long time. (Its chemical half-life has been estimated at about thirty years.61) Even if major industrialized countries agreed to immediate and substantial reductions in CO2 emissions, it would take years to reduce the growth rate of CO2 in the atmosphere, let alone to actually reverse it. “Neither you nor I will ever see a year in which carbon dioxide concentrations have gone down, not ever,” Schmidt told me. “And not your children either.”

The problem comes when, out of frustration that our knowledge of the world is imperfect, we fail to make a forecast at all. An unknown unknown is a contingency that we have not even considered. We have some kind of mental block against it, or our experience is inadequate to imagine it; it’s as though it doesn’t even exist.

The more eagerly we commit to scrutinizing and testing our theories, the more readily we accept that our knowledge of the world is uncertain, the more willingly we acknowledge that perfect prediction is impossible, the less we will live in fear of our failures, and the more liberty we will have to let our minds flow freely. By knowing more about what we don’t know, we may get a few more predictions right.

We have big brains, but we live in an incomprehensibly large universe. The virtue in thinking probabilistically is that you will force yourself to stop and smell the data—slow down, and consider the imperfections in your thinking. Over time, you should find that this makes your decision making better.