Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

The time for Auftragstaktik finally came in the early 1980s. Tensions were mounting and the Soviets had a huge numerical lead in men and tanks, forcing NATO to do more with less. Generals in the United States combed through the writings of historians and theorists and scrutinized Israeli experience. Some even consulted old Wehrmacht generals. In 1982 “mission command” became part of official American doctrine.

The insurgency that blossomed after the fall of Baghdad was not forecast by the military leadership, and for months, even years, they had little idea how to respond.

Petraeus also supports sending officers to top universities for graduate education, not to acquire a body of knowledge, although that is a secondary benefit, but to encounter surprises of another kind. “It teaches you that there are seriously bright people out in the world who have very different basic assumptions about a variety of different topics and therefore arrive at conclusions on issues that are very, very different from one’s own and very different from mainstream kind of thinking, particularly in uniform,” Petraeus said.

“We let our people know what we want them to accomplish. But—and it is a very big ‘but’—we do not tell them how to achieve those goals.”25 That is a near-perfect summary of “mission command.” The speaker is William Coyne, who was senior vice president of research and development at 3M, the famously innovative manufacturing conglomerate.

The humility required for good judgment is not self-doubt—the sense that you are untalented, unintelligent, or unworthy. It is intellectual humility. It is a recognition that reality is profoundly complex, that seeing things clearly is a constant struggle, when it can be done at all, and that human judgment must therefore be riddled with mistakes.

Consider a 2014 interview with General Michael Flynn, who summed up his view of the world shortly before retiring as head of the Defense Intelligence Agency (DIA), the Pentagon’s equivalent of the CIA with seventeen thousand employees. “I come into this office every morning, and other than a short jog to clear my head, I spend two to three hours reading intelligence reports,” he said. “I will frankly tell you that what I see each day is the most uncertain, chaotic, and confused international environment that I’ve witnessed in my entire career. There were probably more dangerous times such as when the Nazis and [Japanese] imperialists were trying to dominate the world, but we’re in another very dangerous era. … I think we’re in a period of prolonged societal conflict that is pretty unprecedented.

I am not belittling Michael Flynn. Quite the opposite: the fact that a man so accomplished made so obvious an error is precisely what makes the error notable. We are all vulnerable. And there’s no way to make ourselves bulletproof, as the famous Müller-Lyer optical illusion illustrates:

Kahneman first documented scope insensitivity thirty years ago when he asked a randomly selected group in Toronto, the capital city of Ontario, how much they would be willing to pay to clean up the lakes in a small region of the province. On average, people said about $10.3 Kahneman asked another randomly selected group how much they would be willing to pay to clean up every one of the 250,000 lakes in Ontario. They too said about $10. Later research got similar results. One study informed people that each year 2,000 migratory birds drown in oil ponds. How much would you be willing to pay to stop that? Other subjects were told 20,000 birds died each year. A third group was told it was 200,000 birds. And yet in each case, the average amount people said they would be willing to pay was around $80. What people were responding to, Kahneman later wrote, was the prototypical image that came to them—a polluted lake or a drowning, oil-soaked duck. “The prototype automatically evokes an affective response, and the intensity of that emotion is then mapped onto the dollar scale.” It’s classic bait and switch. Instead of answering the question asked—a difficult one that requires putting a money value on things we never monetize—people answered “How bad does this make me feel?” Whether the question is about 2,000 or 200,000 dying ducks, the answer is roughly the same: bad. Scope recedes into the background—and out of sight, out of mind.

But notice what’s missing? The time frame. It obviously matters. To use an extreme illustration, the probability of the regime falling in the next twenty-four hours must be less—likely a lot less—than the probability that it will fall in the next twenty-four months. To put this in Kahneman’s terms, the time frame is the “scope” of the forecast.

But another friend and colleague is not as impressed by the superforecasters as I am. Indeed, he suspects this whole research program is misguided.

But Taleb isn’t interested only in surprise. A black swan must be impactful. Indeed, Taleb insists that black swans, and black swans alone, determine the course of history.

Now if you believe that only black swans matter in the long run, the Good Judgment Project should only interest short-term thinkers. But history is not just about black swans. Look at the inch-worm advance in life expectancy. Or consider that an average of 1% annual global economic growth in the nineteenth century and 2% in the twentieth turned the squalor of the eighteenth century and all the centuries that preceded it into the unprecedented wealth of the twenty-first.

This is black swan investing, and it’s similar to how Taleb himself traded—very successfully—before becoming an author.

Taleb has taken this argument further and called for critical systems—like international banking and nuclear weapons—to be made “antifragile,” meaning they are not only resilient to shocks but strengthened by them. In principle, I agree. But a point often overlooked is that preparing for surprises—whether we are shooting for resilience or antifragility—is costly.

Some find it hard wrapping their heads around Taleb’s idea of statistical distributions of possible worlds. It feels like eggheaded nonsense. There is only one reality: what happened in the past, what we’re living in now, and what will happen in the future. But if you are as mathematically inclined as Taleb, you get used to the idea that the world we live in is but one that emerged, quasi-randomly, from a vast population of once-possible worlds. The past did not have to unfold as it did, the present did not have to be what it is, and the future is wide open. History is a virtually infinite array of possibilities. Prudent leaders feel this in their bones, as John F. Kennedy did when he realized the Cuban missile crisis had an array of possible outcomes ranging from peace to nuclear annihilation, which could have pushed the death toll for World War III into the hundreds of millions. Fat tails indeed!

All three of us see history this way. Counterfactuals highlight how radically open the possibilities once were and how easily our best-laid plans can be blown away by flapping butterfly wings.

As it turned out, no won by the surprisingly wide margin of 55.3% to 44.7%. (Superforecasters aced this one, incidentally, even beating British betting markets with real money on the table.)

“Which is to say—what does one do with data points like this to adjust one’s worldview?”

Fuzzy thinking can never be proven wrong. And only when we are proven wrong so clearly that we can no longer deny it to ourselves will we adjust our mental models of the world—producing a clearer picture of reality.

Like many hardball operators before and since, Vladimir Lenin insisted politics, defined broadly, was nothing more than a struggle for power, or as he memorably put it, “kto, kogo?” That literally means “who, whom” and it was Lenin’s shorthand for “Who does what to whom?” Arguments and evidence are lovely adornments but what matters is the ceaseless contest to be the kto, not the kogo. 6 It follows that the goal of forecasting is not to see what’s coming. It is to advance the interests of the forecaster and the forecaster’s tribe.

A century ago, as physicians were slowly professionalizing and medicine was on the cusp of becoming scientific, a Boston doctor named Ernest Amory Codman had an idea similar in spirit to forecaster scorekeeping. He called it the End Result System.

Evidence-based policy is a movement modeled on evidence-based medicine, with the goal of subjecting government policies to rigorous analysis so that legislators will actually know—not merely think they know—whether policies do what they are supposed to do. As a result, in the United States, the United Kingdom, and elsewhere, there is probably more top-quality policy analysis being done than ever before. Of course politics will always be politics and politicians will always factor in partisan advantage and ideological conviction, but there is plenty of evidence that rigorous analysis has made a real difference in government policy.

What would help is a sweeping commitment to evaluation: Keep score. Analyze results. Learn what works and what doesn

In 1976 I was a clueless twenty-two-year-old Canadian who, like countless others, was about to make choices that would shape the rest of my life. I had just graduated from the University of British Columbia.

Implicit within Paul Saffo’s “How does this all turn out?” question were the recent events that had worsened the conflict on the Korean peninsula. North Korea launched a rocket, in violation of a UN Security Council resolution. It conducted a new nuclear test. It renounced the 1953 armistice with South Korea. It launched a cyber attack on South Korea, severed the hotline between the two governments, and threatened a nuclear attack on the United States. Seen that way, it’s obvious that the big question is composed of many small questions. One is “Will North Korea test a rocket?” If it does, it will escalate the conflict a little. If it doesn’t, it could cool things down a little. That one tiny question doesn’t nail down the big question, but it does contribute a little insight.

I call this Bayesian question clustering because of its family resemblance to the Bayesian updating discussed in chapter 7. Another way to think of it is to imagine a painter using the technique called pointillism.

Superforecasters and superquestioners need to acknowledge each other’s complementary strengths, not dwell on each other’s alleged weaknesses.

That’s a tall order. But there’s a much bigger collaboration I’d like to see. It would be the Holy Grail of my research program: using forecasting tournaments to depolarize unnecessarily polarized policy debates and make us collectively smarter.

Bill Flack is perpetual beta.