Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

superforecasters are always just a System 2 slipup away from a blown forecast and a nasty tumble down the rankings. Kahneman and I agree about that. But I am more optimistic that smart, dedicated people can inoculate themselves to some degree against certain cognitive illusions. That may sound like a tempest in an academic teapot, but it has real-world implications.

superforecasters not only paid attention to the time frame in the question but also thought about other possible time frames—and thereby shook off a hard-to-shake bias. I wish I could take credit for that. Our advanced training guidelines urge forecasters to mentally tinker with “the question asked” and explore how their answers to a timing question might change if the cutoff date were six months out instead of twelve, or if the target price for oil were 10% lower, or some other relevant variation.

No matter how physically or cognitively demanding a task may be—cooking, sailing, surgery, operatic singing, flying fighter jets—deliberative practice can make it second nature. Ever watch a child struggling to sound out words and grasp the meaning of a sentence? That was you once. Fortunately, reading this sentence isn’t nearly so demanding for you now.

very different way is to beat competitors by forecasting more accurately—for example, correctly deciding that there is a 68% chance of something happening when others foresee only a 60% chance. This is the approach of the best poker players. It pays off more often, but the returns are more modest, and fortunes are amassed slowly.

ALL THAT SAID… I see Kahneman’s and Taleb’s critiques as the strongest challenges to the notion of superforecasting. We are far enough apart empirically and close enough philosophically to make communication, even collaboration, possible.

On April 11, 2001, Defense Secretary Donald Rumsfeld sent a memo to President George W. Bush and Vice President Dick Cheney. “I ran across this piece on the difficulty of predicting the future,” Rumsfeld wrote. “I thought you might find it interesting.”9 The “piece” looks at the strategic situation at the start of each decade between 1900 and 2000 and shows that, in every case, the reality was a stunning change from ten years earlier. “All of which is to say that I’m not sure what 2010 will look like,” concluded its author, Linton Wells, “but I’m sure that it will be very little like what we expect, so we should plan accordingly.”

Taleb, Kahneman, and I agree there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious—“there will be conflicts”—and the odd lucky hits that are inevitable whenever lots of forecasters make lots of forecasts. These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems.

If you have to plan for a future beyond the forecasting horizon, plan for surprise. That means, as Danzig advises, planning for adaptability and resilience. Imagine a scenario in which reality gives you a smack in the ear and consider how you would respond. Then assume reality will give you a kick in the shin and think about dealing with that. “Plans are useless,” Eisenhower said about preparing for battle, “but planning is indispensable.”11 Taleb has taken this argument further and called for critical systems—like international banking and nuclear weapons—to be made “antifragile,” meaning they are not only resilient to shocks but strengthened by them. In principle, I agree. But a point often overlooked is that preparing for surprises—whether we are shooting for resilience or antifragility—is costly.

Now comes the hardest-to-grasp part of Taleb’s view of the world. He posits that historical probabilities—all the possible ways the future could unfold—are distributed like wealth, not height. That means our world is vastly more volatile than most of us realize and we are at risk of grave miscalculations.

What would have happened if the policy maker had relied on a more realistic fat-tailed distribution of war casualties? He still would have seen the forecast as improbable, but it would now be thousands of times more probable than before.14 The impact would be analogous to your learning that your personal chances of winning the Powerball lottery on any ticket purchase have risen from one in five million to one in five hundred. Wouldn’t you rush to buy tickets? A policy maker in 1914 who knew the true fat-tail risks of a mega-casualty war might well have tried a lot harder to avert the looming catastrophe.

Daniel Kahneman makes the point with a characteristically elegant thought experiment. He invites us to consider three leaders whose impact on the twentieth century was massive—Hitler, Stalin, and Mao. Each came to power backed by a political movement that would never have accepted a female leader, but each man’s origins can be traced to an unfertilized egg that had a 50% chance of being fertilized by different sperm cells and producing a female zygote that would become a female fetus and finally a female baby. That means there was only a 12.5% chance that all three leaders would be born male, and an 87.5% chance that at least one would be born female. The ripple effects of different results are unknowable but potentially enormous. If Anna Hitler had been born on April 20, 1889, in Braunau am Inn, Austria, World War II might never have happened—or a smarter Nazi dictator might have visited even worse horrors upon us by making shrewder decisions.

Counterfactuals highlight how radically open the possibilities once were and how easily our best-laid plans can be blown away by flapping butterfly wings.

Savoring how history could have generated an infinite array of alternative outcomes and could now generate a similar array of alternative futures, is like contemplating the one hundred billion known stars in our galaxy and the one hundred billion known galaxies. It instills profound humility.16 Kahneman, Taleb, and I agree on that much. But I also believe that humility should not obscure the fact that people can, with considerable effort, make accurate forecasts about at least some developments that really do matter. To be sure, in the big scheme of things, human foresight is puny, but it is nothing to sniff at when you live on that puny human scale.

Vague expectations about indefinite futures are not helpful. Fuzzy thinking can never be proven wrong. And only when we are proven wrong so clearly that we can no longer deny it to ourselves will we adjust our mental models of the world—producing a clearer picture of reality. Forecast, measure, revise: it is the surest path to seeing better.

Earlier, I discussed Jonathan Schell’s 1982 warning that a holocaust would certainly occur in the near future “unless we rid ourselves of our nuclear arsenals,” which was clearly not an accurate forecast. Schell wanted to rouse readers to join the swelling nuclear disarmament movement. He did. So his forecast was not accurate, but did it fail? Lenin would say it did exactly what it was supposed to do.

Evidence-based policy is a movement modeled on evidence-based medicine, with the goal of subjecting government policies to rigorous analysis so that legislators will actually know—not merely think they know—whether policies do what they are supposed to do. As a result, in the United States, the United Kingdom, and elsewhere, there is probably more top-quality policy analysis being done than ever before.

It would be another manifestation of a broad and deep shift away from decision making based on experience, intuition, and authority—“Do this because I think it will work and I’m an expert”—toward quantification and analysis. Far from being surprising, one might even think, “What took so long?”

“Not everything that counts can be counted,” goes a famous saying, “and not everything that can be counted counts.”13 In this era of computers and algorithms, some social scientists have forgotten that.

Far too many people treat numbers like sacred totems offering divine insight. The truly numerate know that numbers are tools, nothing more, and their quality can range from wretched to superb. A crude version of Codman’s End Results System that simply tracked patient survival might result in a hospital boasting that 100% of its patients live—without mentioning that the hospital achieves this bragging right by turning away the sickest patients. Numbers must be constantly scrutinized and improved, which can be an unnerving process because it is unending. Progressive improvement is attainable. Perfection is not.

my best scientific guess is that they often are not. The psychological recipe for the ideal superforecaster may prove to be quite different from that for the ideal superquestioner, as superb question generation often seems to accompany a hedgehog-like incisiveness and confidence that one has a Big Idea grasp of the deep drivers of an event. That’s quite a different mindset from the foxy eclecticism and sensitivity to uncertainty that characterizes superb forecasting.