Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs.

They’re also remarkably numerate.

They also tend to be newsjunkies who stay on top of the latest developments and regularly update their forecasts,

matters most is how the forecaster thinks.

superforecasting demands thinking that is open-minded, careful, curious, and—above all—self-critical. It also demands focus. The kind of thinking that produces superior judgment does not come effortlessly. Only the determined can deliver it reasonably consistently, which is why our analyses have consistently found commitment to self-improvement to be the strongest predictor of performance.

“So what I want is that human expert paired with a computer to overcome the human cognitive limitations and biases.”14

If Ferrucci is right—I suspect he is—we will need to blend computer-based forecasting and subjective judgment in the future. So it’s time we got serious about both.

What medicine lacked was doubt. “Doubt is not a fearful thing,” Feynman observed, “but a thing of very great value.”10 It’s what propels science forward.

When the scientist tells you he does not know the answer, he is an ignorant man. When he tells you he has a hunch about how it is going to work, he is uncertain about it. When he is pretty sure of how it is going to work, and he tells you, “This is the way it’s going to work, I’ll bet,” he still is in some doubt. And it is of paramount importance, in order to make progress, that we recognize this ignorance and this doubt. Because we have the doubt, we then propose looking in new directions for new ideas. The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test.11 It was the absence of doubt—and scientific rigor—that made medicine unscientific and caused it to stagnate for so long.

scientists who promoted the modernization

The politicians would be blind men arguing over the colors of the rainbow.

And introspection can only capture a tiny fraction of the complex processes whirling inside your head—and behind your decisions.

A defining feature of intuitive judgment is its insensitivity to the quality of the evidence on which the judgment is based.

These tacit assumptions are so vital to System 1 that Kahneman gave them an ungainly but oddly memorable label: WYSIATI (What You See Is All There Is).14

“It is wise to take admissions of uncertainty seriously,” Daniel Kahneman noted, “but declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”

attribute substitution, but I call it bait and switch:

Only you can see the world from the tip of your own nose. So let’s call it the tip-of-your-nose perspective.

“Without those opportunities to learn, a valid intuition can only be due to a lucky accident or to magic,” Kahneman and Klein conclude, “and we do not believe in magic.”22

collect forecasts, judge their accuracy, add the numbers. That’s it. In no time, we’ll know how good Tom Friedman really is.

Kent wasn’t impressed. “I’d rather be a bookie than a goddamn poet,” was his legendary response.14

To do that, we have to interpret the meaning of the Brier scores, which requires two more things: benchmarks and comparability.

Nor was it what they thought—whether they were liberals or conservatives, optimists or pessimists. The critical factor was how they thought.

The other group consisted of more pragmatic experts who drew on many analytical tools, with the choice of tool hinging on the particular problem they faced. These experts gathered as much information from as many sources as they could. When thinking, they often shifted mental gears, sprinkling their speech with transition markers such as “however,” “but,” “although,” and “on the other hand.” They talked about possibilities and probabilities, not certainties. And while no one likes to say “I was wrong,” these experts more readily admitted it and changed their minds.

“The fox knows many things but the hedgehog knows one big thing.”

Foxes beat hedgehogs.

Foxes beat hedgehogs on both calibration and resolution. Foxes had real foresight. Hedgehogs didn’t.

Animated by a Big Idea, hedgehogs tell tight, simple, clear stories that grab and hold audiences. As anyone who has done media training knows, the first rule is “keep it simple, stupid.”

Now look at how foxes approach forecasting. They deploy not one analytical idea but many and seek out information not from one source but many. Then they synthesize it all into a single conclusion. In a word, they aggregate. They may be individuals working alone, but what they do is, in principle, no different from what Galton’s crowd did. They integrate perspectives and the information contained within them. The only real difference is that the process occurs within one skull.

In the original Star Trek TV series, Mr. Spock was the unflappably logical Vulcan, Dr. McCoy was the hot-headed human, and Captain Kirk was the synthesis of the two.

Stepping outside ourselves and really getting a different view of reality is a struggle.

they tend to engage in the hard work of consulting other perspectives.

My fox/hedgehog model is not a dichotomy. It is a spectrum.

“hybrids”—“fox-hedgehogs,”

“hedgehog-...

a situation like that tempts us with a bait and switch: replace the tough question with the easy one, answer it, and then sincerely believe that we have answered the tough question.

This is not complicated stuff. But it’s easy to misinterpret randomness.

Randomness is invisible from the tip-of-your-nose perspective.

Langer called this the “illusion of control,” but it is also an “illusion of prediction.”

Most things in life involve skill and luck, in varying proportions.

The Success Equation. But as Mauboussin noted, there is an elegant rule of thumb that applies to athletes and CEOs, stock analysts and superforecasters. It involves “regression to the mean.”

Some statistical concepts are both easy to understand and easy to forget. Regression to the mean is one of them.

Regular forecasters scored higher on intelligence and knowledge tests than about 70% of the population. Superforecasters did better, placing higher than about 80% of the population. Note three things. First, the big jumps in intelligence and knowledge are from the public to the forecasters, not from forecasters to superforecasters. Second, although superforecasters are well above average, they did not score off-the-charts high and most fall well short of so-called genius territory, a problematic concept often arbitrarily defined as the top 1%, or an IQ of 135 and up.

Ultimately, it’s not the crunching power that counts. It’s how you use it.

The more sophisticated forecaster knows about confirmation bias and will seek out evidence that cuts both ways.

and superforecasters score high in need-for-cognition tests. An element of personality

is also likely involved. In personality psychology, one of the “Big Five” traits is “openness to experience,”

He is actively open-minded. Active open-mindedness (AOM) is a concept coined by the psychologist Jonathan Baron, who has an office next to mine at the University of Pennsylvania. Baron’s test for AOM asks whether you agree or disagree with statements like: People should take into consideration evidence that goes against their beliefs. It is more useful to pay attention to those who disagree with you than to pay attention to those who agree. Changing your mind is a sign of weakness. Intuition is the best guide in making decisions. It is important to persevere in your beliefs even when evidence is brought to bear against them. Quite predictably, superforecasters score highly on Baron’s test. But more important, superforecasters illustrate the concept. They walk the talk. For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded.

When Bill Flack is asked to forecast something like currency exchange rates he will go into historical changes in the rate and build a Monte Carlo model based on that. That’s

“Nothing is one hundred percent,”

he said several times during our interview. The real Leon Panetta thinks like a superforecaster.