Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

We have all been too quick to make up our minds and too slow to change them.

A defining feature of intuitive judgment is its insensitivity to the quality of the evidence on which the judgment is based.

“What would it take for the answer to be yes? What would it take for it to be no?”

As the legendary investor Charlie Munger sagely observed, “If you don’t get this elementary, but mildly unnatural, mathematics of elementary probability into your repertoire, then you go through a long life like a one-legged man in an ass-kicking contest.”23

So finding meaning in events is positively correlated with well-being but negatively correlated with foresight. That sets up a depressing possibility: Is misery the price of accuracy?

Gaps like that are far from unusual. Research on calibration—how closely your confidence matches your accuracy—routinely finds people are too confident.10 But overconfidence is not an immutable law of human nature. Meteorologists generally do not suffer from it. Neither do seasoned bridge players. That’s because both get clear, prompt feedback. The meteorologist who calls for torrential rain tomorrow will know he was off if he wakes to sunshine. Bridge players, who estimate how many “tricks” they will win, get results at the end of each hand. If their forecasts fail, they know it. That is essential. To learn from failure, we must know when we fail. The baby who flops backward does. So does the boy who skins his knee when he falls off the bike. And the accountant who puts an easy putt into a sand trap. And because they know, they can think about what went wrong, adjust, and try again.

The analogy between forecasting and bicycling is pretty good but, as with all analogies, the fit isn’t perfect. With bike riding, the “try, fail, analyze, adjust, and try again” cycle typically takes seconds. With forecasting, it can take months or years. Plus there is the bigger role of chance in forecasting. Cyclists who follow best cycling practices can usually expect excellent outcomes but forecasters should be more tentative. Following best practices improves their odds of winning but less reliably so than in games where chance plays smaller roles.

PULLING IT ALL TOGETHER We have learned a lot about superforecasters, from their lives to their test scores to their work habits. Taking stock, we can now sketch a rough composite portrait of the modal superforecaster. In philosophic outlook, they tend to be: CAUTIOUS: Nothing is certain HUMBLE: Reality is infinitely complex NONDETERMINISTIC: What happens is not meant to be and does not have to happen In their abilities and thinking styles, they tend to be: ACTIVELY OPEN-MINDED: Beliefs are hypotheses to be tested, not treasures to be protected INTELLIGENT AND KNOWLEDGEABLE, WITH A “NEED FOR COGNITION”: Intellectually curious, enjoy puzzles and mental challenges REFLECTIVE: Introspective and self-critical NUMERATE: Comfortable with numbers In their methods of forecasting they tend to be: PRAGMATIC: Not wedded to any idea or agenda ANALYTICAL: Capable of stepping back from the tip-of-your-nose perspective and considering other views DRAGONFLY-EYED: Value diverse views and synthesize Judge using many grades of maybe THOUGHTFUL UPDATERS: When facts change, they change their minds GOOD INTUITIVE PSYCHOLOGISTS: Aware of the value of checking thinking for cognitive and emotional biases In their work ethic, they tend to have: A GROWTH MINDSET: Believe it’s possible to get better GRIT: Determined to keep at it however long it takes

The strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement. It is roughly three times as powerful a predictor as its closest rival, intelligence. To paraphrase Thomas Edison, superforecasting appears to be roughly 75% perspiration, 25% inspiration.

Groups that get along too well don’t question assumptions or confront uncomfortable facts. So everyone agrees, which is pleasant, and the fact that everyone agrees is tacitly taken to be proof the group is on the right track. We can’t all be wrong, can we? So

And by fostering minicultures that encouraged people to challenge each other respectfully, admit ignorance, and request help.

“Plans are merely a platform for change” was a popular Israeli Defense Forces slogan of the era.

Petraeus also supports sending officers to top universities for graduate education, not to acquire a body of knowledge, although that is a secondary benefit, but to encounter surprises of another kind. “It teaches you that there are seriously bright people out in the world who have very different basic assumptions about a variety of different topics and therefore arrive at conclusions on issues that are very, very different from one’s own and very different from mainstream kind of thinking, particularly in uniform,” Petraeus said. Like encountering shocks on a battlefield, grappling with other ways of thinking trains officers to be mentally flexible. Petraeus speaks from experience. Thirteen years after graduating from West Point, he earned a PhD in international relations from Princeton University, an experience he calls “invaluable.”

But Petraeus sees the divide between doers and thinkers as a false dichotomy. Leaders must be both. “The bold move is the right move except when it’s the wrong move,” he says. A leader “needs to figure out what’s the right move and then execute it boldly.”

Coping with dissonance is hard. “The test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function,” F. Scott Fitzgerald observed in “The Crack-Up.” It requires teasing apart our feelings about the Nazi regime from our factual judgments about the Wehrmacht’s organizational resilience—and to see the Wehrmacht as both a horrific organization that deserved to be destroyed and an effective organization with lessons to teach us.

Probability judgments should be explicit so we can consider whether they are as accurate as they can be. And if they are nothing but a guess, because that’s the best we can do, we should say so. Knowing what we don’t know is better than thinking we know what we don’t.

Forecast, measure, revise: it is the surest path to seeing better.

“There are ‘metrics’ for phenomena that cannot be metrically measured. Numerical values are assigned to things that cannot be captured by numbers.”14 This naive positivism is running rampant, taking over domains it has no business being in. As Wieseltier poetically put it, “Where wisdom once was, quantification will now be.”

Far too many people treat numbers like sacred totems offering divine insight. The truly numerate know that numbers are tools, nothing more, and their quality can range from wretched to superb.

Numbers must be constantly scrutinized and improved, which can be an unnerving process because it is unending. Progressive improvement is attainable. Perfection is not.15

Of course it’s possible that if large numbers of questions are asked, each side may be right on some forecasts but wrong on others and the final outcome won’t generate the banner headlines that celebrity bets sometimes do. But as software engineers say, that’s a feature, not a bug. A major point of view rarely has zero merit, and if a forecasting contest produces a split decision we will have learned that the reality is more mixed than either side thought. If learning, not gloating, is the goal, that is progress.