Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

Computer programmers have a wonderful term for a program that is not intended to be released in a final version but will instead be used, analyzed, and improved without end. It is “perpetual beta.” Superforecasters are perpetual beta.

In philosophic outlook, they tend to be: CAUTIOUS: Nothing is certain HUMBLE: Reality is infinitely complex NONDETERMINISTIC: What happens is not meant to be and does not have to happen

In their abilities and thinking styles, they tend to be: ACTIVELY OPEN-MINDED: Beliefs are hypotheses to be tested, not treasures to be protected INTELLIGENT AND KNOWLEDGEABLE, WITH A “NEED FOR COGNITION”: Intellectually curious, enjoy puzzles and mental challenges REFLECTIVE: Introspective and self-critical NUMERATE: Comfortable with numbers

In their methods of forecasting the...

PRAGMATIC: Not wedded to any idea or agenda ANALYTICAL: Capable of stepping back from the tip-of-your-nose perspective and considering other views DRAGONFLY-EYED: Value diverse views and synthesize them into their own PROBABILISTIC: Judge using many grades of maybe THOUGHTFUL UPDATERS: When facts change, they change their minds GOOD INTUITIVE...

In their work ethic, they tend to have: A GROWTH MINDSET: Believe it’s possible to get better GRIT: Determined to...

The predictive power of perpetual beta does suggest, though, that no matter how high one’s IQ it is difficult to compensate for lack of dedication to the personal project of “growing one’s synapses.

Imagine how events might have gone if the Kennedy team had engaged in precision questioning when planning the Bay of Pigs invasion: “So what happens if they’re attacked and the plan falls apart?

This is a familiar paradox: success can lead to acclaim that can undermine the habits of mind that produced the success.

The usual solutions are those the Kennedy administration implemented after the Bay of Pigs invasion—bring in outsiders, suspend hierarchy, and keep the leader’s views under wraps. There’s also the “premortem,” in which the team is told to assume a course of action has failed and to explain why—which makes team members feel safe to express doubts they may have about the leader’s plan.

the aggregation of different perspectives is a potent way to improve judgment, but the key word is different. Combining uniform perspectives only produces more of the same, while slight variation will produce slight improvement.

Confidence

Decisiveness

vision

The fundamental message: think. If necessary, discuss your orders. Even criticize them.

“Once a course of action has been initiated it must not be abandoned without overriding reason,

Right down to junior officers, NCOs, and the lowliest private, soldiers were told what the commander wanted accomplished but were expected to use their judgment about how best to do that given what they were seeing.

“Great success requires boldness and daring, but good judgment must take precedence,

“Never tell people how to do things,” he wrote, succinctly capturing the spirit of Auftragstaktik: “Tell them what to do, and they will surprise you with their ingenuity.”17

But Petraeus sees the divide between doers and thinkers as a false dichotomy. Leaders must be both. “The bold move is the right move except when it’s the wrong move,” he says. A leader “needs to figure out what’s the right move and then execute it boldly.”23

“Have backbone; disagree and commit” is one of Jeff Bezos’s fourteen leadership principles drilled into every new employee at Amazon. It continues: “Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting.

“There is nothing like danger to focus the mind,

The humility required for good judgment is not self-doubt—the sense that you are untalented, unintelligent, or unworthy. It is intellectual humility. It is a recognition that reality is profoundly complex, that seeing things clearly is a constant struggle, when it can be done at all, and that human judgment must therefore be riddled with mistakes. This

Intellectual humility compels the careful reflection necessary for good judgment; confidence in one’s abilities inspires determined action.

resist a bias of particularly deep relevance to forecasting: scope insensitivity.

“Plans are useless,” Eisenhower said about preparing for battle, “but planning is indispensable.”11

Ten Commandments for Aspiring Superforecasters

(1) Triage.

(2) Break seemingly intractable problems into tractable sub-problems.

(3) Strike the right balance between inside and outside views.

(4) Strike the right balance between

under- and overreacting to evidence.

(5) Look for the clashing causal forces at work in each problem.

(6) Strive to distinguish as many degrees of doubt as the problem permits but no more.

(7) Strike the right balance between under- and overconfidence, between prudence and decisiveness.

(8) Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases

don’t forget to do postmortems on your successes too.

(9) Bring out the best in others and let others bring out the best in you

(10) Master the error-balancing bicycle.

remember that practice is not just going through the motions of making forecasts, or casually

reading the news and tossing out probabilities. Like all other known forms of expertise, superforecasting is the product of deep, deliberative practice.

(11) Don’t treat commandments as commandments