Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

Since Socrates, good teachers have practiced precision questioning, but still it’s often not used when it’s needed most.

bring in outsiders,

suspend hierarchy, and keep the leader’s views under wraps. There’s also the “premortem,” in which the team is told to assume a course of action has failed and to explain why—which makes team members feel safe to express doubts they may have about the leader’s plan.

Ask people to list the qualities an effective leader must have, or consult the cottage industry devoted to leadership coaching, or examine rigorous research on the subject, and you will find near-universal agreement on three basic points. Confidence will be on everyone’s list. Leaders must be reasonably confident, and instill confidence in those they lead, because nothing can be

accomplished without the belief that it can be. Decisiveness is another essential attribute. Leaders can’t ruminate endlessly. They need to size up the situation, make a decision, and move on. And leaders must deliver a vision—the goal that everyone strives together to achieve.

No plan of operations extends with certainty beyond the first encounter with the enemy’s main strength,

What exactly is a black swan? The stringent definition is something literally inconceivable before it happens.

Now comes the hardest-to-grasp part of Taleb’s view of the world. He posits that historical probabilities—all the possible ways the future could unfold—are distributed like wealth, not height. That means our world is vastly more volatile than most of us realize and we are at risk of grave miscalculations.

Daniel Kahneman makes the point with a characteristically elegant thought experiment. He invites us to consider three leaders whose impact on the twentieth century was massive—Hitler, Stalin, and Mao. Each came to power backed by a political movement that would never have accepted a female leader, but each man’s origins can be traced to an unfertilized

egg that had a 50% chance of being fertilized by different sperm cells and producing a female zygote that would become a female fetus and finally a female baby. That means there was only a 12.5% chance that all three leaders would be born male, and an 87.5% chance that at least one would be born female. The ripple effects of different results are unknowable but potentially enormous. If Anna Hitler had been born on April 20, 1889, in Braunau am Inn, Austria, World War II might never have happened—or a smarter Nazi dictator might have visited even worse horrors upon us by making shrewder decisions.

Counterfactuals highlight how radically open the possibilities once were and how easily our best-laid plans can be blown away by flapping butterfly wings. Immersion in what-if history can give us a visceral feeling for Taleb’s vision of radical indeterminacy. Savoring how history could have generated an infinite array of alternative outcomes and could now generate a similar array of alternative futures, is like contemplating the one hu...

Vague expectations about indefinite futures are not helpful. Fuzzy thinking can never be proven wrong. And only when we are proven wrong so clearly that we can no longer deny it to ourselves will we adjust our mental models of the world—producing a clearer picture of reality. Forecast, measure, revise: it is the surest path to seeing better.

Keep score. Analyze results. Learn what works and what doesn’t.

There are good pundits and bad pundits for my purposes, of course. The bad ones issue their predictions with no supporting arguments, expecting their readers to treat their pronouncements like the word from Mt. Sinai; or they back their forecasts with anecdotes rather than useful facts. The good ones argue the case for their forecasts; in fact, I see them as functioning something like lawyers in an adversarial judicial system: they put forth the best argument they can for why X is going to happen, and I consider everybody’s arguments, dig further into background as necessary, and come up with my own forecast as a weighted sum of theirs.