Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

But notice what’s missing? The time frame. It obviously matters. To use an extreme illustration, the probability of the regime falling in the next twenty-four hours must be less—likely a lot less—than the probability that it will fall in the next twenty-four months. To put this in Kahneman’s terms, the time frame is the “scope” of the forecast.

No matter how physically or cognitively demanding a task may be—cooking, sailing, surgery, operatic singing, flying fighter jets—deliberative practice can make it second nature. Ever watch a child struggling to sound out words and grasp the meaning of a sentence? That was you once. Fortunately, reading this sentence isn’t nearly so demanding for you now.

What exactly is a black swan? The stringent definition is something literally inconceivable before it happens. Taleb has implied as much on occasion. If so, many events dubbed black swans are actually gray.

Knowing what we don’t know is better than thinking we know what we don’t.

Kahneman and other pioneers of modern psychology have revealed that our minds crave certainty and when they don’t find it, they impose it. In forecasting, hindsight bias is the cardinal sin.

Now comes the hardest-to-grasp part of Taleb’s view of the world. He posits that historical probabilities—all the possible ways the future could unfold—are distributed like wealth, not height. That means our world is vastly more volatile than most of us realize and we are at risk of grave miscalculations.

In line with Bill Gates’s insistence on having clear goals and measures, the Gates Foundation, one of the world’s largest, is renowned for the rigor of its evaluations.

As the cultural critic Leon Wieseltier put it in the New York Times, “There are ‘metrics’ for phenomena that cannot be metrically measured. Numerical values are assigned to things that cannot be captured by numbers.”14 This naive positivism is running rampant, taking over domains it has no business being in. As Wieseltier poetically put it, “Where wisdom once was, quantification will now be.”

Superforecasters and superquestioners need to acknowledge each other’s complementary strengths, not dwell on each other’s alleged weaknesses.

The catch is that the Kahneman-Klein collaboration presumed good faith. Each side wanted to be right but they wanted the truth more.

They realize that long-term accuracy requires getting good scores on both calibration and resolution—which requires moving beyond blame-game ping-pong. It is not enough just to avoid the most recent mistake. They have to find creative ways to tamp down both types of forecasting errors—misses and false alarms—to the degree a fickle world permits such uncontroversial improvements in accuracy.