Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

want to spoil the joke, so I’ll give away the punch line: the average expert was roughly as accurate as a dart-throwing chimpanzee.

How predictable something is depends on what we are trying to predict, how far into the future, and under what circumstances.

There may be limits to such improvements, however, because weather is the textbook illustration of nonlinearity. The further out the forecaster tries to look, the more opportunity there is for chaos to flap its butterfly wings and blow away expectations.

Superforecasting does require minimum levels of intelligence, numeracy, and knowledge of the world, but anyone who reads serious books about psychological research probably has those prerequisites. So what is it that elevates forecasting to superforecasting? As with the experts who had real foresight in my earlier research, what matters most is how the forecaster thinks. I’ll describe this in detail, but broadly speaking, superforecasting demands thinking that is open-minded, careful, curious, and—above all—self-critical. It also demands focus. The kind of thinking that produces superior judgment does not come effortlessly. Only the determined can deliver it reasonably consistently, which is why our analyses have consistently found commitment to self-improvement to be the strongest predictor of performance.

Ferrucci sees light at the end of this long dark tunnel: “I think it’s going to get stranger and stranger” for people to listen to the advice of experts whose views are informed only by their subjective judgment. Human thought is beset by psychological pitfalls, a fact that has only become widely recognized in the last decade or two. “So what I want is that human expert paired with a computer to overcome the human cognitive limitations and biases.”

The standard histories are usually mute on these scores, but when we use modern science to judge the efficacy of historical treatments, it becomes depressingly clear that most of the interventions were useless or worse.

There is no perfection in our messy world. But it beats wise men stroking their chins.

It was cargo cult science, a term of mockery coined much later by the physicist Richard Feynman to describe what happened after American airbases from World War II were removed from remote South Pacific islands, ending the islanders’ only contact with the outside world. The planes had brought wondrous goods. The islanders wanted more. So they “arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the planes to land.”9 But the planes never returned. So cargo cult science has the outward form of science but lacks what makes it truly scientific.

Physicians and the institutions they controlled didn’t want to let go of the idea that their judgment alone revealed the truth, so they kept doing what they did because they had always done it that way—and they were backed up by respected authority. They didn’t need scientific validation. They just knew. Cochrane despised this attitude. He called it “the God complex.”

The politicians would be blind men arguing over the colors of the rainbow.

Why couldn’t people see that intuition alone was no basis for firm conclusions? It was “bewildering.”

It’s a rare day when a journalist says, “The market rose today for any one of a hundred different reasons, or a mix of them, so no one knows.”

“It is wise to take admissions of uncertainty seriously,” Daniel Kahneman noted, “but declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”17

If you didn’t know the punch line of EPJ before you read this book, you do now: the average expert was roughly as accurate as a dart-throwing chimpanzee.

Foxes beat hedgehogs. And the foxes didn’t just win by acting like chickens, playing it safe with 60% and 70% forecasts where hedgehogs boldly went with 90% and 100%. Foxes beat hedgehogs on both calibration and resolution. Foxes had real foresight. Hedgehogs didn’t.

Keep regression to the mean in mind, however, and it becomes a valuable tool.

Harry Truman once joked that he wanted to hear from a one-armed economist because he was sick of hearing “on the one hand…on the other…”—a joke that bears more than a passing resemblance to Tversky’s.

“When the facts change, I change my mind,” the legendary British economist John Maynard Keynes declared. “What do you do, sir?”

Psycho-logic trumps logic. And when Kahan asks people who feel strongly that gun control increases risk, or diminishes it, to imagine conclusive evidence that shows they are wrong, and then asks if they would change their position if that evidence were handed to them, they typically say no. That belief block is holding up a lot of others. Take it out and you risk chaos, so many people refuse to even imagine it.

Their commitment was massive. Warren was, deep down, a civil libertarian. Admitting to himself that he had unjustly imprisoned 112,000 people would have taken a sledgehammer to his mental tower.

superforecasters not only update more often than other forecasters, they update in smaller increments.

The key factor was mindset. Fixed-mindset kids gave up. Growth-mindset kids knuckled down.

One involves the provocative phrase “diversity trumps ability,” coined by my colleague (and former competitor in the IARPA tournament) Scott Page.10 As we have seen, the aggregation of different perspectives is a potent way to improve judgment, but the key word is different. Combining uniform perspectives only produces more of the same, while slight variation will produce slight improvement.

“No plan of operations extends with certainty beyond the first encounter with the enemy’s main strength,” he wrote. That statement was refined and repeated over the decades and today soldiers know it as “no plan survives contact with the enemy.” That’s much snappier. But notice that Moltke’s original was more nuanced, which is typical of his thinking. “It is impossible to lay down binding rules” that apply in all circumstances, he wrote. In war, “two cases never will be exactly the same.” Improvisation is essential.2

What ties all of this together—from “nothing is certain” to “unwavering determination”—is the command principle of Auftragstaktik. Usually translated today as “mission command,” the basic idea is simple. “War cannot be conducted from the green table,” Moltke wrote, using an expression that referred to top commanders at headquarters. “Frequent and rapid decisions can be shaped only on the spot according to estimates of local conditions.”8 Decision-making power must be pushed down the hierarchy so that those on the ground—the first to encounter surprises on the evolving battlefield—can respond quickly.

“Never tell people how to do things,” he wrote, succinctly capturing the spirit of Auftragstaktik: “Tell them what to do, and they will surprise you with their ingenuity.”17

Taleb has taken this argument further and called for critical systems—like international banking and nuclear weapons—to be made “antifragile,” meaning they are not only resilient to shocks but strengthened by them.

The status quo seemed unassailable.

The truly numerate know that numbers are tools, nothing more, and their quality can range from wretched to superb.

would be the Holy Grail of my research program: using forecasting tournaments to depolarize unnecessarily polarized policy debates and make us collectively smarter.