Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

In forecasting, as in other fields, we will continue to see human judgment being displaced—to the consternation of white-collar workers—but we will also see more and more syntheses, like “freestyle chess,” in which humans with computers compete as teams, the human drawing on the computer’s indisputable strengths but also occasionally overriding the computer. The result is a combination that can (sometimes) beat both humans and machines. To reframe the man-versus-machine dichotomy, combinations of Garry Kasparov and Deep Blue may prove more robust than pure-human or pure-machine approaches.

Consider Galen, the second-century physician to Roman emperors. No one has influenced more generations of physicians. Galen’s writings were the indisputable source of medical authority for more than a thousand years. “It is I, and I alone, who has revealed the true path of medicine,” Galen wrote with his usual modesty. And yet Galen never conducted anything resembling a modern experiment. Why should he? Experiments are what people do when they aren’t sure what the truth is. And Galen was untroubled by doubt. Each outcome confirmed he was right, no matter how equivocal the evidence might look to someone less wise than the master. “All who drink of this treatment recover in a short time, except those whom it does not help, who all die,” he wrote. “It is obvious, therefore, that it fails only in incurable cases.”

A defining feature of intuitive judgment is its insensitivity to the quality of the evidence on which the judgment is based. It has to be that way. System 1 can only do its job of delivering strong conclusions at lightning speed if it never pauses to wonder whether the evidence at hand is flawed or inadequate, or if there is better evidence elsewhere.

“Many investors move from stock to stock or from mutual fund to mutual fund as if they were selecting and discarding cards in a game of gin rummy,” observed the Princeton economist Burton Malkiel.13 And they pay a price. Many studies have found that those who trade more frequently get worse returns than those who lean toward old-fashioned buy-and-hold strategies. Malkiel cited one study of sixty-six thousand American households over a five-year period in the 1990s when the market had a 17.9% annual return: households that traded the most had an annual return of only 11.4%.14 Massive time and effort went into those trades and yet the people who made them would have been better off if they had gone golfing.

Leaders must decide, and to do that they must make and use forecasts. The more accurate those forecasts are, the better, so the lessons of superforecasting should be of intense interest to them. But leaders must also act and achieve their goals. In a word, they must lead. And anyone who has led people may have doubts about how useful the lessons of superforecasting really are for leaders.

But look at the style of thinking that produces superforecasting and consider how it squares with what leaders must deliver. How can leaders be confident, and inspire confidence, if they see nothing as certain? How can they be decisive and avoid “analysis paralysis” if their thinking is so slow, complex, and self-critical? How can they act with relentless determination if they readily adjust their thinking in light of new information or even conclude they were wrong? And underlying superforecasting is a spirit of humility—a sense that the complexity of reality is staggering, our ability to comprehend limited, and mistakes inevitable. No one ever described Winston Churchill, Steve Jobs, or any other great leader as “humble.” Well, maybe Gandhi. But try to name a second and a third.

In 1758, when Prussia’s King Frederick the Great battled Russian forces at Zorndorf, he sent a messenger to the youngest Prussian general, Friedrich Wilhelm von Seydlitz, who commanded a cavalry unit. “Attack,” the messenger said. Seydlitz refused. He felt the time wasn’t right and his forces would be wasted. The messenger left but later returned. Again he told Seydlitz the king wanted him to attack. Again Seydlitz refused. A third time the messenger returned and he warned Seydlitz that if he didn’t attack immediately, the king would have his head. “Tell the King that after the battle my head is at his disposal,” Seydlitz responded, “but meanwhile I will make use of it.” Finally, when Seydlitz judged the time right, he attacked and turned the battle in Prussia’s favor. Frederick the Great congratulated his general and let him keep his head. This story, and others like it, notes Muth, “were collective cultural knowledge within the Prussian officer corps, recounted and retold countless times in an abundance of variations during official lectures, in the officer’s mess, or in the correspondence between comrades.” The fundamental message: think. If necessary, discuss your orders. Even criticize them. And if you absolutely must—and you better have a good reason—disobey them.

Shortly after World War I, Eisenhower, then a junior officer who had some experience with the new weapons called tanks, published an article in the US Army’s Infantry Journal making the modest argument that “the clumsy, awkward and snail-like progress of the old tanks must be forgotten, and in their place we must picture this speedy, reliable and efficient engine of destruction.” Eisenhower was dressed down. “I was told my ideas were not only wrong but dangerous, and that henceforth I was to keep them to myself,” he recalled. “Particularly, I was not to publish anything incompatible with solid infantry doctrine. If I did, I would be hauled before a court-martial.”15

There were smart and creative American officers, but they valued individual initiative despite their training, not because of it. George Patton was one. “Never tell people how to do things,” he wrote, succinctly capturing the spirit of Auftragstaktik: “Tell them what to do, and they will surprise you with their ingenuity.

You don’t have to be the head of the DIA to find those reports. Googling “global conflict trends” will suffice. But Flynn saw no need to do so before drawing his sweeping conclusion and sharing it. Why not? The same reason Peggy Noonan saw no need to consult the data on other former presidents’ approval ratings when she judged the meaning of President Bush’s rising appeal. At work in each case was Kahneman’s WYSIATI—What You See Is All There Is—the mother of all cognitive illusions, the egocentric worldview that prevents us from seeing any world beyond the one visible from the tips of our noses. Flynn saw mountains of bad news on his desk every day, and his conclusion felt right—so That Was All There Was. As a lifelong intelligence officer, Flynn knew the importance of checking assumptions, no matter how true they feel, but he didn’t because it didn’t feel like an assumption. It felt true. It’s the oldest trick in the psychological book and Flynn fell for it.

It was an extreme illustration of a problem I swept under the rug in chapter 1, when I said the exclusive goal of forecasting should be accuracy and that would be the sole concern in this book. In reality, accuracy is often only one of many goals. Sometimes it is irrelevant. This unseemly truth is usually disguised but occasionally the mask falls, as it did when an analyst with the Banco Santander Brasil SA warned wealthy clients that Brazil’s stock market and currency would likely decline if a left-wing candidate continued her ascent in the polls. The candidate and her party were livid and demanded the analyst be fired. She was, immediately, because in Brazil it’s not good for a bank to be on bad terms with potential future presidents. Whether the analyst’s forecast was accurate did not matter in the slightest.5