Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

Soros’s most famous bet, shorting the British pound in 1992, which earned him an estimated $1.1 billion,

After World War II, Eisenhower was immensely popular and both parties all but begged him to accept their nominations for the presidency. President Truman even offered to step aside for him. Eisenhower always refused. He sincerely did not want the job. But as the race for the 1952 election shaped up, it became clear that the Republican nomination, and likely the presidency, would be won by an isolationist who promised to bring all American troops home to “the Gibraltar of freedom.” Eisenhower thought this would be a catastrophe that he—and he alone—could stop. “He wanted what was best for his country,” one biographer wrote, “and in the end he decided that he was the best and would have to serve.”29 Clearly, he had no shortage of self-regard.

The humility required for good judgment is not self-doubt—the sense that you are untalented, unintelligent, or unworthy. It is intellectual humility. It is a recognition that reality is profoundly complex, that seeing things clearly is a constant struggle, when it can be done at all, and that human judgment must therefore be riddled with mistakes.

the Wehrmacht? Other organizations illustrate how thinking like a superforecaster can improve leader performance. So why make the point with an army that served the evilest cause in modern history?

The Wehrmacht was Hitler’s army but was created before him

There is no divinely mandated link between morality and competence. If the Puritan poet John Milton could portray Satan as both evil and resourceful in Paradise Lost, is it too much to acknowledge the same could be true of the Wehrmacht?

“The test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function,” F. Scott Fitzgerald observed in “The Crack-Up.”

the Defense Intelligence Agency (DIA), the Pentagon’s equivalent of the CIA with seventeen thousand employees.

the famous Müller-Lyer optical illusion illustrates:

scope insensitivity. Kahneman first documented scope insensitivity thirty years ago when he asked a randomly selected group in Toronto, the capital city of Ontario, how much they would be willing to pay to clean up the lakes in a small region of the province. On average, people said about $10.3 Kahneman asked another randomly selected group how much they would be willing to pay to clean up every one of the 250,000 lakes in Ontario. They too said about $10. Later research got similar results.

The time frame would make no difference to the final answers, just as it made no difference whether 2,000, 20,000, or 200,000 migratory birds died. Mellers ran several studies and found that, exactly as Kahneman expected, the vast majority of forecasters were scope insensitive. Regular forecasters said there was a 40% chance Assad’s regime would fall over three months and a 41% chance it would fall over six months.

Time frame makes a huge difference in forecasting and we must ensure that we aren’t scope insensitive to it

No matter how physically or cognitively demanding a task may be—cooking, sailing, surgery, operatic singing, flying fighter jets—deliberative practice can make it second nature.

But 9/11 was not unimaginable. In 1994 a plot to hijack a jet and crash it into the Eiffel Tower was broken up. In 1998 the US Federal Aviation Administration examined a scenario in which terrorists hijacked FedEx cargo planes and crashed them into the World Trade Center.

Or consider that an average of 1% annual global economic growth in the nineteenth century and 2% in the twentieth turned the squalor of the eighteenth century and all the centuries that preceded it into the unprecedented wealth of the twenty-first.7 History does sometimes jump. But it also crawls, and slow, incremental change can be profoundly important.

This is wild - global economic growth only grew by 1 and 2 percent in the last couple centuries - you’d think it would be way more

On July 14, 1789, a mob took control of a prison in Paris known as the Bastille, but we mean something much bigger than what happened that day when we refer today to “the storming of the Bastille.” We mean the actual event plus the events it triggered that spiraled into the French Revolution. That’s why, centuries later, July 14 is the national holiday of France.

Three days after terrorists flew jets into the World Trade Center and the Pentagon, the US government demanded that the Taliban rulers of Afghanistan hand over Osama bin Laden and other al-Qaeda terrorists. The Taliban said they would comply if the US government produced satisfactory evidence of al-Qaeda’s guilt. The United States prepared for invasion. Still the Taliban refused to hand over bin Laden. Finally, almost a month after 9/11, the United States attacked. Today, when we refer to the black swan of 9/11, we mean the attacks plus the consequences, which include the invasion of Afghanistan. But that sequence of events could arguably have gone differently. Bin Laden and al-Qaeda were Arabic-speaking foreigners in Afghanistan and the Taliban could have decided that sheltering them just wasn’t worth incurring the wrath of the world’s sole superpower right when they were on the cusp of finally destroying the Northern Alliance, their hated rivals.

Great summary of what transpired post 9/11

“Plans are useless,” Eisenhower said about preparing for battle, “but planning is indispensable.”

Many are sure it will. And it might. But in the 1980s and early 1990s, there was an even more prevalent belief that Japan would soon dominate the global economy, and its subsequent decline should at least give pause to those asserting China’s ascendancy.12 But it often doesn’t because when looking back it seems strange that anyone ever thought Japan would take the lead. Of course Japan would falter! It’s obvious—in hindsight—just as the prediction that China will not seems obvious today.

World War II could have claimed far more than sixty million lives

10 million killed in WWI, 60 million in WWII

Daniel Kahneman makes the point with a characteristically elegant thought experiment. He invites us to consider three leaders whose impact on the twentieth century was massive—Hitler, Stalin, and Mao. Each came to power backed by a political movement that would never have accepted a female leader, but each man’s origins can be traced to an unfertilized egg that had a 50% chance of being fertilized by different sperm cells and producing a female zygote that would become a female fetus and finally a female baby. That means there was only a 12.5% chance that all three leaders would be born male, and an 87.5% chance that at least one would be born female. The ripple effects of different results are unknowable but potentially enormous.

Immersion in what-if history can give us a visceral feeling for Taleb’s vision of radical indeterminacy. Savoring how history could have generated an infinite array of alternative outcomes

Radical indeterminacy- cool term

our expectations of the future are derived from our mental models of how the world works, and every event is an opportunity to learn and improve those models.

Prior to the Scottish referendum, he had thought no would win but he wrote a post about why yes might win instead. So what had he forecast? It wasn’t clear. How should he change his thinking in light of the outcome? That too wasn’t clear.

That’s really interesting - we won’t know how to change our own thinking if we don’t commit ourselves to one discrete prediction over another - with a confidence interval

Boston doctor named Ernest Amory Codman had an idea similar in spirit to forecaster scorekeeping. He called it the End Result System. Hospitals should record what ailments incoming patients had, how they were treated, and—most important—the end result of each case. These records should be compiled and statistics released so consumers could choose hospitals on the basis of good evidence.

Evidence-based medicine

ramparts.

Ramparts = wall of castle, with a high broad top for walking on

This blame-game ping-pong does nothing to make the intelligence community improve. It even prevents the long-term investments necessary to get better at forecasting.12 What would help is a sweeping commitment to evaluation: Keep score. Analyze results. Learn what works and what doesn’t. But that requires numbers, and numbers would leave the intelligence community vulnerable to the wrong-side-of-maybe fallacy, lacking any cover the next time they blow a big call.

North Korea launched a rocket, in violation of a UN Security Council resolution. It conducted a new nuclear test. It renounced the 1953 armistice with South Korea. It launched a cyber attack on South Korea, severed the hotline between the two governments, and threatened a nuclear attack on the United States.

I call this Bayesian question clustering because of its family resemblance to the Bayesian updating discussed in chapter 7.

Friedman became a strong supporter of the invasion in part because his expectations of how things would unfold were presumably quite different from what actually happened.

That’s quite a different mindset from the foxy eclecticism and sensitivity to uncertainty that characterizes superb forecasting.

GP - eclecticism

Friedman poses provocative questions that superforecasters should use to sharpen their foresight; superforecasters generate well-calibrated answers that superquestioners should use to fine-tune and occasionally overhaul their mental models of reality.

the president of the Federal Reserve Bank of Minneapolis, an Austerian, announced publicly that events had shown the Keynesians were closer to the truth, it made headlines—that could have read “Man Changes His Mind in Response to Evidence.”

In the mindset of the 2008 financial crisis

bully pulpit

Term Originally used by Theo Roosevelt to describe the presidency

Remember the “adversarial collaboration” between Daniel Kahneman and Gary Klein?

Great ideal - an adversarial collaboration

But how much inflation? Measured by what benchmark? Over what time period? The result must be a forecast question that reduces ambiguity to an absolute minimum.

Mt. Sinai;

Where Moses purportedly received the 10 commandments

Ten Commandments for Aspiring Superforecasters

Consider Peter Backus, a lonely guy in London, who guesstimated the number of potential female partners in his vicinity by starting with the population of London (approximately six million) and winnowing that number down by the proportion of women in the population (about 50%), by the proportion of singles (about 50%), by the proportion in the right age range (about 20%), by the proportion of university graduates (about 26%), by the proportion he finds attractive (only 5%), by the proportion likely to find him attractive (only 5%), and by the proportion likely to be compatible with him (about 10%). Conclusion: roughly twenty-six women in the pool, a daunting but not impossible search task.1

HAHA! love it

(Joseph Kony)

The leader of the lord’s resistance army in Uganda

Superforecasters are in the habit of posing the outside-view question: How often do things of this sort happen in situations of this sort?

Summers’s strategy: he doubled the employee’s estimate, then moved to the next higher time unit. “So, if the research assistant says the task will take an hour, it will take two days. If he says two days, it will take four weeks.”2 It’s a nerd joke:

For instance, if you are a devout dove who believes that threatening military action never brings peace, be open to the possibility that you might be wrong about Iran. And the same advice applies if you are a devout hawk who believes that soft “appeasement” policies never pay off. Each side should list, in advance, the signs that would nudge them toward the other.

Synthesis is an art that requires reconciling irreducibly subjective judgments. If you do it well, engaging in this process of synthesizing should transform you from a cookie-cutter dove or hawk into an odd hybrid creature, a dove-hawk, with a nuanced view of when tougher or softer policies are likelier to work.

Try to find the central perspective - take into account the side opposite your intuition may be right

Superforecasters understand the risks both of rushing to judgment and of dawdling too long near “maybe.”

They routinely manage the trade-off between the need to take decisive stands (who wants to listen to a waffler?) and the need to qualify their stands (who wants to listen to a blowhard?).

Master the fine arts of team management, especially perspective taking (understanding the arguments of the other side so well that you can reproduce them to the other’s satisfaction),

I like this - ask the other side if you’ve got their argument right