Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

The person who believes he is bad at math, and always will be, won’t try hard to improve, because that would be pointless, and if he is compelled to study math—as we all are in school—he will take any setback as further proof that his limits have been revealed and he should stop wasting his time as soon as possible. Whatever potential he had for improvement will never be realized. Thus, the belief “I am bad at math” becomes self-fulfilling.

Even when the fixed-minded try, they don’t get as much from the experience as those who believe they can grow.

“Only people with a growth mindset paid close attention to information that could stretch their knowledge. Only for them was learning a priority.”

To be a top-flight forecaster, a growth mindset is essential.

habit that sometimes required him to change his mind. He did so ungrudgingly. Indeed,

Ungrudgingly = opposite of begrudgingly; I like it

Stock prices do not always reflect the true value of companies, so an investor should study a company thoroughly and really understand its business, capital, and management when deciding whether it had sufficient underlying value to make an investment for the long term worthwhile. In the United States, about the same time, this approach was developed by Benjamin Graham, who called it “value investing.” It became the cornerstone of Warren Buffett’s fortune.

The knowledge required to ride a bicycle can’t be fully captured in words and conveyed to others. We need “tacit knowledge,” the sort we only get from bruising experience. To learn to ride a bicycle, we must try to ride one.

But not all practice improves skill. It needs to be informed practice. You need to know which mistakes to look out for—and which best practices really are best.

The training guidelines help us draw the right lessons from our personal experiences and to strike the right balances between the outside and inside views. And our personal experiences help us infuse pallid public-knowledge abstractions with real-world content.

Pallid= Lacking intensity; We need real world experience AND classroom education

To learn from failure, we must know when we fail. The baby who flops backward does. So does the boy who skins his knee when he falls off the bike.

Consider the Forer effect, named for the psychologist Bertram Forer, who asked some students to complete a personality test, then gave them individual personality profiles based on the results and asked how well the test captured their individual personalities. People were impressed by the test, giving it an average rating of 4.2 out of 5—which was remarkable because Forer had actually taken vague statements like “you have a great need for people to like and admire you” from a book on astrology, assembled them into a profile, and given the same profile to everyone.11 Vague language is elastic language. The students stretched it to fit their self-images, even though they thought they were judging the test objectively.

Once we know the outcome of something, that knowledge skews our perception of what we thought before we knew the outcome: that’s hindsight bias. Baruch Fischhoff was the first to document the phenomenon in a set of elegant experiments.

asked experts to estimate how likely it was that the Communist Party would lose its monopoly on power in the Soviet Union in the next five years. In 1991 the world watched in shock as the Soviet Union disintegrated. So in 1992–93 I returned to the experts, reminded them of the question in 1988, and asked them to recall their estimates.

Hindsight bias can be hugely significant even among experts

People often assume that when a decision is followed by a good outcome, the decision was good, which isn’t always true, and can be dangerous if it blinds us to the flaws in our thinking.13 The successful are not

the coup against Gorbachev in 1991 had

Didn’t know there was a coup against Gorbachev

Grit is passionate perseverance of long-term goals, even in the face of frustration and failure. Married with a growth mindset, it is a potent force for personal progress.

If I gritty and have a growth mindset - I will make tremendous personal progress towards my mission

Why did she do it? For the same reason a college student might take all the toughest courses with the hardest-grading professors: she cared more about learning than getting top grades. “I am always trying to grow, to learn, to change,”

So many students get this wrong and I have at times too - take classes to learn and grow, NOT just because you think you can ace them or you think they look good on a resume

is “perpetual beta.” Superforecasters are perpetual beta.

Taking stock, we can now sketch a rough composite portrait of the modal superforecaster. In philosophic outlook, they tend to be: CAUTIOUS: Nothing is certain HUMBLE: Reality is infinitely complex NONDETERMINISTIC: What happens is not meant to be and does not have to happen In their abilities and thinking styles, they tend to be: ACTIVELY OPEN-MINDED: Beliefs are hypotheses to be tested, not treasures to be protected INTELLIGENT AND KNOWLEDGEABLE, WITH A “NEED FOR COGNITION”: Intellectually curious, enjoy puzzles and mental challenges

The strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement. It is roughly three times as powerful a predictor as its closest rival, intelligence. To paraphrase Thomas Edison, superforecasting appears to be roughly 75% perspiration, 25% inspiration.

the Bay of Pigs invasion.1 When the CIA-trained guerrillas landed, the Cuban army was waiting and the fourteen hundred men onshore were quickly surrounded by twenty thousand soldiers. Within three days they were all dead or taken prisoner.

Ok so in the bay of pigs invasion - it was Cuban exiles that had been trained by the CIA trying to invade Cuba

If the Bay of Pigs was the Kennedy administration’s nadir, the Cuban missile crisis was its zenith, a moment when Kennedy and his team creatively engineered a positive result under extreme pressure.

esprit de corps

A feeling of mutual pride; group bonding

Groups that get along too well don’t question assumptions or confront uncomfortable facts.

So if a secret American plan to invade Cuba without apparent American involvement happens to be published on the front page of the New York Times, the plan can still go ahead—just make sure there are no American soldiers on the beach and deny American involvement. The world will believe it. And if that sounds implausible…well, not to worry, no one in the group has objected, which means everyone thinks it’s perfectly reasonable, so it must be.

So the bay of pigs invasion was reported by the NYT ahead of time - no wonder it failed

Teams can cause terrible mistakes. They can also sharpen judgment and accomplish together what cannot be done alone.

Unanimity among group is often a bad sign - you need to encourage thoughtful disagreement - try to poke holes and imagine what might go wrong with any given plan

Dutch investors in the seventeenth century, who became collectively convinced that a tulip bulb was worth more than a laborer’s annual salary, or American home buyers in 2005, talking themselves into believing that real estate prices could only go up.

On the other hand, the opposite of groupthink—rancor and dysfunction—is also a danger. Team members must disagree without being disagreeable, we advised. Practice “constructive confrontation,” to use the phrase of Andy Grove, the former CEO of Intel.

You might say, “What do you mean by ‘pastime’?” or “What evidence is there that soccer’s popularity is declining? Over what time frame?”

When someone says something that sounds wrong to you - get curious. Ask probing questions so you can test their assumptions.

a familiar paradox: success can lead to acclaim that can undermine the habits of mind that produced the success. Such hubris often afflicts highly accomplished individuals. In business circles, it is called CEO disease.

harangues.

Lengthy and aggressive speech

“Everybody has said, ‘I want push-back from you if you see something I don’t,’ ” said Rosenthal. That made a difference. So did offering thanks for constructive criticism. Gradually, the dancing around diminished.

Tell people I want constructive criticism and thank them when they give it to me

On average, when a forecaster did well enough in year 1 to become a superforecaster, and was put on a superforecaster team in year 2, that person became 50% more accurate.

That’s the strong version of what economists call the efficient market hypothesis (EMH),

Prediction markets are simply markets that trade in predictions, meaning traders buy and sell contracts on specified outcomes—such as “Hillary Clinton will be elected president of the United States in 2016.”

If she wins, it pays out $1. If the contract is currently selling for 40 cents and I think Clinton has a 60% or 70% chance of winning, I should buy.

Because expected value is then 60 or 70 cents against the 40 cent cost

By aggregating all these judgments, the contract price should, in theory, closely track the true probability of Hillary Clinton winning.

superteams resembled the best surgical teams identified by Harvard’s Amy Edmondson, in which the nurse doesn’t hesitate to tell the surgeon he left a sponge behind the pancreas because she knows it is “psychologically safe” to correct higher-ups.

You want to create psychological safety for people lower in the superiority chain so to speak

So did our superteams. One sign of that was linguistic: they said “our” more than “my.”

But Grant’s research shows that the pro-social example of the giver can improve the behavior of others, which helps everyone, including the giver—which explains why Grant has found that givers tend to come out on top.

He wasn’t indiscriminately generous with his time and effort. He was generous in a deliberate effort to change the behavior of others for the benefit of all.

Helmuth von Moltke.

Moltke was famous the world over after he led Prussian forces to victory against Denmark in 1864, Austria in 1866, and France in 1871—victories that culminated in the unification of Germany.

That statement was refined and repeated over the decades and today soldiers know it as “no plan survives contact with the enemy.” That’s much snappier. But notice that Moltke’s original was more nuanced, which is typical

In Germany’s war academies, scenarios were laid out and students were invited to suggest solutions and discuss them collectively. Disagreement was not only permitted, it was expected, and even the instructor’s views could be challenged because he “understood himself to be a comrade among others,”

Hitler, took direct control of operations in violation of Helmuth von Moltke’s principles, nowhere with more disastrous effect than during the invasion of Normandy. The Allies feared that after their troops landed, German tanks would drive them back to the beaches and into the sea, but Hitler had directed that the reserves could only move on his personal command. Hitler slept late. For hours after the Allies landed on the beaches, the dictator’s aides refused to wake him to ask if he wanted to order the tanks into battle. Ironically, a nineteenth-century German general was vindicated by the German defeat

Wow! To think that the world could be completely different if Hitler just didn’t sleep in that day

George Patton was one. “Never tell people how to do things,” he wrote, succinctly capturing the spirit of Auftragstaktik: “Tell them what to do, and they will surprise you with their ingenuity.”17

An Israeli officer commenting on the performance of a division in the 1956 war with Egypt proudly noted that “almost all the plans were foiled during the fighting but all objectives were attained in full—and faster than expected.”

In the northern Iraqi city of Mosul, General David Petraeus, commander of the 101st Airborne, drew on his extensive knowledge of military history to improvise strategies he hoped would “secure and serve” the people of the city and thereby deny the insurgents popular support.

“Have backbone; disagree and commit” is one of Jeff Bezos’s fourteen leadership principles drilled into every new employee at Amazon. It continues: “Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting. Leaders have conviction and are tenacious. They do not compromise for the sake of social cohesion. Once a decision is determined, they commit wholly.”