Superforecasting Quotes

“There is simply no way that any individual could cover as much ground as a good team does. Even if you had unlimited hours, it would be less fruitful, given different research styles. Each team member brings something different.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“explore the similarities and differences between your views and those of others—and pay special attention to prediction markets and other Methods of extracting wisdom from crowds. Synthesize all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgment as precisely as you can, using a finely grained scale of probability.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“It was the earliest demonstration of a phenomenon popularized by—and now named for—James Surowiecki’s bestseller The Wisdom of Crowds. Aggregating the judgment of many consistently beats the accuracy of the average member of the group, and is often as startlingly accurate as Galton’s weight-guessers.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Popular books often draw a dichotomy between intuition and analysis—“blink” versus “think”—and pick one or the other as the way to go. I am more of a thinker than a blinker, but blink-think is another false dichotomy. The choice isn’t either/or, it is how to blend them in evolving situations.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“then you go through a long life like a one-legged man in an ass-kicking contest.”23”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Assemble forecasters. Ask them large numbers of questions with precise time frames and unambiguous language. Require that forecasts be expressed using numerical probability scales. And wait for”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“We also gave teams a primer on teamwork based on insights gleaned from research in group dynamics. On the one hand, we warned, groupthink is a danger. Be cooperative but not deferential. Consensus is not always good; disagreement is not always bad. If you do happen to agree, don't take the agreement—in itself—as proof that you are right. Never stop doubting. Pointed questions are as essential to a team as vitamins are to a human body.

On the other hand, the opposite of groupthink—rancor and dysfunction—is also a danger. Team members must disagree without being disagreeable, we advised. Practice 'constructive confrontation' to use the phrase of Andy Grove, the former CEO of Intel. Precision questioning is one way to do that. Drawing on the work of Dennis Matthies and Monica Worline, we showed them how to tactfully dissect the vague claims people often make. Suppose someone says, 'Unfortunately, the popularity of soccer, the world's favorite pastime, is starting to decline.' You suspect [they] are wrong. ... Zero in. You might say, 'What do you mean by 'pastime?' or 'What evidence is there that soccer's popularity is declining? Over what time frame.' The answers to these precise questions won't settle the matter, but they will reveal the thinking behind the conclusion so it can be probed and tested.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Maxims Hidden in the Text

Try, fail, analyze, adjust, try again. John Maynard Keynes cycled through these steps ceaselessly. 178

An imperfect decision made in time is better than a perfect one made too late. 215-216

Plans are merely a platform for change. Israeli Defense Forces slogan. 222

If we ask many tiny pertinent questions, we can close in on an answer for the big question. 263”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“What make the difference is correctly identifying and responding to subtler information so you zero in on the eventual outcome faster than others.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Researchers have found that merely asking people to assume their initial judgment is wrong, to seriously consider why that might be, and then make another judgment, produces a second estimate which, when combined with the first, improves accuracy almost as much as getting a second estimate from another person.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Need for cognition" is the psychological term for the tendency to engage in and enjoy hard mental slogs. [...] superforecasters score high in need-for-cognition tests.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“slow regression is more often seen in activities dominated by skill, while faster regression is more associated with chance.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Mauboussin notes that slow regression is more often seen in activities dominated by skill, while faster regression is more associated with chance.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Most things in life involve skill and luck, in varying proportions. The mix may be almost all luck and a little skill, or almost all skill and a little luck, or it could be one of a thousand other possible variations. That complexity makes it hard to figure out what to chalk up to skill and what to”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“A single dot on a canvas is not a painting and a single bet cannot resolve a complex theoretical dispute. This will take many questions and question clusters. Of course it’s possible that if large numbers of questions are asked, each side may be right on some forecasts but wrong on others and the final outcome won’t generate the banner headlines that celebrity bets sometimes do. But as software engineers say, that’s a feature, not a bug. A major point of view rarely has zero merit, and if a forecasting contest produces a split decision we will have learned that the reality is more mixed than either side thought. If learning, not gloating, is the goal, that is progress.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“When a debate like “Keynesians versus Austerians” emerges, key figures could work together—with the help of trusted third parties—to identify what they disagree about and which forecasts would meaningfully test those disagreements. The key is precision. It’s one thing for Austerians to say a policy will cause inflation and Keynesians to say it won’t. But how much inflation? Measured by what benchmark? Over what time period? The result must be a forecast question that reduces ambiguity to an absolute minimum.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“What qualifies as a good question? It’s one that gets us thinking about something worth thinking about. So one way to identify a good question is what I call the smack-the-forehead test: when you read the question after time has passed, you smack your forehead and say, “If only I had thought of that before!”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“A century ago, as physicians were slowly professionalizing and medicine was on the cusp of becoming scientific, a Boston doctor named Ernest Amory Codman had an idea similar in spirit to forecaster scorekeeping. He called it the End Result System. Hospitals should record what ailments incoming patients had, how they were treated, and—most important—the end result of each case. These records should be compiled and statistics released so consumers could choose hospitals on the basis of good evidence. Hospitals would respond to consumer pressure by hiring and promoting doctors on the same basis. Medicine would improve, to the benefit of all. “Codman’s plan disregarded a physician’s clinical reputation or social standing as well as bedside manner or technical skills,” noted the historian Ira Rutkow. “All that counted were the clinical consequences of a doctor’s effort.”8 Today, hospitals do much of what Codman demanded, and more, and physicians would find it flabbergasting if anyone suggested they stop. But the medical establishment saw it differently when Codman first proposed the idea.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“effective learning from experience can’t happen without clear feedback, and you can’t have clear feedback unless your forecasts are unambiguous and scorable. Sound familiar? It should. Drezner cited an article about the IARPA tournament.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Counterfactuals highlight how radically open the possibilities once were and how easily our best-laid plans can be blown away by flapping butterfly wings. Immersion in what-if history can give us a visceral feeling for Taleb’s vision of radical indeterminacy.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Taleb, Kahneman, and I agree there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious—“there will be conflicts”—and the odd lucky hits that are inevitable whenever lots of forecasters make lots of forecasts. These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out. And yet, this sort of forecasting is common, even within institutions that should know better. Every four years, Congress requires the Department of Defense to produce a twenty-year forecast of the national security environment.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“The more you want to explain about a black swan event like the storming of the Bastille,” wrote the sociologist Duncan Watts, “the broader you have to draw the boundaries around what you consider to be the event itself.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“How skillfully leaders perform this balancing act determines how successfully their organizations can cultivate superteams that can replicate the balancing act down the chain of command. And this is not something that one isolated leader can do on his own. It requires a wider willingness to hear unwelcome words from others—and the creation of a culture in which people feel comfortable speaking such words.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“what is often forgotten is that the Nazis did not create the Wehrmacht. They inherited it. And it could not have been more different from the unthinking machine we imagine—as the spectacular attack on the Belgian fortress of Eben Emael demonstrated.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“In Germany’s war academies, scenarios were laid out and students were invited to suggest solutions and discuss them collectively. Disagreement was not only permitted, it was expected, and even the instructor’s views could be challenged because he “understood himself to be a comrade among others,”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“people realized that excessive politeness was hindering the critical examination of views, so they made special efforts to assure others that criticism was welcome. “Everybody has said, ‘I want push-back from you if you see something I don’t,’ ” said Rosenthal. That made a difference. So did offering thanks for constructive criticism. Gradually, the dancing around diminished.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Drawing on the work of Dennis Matthies and Monica Worline, we showed them how to tactfully dissect the vague claims people often make. Suppose someone says, “Unfortunately, the popularity of soccer, the world’s favorite pastime, is starting to decline.” You suspect he is wrong. How do you question the claim? Don’t even think of taking a personal shot like “You’re silly.” That only adds heat, not light. “I don’t think so” only expresses disagreement without delving into why you disagree. “What do you mean?” lowers the emotional temperature with a question but it’s much too vague. Zero in. You might say, “What do you mean by ‘pastime’?” or “What evidence is there that soccer’s popularity is declining? Over what time frame?” The answers to these precise questions won’t settle the matter, but they will reveal the thinking behind the conclusion so it can be probed and tested.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“members of any small cohesive group tend to maintain esprit de corps by unconsciously developing a number of shared illusions and related norms that interfere with critical thinking and reality testing.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Once we know the outcome of something, that knowledge skews our perception of what we thought before we knew the outcome: that’s hindsight bias.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Research on calibration—how closely your confidence matches your accuracy—routinely finds people are too confident.10 But overconfidence is not an immutable law of human nature. Meteorologists generally do not suffer from it. Neither do seasoned bridge players. That’s because both get clear, prompt feedback.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction