Superforecasting Quotes

“Like everyone else, scientists have intuitions. Indeed, hunches and flashes of insight—the sense that something is true even if you can’t prove it—have been behind countless breakthroughs. The interplay between System 1 and System 2 can be subtle and creative. But scientists are trained to be cautious. They know that no matter how tempting it is to anoint a pet hypothesis as The Truth, alternative explanations must get a hearing. And they must seriously consider the possibility that their initial hunch is wrong. In fact, in science, the best evidence that a hypothesis is true is often an experiment designed to prove the hypothesis is false, but which fails to do so. Scientists must be able to answer the question “What would convince me I am wrong?” If they can’t, it’s a sign they have grown too attached to their beliefs.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“In one of history’s great ironies, scientists today know vastly more than their colleagues a century ago, and possess vastly more data-crunching power, but they are much less confident in the prospects for perfect predictability.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“The fundamental message: think. If necessary, discuss your orders. Even criticize them. And if you absolutely must—and you better have a good reason—disobey them.4”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“It is wise to take admissions of uncertainty seriously,” Daniel Kahneman noted, “but declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”17”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“statisticians sleeping with their feet in an oven and their head in a freezer because the average temperature is comfortable.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“The one undeniable talent that talking heads have is their skill at telling a compelling story with conviction, and that is enough.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“All who drink of this treatment recover in a short time, except those whom it does not help, who all die,” he wrote. “It is obvious, therefore, that it fails only in incurable cases.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“This approach, built on the “wisdom of the crowd” concept, has been called “the crowd within.” The billionaire financier George Soros exemplifies it. A key part of his success, he has often said, is his mental habit of stepping back from himself so he can judge his own thinking and offer a different perspective—to himself.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“When you combine the judgments of a large group of people to calculate the “wisdom of the crowd” you collect all the relevant information that is dispersed among all those people. But none of those people has access to all that information. One person knows only some of it, another knows some more, and so on.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Like everyone else, scientists have intuitions. Indeed, hunches and flashes of insight—the sense that something is true even if you can’t prove it—have been behind countless breakthroughs. The interplay between System 1 and System 2 can be subtle and creative.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Tell someone they’re exceptionally good at something and they may start taking their superiority for granted. Surround them with others who are similarly accomplished, tell them how special they are, and egos may swell even more. Rather than spur a superforecaster to take his game to the next level, it might make him so sure of himself that he is tempted to think his judgment must be right because it is his judgment. This is a familiar paradox: success can lead to acclaim that can undermine the habits of mind that produced the success.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Once we know the outcome of something, that knowledge skews our perception of what we thought before we knew the outcome: that’s hindsight bias. Baruch Fischhoff was the first to document the phenomenon in a set of elegant experiments.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Scientists must be able to answer the question “What would convince me I am wrong?” If they can’t, it’s a sign they have grown too attached to their beliefs.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Take the price of oil, long a graveyard topic for forecasting reputations.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“There are no certainties in life—not even death and taxes if we assign a nonzero probability to the invention of technologies that let us upload the contents of our brains into a cloud-computing network and the emergence of a future society so public-spirited and prosperous that the state can be funded with charitable donations.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“But when big events happen—markets crash, wars loom, leaders tremble—we turn to the experts, those in the know. We look to people like Tom Friedman.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“We are all forecasters. When we think about changing jobs, getting married, buying a home, making an investment, launching a product, or retiring, we decide based on how we expect the future will unfold.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Consumers of forecasting will stop being gulled by pundits with good stories and start asking pundits how their past predictions fared—and reject answers that consist of nothing but anecdotes and credentials.”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“Thoughts for the 2001 Quadrennial Defense Review If you had been a security policy-maker in the world’s greatest power in 1900, you would have been a Brit, looking warily at your age-old enemy, France. By 1910, you would be allied with France and your enemy would be Germany. By 1920, World War I would have been fought and won, and you’d be engaged in a naval arms race with your erstwhile allies, the U.S. and Japan. By 1930, naval arms limitation treaties were in effect, the Great Depression was underway, and the defense planning standard said ‘no war for ten years.’ Nine years later World War II had begun. By 1950, Britain no longer was the world’s greatest power, the Atomic Age had dawned, and a ‘police action’ was underway in Korea. Ten years later the political focus was on the ‘missile gap,’ the strategic paradigm was shifting from massive retaliation to flexible response, and few people had heard of Vietnam. By 1970, the peak of our involvement in Vietnam had come and gone, we were beginning détente with the Soviets, and we were anointing the Shah as our protégé in the Gulf region. By 1980, the Soviets were in Afghanistan, Iran was in the throes of revolution, there was talk of our ‘hollow forces’ and a ‘window of vulnerability,’ and the U.S. was the greatest creditor nation the world had ever seen. By 1990, the Soviet Union was within a year of dissolution, American forces in the Desert were on the verge of showing they were anything but hollow, the U.S. had become the greatest debtor nation the world had ever known, and almost no one had heard of the internet. Ten years later, Warsaw was the capital of a NATO nation, asymmetric threats transcended geography, and the parallel revolutions of information, biotechnology, robotics, nanotechnology, and high density energy sources foreshadowed changes almost beyond forecasting. All of which is to say that I’m not sure what 2010 will look like, but I’m sure that it will be very little like we expect, so we should plan accordingly. Lin Wells”
― Philip E. Tetlock, Superforecasting: The Art and Science of Prediction

“In the Paleolithic world in which our brains evolved, that’s not a bad way of making decisions. Gathering all evidence and mulling it over may be the best way to produce accurate answers, but a hunter-gatherer who consults statistics on lions before deciding whether to worry about the shadow moving in the grass isn’t likely to live long enough to bequeath his accuracy-maximizing genes to the next generation. Snap judgments are sometimes essential. As Daniel Kahneman puts it, “System 1 is designed to jump to conclusions from little evidence.”13”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“THINKING ABOUT THINKING It is natural to identify our thinking with the ideas, images, plans, and feelings that flow through consciousness. What else could it be? If I ask, “Why did you buy that car?” you can trot out reasons: “Good mileage. Cute style. Great price.” But you can only share thoughts by introspecting; that is, by turning your attention inward and examining the contents of your mind. And introspection can only capture a tiny fraction of the complex processes whirling inside your head—and behind your decisions.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“When the scientist tells you he does not know the answer, he is an ignorant man. When he tells you he has a hunch about how it is going to work, he is uncertain about it. When he is pretty sure of how it is going to work, and he tells you, “This is the way it’s going to work, I’ll bet,” he still is in some doubt. And it is of paramount importance, in order to make progress, that we recognize this ignorance and this doubt. Because we have the doubt, we then propose looking in new directions for new ideas. The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test.11”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“if you have the time to think before making a big decision, do so—and be prepared to accept that what seems obviously true now may turn out to be false later.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“The foundations of our decision making were gravely flawed,” McNamara wrote in his autobiography. “We failed to analyze our assumptions critically, then or later.”5”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“And only when we are proven wrong so clearly that we can no longer deny it to ourselves will we adjust our mental models of the world—producing a clearer picture of reality. Forecast,”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“This is painfully obvious at a poker table. Even weak players know, in principle, that seeing through the eyes of opponents is critical. She raised the bet $20? What does that tell me about her thinking—and the cards she has? Each bet is another clue to what your opponent is holding, or wants you to think she is holding, and the only way to piece it together is to imagine yourself in her seat. Good perspective-takers can make a lot of money. So you might suppose that anyone who takes poker seriously would get good at it, quickly, or take up another hobby. And yet they so often don’t. “Here’s a very simple example,” says Annie Duke, an elite professional poker player, winner of the World Series of Poker, and a former PhD-level student of psychology. “Everyone who plays poker knows you can either fold, call, or raise [a bet]. So what will happen is that when a player who isn’t an expert sees another player raise, they automatically assume that that player is strong, as if the size of the bet is somehow correlated at one with the strength of the other person’s hand.” This is a mistake. Duke teaches poker and to get her students to see like dragonflies she walks them through a game situation. A hand is dealt. You like your cards. In the first of several rounds of betting, you wager a certain amount. The other player immediately raises your bet substantially. Now, what do you think the other player has? Duke has taught thousands of students “and universally, they say ‘I think they have a really strong hand.’” So then she asks them to imagine the same situation, except they’re playing against her. The cards are dealt. Their hand is more than strong—it’s unbeatable. Duke makes her bet. Now, what will you do? Will you raise her bet? “And they say to me, ‘Well, no.’” If they raise, Duke may conclude their hand is strong and fold. They don’t want to scare her off. They want Duke to stay in for each of the rounds of betting so they can expand the pot as much as possible before they scoop it up. So they won’t raise. They’ll only call. Duke then walks them through the same hypothetical with a hand that is beatable but still very strong. Will you raise? No. How about a little weaker hand that is still a likely winner? No raise. “They would never raise with any of these really great hands because they don’t want to chase me away.” Then Duke asks them: Why did you assume that an opponent who raises the bet has a strong hand if you would not raise with the same strong hand? “And it’s not until I walk them through the exercise,” Duke says, that people realize they failed to truly look at the table from the perspective of their opponent. If Duke’s students were all vacationing retirees trying poker for the first time, this would only tell us that dilettantes tend to be naive. But “these are people who have played enough poker, and are passionate about the game, and consider themselves good enough, that they’re paying a thousand dollars for a seminar with me,” Duke says. “And they don’t understand this basic concept.”22”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“but declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“It’s a rare day when a journalist says, “The market rose today for any one of a hundred different reasons, or a mix of them, so no one knows.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“The difference between heavyweights and amateurs, she said, is that the heavyweights know the difference between a 60⁄40 bet and a 40⁄60 bet.”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction

“Also, even when people feel they know nothing, they typically know a bit and that bit should tip them away from maximum uncertainty, at least a bit. The astrophysicist J. Richard Gott shows us what forecasters should do when all they know is how long something—a civil war or a recession or an epidemic—has thus far lasted. The right thing is to adopt an attitude of “Copernican humility” and assume there is nothing special about the point in time at which you happen to be observing the phenomenon. For instance, if the Syrian civil war has been going on for two years when IARPA poses a question about it, assume it is equally likely you are close to the beginning—say, we are only 5% into the war—or close to the end—say, the war is 95% complete. Now you can construct a crude 95% confidence band of possibilities: the war might last as little as 1/39 of 2 years (or less than another month), or as long as about 39 × 2 years, or 78 years. This may not seem to be a great achievement but it beats saying “zero to infinity.” And if 78 years strikes you as ridiculously long that is because you cheated by violating the ground rule of you must know “nothing.” You just introduced outside-view base-rate knowledge about wars in general (e.g., you know that very few wars have ever lasted that long). You are now on the long road to becoming a better forecaster. See Richard Gott, “Implications of the Copernican Principle for Our Future Prospects,” Nature”
― Philip Tetlock, Superforecasting: The Art and Science of Prediction