Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

What did we find? Regular forecasters scored higher on intelligence and knowledge tests than about 70% of the population. Superforecasters did better, placing higher than about 80% of the population. Note three things. First, the big jumps in intelligence and knowledge are from the public to the forecasters, not from forecasters to superforecasters. Second, although superforecasters are well above average, they did not score off-the-charts high and most fall well short of so-called genius territory, a problematic concept often arbitrarily defined as the top 1%, or an IQ of 135 and up.

In personality psychology, one of the “Big Five” traits is “openness to experience,” which has various dimensions, including preference for variety and intellectual curiosity. It’s unmistakable in many superforecasters. Most people who are not from Ghana would find a question like “Who will win the presidential election in Ghana?” pointless. They wouldn’t know where to start, or why to bother. But when I put that hypothetical question to Doug Lorch and asked for his reaction, he simply said, “Well, here’s an opportunity to learn something about Ghana.”16

Doug knows that when people read for pleasure they naturally gravitate to the like-minded. So he created a database containing hundreds of information sources—from the New York Times to obscure blogs—that are tagged by their ideological orientation, subject matter, and geographical origin, then wrote a program that selects what he should read next using criteria that emphasize diversity. Thanks to Doug’s simple invention, he is sure to constantly encounter different perspectives. Doug is not merely open-minded. He is actively open-minded. Active open-mindedness (AOM) is a concept coined by the psychologist Jonathan Baron, who has an office next to mine at the University of Pennsylvania. Baron’s test for AOM asks whether you agree or disagree with statements like: People should take into consideration evidence that goes against their beliefs. It is more useful to pay attention to those who disagree with you than to pay attention to those who agree. Changing your mind is a sign of weakness. Intuition is the best guide in making decisions. It is important to persevere in your beliefs even when evidence is brought to bear against them.

For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded.

PULLING IT ALL TOGETHER We have learned a lot about superforecasters, from their lives to their test scores to their work habits. Taking stock, we can now sketch a rough composite portrait of the modal superforecaster. In philosophic outlook, they tend to be: CAUTIOUS: Nothing is certain HUMBLE: Reality is infinitely complex NONDETERMINISTIC: What happens is not meant to be and does not have to happen In their abilities and thinking styles, they tend to be: ACTIVELY OPEN-MINDED: Beliefs are hypotheses to be tested, not treasures to be protected INTELLIGENT AND KNOWLEDGEABLE, WITH A “NEED FOR COGNITION”: Intellectually curious, enjoy puzzles and mental challenges REFLECTIVE: Introspective and self-critical NUMERATE: Comfortable with numbers In their methods of forecasting they tend to be: PRAGMATIC: Not wedded to any idea or agenda ANALYTICAL: Capable of stepping back from the tip-of-your-nose perspective and considering other views DRAGONFLY-EYED: Value diverse views and synthesize them into their own PROBABILISTIC: Judge using many grades of maybe THOUGHTFUL UPDATERS: When facts change, they change their minds GOOD INTUITIVE PSYCHOLOGISTS: Aware of the value of checking thinking for cognitive and emotional biases In their work ethic, they tend to have: A GROWTH MINDSET: Believe it’s possible to get better GRIT: Determined to keep at it however long it takes