Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

If a meteorologist says there is a 70% chance of rain and it doesn’t rain, is she wrong? Not necessarily. Implicitly, her forecast also says there is a 30% chance it will not rain. So if it doesn’t rain, her forecast may have been off, or she may have been exactly right. It’s not possible to judge with only that one forecast in hand. The only way to know for sure would be to rerun the day hundreds of times. If it rained in 70% of those reruns, and didn’t rain in 30%, she would be bang on. Of course we’re not omnipotent beings, so we can’t rerun the day—and we can’t judge.

This particular bait and switch—replacing “Was it a good decision?” with “Did it have a good outcome?”—is both popular and pernicious. Savvy poker players see this mistake as a beginner’s blunder. A novice may overestimate the probability that the next card will win her the hand, bet big, get lucky, and win, but winning doesn’t retroactively make her foolish bet wise. Conversely, a pro may correctly see that there is a high probability of winning the hand, bet big, get unlucky, and lose, but that doesn’t mean her bet was unwise. Good poker players, investors, and executives all understand this. If they don’t, they can’t remain good at what they do—because they will draw false lessons from experience, making their judgment worse over time.