Superforecasting: The Art and Science of Prediction

by Philip E. Tetlock

Cochrane cited the Thatcher government’s “short, sharp, shock” approach to young offenders, which called for brief incarceration in spartan jails governed by strict rules. Did it work? The government had simply implemented it throughout the justice system, making it impossible to answer. If the policy was introduced and crime went down, that might mean the policy worked, or perhaps crime went down for any of a hundred other possible reasons. If crime went up, that might show the policy was useless or even harmful, or it might mean crime would have risen even more but for the beneficial effects of the policy. Naturally, politicians would claim otherwise. Those in power would say it worked; their opponents would say it failed. But nobody would really know. The politicians would be blind men arguing over the colors of the rainbow. If the government had subjected its policy “to a randomized controlled trial then we might, by now, have known its true worth and be some way ahead in our thinking,” Cochrane observed. But it hadn’t. It had just assumed that its policy would work as expected. This was the same toxic brew of ignorance and confidence that had kept medicine in the dark ages for millennia.