The Data Detective: Ten Easy Rules to Make Sense of Statistics

by Tim Harford

In 2013, on the auspicious date of April 1, I received an email from Tetlock inviting me to join what he described as “a major new research program funded in part by Intelligence Advanced Research Projects Activity, an agency within the U.S. intelligence community.” The core of the program, which had been running since 2011, was a collection of quantifiable forecasts much like Tetlock’s long-running study. The forecasts would be of economic and geopolitical events, “real and pressing matters of the sort that concern the intelligence community—whether Greece will default, whether there will be a military strike on Iran, etc.” These forecasts took the form of a tournament with thousands of contestants; the tournament ran for four annual seasons. “You would simply log on to a website,” Tetlock’s email continued, “give your best judgment about matters you may be following anyway, and update that judgment if and when you feel it should be. When time passes and forecasts are judged, you could compare your results with those of others.” I did not participate. I told myself I was too busy; perhaps I was too much of a coward as well. But the truth is that I did not participate because, largely thanks to Tetlock’s work, I had concluded that the forecasting task was impossible. Still, more than 20,000 people embraced the idea. Some could reasonably be described as having some professional standing, with experience in intelligence analysis, think tanks, or academia. Others were pure a...