The Data Detective: Ten Easy Rules to Make Sense of Statistics

by Tim Harford

The first simple step is to notice those emotions. When you see a statistical claim, pay attention to your own reaction. If you feel outrage, triumph, denial, pause for a moment. Then reflect. You don’t need to be an emotionless robot, but you could and should think as well as feel.

If we don’t understand the statistics, we’re likely to be badly mistaken about the way the world is. It is all too easy to convince ourselves that whatever we’ve seen with our own eyes is the whole truth; it isn’t. Understanding causation is tough even with good statistics, but hopeless without them. And yet, if we understand only the statistics, we understand little. We need to be curious about the world that we see, hear, touch, and smell, as well as the world we can examine through a spreadsheet. My second piece of advice, then, is to try to take both perspectives—the worm’s-eye view as well as the bird’s-eye view. They will usually show you something different, and they will sometimes pose a puzzle: How could both views be true?

What the psychologist Steven Pinker calls the “curse of knowledge” is a constant obstacle to clear communication: once you know a subject fairly well, it is enormously difficult to put yourself in the position of someone who doesn’t know it. My colleagues and I weren’t immune. When we started researching some statistical confusion, we’d habitually start by pinning down the definitions—but as we quickly took them for granted, we always had to remind ourselves to explain them to our listeners, too.

What counts as “news” depends very much on the frequency with which we pay attention.2 If media outlets know most of their audience is checking in every day, or every few hours, they will naturally tell us the most attention-grabbing event that’s happened in that time.

In the first chapter, I advised trying to notice your feelings about the claim; in the second chapter, constructively sense-checking the claim against your personal experience; in the third chapter, asking yourself if you really understand what the claim means. These are all simple, commonsense suggestions, and in this chapter I’ve added a fourth: Step back and look for information that can put the claim into context. Try to get a sense of the trend. “Another terrible crime has occurred!” is perfectly consistent with “Overall, crime is way down.” Look for something that will give you a sense of scale, such as comparing the situation in one country with the situation in other countries, or figuring out the cost per person of some proposed government expenditure. None of these methods is technical; anyone can use them. Together they can go a long way toward providing statistical illumination. But sometimes we need to dig a little deeper into how a statistic was produced.

This, then, is a survivorship bias as strong as press coverage of Kickstarter projects or trying to deduce the vulnerabilities of planes by examining only the ones whose vulnerabilities weren’t fatal. Out of all the possible studies that could have been conducted, it’s reasonable to guess that the journal was interested only in the ones that demonstrated precognition. This wasn’t because of a bias in favor of precognition. It was because of a bias in favor of novel and surprising discoveries.

This leads to yet another kind of publication bias: if a particular way of analyzing the data produces no result, and a different way produces something more intriguing, then of course the more interesting method is likely to be what is reported and then published. Scientists sometimes call this practice “HARKing”—HARK is an acronym for Hypothesizing After Results Known. To be clear, there’s nothing wrong with gathering data, poking around to find the patterns, and then constructing a hypothesis. That’s all part of science. But you then have to get new data to test the hypothesis. Testing a hypothesis using the numbers that helped form the hypothesis in the first place is not OK.15

We should not simply trust that algorithms are doing a better job than humans, nor should we assume that if the algorithms are flawed, the humans would be flawless.

dazzle camouflage. Dazzle was a defense mechanism for battleships in the First World War, always at risk of being torpedoed by a lurking submarine. The usual “blending in” method of camouflage wasn’t an option for a huge steel vessel that advertised its presence against an ever-changing sea and sky with bow waves and smokestacks. Dazzle camouflage flipped the idea of camouflage on its head. It was an abstract riot of squiggles and harlequin patterns—in fact, it bore enough of a resemblance to Cubist art that Picasso himself impishly tried to claim the credit.7

Stocks are compared with flows. That’s the equivalent of comparing the total cost of buying a house with the annual cost of renting one; it’s not a trivial confusion. Net measures are put alongside gross ones—the equivalent of comparing a firm’s profit with its turnover. The shocking difference between the before-and-after comparisons of the cost of the war in Iraq turns out to be based on an unfair comparison. (Admittedly, a fair comparison might also show a shocking difference.) The prewar number is a narrow estimate: the cost to the US military budget. The postwar number is very broad, including a figure for the cost of lost life, the cost of high oil prices, and a huge sum for the cost of macroeconomic instability, putting part of the blame for the 2008 financial crisis on the war. That broad estimate of cost is not unreasonable, but what is unreasonable is to put it alongside a very different kind of estimate without comment. What seems to be a pure before-and-after contrast is actually narrow-and-before versus broad-and-after, measuring a different thing at a different time. Nobody looking at the Debtris animation would realize that. Debtris was published in 2010, and quickly became my favorite cautionary example—the visualization is so good but the data are so bad. A

combine the outside view and the inside view—or, similarly, statistics plus personal experience. But it’s much better to start with the statistical view, the outside view, and then modify it in the light of personal experience than it is to go the other way around. If you start with the inside view you have no real frame of reference, no sense of scale—and can easily come up with a probability that is ten times too large, or ten times too small.

This willingness to adjust predictions is correlated with making better predictions in the first place: it wasn’t just that the superforecasters beat the others because they were news junkies with too much time on their hands, prospering by endlessly tweaking their forecasts with each new headline. Even if the tournament rules had demanded a one-shot forecast, the superforecasters would have come top of the heap.

having an open-minded personality.

First, we should learn to stop and notice our emotional reaction to a claim, rather than accepting or rejecting it because of how it makes us feel. Second, we should look for ways to combine the “bird’s eye” statistical perspective with the “worm’s eye” view from personal experience. Third, we should look at the labels on the data we’re being given, and ask if we understand what’s really being described. Fourth, we should look for comparisons and context, putting any claim into perspective. Fifth, we should look behind the statistics at where they came from—and what other data might have vanished into obscurity. Sixth, we should ask who is missing from the data we’re being shown, and whether our conclusions might differ if they were included. Seventh, we should ask tough questions about algorithms and the big datasets that drive them, recognizing that without intelligent openness they cannot be trusted. Eighth, we should pay more attention to the bedrock of official statistics—and the sometimes heroic statisticians who protect it. Ninth, we should look under the surface of any beautiful graph or chart. And tenth, we should keep an open mind, asking how we might be mistaken, and whether the facts have changed.

About a decade ago, researchers led by Yale University’s Dan Kahan showed students some footage of a protest outside an unidentified building. Some of the students were told that it was a pro-life demonstration outside an abortion clinic. Others were informed that it was a gay rights demonstration outside an army recruitment office. The students were asked some factual questions. Was it a peaceful protest? Did the protesters try to intimidate people passing by? Did they scream or shout? Did they block the entrance to the building? The answers people gave depended on the political identities they embraced. Conservative students who believed they were looking at a demonstration against abortion saw no problems with the protest: no abuse, no violence, no obstruction. Students on the left who thought they were looking at a gay rights protest reached the same conclusion: the protesters had conducted themselves with dignity and restraint. But right-wing students who thought they were looking at a gay rights demonstration reached a very different conclusion, as did left-wing students who believed they were watching an anti-abortion protest. Both these groups concluded that the protesters had been aggressive, intimidating, and obstructive.3

It’s a rather beautiful discovery: in a world where so many people seem to hold extreme views with strident certainty, you can deflate somebody’s overconfidence and moderate their politics simply by asking them to explain the details. Next time you’re in a politically heated argument, try asking your interlocutor not to justify herself, but simply to explain the policy in question. She wants to introduce a universal basic income, or a flat tax, or a points-based immigration system, or Medicare for all. OK, that’s interesting. So what exactly does she mean by that? She may learn something as she tries to explain. So may you. And you may both find that you understand a little less, and agree a little more, than you had assumed.

Those of us in the business of communicating ideas need to go beyond the fact-check and the statistical smackdown. Facts are valuable things, and so is fact-checking. But if we really want people to understand complex issues, we need to engage their curiosity. If people are curious, they will learn.*