The Data Detective: Ten Easy Rules to Make Sense of Statistics

by Tim Harford

The motto of one of the oldest scientific societies, the Royal Society, is Nullius in verba—“Take nobody’s word for it.”

When we encounter evidence that we dislike, we ask ourselves, “Must I believe this?” More detail will often give us more opportunity to find holes in the argument. And when we encounter evidence that we approve of, we ask a different question: “Can I believe this?” More detail means more toeholds on to which that belief can cling.[19] The counterintuitive result is that presenting people with a detailed and balanced account of both sides of the argument may actually push people away from the center rather than pull them in. If we already have strong opinions, then we’ll seize upon welcome evidence, but we’ll find opposing data or arguments irritating. This biased assimilation of new evidence means that the more we know, the more partisan we’re able to be on a fraught issue.

A bat and ball cost $1.10, and the bat costs a dollar more than the ball. How much does the ball cost? and: A lake contains a patch of lily pads which doubles in size each day. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?[*] Many people get the answers to these questions wrong the first time they hear them, but what’s required to reach the correct solution isn’t intelligence or mathematical training, but pausing for a moment to double-check your gut reaction. Shane Frederick points out that noticing your initial error is usually all that’s necessary to solve the problem.[24] The cognitive reflection questions invite us to leap to the wrong conclusion without thinking. But so, too, do inflammatory memes or tub-thumping speeches. That’s why we need to be calm. And that is also why so much persuasion is designed to arouse us—our lust, our desire, our sympathy, or our anger. When was the last time Donald Trump, or for that matter Greenpeace, tweeted something designed to make you pause in calm reflection? Today’s persuaders don’t want you to stop and think. They want you to hurry up and feel. Don’t be rushed.

The United States has a notoriously high infant mortality rate for a rich country—6.1 deaths per thousand live births in 2010. In Finland, by comparison, it is just 2.3. But it turns out that physicians in America, like those in the UK’s Midlands, seem to be far more likely to record a pregnancy that ends at twenty-two weeks as a live birth, followed by an early death, than as a late miscarriage. Perhaps this is for cultural reasons, or perhaps it reflects different legal or financial considerations. Whatever the reason, some—by no means all—of the high infant mortality rate in the United States seems to be the result of recording births before twenty-four weeks as live when in other countries they would be recorded as miscarried pregnancies. For babies born after twenty-four weeks, the US infant mortality rate falls from 6.1 to 4.2 deaths per thousand live births. The rate in Finland barely shifts, from 2.3 to 2.1.[2]

Statisticians are sometimes dismissed as bean counters. The sneering term is misleading as well as unfair. Most of the concepts that matter in policy are not like beans; they are not merely difficult to count, but difficult to define. Once you’re sure what you mean by “bean,” the bean counting itself may come more easily. But if we don’t understand the definition, then there is little point in looking at the numbers. We have fooled ourselves before we have begun. The solution, then: Ask what is being counted, what stories lie behind the statistics. It is natural to think that the skills required to evaluate numbers are numerical—understanding how to compute a percentage, or how to disentangle your millions from your billions from your zillions. It’s a question of mathematics, is it not? What I hope we’ve learned over the past few pages is that the truth is more subtle yet in some ways easier: our confusion often lies less in numbers than in words. Before we figure out whether nurses have had a pay raise, first find out what is meant by “nurse.” Before lamenting the prevalence of self-harm in young people, stop to consider whether you know what “self-harm” is supposed to mean. Before concluding that inequality has soared, ask, “Inequality of what?” Demanding a short, sharp answer to the question “Has inequality risen?” is not only unfair, but strangely incurious. If we are curious instead, and ask the right questions, deeper insight is within easy reach.

Another way to step back and enjoy the view is to give yourself a sense of scale. Faced with a statistic, simply ask yourself, “Is that a big number?” The creators of More or Less, Michael Blastland and Andrew Dilnot, made a habit of asking this unassuming but powerful question.[10]

Andrew Elliott—an entrepreneur who likes the question so much he published a book with the title Is That a Big Number?—suggests that we should all carry a few “landmark numbers” in our heads to allow easy comparison.[11] A few examples: The population of the United States is 325 million. The population of the United Kingdom is 65 million. The population of the world is 7.5 billion. Name any particular age (under the age of sixty). There are about 800,000 people of that age in the UK. If a policy involves all three-year-olds, for example, there are 800,000 of them. In the United States, there are about 4 million people of any particular age (under the age of sixty). Distance around the Earth: 40,000 kilometers, or 25,000 miles. It varies depending on whether you go around the poles or around the equator, but not much. The drive from Boston to Seattle: 3,000 miles. Length of a bed: 2 meters (or 7 feet). As Elliott points out, this helps you visualize the size of a room: How many beds is that? The gross domestic product of the United States: about $20 trillion (or $20,000 billion). It’s a lot of walls, if that’s really how you want to spend it. 100,000 words: the length of a medium-size novel. 1,454 feet: the height of the Empire State Building to its tip. (It’s also about 102 stories.)

Most of the books people read are bestsellers—but most books are not bestsellers, and most book projects never become books at all. There’s a similar tale to tell about music, films, and business ventures. Even cases of COVID-19 are subject to selective attention: people who feel terrible go to the hospital and are tested for the disease; people who feel fine do not. As a result, the disease looks even more dangerous than it really is. Even though statisticians understand this problem perfectly well, there’s no easy way to solve it without systematic testing. And in the early stages of the pandemic, when the most difficult policy decisions were being made, systematic testing was elusive.

If the story you’re reading is about health, there’s one place you should be sure to look for a second opinion: the Cochrane Collaboration. It’s named after Archie Cochrane, a doctor, epidemiologist, and campaigner for better evidence in medicine. In 1941, when Cochrane was captured by the Germans and became a prisoner of war, he improvised a clinical trial. It was an astonishing combination of bravery, determination, and humility. The prison camp was full of sick men—Cochrane was one of them—and he suspected that the illness was caused by a dietary deficiency, but he knew that he didn’t know enough to confidently prescribe a treatment. Rather than slump into despair or follow a hunch, he managed to organize his fellow prisoners to test the effects of different diets, discovered what they were lacking, and provided incontrovertible evidence to the camp commandant. Vitamin supplements were duly procured, and many lives were saved as a result.[39]

Alchemy did not evolve into chemistry. It stagnated, and in due course science elbowed it to one side. For a while the two disciplines existed in parallel. So what distinguished them? Of course, modern science uses the experimental method, so clearly demonstrated by Pascal’s hardworking brother-in-law, by Torricelli, Boyle, and others. But so did alchemy. The alchemists were unrelenting experimenters. It’s just that their experiments yielded no information that advanced the field as a whole. The use of experiments does not explain why chemistry flourished and alchemy died. Perhaps, then, it was down to the characters involved? Perhaps the great early scientists such as Robert Boyle and Isaac Newton were sharper, wiser, more creative men than the alchemists they replaced? This is a spectacularly unpersuasive explanation. Two of the leading alchemists of the 1600s were Robert Boyle and Isaac Newton. They were energetic, even fervent, practitioners of alchemy, which thankfully did not prevent their enormous contributions to modern science.[25] No, the alchemists were often the very same people using the same experimental methods to try to understand the world around them. What accounts for the difference, says David Wootton, a historian of science, is that alchemy was pursued in secret, while science depended on open debate. In the late 1640s, a small network of experimenters across France, including Pascal, worked simultaneously on vacuum experiments. At least a hundred peop...

As Nightingale quipped when sending one of her analytical books to the Queen, “She may look at it because it has pictures.”[3] It’s a cynical, almost contemptuous, thing to write. But it is true. A chart has a special power. Our visual sense is potent, perhaps too potent. The word “see” is often used as a direct synonym for “understand”—“I see what you mean.” Yet sometimes we see but we don’t understand; worse, we see, then “understand” something that isn’t true at all. Done well, a picture of data is worth the proverbial thousand words. It is more than persuasive; it shows us things we could not have seen before, revealing patterns amid chaos. However, much depends on the intent of the chart’s creator, and the wisdom of the reader.

If you’re handling a complex dataset, you’ll learn a lot by turning it into a few different graphs to see what they show. Trends and patterns will often leap out immediately if plotted in the right way. For example, visualization expert Robert Kosara suggests plotting linear data on a spiral. If there’s a periodic pattern to the data—say, repeating every seven days or every three months—that may be concealed by other fluctuations in a conventional plot but will leap out in a spiral plot.

Millikan was a physicist. In 1923, as Fisher’s stock tips were being devoured across the United States, Millikan was collecting a Nobel Prize. For all his achievements, Millikan is most famous for an experiment so simple that a schoolkid can attempt it: the “oil drop” experiment, in which a mist of oil droplets from a perfume spritzer is given an electrical charge while floating between two electrified plates. Millikan could adjust the voltage between the plates until the droplets were suspended, without moving—and since he could measure the diameter of the droplets, he could calculate their mass, and thus also the electrical charge that was precisely offsetting the pull of gravity. This, in effect, allowed Millikan to calculate the electrical charge of a single electron. I was one of countless students who attempted this experiment in school, but in all honesty I was unable to get my results quite as neat as Millikan’s. There are a lot of details to get right—in particular, the experiment depends on correctly measuring the diameter of the tiny oil droplet. Mismeasure that, and all your other calculations will be off. We now know that even Millikan didn’t get his answers quite as neat as he claimed he did. He systematically omitted observations that didn’t suit him, and lied about those omissions. (He also minimized the contribution of a junior colleague, Harvey Fletcher.) Historians of science argue about the seriousness of this cherry-picking, ethically and practically. ...

Subjectivity in physics!

After a long and fruitless search for an antidote to tribalism, Kahan could be forgiven for becoming jaded.[5] Yet a few years ago, to his surprise, he and his colleagues stumbled upon a trait that some people have—and that other people can be encouraged to develop—that inoculates us against this toxic polarization. On the most politically polluted, tribal questions, where intelligence and education fail, this trait does not. And if you’re desperately, burningly curious to know what it is—congratulations. You may be inoculated already. Curiosity breaks the relentless pattern. Specifically, Kahan identified “scientific curiosity.” That’s different from scientific literacy. The two qualities are correlated, of course, but there are curious people who know rather little about science (yet), and highly trained people with little appetite to learn more. More scientifically curious Republicans aren’t further apart from Democrats on these polarized issues. If anything, they’re slightly closer together. It’s important not to exaggerate the effect. Curious Republicans and Democrats still disagree on issues such as climate change—but the more curious they are, the more they converge on what we might call an evidence-based view of the issues in question. Or to put it another way, the more curious we are, the less our tribalism seems to matter.

There’s a sweet spot for curiosity: if we know nothing, we ask no questions; if we know everything, we ask no questions either. Curiosity is fueled once we know enough to know that we do not know.[7] Alas, all too often we don’t even think about what we don’t know. There’s a beautiful little experiment about our incuriosity, conducted by the psychologists Leonid Rozenblit and Frank Keil. They gave their experimental subjects a simple task: to look through a list of everyday objects such as a flush lavatory, a zipper, and a bicycle, and to rate their understanding of each object on a scale of one to seven.[8] After people had written down their ratings, the researchers would gently launch a devastating ambush. They asked the subjects to elaborate. Here’s a pen and paper, they would say; please write out your explanation of a flush lavatory in as much detail as you can. By all means include diagrams. It turns out that this task wasn’t as easy as people had thought. People stumbled, struggling to explain the details of everyday mechanisms. They had assumed that those details would readily spring to mind, and they did not. And to their credit, most experimental subjects realized that they’d been lying to themselves. They had felt they understood zippers and lavatories, but when invited to elaborate, they realized they didn’t understand at all. When people were asked to reconsider their previous one-to-seven rating, they marked themselves down, acknowledging that their knowledg...

Be curious!