The Data Detective: Ten Easy Rules to Make Sense of Statistics

by Tim Harford

But sometimes the problem is not that we are too eager to believe something, but that we find reasons not to believe anything.

it turns out that doubt is a really easy product to make.

The experimental subjects found it much easier to argue against positions they disliked than in favor of those they supported. There was a special power in doubt.

I worry about a world in which many people will believe anything, but I worry far more about one in which people believe nothing beyond their own preconceptions.

when it comes to interpreting the world around us, we need to realize that our feelings can trump our expertise.

the more detail people were presented with—graphs, research methods, commentary by other fictional academics—the easier they found it to disbelieve unwelcome evidence. If doubt is the weapon, detail is the ammunition.

The counterintuitive result is that presenting people with a detailed and balanced account of both sides of the argument may actually push people away from the center rather than pull them in. If we already have strong opinions, then we’ll seize upon welcome evidence, but we’ll find opposing data or arguments irritating.

This biased assimilation of new evidence means that the more we know, the more partisan we’re able to be on a fraught issue.

Hundreds of thousands of years of human evolution have wired us to care deeply about fitting in with those around us. This helps to explain the findings of Taber and Lodge that better-informed people are actually more at risk of motivated reasoning on politically partisan topics: the more persuasively we can make the case for what our friends already believe, the more our friends will respect us.

it’s far easier to lead ourselves astray when the practical consequences of being wrong are small or nonexistent, while the social consequences of being “wrong” are severe.

so much persuasion is designed to arouse us—our lust, our desire, our sympathy, or our anger. When was the last time Donald Trump, or for that matter Greenpeace, tweeted something designed to make you pause in calm reflection? Today’s persuaders don’t want you to stop and think. They want you to hurry up and feel.

The world is full of patterns that are too subtle or too rare to detect by eyeballing them, and a pattern doesn’t need to be very subtle or rare to be hard to spot without a statistical lens.

Psychologists have a name for our tendency to confuse our own perspective with something more universal: it’s called “naive realism,” the sense that we are seeing reality as it truly is, without filters or errors.

Naive realism can lead us badly astray when we confuse our personal perspective on the world with some universal truth.

As the great psychologist Daniel Kahneman explained in Thinking, Fast and Slow: “When faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution.”

Much of what we think of as cultural differences turn out to be differences in income.

Understanding causation is tough even with good statistics, but hopeless without them.

Often, looking for an explanation really means looking for someone to blame.

In truth, there may never have been anything to blame anybody for at all.

Premature enumeration is not just an intellectual failure. Not asking what a statistic actually means is a failure of empathy, too.

What counts as “news” depends very much on the frequency with which we pay attention.2 If media outlets know most of their audience is checking in every day, or every few hours, they will naturally tell us the most attention-grabbing event that’s happened in that time.

How about a fifty-year newspaper? Max Roser, a young economist who created the Our World in Data website, proposed that idea, inspired by Galtung and Ruge. Roser imagines a newspaper published in 1918, 1968, and 2018. Topics that seemed earth-shattering to the daily newspapers of the time might not be mentioned at all, while huge changes in the world would scream from the front pages.3

For a reader picking up a newspaper in 2018 for the first time since 1968, it would be big news that the Cold War had simply ended without a nuclear exchange of any kind—even if no daily newspaper would have been tempted in the meantime to run with the headline “No H-Bombs Dropped Today.”

most people think that all is well for them personally but are worried about the society they live in.20 Presumably this is because we personally experience our own localities, but we rely on the news for information about the wider world.

Good things happen so often that they cannot seriously be considered for inclusion in a newspaper. “An Estimated 154,000 People Escaped from Poverty Yesterday!” True, but not news.

“To be completely cured of newspapers, spend a year reading the previous week’s newspapers.”

So however much news you choose to read, make sure you spend time looking for longer-term, slower-paced information. You will notice things—good and bad—that others ignore.

Testing a hypothesis using the numbers that helped form the hypothesis in the first place is not OK.

The standard statistical methods are designed to exclude most chance results.19 But a combination of publication bias and loose research practices means we can expect that mixed in with the real discoveries will be a large number of statistical accidents.

we should draw conclusions about human nature only after studying a broad range of people. Psychologists are increasingly acknowledging the problem of experiments that study only “WEIRD” subjects—that is, Western, Educated, and from Industrialized Rich Democracies.

A hammer looks like a useful tool to a carpenter; the nail has a different impression altogether.

Independent statistical agencies, like sewers, are an essential part of modern life. As with sewers, we tend to take them for granted until something goes wrong. And like sewers, they can suffer badly from neglect—or because someone tries to force something unsuitable through them for their own selfish or foolish reasons.

To justify its existence, the CBO would need to improve the effectiveness of government spending decisions by a mere 0.00125 percent. It’s hard to imagine how the CBO could fail to clear that bar.

Our visual sense is potent, perhaps too potent. The word “see” is often used as a direct synonym for “understand”—“I see what you mean.” Yet sometimes we see but we don’t understand; worse, we see, then “understand” something that isn’t true at all.

A man with a conviction is a hard man to change. Tell him you disagree and he turns away. Show him facts or figures and he questions your sources. Appeal to logic and he fails to see your point.

superforecasting is a matter of having an open-minded personality. The superforecasters are what psychologists call “actively open-minded thinkers”—people who don’t cling too tightly to a single approach, are comfortable abandoning an old view in the light of fresh evidence or new arguments, and embrace disagreements with others as an opportunity to learn. “For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded,”

superforecasting means being willing to change your mind.

the more scientifically literate opponents are, the more they disagree. The same is true for numeracy. “The greater the proficiency, the more acute the polarization,”

the more curious we are, the less our tribalism seems to matter.

Neuroscientific studies suggest that the brain responds in much the same anxious way to facts that threaten our preconceptions as it does to wild animals that threaten our lives.

if we know nothing, we ask no questions; if we know everything, we ask no questions either. Curiosity is fueled once we know enough to know that we do not know.

in a world where so many people seem to hold extreme views with strident certainty, you can deflate somebody’s overconfidence and moderate their politics simply by asking them to explain the details.