The Data Detective: Ten Easy Rules to Make Sense of Statistics

by Tim Harford

Motivated reasoning is thinking through a topic with the aim, conscious or unconscious, of reaching a particular kind of conclusion. In a football game, we see the fouls committed by the other team but overlook the sins of our own side. We are more likely to notice what we want to notice.[11]

When invited to search for more information, people would seek out data that backed their preconceived ideas. When invited to assess the strength of an opposing argument, they would spend considerable time thinking up ways to shoot it down.

Once you start using that proxy as a target to be improved, or a metric to control others at a distance, it will be distorted, faked, or undermined. The value of the measure will evaporate.

So not only are journals predisposed to publish surprising results, researchers facing “publish or perish” incentives are more likely to submit surprising results that may not stand up to scrutiny.

Scientists sometimes call this practice “HARKing”—HARK is an acronym for Hypothesizing After Results Known.

A major investment house will offer many different funds, and will advertise the ones that have been successful in the past.

Psychologists are increasingly acknowledging the problem of experiments that study only “WEIRD” subjects—that is, Western, Educated, and from Industrialized Rich Democracies.

If an algorithm’s creators claim that it will sack the right teachers or recommend bail for the right criminal suspects, our response should be “Prove it.”

If they put some event at a 25 percent likelihood and then it happened, they might then remember they’d called it as a 50/50 proposition. If a subject had put a 60 percent probability on an event that later failed to happen, she might later recall that she’d forecast a 30 percent probability.

Ideally, a decision maker or a forecaster will combine the outside view and the inside view—or, similarly, statistics plus personal experience.

The superforecasters are what psychologists call “actively open-minded thinkers”—people who don’t cling too tightly to a single approach, are comfortable abandoning an old view in the light of fresh evidence or new arguments, and embrace disagreements with others as an opportunity to learn.

you can deflate somebody’s overconfidence and moderate their politics simply by asking them to explain the details.