Calling Bullshit: The Art of Skepticism in a Data-Driven World

by Carl T. Bergstrom

But look at it from the researcher’s perspective. Imagine that you have just spent months collecting a massive data set. You test your main hypothesis and end up with results that are promising—but nonsignificant. You know you won’t be able to publish your work in a good journal this way, and you may not be able to publish at all. But surely the hypothesis is true, you think—maybe you just don’t have quite enough data. So you keep collecting data until your p-value drops below 0.05, and then stop right away lest it drift back above that threshold. Or maybe you try a few other statistical tests. Since the data are close to significant, the right choice of measures and of tests might allow you to clear the critical p = 0.05 bar. Sure enough, with a bit of tinkering, you find an approach that gives you a significant result. Or maybe your hypothesis appears to be correct only for men, and the meaningful pattern is getting swamped out by including women in your sample. You look—and lo and behold, once you look at men only, you’ve got a statistically significant result. What are you going to do? Scrap the whole project, abandon thousands of dollars invested, ask your graduate student to delay her graduation by another six months…or just write up the result for men and submit to a top journal? Under those circumstances it may feel easy to rationalize the latter approach. “I’m sure the trend is really there,” you might tell yourself. “I was thinking about omitting women from the s...