Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth

by Stuart Ritchie

Perhaps the scientists who ran these studies thought something like: ‘well, it was only a small study, and the small effect I found is probably just due to noisy data. Come to think of it, I was silly even to expect to find an effect here! There’s no point in trying to publish this.’ Crucially, though, this post hoc rationalisation wouldn’t have occurred to them if the same small-sample study, with its potentially noisy data, happened to show a large effect: they’d have eagerly sent off their positive results to a journal. This double standard, based on the entrenched human tendency towards confirmation bias (interpreting evidence in the way that fits our pre-existing beliefs and desires), is what’s at the root of publication bias. If you consider the overall conclusions of a meta-analysis based on Figure 2B rather than 2A, you can see how publication bias deranges the scientific literature. If the studies with small effects have been removed from the funnel shape, the overall effect that shows up in the meta-analysis will by definition be larger than is justified. We get an exaggerated view of the importance of the effects and can be misled into believing something exists when it doesn’t. By failing to publish null or ambiguous studies, researchers force blinkers onto anyone who reads the scientific literature.