Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth

by Stuart Ritchie

The remedy proposed most often is simply to abandon the idea of statistical significance. In 2019, over 850 scientists signed an open letter in Nature arguing just that. ‘It’s time,’ they wrote, ‘for statistical significance to go.’23 Rather than emphasising significance, they argued, researchers should be clearer about the uncertainty of their findings, reporting instead the margin of error around each number and generally having more humility about what can be derived from often-blurry statistical results.24 There’s a lot to be said for this, although it should be borne in mind that the most common margin of error that’s calculated – the so-called ‘confidence interval’ – just provides a different perspective on the data than that of the p-value, rather than much new statistical information.25 But there’s also a baby in the bathwater of statistical significance. By providing scientists with an objective measure, albeit an arbitrary one, it ties their hands. Doing away with p-values wouldn’t necessarily improve matters; in fact, by introducing another source of subjectivity, it might make the situation a lot worse.26 With tongue only partly in cheek, John Ioannidis has noted that if we remove all such objective measures we invite a situation where ‘all science will become like nutritional epidemiology’ – a scary prospect indeed.27