Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth

by Stuart Ritchie

You’d think that if you obtained the exact same dataset as was used in a published study, you’d be able to derive the exact same results that the study reported. Unfortunately, in many subjects, researchers have had terrible difficulty with this seemingly straightforward task. This is a problem sometimes described as a question of reproducibility, as opposed to replicability (the latter term being usually reserved to mean studies that ask the same questions of different data). How is it possible that some results can’t even be reproduced? Sometimes it’s due to errors in the original study. Other times, the original scientists weren’t clear enough with reporting their analysis: they took twists and turns with the statistics that weren’t declared in their scientific paper, and thus their exact steps can’t be retraced by independent researchers. When new scientists run the statistics in their own way, the results come out differently. Those studies are like a cookbook including mouth-watering photographs of meals but providing only the patchiest details of the ingredients and recipe needed to produce them.