Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth

by Stuart Ritchie

Another option is to use an even more rigorous version of pre-registration. In this scenario, a scientist submits the registration itself to peer review and, if it’s approved and the reviewers agree that the study design is sound, the journal commits to publishing the eventual paper no matter how its results come out. Only then do the scientists start to collect their data.49 Not only does this type of study, called a ‘Registered Report’, kill publication bias stone dead, by removing the pernicious link between the statistical significance of the results and the decision to publish, but it reduces p-hacking as well, since you have to agree to your analysis with the reviewers beforehand and can’t just alter it post hoc without making it very clear what you’ve done. Best of all, it nullifies many of the perverse incentives that lead to bias and fraud in the first place. You know you’re going to get a publication in any case, so there’s no longer so much pressure to beautify your findings.50

Open Science is the idea that as far as possible, every part of the scientific process should be made freely accessible.52 The perfect Open Science study would have an associated webpage where you could download all its data, all the statistical code that the scientists used to analyse those data, and all the materials they used to gather the data in the first place.53 The peer reviews and previous drafts of the article could be published alongside the article (even if the identity of the reviewers isn’t revealed), allowing the reader to see the whole publication process.54 Providing free access to all your data and methods is an embodiment of the Mertonian norm of communalism: allowing other scientists to re-use your work to make their own research more efficient. It also serves the norm of organised scepticism: it makes it easier for other researchers to check your paper for negligent errors, attempt to replicate your results, and generally dig into your findings in more detail. They can do that far more readily than they could were they using just your study’s summary tables and graphs, which are typically all that’s included in a journal publication; and they can do it without having to write to you directly to beg for your dataset, which as we know rarely works. Taking the Open Science approach demonstrates to your scientific colleagues that they don’t need to have blind faith that everything’s been reported accurately: you have nothing to hide.

Open Science doesn’t stop at the scientific community: a final part of making science more transparent is to open it to the general public. After all, the Mertonian norm of communalism, which emphasises the shared ownership of scientific findings, captures an important truth: a huge chunk of science is funded by taxpayer money. Most journals charge anyone who doesn’t have a subscription something close to $35 for access to an individual paper. It seems problematic and even undemocratic that members of the public, who’ve funded scientific research through their taxes, should be charged to access the results of that research. Of course, there are many things that the taxpayer funds to which they don’t have access, and shouldn’t: for example, the secret patrol routes of the submarines that form the UK’s continuous at-sea nuclear deterrent. Like the sensitive data on viruses we saw above, however, this kind of information is an exception. Most governments have now passed Freedom of Information Acts that allow members of the public to access their documents and statistics; in the United States, the work of all government employees is automatically in the public domain.64 The branch of Open Science that tries to bring science closer to this free-information default is the Open Access movement. Its impact is already obvious: many journals have begun allowing scientists to pay a fee upon publication that lets their paper be downloaded by anyone, in perpetuity, at no cost. Paying t...

In 2019 the Nobel Prize-winning biologist and journal editor Randy Schekman compared the fees paid by the University of California to two different publishers: the non-profit National Academy of Sciences and the for-profit Elsevier.67 Per individual scientific paper downloaded, the National Academy of Science subscription cost the university $0.04. The Elsevier per-paper fee was $1.06. More than twenty-six times more expensive. Researchers from across the world are pouring millions into Elsevier’s coffers, and indeed billions into those of for-profit publishers in general. What additional service are they getting for all this money? It’s not obvious: the National Academy of Sciences offers essentially the same services at its journals as Elsevier, or even better. The time and expertise of peer reviewers is the crux of the whole scientific system, providing the critical analysis that’s required for science to progress. Yet those reviewers are volunteers and aren’t employed by the journals. They are free labour, so they clearly can’t be what justifies the cost. Rather, Elsevier and other for-profit publishers are engaging in what economists call rent-seeking: gaining more profits without providing more value.68 One highly critical investigation of the scientific publication system noted its absurdities: It is as if the New Yorker or the Economist demanded that journalists write and edit each other’s work for free, and asked the government to foot the bill. Outside observers ...

The journals, too, should adopt new standards that promote openness and replicability: explicitly inviting scientists to submit replications, to pre-register their plans, and to attach their datasets to their papers would be a great start.104 This extends the list of ways in which scientists can endear their paper to journal editors beyond the narrow focus on positive studies and means that researchers with null results won’t give up at the first hurdle.105 Journals could also institute policies of scanning studies for basic errors or employ data integrity officers to do random spot checks using methods like the GRIM test.106 They should certainly push authors to present their findings with humility: just as they require scientists to disclose their conflicts of interest, they could require them to write a ‘study limitations’ section that’s displayed in a little box on the final published paper.107 They could even try to discourage salami-slicing by making it more costly (in terms of time, effort, or money) to submit papers.108

For decades, even centuries, many scientists have harboured serious suspicions about the reliability of results and about the system in which those results are produced. But only in the last few years have they amassed stacks of hard data that back up their case. Now, across many fields, we can place high-profile single instances of science going wrong within the wider context of systemic problems, and point to strong meta-scientific data to back up the anecdotes. It’s only recently, in other words, that we’ve had the data that provides a concrete basis for fixing science. No university wants their scientists to be known for their work failing to replicate, or to become embroiled in fraud investigations. Nor do funders: they recognise that if they don’t fund high-quality, replicable research, their money is being wasted and in the long run, they’ll end up looking foolish. Nor do journals: the last thing they or their publishers want is to become a laughing stock, as has happened to some journals that became notorious for publishing flimsy research.109 Reputational concerns are partly what got us into this mess, but if the scandals and shortcomings become sufficiently well known, those same concerns might also help us out of it.

Linking bottom-up demands and top-down policies can result in positive feedback loops – virtuous cycles that create beneficial self-sustaining norms, rather than perverse ones. For instance, the UK Reproducibility Network, formed by academics concerned about the replication crisis and represented by grassroots groups at universities, is now talking with those universities about ways to change their hiring practices to better reward openness and transparency.110 Such efforts can serve as an antidote to the ‘natural selection of bad science’ that we saw in the previous chapter. Rewarding scientists who value openness and transparency can create its own virtuous cycle: as the number of Open Science practitioners increases, it’ll help to drive further bottom-up changes and encourage other researchers to buy into the movement and its associated reforms. It’ll also produce even more meta-science, helping us understand which kinds of studies replicate, which academic structures foster reliable research, and which results we should generally view with suspicion.111

The cancer biologist Florian Markowetz, in a paper about how scientists should take advantage of new automated tools to make the links between their data, their analysis and their papers crystal clear, gives ‘five selfish reasons to work reproducibly’: 1. Making your data open and transparent helps you and your co-authors to spot errors that might undermine your results, stopping your study from becoming the next Reinhart-Rogoff Excel-typo disaster. 2. The new automated methods make the paper easier to write. 3. If anyone can see how you analysed your data, it’s easier to convince reviewers that you did the right thing (and if the data are available, they can even try running your analysis for themselves). 4. Documenting every step of the analysis openly helps you continue your work a few months down the line: ‘future you’ won’t have to rely on memory of what ‘past you’ did. 5. Making everything available shows the scientific community that you’ve nothing to hide and that you’ve done everything in good faith: in other words, it builds your reputation as an honest researcher.114 We clearly see the problem. The solutions are within our grasp. All we need to fix science is to give people the right motivation.

Henryk Górecki’s Third Symphony, the Symphony of Sorrowful Songs, has sold more than a million copies, completely unheard of for recordings of modern classical music.115 Its popularity comes partly from its simplicity: it’s slow moving, almost like a film score, with none of the harsh atonality of Górecki’s previous works. Despite its eventual incredible sales, its 1977 live premiere apparently didn’t go down well with all audience members. As the final movement closed, with its twenty-one repetitions of an A major chord, Górecki overheard ‘a prominent French musician’ sitting in the front row (most assume it was the irascible avant-gardist Pierre Boulez) hiss ‘Merde!’.116 Even though it’s now recognised as a modern masterpiece, the Third Symphony was just not novel or experimental enough for the musical revolutionaries. Scientists think more like Boulez than Górecki. They’ve taken excitement about novelty in science too far, producing a perverse neophilia where every study needs to be a massive breakthrough that revolutionises the way we think about the world – perhaps announced, as though in the movies, by a scientist in a white lab coat bursting excitedly into the room waving a sheaf of paper. Scientists want their research to sound as though it’s of this Eureka-shouting kind, so they analyse it, write it up, and publicise it accordingly. But although unexpected breakthroughs do occur from time to time, most science is incremental and cumulative, building slowly towards...

Ideally, the meta-scientific evidence adduced in this book will convince almost everyone that something has gone very wrong with science, and that there’s a dire need for change. But even if you think the talk of a ‘crisis’ is grandiose or exaggerated, there’s one final argument in my quiver.122 It’s this: the reforms we’ve discussed in this chapter would all be beneficial for science even if there weren’t a replication crisis.

The argument that airing our dirty laundry in public will reduce trust in science seems even more misconceived when we look at the sheer amount of worthless, misleading and fundamentally untrustworthy research we’re putting out into the world. Every time we allow a flawed or obviously biased study to be published; every time we write another boy-who-cried-wolf press release that can’t be backed up by the data; every time a scientist writes a popular book full of feel-good-but-flimsy advice, we hand science’s critics another round of ammunition. Fix the science, I’d suggest, and the trust will follow.

A task as grand as revolutionising science was never going to be straightforward. There’ll be trial, error and, appropriately, experimentation along the way. This isn’t just about discarding some faulty theory, like geocentrism, or phlogiston, or alchemy, or any of the menagerie of incorrect ideas that litter the history of science. This is about root-and-branch (or lab-and-journal) reform of the way we do research, and of scientific culture – an attempt to master the faults and biases that have crept in, largely unnoticed. The world is rightly proud of where science has brought us. To retain that pride, we owe it something far better than the product of our flawed human temperaments. We owe it the truth.