More on this book
Community
Kindle Notes & Highlights
Read between
April 26 - June 15, 2021
On average, the use of positive words in abstracts increased by nearly nine-fold over the analysed period:
The authors wryly concluded that by ‘extrapolating the upward trend of positive words over the past forty years to the future, we predict that the word ‘novel’ will appear in every [abstract] by the year 2123’.57
68 per cent of the papers’ abstracts and 61 per cent of their main texts contained attempts to highlight the benefits of the treatment, even though it had failed the trial.
‘a trend that approached significance’ (for a result reported as ‘p < 0.06’) • ‘fairly significant’ (p = 0.09) • ‘significantly significant’ (p = 0.065) • ‘narrowly eluded statistical significance’ (p = 0.0789) • ‘hovered around significance’ (p = 0.061) • ‘very closely brushed the limit of statistical significance’ (p = 0.051) • ‘not absolutely significant but very probably so’ (p > 0.05)
There’s also increasing interest in a therapy known as a ‘faecal transplant’.73 This is where stool samples from a healthy donor, replete with their various microbes, are transferred to a patient – usually via a colonoscopy but sometimes via swallowable capsules.
In a now-classic paper entitled ‘Is everything we eat associated with cancer?’, researchers Jonathan Schoenfeld and John Ioannidis randomly selected fifty ingredients from a cookbook, then checked the scientific literature to see whether they had been said to affect the risk of cancer.100 Forty of them had, including bacon, pork, eggs, tomatoes, bread, butter and tea (essentially all the aspects of that Killer Full English).
The system incentivises scientists not to practise science, but simply to meet its own perverse demands. These incentives are at the root of so many of the dubious practices that undermine our research.
the entire scientific literature will double in size every nine years.
In 2014, the computer scientist Peter Vamplew became so irritated by the constant stream of junk emails from the predatory journal International Journal of Advanced Computer Technology that he submitted a joke article entitled ‘Get Me Off Your Fucking Mailing List’. The paper consisted entirely of the sentence ‘Get me off your fucking mailing list’ repeated over eight hundred times (including a helpful flowchart figure with boxes and arrows portraying the message Get → me → off → Your → Fucking → Mail → ing → List). The journal rated it as ‘excellent’ and accepted it for publication.
In 2016, the major scientific publisher Springer despaired so deeply of the rampant peer-review fraud occurring at one of their journals, Tumor Biology, that, after retracting 107 tainted articles from just four years’ worth of issues, they gave up publishing the journal and sold it off to another company.
One analysis showed that in the five years following publication, approximately 12 per cent of medical research papers and around 30 per cent of natural-and social-science papers had zero citations.
Self-citation, which one analysis found makes up around a third of all citations in the first three years after a paper’s publication, is a grey area.
a study of peer reviews that included citation ‘suggestions’ found that 29 per cent were to the reviewer’s own work, and that self-citation suggestions were more common in positive than negative reviews (that is, reviewers were more likely to throw in the suggestion to cite their own work in reviews of papers that they were endorsing for publication).
One relatively small-scale analysis of papers from a sample of Australian academics found that if self-plagiarism is defined as reusing 10 per cent or more of the text from a previous paper in a subsequent one without attribution, six out of ten of the authors examined were guilty of it.
‘coercive citation’: demanding during peer review that authors cite a list of previous papers published in that journal, whether or not they’re strictly relevant to the work at hand. In one survey, around a fifth of scientists said this had happened to them.
In 2010, a review article was published in the Medical Science Monitor citing 490 articles, 445 of which were to papers published in Cell Transplantation. All 445 citations pointed to papers published in 2008 or 2009 … Of the remaining 45 citations, 44 cited the Medical Science Monitor, again, to papers published in 2008 and 2009 … Three of the four authors of this paper sit on the editorial board of Cell Transplantation. In the same year, 2010, two of these editors also published a review article in The Scientific World Journal citing 124 papers, 96 of which were published in Cell
...more
Goodhart’s Law: ‘when a measure becomes the target, it ceases to be a good measure’.
Scientists are human, and humans respond to incentives.
‘the natural selection of bad science’.
Perverse incentives work like an ill-tempered genie, giving you exactly what you asked for but not necessarily what you wanted.
On the surface, our current system of science funding and publication might seem like it promotes productivity and innovation, but instead it often rewards those who are following only the letter, rather than the spirit, of the endeavour.
a process of literature-laundering that transformed a messy, varied set of trials into a much cleaner-looking story of scientific discovery
To paraphrase the biologist Ottoline Leyser, the point of breaking ground is to begin to build something; if all you do is groundbreaking, you end up with a lot of holes in the ground but no buildings.
‘sizeless stare of statistical significance’, where scientists develop a laser-like focus on p-values at the expense of considering, as Ziliak and McCloskey put it, the ‘oomph’ of their effect.
over 55 per cent of trials had their results reported late to the US government’s trial registry.
It is as if the New Yorker or the Economist demanded that journalists write and edit each other’s work for free, and asked the government to foot the bill. Outside observers tend to fall into a sort of stunned disbelief when describing this setup … A 2005 Deutsche Bank report referred to it as a ‘bizarre’ ‘triplepay’ system, in which ‘the state funds most research, pays the salaries of most of those checking the quality of research, and then buys most of the published product’.
To help get the scientists themselves on board with necessary reforms, we should admit that the Mertonian norm of disinterestedness, admirable as it might be, only goes so far. In practice, personal reward is and always has been an important spur towards scientific progress.90 If we want to encourage researchers to collaborate more with each other, and to share their data online freely, we will need to make doubly sure they get the credit they deserve.
before tackling ingrained norms, incentives and policies, the first, deceptively simple step to effecting cultural change is to make it possible and easy for people to go along with your new ideas.
over 70,000 Russian-language journal articles that have been published at least twice, with some of them appearing in up to seventeen different journals.
a US biophysicist, was barred from his position on the board of a biology journal after he was found to have regularly coerced authors whose papers he edited into citing his own publications – sometimes more than fifty of them at a time, apparently giving him a massive spike in his citation count.
Anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.
More importantly, though, a sophisticated view on science isn’t one of unquestioning trust. It’s one that’s pithily summed up by the motto of the UK’s Royal Society: nullius in verba, or ‘take nobody’s word for it’.
Even if politicians use concerns about replicability as a disingenuous pretext for their scepticism of climate change, it doesn’t justify scientists overstating how much confidence we should have in our results.
In fact, a frank admission of science’s weaknesses is the best way to pre-empt attacks by science’s critics and to be honest more generally about how the uncertainty-filled process of science really works.
Émile Zola defined art as ‘a corner of nature seen through a temperament’.
