Bad Science

by Ben Goldacre

3. We overvalue confirmatory information for any given hypothesis. 4. We seek out confirmatory information for any given hypothesis.

5. Our assessment of the quality of new evidence is biased by our previous beliefs.

When people learn no tools of judgement and merely follow their hopes, the seeds of political manipulation are sown.

Most of us exhibit something called ‘attributional bias’: we believe our successes are due to our own internal faculties, and our failures are due to external factors; whereas for others, we believe their successes are due to luck, and their failures to their own flaws. We can’t all be right.

Natural frequencies are readily understandable, because instead of using probabilities, or percentages, or anything even slightly technical or difficult, they use concrete numbers, just like the ones you use every day to check if you’ve lost a kid on a coach trip, or got the right change in a shop.

(there are excellent best-practice guidelines from the Royal Society on communicating research, if you are interested).

Over a hundred years ago, H.G. Wells said that statistical thinking would one day be as important as the ability to read and write in a modern technological society. I disagree; probabilistic reasoning is difficult for everyone, but everyone understands normal numbers. This is why ‘natural frequencies’ are the only sensible way to communicate risk.

For murder, the extremely rare crime in question in this report, for which more action was demanded, occurring at one in 10,000 a year among patients with psychosis, the false positive rate is so high that the best predictive test is entirely useless.

It was calculated as 8,543 × 8,543, as if the chances of two SIDS episodes in this one family were independent of each other.

If we really wanted to play statistics, we would need to know which is relatively more rare, double SIDS or double murder.

This breaks a cardinal rule of any research involving statistics: you cannot find your hypothesis in your results. Before you go to your data with your statistical tool, you have to have a specific hypothesis to test. If your hypothesis comes from analysing the data, then there is no sense in analysing the same data again to confirm it.

In the three years before Lucia worked on the ward in question, there were seven deaths. In the three years that she did work on the ward, there were six deaths. Here’s a thought: it seems odd that the death rate should go down on a ward at the precise moment that a serial killer—on a killing spree—arrives. If Lucia killed them all, then there must have been no natural deaths on that ward at all in the whole of the three years that she worked there. Ah,

I first worked out what was going on when I got a phone call from a friend who works as an undercover journalist for television. ‘I just got a job as a cleaner to take some MRSA swabs for my filthy hospital superbug scandal,’ he said, ‘but they all came back negative. What am I doing wrong?’ Happy to help, I explained that MRSA doesn’t survive well on windows and doorknobs. The stories he had seen elsewhere were either rigged or rigged.

‘the lab that always gives positive results’ were the words she used, and it turned out to be Northants-based Chemsol Consulting, run by a man called Dr Christopher Malyszewicz. If you have ever seen an undercover MRSA superbug positive swab scandal, it definitely came from here. They all do.

Britain has more MRSA than many other countries, and this could be for any number of reasons, including infection control measures, cleanliness, prescribing patterns, or things we’ve not thought of yet (I’m talking off the top of my head here).

He agreed that his PhD was not recognised in the UK. He had no microbiology qualifications or training (as many journalists were repeatedly told by professional microbiologists). He was charming and very pleasant to talk to: eager to please.

The report from this visit describes the Chemsol laboratory as ‘a freestanding, single storey wooden building, approximately 6m × 2m in the back garden’.

‘Health Secretary John Reid was accused last night of trying to gag Britain’s leading expert on the killer bug MRSA.’ Britain’s leading expert who has no microbiology qualifications, runs his operation from a shed in the garden, mispronounces the names of common bacteria, and demonstrably doesn’t understand the most basic aspects of microbiology.

The accompanying editorial in the Sunday Mirror heroically managed to knit three all-time classic bogus science stories together, into one stirring eulogy:

First, Frankenstein foods expert Arpad Puzstai felt Labour’s wrath when he dared to raise the alarm over genetically-modified crops. Then Dr Andrew Wakefield suffered the same fate when he suggested a link between the single-jab MMR vaccine and autism. Now it’s the turn of Dr Chris Malyszewicz, who has publicly exposed alarmingly high rates of the killer bug MRSA in NHS hospitals.

We stand by the accuracy and integrity of our articles. The research was carried out by a competent person using current testing media. Chris Malyszewicz … is a fully trained microbiologist with eighteen years’ experience… We believe the test media used … were sufficient to detect the presence of pathogenic type MRSA.

‘Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self Assessments’, by Justin Kruger and David Dunning. They noted that people who are incompetent suffer a dual burden: not only are they incompetent, but they may also be too incompetent to assay their own incompetence, because the skills which underlie an ability to make a correct judgement are the same as the skills required to recognise a correct judgement.

Their findings were twofold: people who performed particularly poorly relative to their peers were unaware of their own incompetence; but more than that, they were also less able to recognize competence in others, because this, too, relied on ‘meta-cognition’, or knowledge about the skill.

The drug went to market, and between 1958 and 1962 around 10,000 children were born with severe malformations, all around the world, caused by this same drug, thalidomide.

In fact Philip Knightley—a god of investigative journalism from the Sunday Times’ legendary Insight team, and the man most associated with heroic coverage on thalidomide—specifically writes in his autobiography about his shame over not covering the thalidomide story sooner.

It has every ingredient, every canard, every sleight of hand, and every aspect of venal incompetence and hysteria, systemic and individual.

As we will see, they overextrapolated from one study into absurdity, while studiously ignoring all reassuring data, and all subsequent refutations.

clinical anecdotes, a type of paper called a ‘case series’—and a case series, by design, wouldn’t demonstrate such a relationship between an exposure and an outcome with any force.

if some children with a combination of fairly common things (vaccination, autism, bowel problems) all come together in one place which is already acting as beacon for such a combination, as this clinic was, we should not naturally be impressed.

At the publication of this paper, a press conference was held at the Royal Free Hospital, and to the visible surprise of many other clinicians and academics present, Andrew Wakefield announced that he thought it would be prudent to use single vaccines instead of the MMR triple vaccine.

For example, it is investigating whether Wakefield failed to disclose to the editor of the Lancet his involvement in a patent relating to a new vaccine;

Wakefield himself eventually received £435,643 plus expenses from the legal aid fund for his role in the case against MMR.

In 2004 Chadwick gave an interview to Channel 4’s Dispatches, and in 2007 he gave evidence at a US case on vaccines, stating that there was no measles RNA to be found in these samples.

Now, you will see news reporters—including the BBC—saying stupid things like ‘The research has since been debunked.’ Wrong. The research never justified the media’s ludicrous over-interpretation. If they had paid attention, the scare would never have even started.

False positives are common in PCR, because it works by using enzymes to replicate RNA, so you start with a small amount in your sample, which is then ‘amplified up’, copied over and over again, until you have enough to measure and work with.

But I think we can safely assume there is no Panorama documentary in the pipeline covering the startling new research data suggesting that mercury fillings may not be harmful after all.

It’s hardly surprising that the MMR vaccination rate has fallen from 92 per cent in 1996 to 73 per cent today. In some parts of London it’s down to 60 per cent, and figures from 2004–05 showed that in Westminster only 38 per cent of children had both jabs by the age of five.