Bad Science

by Ben Goldacre

Today, scientists and doctors find themselves outnumbered and outgunned by vast armies of individuals who feel entitled to pass judgement on matters of evidence—an admirable aspiration—without troubling themselves to obtain a basic understanding of the issues.

You cannot reason people out of positions they didn’t reason themselves into.

People tend, for example, to rate longer explanations as being more similar to ‘experts’ explanations’.

This process of professionalising the obvious fosters a sense of mystery around science, and health advice, which is unnecessary and destructive. More than anything, more than the unnecessary ownership of the obvious, it is disempowering. All too often this spurious privatisation of common sense is happening in areas where we could be taking control, doing it ourselves, feeling our own potency and our ability to make sensible decisions; instead we are fostering our dependence on expensive outside systems and people.

No free lunch, and no effects without side-effects, as usual.

In general, you don’t absorb things very well through your skin, because its purpose is to be relatively impermeable. When you sit in a bath of baked beans for charity you do not get fat, nor do you start farting.

I can very happily view posh cosmetics—and other forms of quackery—as a special, self-administered, voluntary tax on people who don’t understand science properly.

It’s not about the dilution, he said: it’s the succussion. You have to bang the flask of water briskly ten times on a leather and horsehair surface, and that’s what makes the water remember a molecule. Because I did not mention this, he explained, I had deliberately made homeopaths sound stupid. This is another universe of foolishness.

As Voltaire said: ‘The art of medicine consists in amusing the patient while nature cures the disease.’

Some of the biggest figures in evidence-based medicine got together and did a review of blinding in all kinds of trials of medical drugs, and found that trials with inadequate blinding exaggerated the benefits of the treatments being studied by 17 per cent. Blinding is not some obscure piece of nitpicking, idiosyncratic to pedants like me, used to attack alternative therapies.

In reality, the biggest problem with poor-quality trials is not that they’ve used an inadequate method of randomisation, it’s that they don’t tell you how they randomised the patients at all. This is a classic warning sign, and often means the trial has been performed badly.

This phenomenon has been carefully studied, and there is an almost linear relationship between the methodological quality of a homeopathy trial and the result it gives. The worse the study—which is to say, the less it is a ‘fair test’—the more likely it is to find that homeopathy is better than placebo.

Why is evidence so complicated? Why do we need all of these clever tricks, these special research paradigms? The answer is simple: the world is much more complicated than simple stories about pills making people get better. We are human, we are irrational, we have foibles, and the power of the mind over the body is greater than anything you have previously imagined.

Mitchel in the early twentieth century was performing full amputations and mastectomies, entirely without anaesthesia; and surgeons from before the invention of anaesthesia often described how some patients could tolerate knife cutting through muscle, and saw cutting through bone, perfectly awake, and without even clenching their teeth. You might be tougher than you think.

In 1970 a sedative—chlordiazepoxide—was found to be more effective in capsule form than pill form, even for the very same drug, in the very same dose: capsules at the time felt newer, somehow, and more sciencey.

People I know still insist on buying brand-name painkillers. As you can imagine, I’ve spent half my life trying to explain to them why this is a waste of money: but in fact the paradox of Branthwaite and Cooper’s experimental data is that they were right all along. Whatever pharmacology theory tells you, that brand-named version is better, and there’s just no getting away from it.

To me, this is yet another example of a fascinating paradox in the philosophy of alternative therapists: when they claim that their treatments are having a specific and measurable effect on the body, through specific technical mechanisms rather than ritual, they are championing a very old-fashioned and naïve form of biological reductionism, where the mechanics of their interventions, rather than the relationship and the ceremony, have the positive effect on healing. Once again, it’s not just that they have no evidence for their claims about how their treatments work: it’s that their claims are mechanistic, intellectually disappointing, and simply less interesting than the reality.

In defence of almost all nutritionists, I would argue that they lack the academic experience, the ill-will, and perhaps even the intellectual horsepower necessary to be fairly derided as liars.

The philosopher Professor Harry Frankfurt of Princeton University discusses this issue at length in his classic 1986 essay ‘On Bullshit’. Under his model, ‘bullshit’ is a form of falsehood distinct from lying: the liar knows and cares about the truth, but deliberately sets out to mislead; the truth-speaker knows the truth and is trying to give it to us; the bullshitter, meanwhile, does not care about the truth, and is simply trying to impress us:

Does the cock’s crow cause the sun to rise? No. Does this light switch make the room get brighter? Yes. Things can happen at roughly the same time, but that is weak, circumstantial evidence for causation. Yet it’s exactly this kind of evidence that is used by media nutritionists as confident proof of their claims in our second major canard.

Anything can kill cells in a test tube. Fairy Liquid will kill cells in a test tube, but you don’t take it to cure cancer. This is just another example of how nutritionism, despite the ‘alternative medicine’ rhetoric and phrases like ‘holistic’, is actually a crude, unsophisticated, old fashioned, and above all reductionist tradition.

The public don’t generally have to deal with drug-company propaganda, because the companies are not currently allowed to talk to patients in Europe—thankfully—but they badger doctors incessantly, and they use many of the same tricks as the miracle-cure industries. You’re taught about these tricks at medical school, which is how I’m able to teach you about them now.

In the nineteenth century, as the public-health doctor Muir Gray has said, we made great advances through the provision of clean, clear water; in the twenty-first century we will make the same advances through clean, clear information. Systematic reviews are one of the great ideas of modern thought. They should be celebrated.

It’s a chilling thought that when we think we are doing good, we may actually be doing harm, but it is one we must always be alive to, even in the most innocuous situations.

The food supplement industry has engineered itself a beneficent public image, but this is not borne out by the facts. Firstly, there is essentially no difference between the vitamin industry and the pharmaceutical and biotech industries (that is one message of this book, after all: the tricks of the trade are the same the world over).

Whenever a piece of evidence is published suggesting that the $50-billion food supplement pill industry’s products are ineffective, or even harmful, an enormous marketing machine lumbers into life, producing spurious and groundless methodological criticisms of the published data in order to muddy the waters—not enough to be noteworthy in a meaningful academic discussion, but that is not their purpose.

If I was writing a lifestyle book it would have the same advice on every page, and you’d know it all already. Eat lots of fruit and vegetables, and live your whole life in every way as well as you can: exercise regularly as part of your daily routine, avoid obesity, don’t drink too much, don’t smoke, and don’t get distracted from the real, basic, simple causes of ill health. But as we will see, even these things are hard to do on your own, and in reality require wholesale social and political changes.

The story was unanimously ignored by the British news media, despite their preoccupation with both antisocial behaviour and miracle cures, for one simple reason: the research was not about a pill. It was about a cheap, practical parenting programme.

We are about to learn some very interesting lessons about the media, about how not to conduct a trial, and about our collective desire for medicalised, sciencey-sounding explanations for everyday problems.

They did literally everything which, in my view, would guarantee them a false positive result, and ruin any chance of their study giving meaningful and useful new information. How often does this happen? In the world of nutritionism, sadly, it seems to be standard research protocol.

The devil is in the detail, and this is why scientists report their full methods and results in academic papers, not in newspapers or on television programmes, and it is why experimental research cannot be reported in the mainstream media alone.

as George Orwell first noted, the true genius in advertising is to sell you the solution and the problem.

in reality it seems that these reductionist biomedical stories can appeal to us all, because complex problems often have depressingly complex causes, and the solutions can be taxing and unsatisfactory.

The scientific research is one thing. What you claim it shows—your interpretation—is entirely separate.

You’ll see that Holford is explaining the need to eat pills. This might be an apposite moment to mention that Professor Patrick Holford currently has his own range of best-selling pills, at least twenty different varieties, all featuring a photograph of his smiling face on the label.

It’s a popular myth among self-declared nutritionists (there is no other kind), and those who sell food-supplement pills, that our food is becoming less nutritious: in reality, many argue it may be more nutritious overall, because we eat more fresh and frozen fruit and veg, less tinned or dried stuff, and so they all get to the shops quicker, and thus with more nutrients (albeit at phenomenal cost to the environment). But Holford’s vitamin claim is somewhat more extreme than the usual fare. These oranges are not just less nutritious: ‘Yes, some supermarket oranges contain no vitamin C!’* Frightening stuff! Buy pills!

In the conclusion to this book, written before I was able to include this chapter, I will argue that the biggest dangers posed by the material we have covered are cultural and intellectual. I may be mistaken.

Doctors can be awful, and mistakes can be murderous, but the philosophy driving evidence-based medicine is not. How well does it work?

‘Big pharma is evil,’ goes the line of reasoning, ‘therefore homeopathy works and the MMR vaccine causes autism.’ This is probably not helpful.

A key feature of science is that findings should be replicated, but if only one organisation is doing the funding, then this feature is lost.

One 3,000-subject review on SSRIs simply did not list any sexual side-effects on its twenty-three-item side-effect table. Twenty-three other things were more important, according to the researchers, than losing the sensation of orgasm. I have read them. They are not.

Now you’ve done your trial, and despite your best efforts things have come out negative. What can you do? Well, if your trial has been good overall, but has thrown out a few negative results, you could try an old trick: don’t draw attention to the disappointing data by putting it on a graph. Mention it briefly in the text, and ignore it when drawing your conclusions. (I’m so good at this I scare myself. Comes from reading too many rubbish trials.)

If your results are completely negative, don’t publish them at all, or publish them only after a long delay. This is exactly what the drug companies did with the data on SSRI antidepressants: they hid the data suggesting they might be dangerous, and they buried the data showing them to perform no better than placebo.

Ignore the protocol entirely Always assume that any correlation proves causation. Throw all your data into a spreadsheet programme and report—as significant—any relationship between anything and everything if it helps your case. If you measure enough, some things are bound to be positive just by sheer luck.

Play with the baseline Sometimes, when you start a trial, quite by chance the treatment group is already doing better than the placebo group. If so, then leave it like that. If, on the other hand, the placebo group is already doing better than the treatment group at the start, then adjust for the baseline in your analysis.

Ignore dropouts People who drop out of trials are statistically much more likely to have done badly, and much more likely to have had side-effects. They will only make your drug look bad. So ignore them, make no attempt t...

Clean up the data Look at your graphs. There will be some anomalous ‘outliers’, or points which lie a long way from the others. If they are making your drug look bad, just delete them. But if they are helping your drug loo...

‘The best of five … no … seven … no … nine!’ If the difference between your drug and placebo becomes significant four and a half months into a six-month trial, stop the trial immediately and start writing up the results: things might get less impressive if you carry on. Alternatively, if at six months...

Torture the data If your results are bad, ask the computer to go back and see if any particular subgroups behaved differently. You might find that your drug works very well in Chinese women aged fiftytwo to sixty-one. ‘Torture the data a...

remember, the tricks we have just described hide nothing, and will be obvious to anyone who reads your paper, but only if they read it very attentively, so it’s in your interest to make sure it isn’t read beyond the abstract.