Bad Science

by Ben Goldacre

Today, scientists and doctors find themselves outnumbered and outgunned by vast armies of individuals who feel entitled to pass judgement on matters of evidence—an admirable aspiration—without troubling themselves to obtain a basic understanding of the issues.

Then we move on to the bigger fish. Nutritionists are alternative therapists, but have somehow managed to brand themselves as men and women of science. Their errors are much more interesting than those of the homeopaths, because they have a grain of real science to them, and that makes them not only more interesting, but also more dangerous, because the real threat from cranks is not that their customers might die—there is the odd case, although it seems crass to harp on about them—but that they systematically undermine the public’s understanding of the very nature of evidence.

You cannot reason people out of positions they didn’t reason themselves into.

That burgers and beer can have negative effects on your body is certainly true, for a number of reasons; but the notion that they leave a specific residue, which can be extruded by a specific process, a physiological system called detox, is a marketing invention.

How can this nonsense be so widespread in schools? One obvious explanation is that the teachers have been blinded by all these clever long phrases like ‘reticular formation’ and ‘increased oxidation’. As it happens, this very phenomenon has been studied in a fascinating set of experiments from the March 2008 edition of the Journal of Cognitive Neuroscience, which elegantly demonstrated that people will buy into bogus explanations much more readily when they are dressed up with a few technical words from the world of neuroscience.

Most people know what constitutes a healthy diet already. If you want to make money out of it, you have to make a space for yourself in the market: and to do this, you must overcomplicate it, attach your own dubious stamp.

If you really want to, you can replicate this by making your own moisturiser at home: you’re aiming for a mix of water and oil, but one that’s ‘emulsified’, which is to say, nicely mixed up. When I was involved in hippy street theatre—and I’m being entirely serious here—we made moisturiser from equal parts of olive oil, coconut oil, honey and rosewater (tap water is fine too).

Get all your ingredients lightly heated, but separately, stir the oil into the wax, beating all the time, and then stir in the water. Stick it in a jar, and keep for three months in the fridge.

There are basically three groups of ingredients in moisturising cream. Firstly there are powerful chemicals, like alpha-hydroxy acids, high levels of vitamin C, or molecular variations on the theme of vitamin A. These have genuinely been shown to make your skin seem more youthful, but they are only effective at such high concentrations, or high acidity levels, that the creams cause irritation, stinging, burning and redness. They were the great white hope in the 1990s, but now they’ve all had to be massively watered down by law, unless on prescription. No free lunch, and no effects without side-effects, as usual.

or that smearing it on your face will make any difference to your appearance. In general, you don’t absorb things very well through your skin, because its purpose is to be relatively impermeable. When you sit in a bath of baked beans for charity you do not get fat, nor do you start farting.

The simple theme running through all these products is that you can hoodwink your body, when in reality there are finely tuned ‘homeostatic’ mechanisms, huge, elaborate systems with feedback and measuring devices, constantly calibrating and recalibrating the amounts of various different chemical constituents being sent to different parts of your body. If anything, interfering with that system is likely to have the opposite of the simplistic effects claimed.

Even if the claim about oxygen in cream penetrating your tissues were true, your body would simply downregulate the supply of blood to that part of skin, scoring a homeostatic own goal.

I can very happily view posh cosmetics—and other forms of quackery—as a special, self-administered, voluntary tax on people who don’t understand science properly.

We know that two sugar pills are a more effective treatment than one sugar pill, for example, and we know that salt-water injections are a more effective treatment for pain than sugar pills, not because salt-water injections have any biological action on the body, but because an injection feels like a more dramatic intervention.

As Voltaire said: ‘The art of medicine consists in amusing the patient while nature cures the disease.’

They died, even when there was enough information available to know what would save them, because that information had not been synthesised together, and analysed systematically, in a meta-analysis.

survey of the colour of pills currently on the market found that stimulant medication tends to come in red, orange or yellow tablets, while antidepressants and tranquillisers are generally blue, green or purple.

that aspirin had more of an effect on headaches than sugar pills; but more than that, they found that the packaging itself had a beneficial effect, enhancing the benefit of both the placebo and the aspirin.

Part of that might be the cost: a recent study looking at pain caused by electric shocks showed that a pain relief treatment was stronger when subjects were told it cost $2.50 than when they were told it cost 10c. (And a paper currently in press shows that people are more likely to take advice when they have paid for it.)

operation was performed: in some operations they did the whole thing properly, but in the ‘placebo’ operations they went through the motions but didn’t tie off any arteries. It was found that the placebo operation was just as good as the real one—people seemed to get a bit better in both cases, and there was little difference between the groups—but the most strange thing about the whole affair was that nobody made a fuss at the time: the real operation wasn’t any better than a sham operation, sure, but how could we explain the fact that people had been sensing an improvement from the operation for a very long time? Nobody thought of the power of placebo. The operation was simply binned.

‘Electrical machines have great appeal to patients,’ wrote Dr Alan Johnson in the Lancet in 1994 about this trial, ‘and recently anything to do with the word LASER attached to it has caught the imagination.’

As you will now be realising, in the study of expectation and belief, we can move away from pills and devices entirely. It turns out, for example, that what the doctor says, and what the doctor believes, both have an effect on healing. If that sounds obvious, I should say they have an effect which has been measured, elegantly, in carefully designed trials.

Two hundred patients with abnormal symptoms, but no signs of any concrete medical diagnosis, were divided randomly into two groups. The patients in one group were told, ‘I cannot be certain of what the matter is with you,’ and two weeks later only 39 per cent were better; the other group were given a firm diagnosis, with no messing about, and confidently told they would be better within a few days. Sixty-four per cent of that group got better in two weeks.

And here, it seems that this ‘placebo’ explanation—even if grounded in sheer fantasy—can be beneficial to a patient, although interestingly, perhaps not without collateral damage, and it must be done delicately: assertively and authoritatively giving someone access to the sick role can also reinforce destructive illness beliefs and behaviours, unnecessarily medicalise symptoms like aching muscles (which for many people are everyday occurrences), and militate against people getting on with life and getting better. It’s a very tricky area.

doctors who adopt a warm, friendly and reassuring manner are more effective than those who keep consultations formal and do not offer reassurance.

‘The drive to keep patients fully informed has led to exponential increases in the formal requirements for consent that only serve to confuse and frighten patients while delaying their access to needed medical attention.’

Not only did the patients’ symptoms improve, but their gastric contractions—which ipecac should worsen—were reduced. His results suggest—albeit it in a very small sample—that a drug could be made to have the opposite effect to what you would predict from the pharmacology, simply by manipulating people’s expectations. In this case, the placebo effect outgunned even the pharmacological influences.

If we delve further into theoretical work from the animal kingdom, we find that animals’ immune systems can be conditioned to respond to placebos, in exactly the same way that Pavlov’s dog began to salivate in response to the sound of a bell. Researchers have measured immune system changes in dogs using just flavoured sugar water, once that flavoured water has been associated with immunosuppression, by administering it repeatedly alongside cyclophosphamide, a drug that suppresses the immune system.

similar effect has been demonstrated in humans, when the researchers gave healthy subjects a distinctively flavoured drink at the same time as cyclosporin A (a drug which measurably reduces your immune function). Once the association was set up with sufficient repetition, they found that the flavoured drink on its own could induce modest immune suppression. Researchers have even managed to elicit an association between sherbet and natural killer cell activity.

world without those cultural preconditions to set up the dominoes, you would expect aspirin to do the same job no matter what box it came out of.

just as prescribing a pill in itself carries risks: it medicalises problems, as we will see, it can reinforce destructive beliefs about illness, and it can promote the idea that a pill is an appropriate response to a social problem, or a modest viral illness.

The philosopher Professor Harry Frankfurt of Princeton University discusses this issue at length in his classic 1986 essay ‘On Bullshit’. Under his model, ‘bullshit’ is a form of falsehood distinct from lying: the liar knows and cares about the truth, but deliberately sets out to mislead; the truth-speaker knows the truth and is trying to give it to us; the bullshitter, meanwhile, does not care about the truth, and is simply trying to impress us:

me this is not a great surprise, and it illustrates a very simple issue in epidemiological research called ‘confounding variables’: these are things which are related both to the outcome you’re measuring (wrinkles) and to the exposure you are measuring (food), but which you haven’t thought of yet.

They can confuse an apparently causal relationship,

you have to think of ways to exclude or minimise confounding variables to get to the right answer, or at least b...

Drug companies are very keen to promote theoretical advantages (‘It works more on the Z4 receptor, so it must have fewer side-effects!’), animal experiment data or ‘surrogate outcomes’ (‘It improves blood test results, it must be protective against heart attacks!’) as evidence of the efficacy or superiority of their product.

There are problems with using such surrogate outcomes. They are often only tenuously associated with the real disease, in a very abstract theoretical model, and often developed in the very idealised world of an experimental animal, genetically inbred, kept under conditions of tight physiological control.

Both the tissue and the disease in an animal model, after all, may be very different to those in a living human system, and these problems are even greater with a lab-dish model. Giving unusually high doses of chemicals to animals can distort the usual metabolic pathways, and give misleading results—and so on.

There have been an estimated fifteen million medical academic articles published so far, and 5,000 journals are published every month. Many of these articles will contain contradictory claims: picking out what’s relevant—and what’s not—is a gargantuan task.

That solution is a process called ‘systematic review’. Instead of just mooching around online and picking out your favourite papers to back up your prejudices and help you sell a product, in a systematic review you have an explicit search strategy for seeking out data (openly described in your paper, even including the search terms you used on databases of research papers), you tabulate the characteristics of each study you find, you measure—ideally blind to the results—the methodological quality of each one (to see how much of a ‘fair test’ it is), you compare alternatives, and then finally you give a critical, weighted summary.

This careful sifting of information has revealed huge gaps in knowledge, it has revealed that ‘best practices’ were sometimes murderously flawed, and simply by sifting methodically through pre-existing data, it has saved more lives than you could possibly imagine.

Free radicals are highly chemically reactive, as are many things in the body.

if you have an infection, and there are some harmful bacteria in your body, then a phagocytic cell from your immune system might come along, identify the bacteria as unwelcome, build a strong wall around as many of them as it can find, and blast them with destructive free radicals. Free radicals are basically like bleach, and this process is a lot like pouring bleach down the toilet.

free radicals in the wrong places can damage the desirable components of cells. They can damage the lining of your arteries, and they can damage DNA; and damaged DNA leads to ageing or cancer, and so on. For this reason, it has been suggested that free radicals are responsible for ageing and various diseases. This is a theory, and it may or may not be correct.

Antioxidants are compounds which can—and do—‘mop up’ these free radicals, by reacting with them.

Whenever a piece of evidence is published suggesting that the $50-billion food supplement pill industry’s products are ineffective, or even harmful, an enormous marketing machine lumbers into life, producing spurious and groundless methodological criticisms of the published data in order to muddy the waters—not enough to be noteworthy in a meaningful academic discussion, but that is not their purpose. This is a well-worn risk-management tactic from many industries, including those producing tobacco, asbestos, lead, vinyl chloride, chromium and more. It is called ‘manufacturing doubt’, and in 1969 one tobacco executive was stupid enough to commit it to paper in a memo: ‘Doubt is our product,’ he wrote, ‘since it is the best means of competing with the “body of fact” that exists in the minds of the general public. It is also the means of establishing a controversy.’

two new antioxidant products came on the market, the ultimate expression of how nutritionism has perverted and distorted our common sense about food.

The most important take-home message with diet and health is that anyone who ever expresses anything with certainty is basically wrong, because the evidence for cause and effect in this area is almost always weak and circumstantial, and changing an individual person’s diet may not even be where the action is.

but here it is enough to remind you that the placebo effect is very powerful: consciously or unconsciously, the children will expect themselves to improve, and so will their parents and their teachers.

Children are exquisitely sensitive to our expectations of them,