Bad Science

by Ben Goldacre

At the time of C.P. Snow’s famous lecture on the ‘Two Cultures’ of science and the humanities half a century ago, arts graduates simply ignored us. Today, scientists and doctors find themselves outnumbered and outgunned by vast armies of individuals who feel entitled to pass judgement on matters of evidence—an admirable aspiration—without troubling themselves to obtain a basic understanding of the issues.

That burgers and beer can have negative effects on your body is certainly true, for a number of reasons; but the notion that they leave a specific residue, which can be extruded by a specific process, a physiological system called detox, is a marketing invention.

Like the best pseudoscientific inventions, it deliberately blends useful common sense with outlandish, medicalised fantasy.

people will buy into bogus explanations much more readily when they are dressed up with a few technical words from the world of neuroscience.

Experiments have shown that people are quite bad at estimating the knowledge of others: if we know the answer to a question about a piece of trivia, we overestimate the extent to which other people will know that answer too.

‘Something about seeing neuroscience information may encourage people to believe they have received a scientific explanation when they have not.’

to rate longer explanations as being more similar to ‘experts’ explanations’.

children are predisposed to learn about the world from adults, and specifically from teachers; they are sponges for information, for ways of seeing, and authority figures who fill their heads with nonsense are sowing the ground, I would say, for a lifetime of exploitation.

This process of professionalising the obvious fosters a sense of mystery around science, and health advice, which is unnecessary and destructive. More than anything, more than the unnecessary ownership of the obvious, it is disempowering. All too often this spurious privatisation of common sense is happening in areas where we could be taking control, doing it ourselves, feeling our own potency and our ability to make sensible decisions; instead we are fostering our dependence on expensive outside systems and people.

In general, you don’t absorb things very well through your skin, because its purpose is to be relatively impermeable. When you sit in a bath of baked beans for charity you do not get fat, nor do you start farting.

can very happily view posh cosmetics—and other forms of quackery—as a special, self-administered, voluntary tax on people who don’t understand science properly.

a homeopath complained to the Press Complaints Commission. It’s not about the dilution, he said: it’s the succussion. You have to bang the flask of water briskly ten times on a leather and horsehair surface, and that’s what makes the water remember a molecule. Because I did not mention this, he explained, I had deliberately made homeopaths sound stupid. This is another universe of foolishness.

‘OK, I accept that, but perhaps, also, you feel you’re getting better because of “regression to the mean”.’ This is just one of the many ‘cognitive illusions’ described in this book, the basic flaws in our reasoning apparatus which lead us to see patterns and connections in the world around us, when closer inspection reveals that in fact there are none.

As Voltaire said: ‘The art of medicine consists in amusing the patient while nature cures the disease.’

In fact, it’s not really money that’s missing from the alternative therapy research community, especially in Britain: it’s knowledge of evidence-based medicine, and expertise in how to do a trial. Their literature and debates drip with ignorance, and vitriolic anger at anyone who dares to appraise the trials. Their university courses, as far as they ever even dare to admit what they teach on them (it’s all suspiciously hidden away), seem to skirt around such explosive and threatening questions.

Just as the Big Bang theory is far more interesting than the creation story in Genesis, so the story that science can tell us about the natural world is far more interesting than any fable about magic pills concocted by an alternative therapist.

Mitchel in the early twentieth century was performing full amputations and mastectomies, entirely without anaesthesia; and surgeons from before the invention of anaesthesia often described how some patients could tolerate knife cutting through muscle, and saw cutting through bone, perfectly awake, and without even clenching their teeth. You might be tougher than you think.

But these are just stories, and the plural of anecdote is not data.

In the real world of clinical practice, patients and doctors aren’t so interested in whether a new drug works better than nothing, they’re interested in whether it works better than the best treatment they already have.

For the moment we will focus on ‘nutritionists’, members of a newly invented profession who must create a commercial space to justify their own existence. In order to do this, they must mystify and overcomplicate diet, and foster your dependence upon them. Their profession is based on a set of very simple mistakes in how we interpret scientific literature: they extrapolate wildly from ‘laboratory bench data’ to make claims about humans; they extrapolate from ‘observational data’ to make ‘intervention claims’; they ‘cherry-pick’; and, lastly, they quote published scientific research evidence which seems, as far as one can tell, not to exist.

The philosopher Professor Harry Frankfurt of Princeton University discusses this issue at length in his classic 1986 essay ‘On Bullshit’. Under his model, ‘bullshit’ is a form of falsehood distinct from lying: the liar knows and cares about the truth, but deliberately sets out to mislead; the truth-speaker knows the truth and is trying to give it to us; the bullshitter, meanwhile, does not care about the truth, and is simply trying to impress us:

Does the cock’s crow cause the sun to rise? No. Does this light switch make the room get brighter? Yes. Things can happen at roughly the same time, but that is weak, circumstantial evidence for causation. Yet it’s exactly this kind of evidence that is used by media nutritionists as confident proof of their claims in our second major canard.

‘confounding variables’: these are things which are related both to the outcome you’re measuring (wrinkles) and to the exposure you are measuring (food), but which you haven’t thought of yet. They can confuse an apparently causal relationship, and you have to think of ways to exclude or minimise confounding variables to get to the right answer, or at least be very wary that they are there. In the case of this study, there are almost too many confounding variables to describe.

you are almost certainly witnessing a journalist of limited intellect, overinterpreting a study with huge confounding variables.

media nutritionists, in their eagerness to make lifestyle claims, are all too often blind to the problems of applying these isolated theoretical nuggets to humans, and anyone would think they were just trawling the internet looking for random bits of science to sell their pills and expertise (imagine that).

The idea is to try and give all the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.

solution is a process called ‘systematic review’. Instead of just mooching around online and picking out your favourite papers to back up your prejudices and help you sell a product, in a systematic review you have an explicit search strategy for seeking out data (openly described in your paper, even including the search terms you used on databases of research papers), you tabulate the characteristics of each study you find, you measure—ideally blind to the results—the methodological quality of each one (to see how much of a ‘fair test’ it is), you compare alternatives, and then finally you give a critical, weighted summary.

but again (although nutritionists are keen to ignore this fact), these are simply surveys of people who have already chosen to take vitamin pills. These are people who are more likely to care about their health, and are different from the everyday population—and perhaps from you—in lots of other ways, far beyond their vitamin pill consumption: they may take more exercise, have more social supports, smoke less, drink less, and so on.

Eat lots of fruit and vegetables, and live your whole life in every way as well as you can: exercise regularly as part of your daily routine, avoid obesity, don’t drink too much, don’t smoke, and don’t get distracted from the real, basic, simple causes of ill health.

Diet has been studied very extensively, and there are some things that we know with a fair degree of certainty: there is reasonably convincing evidence that having a diet rich in fresh fruit and vegetables, with natural sources of dietary fibre, avoiding obesity, moderating one’s intake of alcohol, cutting out cigarettes and taking physical exercise are protective against things such as cancer and heart disease.

Nutritionists don’t stop there, because they can’t: they have to manufacture complication, to justify the existence of their profession. These new nutritionists have a major commercial problem with the evidence. There’s nothing very professional or proprietary about ‘Eat your greens,’ so they have had to push things further. But unfortunately for them, the technical, confusing, overcomplicated, tinkering interventions that they promote—the enzymes, the exotic berries—are very frequently not supported by convincing evidence.

Sensible dietary practices, which we all know about, still stand. But the unjustified, unnecessary overcomplication of this basic dietary advice is, to my mind, one of the greatest crimes of the nutritionist movement.

individual people do not, in reality, change their diets at the drop of a hat, alone, as individuals, for the long term. A dietary change probably requires a change in lifestyle, shopping habits, maybe even what’s in the shops, how you use your time, it might even require that you buy some cooking equipment, how your family relates to each other, change your work style, and so on.

Children are exquisitely sensitive to our expectations of them, and anyone who doubts that fact should have their parenting permit revoked.

the placebo is not about the mechanics of a sugar pill, it is about the cultural meaning of an intervention, which includes, amongst other things, your expectations, and the expectations of the people tending to you and measuring you.

George Orwell first noted, the true genius in advertising is to sell you the solution and the problem.

they cannot find new treatments for the diseases we already have, the pill companies instead invent new diseases for the treatments they already have. Recent favourites include Social Anxiety Disorder (a new use for SSRI drugs), Female Sexual Dysfunction (a new use for Viagra in women), night eating syndrome (SSRIs again) and so on: problems, in a real sense, but perhaps not necessarily the stuff of pills, and perhaps not best conceived of in reductionist biomedical terms.

they have sold children, at the most impressionable time of their lives, one very compelling message: that you need to take pills to lead a healthy normal life, that a sensible diet and lifestyle are not enough in themselves, and that a pill can even make up for failings elsewhere.

(although I can’t imagine the point of taking health advice from someone who is only occasionally correct).

I welcome other people challenging my ideas: it helps me to refine them.

If you secure the legal rights of prostitutes to be free from violence and discrimination, you empower them to demand universal condom use, and that way you can prevent HIV from being spread into the whole community. This is where science meets culture. But perhaps even to your own friends and neighbours, in whatever suburban idyll has become your home, the moral principle of abstinence from sex and drugs is more important than people dying of AIDS; and perhaps, then, they are no less irrational than Thabo Mbeki.

The alternative therapy movement as a whole has demonstrated itself to be so dangerously, systemically incapable of critical self-appraisal that it cannot step up even in a case like that of Rath: in that count I include tens of thousands of practitioners, writers, administrators and more.

the biggest dangers posed by the material we have covered are cultural and intellectual.

you will also have to rise above your own narcissism.

First of all, you need an idea for a drug. This can come from any number of places: a molecule in a plant; a receptor in the body that you think you can build a molecule to interface with; an old drug that you’ve tinkered with; and so on. This part of the story is extremely interesting, and I recommend doing a degree in it. When you think you have a molecule that might be a runner, you test it in animals, to see if it works for whatever you think it should do (and to see if it kills them, of course).

Then you do Phase I, or ‘first in man’, studies on a small number of brave, healthy young men who need money, firstly to see if it kills them, and also to measure basic things like how fast the drug is excreted from the body (this is the phase that went horribly wrong in the TGN1412 tests in 2006, where several young men were seriously injured). If this works, you move to a Phase II trial, in a couple of hundred people with the relevant illness, as a ‘proof of concept’, to work out the dose, and to get an idea if it is effective or not. A lot of drugs fail at this point, which is a shame, since this is no GCSE science project: bringing a drug to market costs around $500 million in total.

Then you do a Phase III trial, in hundreds or thousands of patients, randomised, blinded, comparing your drug against placebo or a comparable treatment, and collect much more data on efficacy and safety. You might need to do a few of these, and then you can apply for a licence to sell your drug. After it goes to market, you should be doing more trials, and other people will probably do trials and other studies on your drug too; and hopefully everyone wi...

A key feature of science is that findings should be replicated, but if only one organisation is doing the funding, then this feature is lost.

‘Torture the data and it will confess to anything,’

you should always read the methods and results section of a trial to decide what its findings are, because the discussion and conclusion pages at the end are like the comment pages in a newspaper. They’re not where you get your news from.