Bad Science

by Ben Goldacre

Put simply, the subjects’ faith in research data was not predicated on an objective appraisal of the research methodology, but on whether the results validated their pre-existing views.

Our assessment of the quality of new evidence is biased by our previous beliefs.

The anecdotal success stories about CAM—and the tragic anecdotes about the MMR vaccine—are disproportionately misleading, not just because the statistical context is missing, but because of their ‘high availability’: they are dramatic, associated with strong emotion, and amenable to strong visual imagery. They are concrete and memorable, rather than abstract. No matter what you do with statistics about risk or recovery, your numbers will always have inherently low psychological availability, unlike miracle cures, scare stories, and distressed parents.

This is why statisticians do a ‘correction for multiple comparisons’, a correction for ‘rolling the dice’ lots of times. This, like correcting for clustering, is particularly brutal on the data, and often reduces the significance of findings dramatically.

This breaks a cardinal rule of any research involving statistics: you cannot find your hypothesis in your results.

Before you go to your data with your statistical tool, you have to have a specific hypothesis to test. If your hypothesis comes from analysing the data, then there is no sense in analysing the same data again to confirm it.

There are lots of ways to distinguish one type of bacteria from another, and you can learn some of the tricks at home with a cheap toy microscope: you might look at them, to see what shape they are, or what kinds of dyes and stains they pick up. You can see what shapes and colours the colonies make as they grow on ‘culture media’ in a glass dish, and you can look at whether certain things in the culture media affect their growth (like the presence of certain antibiotics, or types of nutrient). Or you can do genetic fingerprinting on them. These are just a few examples.