The Skeptics' Guide to the Universe: How to Know What's Really Real in a World Increasingly Full of Fake

by Steven Novella

Just as with breast cancer, you have to know what the base rate is. We also call this the prior probability. With a scientific hypothesis this often means the scientific plausibility. The lower the plausibility (just like the lower the rate of cancer in the population), the smaller the percentage of positive or statistically significant studies will be true positives. This means that we can’t know the probability that a hypothesis is true just from the p-value of one study. We need to know the plausibility of the hypothesis, and we need to know what all the other relevant studies showed. We call this a Bayesian approach—you take the new information, you add it to the prior existing information, and you come up with a new probability that the idea is true. While it’s possible to debate about how plausible a hypothesis is, one thing is clear: p-values (statistical significance) don’t mean nearly as much as most people think they do. Even a significant study doesn’t alter the prior probability by that much. It takes a convergence of multiple studies, and multiple independent lines of evidence, before we can be confident that a hypothesis is likely to be true.