The Skeptics' Guide to the Universe: How to Know What's Really Real in a World Increasingly Full of Fake

by Steven Novella

The term derives from the statistical calculation known as the p-value. The p-value is just one way to look at scientific data. It first assumes a specific null hypothesis (such as: There is no correlation between these two variables) and then asks, What is the probability that the data would be at least as extreme as they are if the null hypothesis were true? A p-value of 0.05 (a typical threshold for being considered “significant”) indicates a 5 percent probability that the data are due to chance rather than a real effect. Except—that’s not actually true. That is how most people interpret the p-value, but that’s not what it actually says. P-values don’t consider many other important variables, like prior probability, effect size, confidence intervals, and alternative hypotheses. For example, if we ask, “What is the probability of a new set of data replicating the results of a study with a p-value of 0.05?” we get a very different answer.