How Not to Be Wrong: The Power of Mathematical Thinking

by Jordan Ellenberg

So here’s the procedure for ruling out the null hypothesis, in executive bullet-point form: Run an experiment. Suppose the null hypothesis is true, and let p be the probability (under that hypothesis) of getting results as extreme as those observed. The number p is called the p-value. If it is very small, rejoice; you get to say your results are statistically significant. If it is large, concede that the null hypothesis has not been ruled out. How small is “very small”? There’s no principled way to choose a sharp dividing line between what is significant and what is not; but there’s a tradition, which started with Fisher himself and is now widely adhered to, of taking p= 0.05, or 1/20, to be the threshold. Null hypothesis significance testing is popular because it captures our intuitive way of reasoning about uncertainty. Why do we find the Bible codes compelling, at least at first glance? Because codes like the ones Witztum uncovered are very unlikely under the null hypothesis that the Torah doesn’t know the future. The value of p—the likelihood of finding so many equidistant letter sequences, so accurate in their demographic profiling of notable rabbis—is very close to 0.