How Not to Be Wrong: The Power of Mathematical Thinking

by Jordan Ellenberg

One thing the American defense establishment has traditionally understood very well is that countries don’t win wars just by being braver than the other side, or freer, or slightly preferred by God. The winners are usually the guys who get 5% fewer of their planes shot down, or use 5% less fuel, or get 5% more nutrition into their infantry at 95% of the cost. That’s not the stuff war movies are made of, but it’s the stuff wars are made of. And there’s math every step of the way.

To paraphrase Clausewitz: Mathematics is the extension of common sense by other means.

A significance test is a scientific instrument, and like any other instrument, it has a certain degree of precision. If you make the test more sensitive—by increasing the size of the studied population, for example—you enable yourself to see ever-smaller effects. That’s the power of the method, but also its danger.

But impossible and improbable are not the same—not even close. Impossible things never happen. But improbable things happen a lot.

Hacking the p isn’t usually as crude as I’ve made it out to be, and it’s seldom malicious. The p-hackers truly believe in their hypotheses, just as the Bible coders do, and when you’re a believer, it’s easy to come up with reasons that the analysis that gives a publishable p-value is the one you should have done in the first place.

In the Bayesian framework, how much you believe something after you see the evidence depends not just on what the evidence shows, but on how much you believed it to begin with. That may seem troubling. Isn’t science supposed to be objective? You’d like to say that your beliefs are based on evidence alone, not on some prior preconceptions you walked in the door with. But let’s face it—no one actually forms their beliefs this way. If an experiment provided statistically significant evidence that a new tweak of an existing drug slowed the growth of certain kinds of cancer, you’d probably be pretty confident the new drug was actually effective. But if you got the exact same results by putting patients inside a plastic replica of Stonehenge, would you grudgingly accept that the ancient formations were actually focusing vibrational earth energy on the body and stunning the tumors? You would not, because that’s nutty. You’d think Stonehenge probably got lucky.

Relying purely on null hypothesis significance testing is a deeply non-Bayesian thing to do—strictly speaking, it asks us to treat the cancer drug and the plastic Stonehenge with exactly the same respect.

You probably know what Sherlock Holmes had to say about inference, the most famous thing he ever said that wasn’t “Elementary!”: “It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth.” Doesn’t that sound cool, reasonable, indisputable? But it doesn’t tell the whole story. What Sherlock Holmes should have said was: “It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth, unless the truth is a hypothesis it didn’t occur to you to consider.”

But what can we mean when we say, “There’s a 20% chance that God created the universe?” It can’t be that one in five universes was made by God and the rest popped up on their own. The truth is, I’ve never seen a method I find satisfying for assigning numbers to our uncertainty about ultimate questions of this kind. As much as I love numbers, I think people ought to stick to “I don’t believe in God,” or “I do believe in God,” or just “I’m not sure.” And as much as I love Bayesian inference, I think people are probably best off arriving at their faith, or discarding it, in a non-quantitative way. On this matter, math is silent.

usual, the mathematical approach is a formalized version of our natural mental reckonings, an extension of common sense by other means.

Galton observed that the curves he drew by hand looked like ellipses, but was not quite geometer enough to be sure that this precise curve, and not some other more or less ovoid figure, was actually in charge. Was he letting his desire for an elegant and universal theory affect his perception of the data he’d collected? He wouldn’t be the first or last scientist to make that mistake.

mean to restrict its sphere of influence over our actions. There is real danger that, by strengthening our abilities to analyze some questions mathematically, we acquire a general confidence in our beliefs, which extends unjustifiably to those things we’re still wrong about. We become like those pious people who, over time, accumulate a sense of their own virtuousness so powerful as to make them believe the bad things they do are virtuous too.

Mathematics, mostly, is a communal enterprise, each advance the product of a huge network of minds working toward a common purpose, even if we accord special honor to the person who places the last stone in the arch. Mark Twain is good on this: “It takes a thousand men to invent a telegraph, or a steam engine, or a phonograph, or a telephone or any other important thing—and the last man gets the credit and we forget the others.”