How Not to Be Wrong: The Power of Mathematical Thinking

by Jordan Ellenberg

It feels like something is making it happen. Indeed, de Moivre himself might have felt this way. By many accounts, he viewed the regularities in the behavior of repeated coin flips (or any other experiment subject to chance) as the work of God’s hand itself, which turned the short-term irregularities of coins, dice, and human life into predictable long-term behavior, governed by immutable laws and decipherable formulae. It’s dangerous to feel this way. Because if you think somebody’s transcendental hand—God, Lady Luck, Lakshmi, doesn’t matter—is pushing the coins to come up half heads, you start to believe in the so-called law of averages: five heads in a row and the next one’s almost sure to land tails. Have three sons, and a daughter is surely up next.

I might start flipping and get 10 heads in a row. What happens next? Well, one thing that might happen is you’d start to suspect something was funny about the coin. We’ll return to that issue in part II, but for now let’s assume the coin is fair. So the law demands that the proportion of heads must approach 50% as I flip the coin more and more times. Common sense suggests that, at this point, tails must be slightly more likely, in order to correct the existing imbalance. But common sense says much more insistently that the coin can’t remember what happened the first ten times I flipped it!

The law of averages is not very well named, because laws should be true, and this one is false. Coins have no memory. So the next coin you flip has a 50-50 chance of coming up heads, the same as any other. The way the overall proportion settles down to 50% isn’t that fate favors tails to compensate for the heads that have already landed; it’s that those first ten flips become less and less important the more flips we make.

That’s how the Law of Large Numbers works: not by balancing out what’s already happened, but by diluting what’s already happened with new data, until the past is so proportionally negligible that it can safely be forgotten.

If you rate your bloodshed by proportion of national population eliminated, the worst offenses will tend to be concentrated in the smallest countries. Matthew White, author of the agreeably morbid Great Big Book of Horrible Things, ranked the bloodlettings of the twentieth century in this order, and found that the top three were the massacre of the Herero of Namibia by their German colonists, the slaughter of Cambodians by Pol Pot, and King Leopold’s war in the Congo. Hitler, Stalin, Mao, and the big populations they decimated don’t make the list.

Most mathematicians would say that, in the end, the disasters and atrocities of history form what we call a partially ordered set. That’s a fancy way of saying that some pairs of disasters can be meaningfully compared and others cannot.

the question of whether one war was worse than another is fundamentally unlike the question of whether one number is bigger than another. The latter question always has an answer. The former does not.

Between 1990 and 2008, the U.S. economy gained a net 27.3 million jobs. Of those, 26.7 million, or 98%, came from the “nontradable sector”: the part of the economy including things like government, health care, retail, and food service, which can’t be outsourced and which don’t produce goods to be shipped overseas.

Jobs in the tradable sector grew by a mere 620,000 between 1990 and 2008, that’s true. But it could have been worse—they could have declined! That’s what happened between 2000 and 2008; the tradable sector lost about 3 million jobs, while the nontradable sector added 7 million. So the nontradable sector accounted for 7 million jobs out of the total gain of 4 million, or 175%!

Don’t talk about percentages of numbers when the numbers might be negative.

The great sixteenth-century algebraists, like Cardano and François Viète, argued furiously about whether a negative times a negative equaled a positive; or rather, they understood that consistency seemed to demand that this be so, but there was real division about whether this had been proved factual or was only a notational expedient.

As you might remember, there was at least one other part of the U.S. economy that added a lot of jobs between 1990 and today: the sector classified as “computer systems design and related services,” which tripled its job numbers, adding more than a million jobs all by itself. The total jobs added by finance and computers were way over the 620,000 jobs added by the tradable sector as a whole; those gains were balanced out by big losses in manufacturing. The combination of positive and negative allows you, if you’re not careful, to tell a fake story, in which the whole work of job creation in the tradable sector was done by the financial industry.

It had been another weak month for the U.S. economy as a whole, which added only eighteen thousand jobs nationally. But the state employment numbers looked much better: a net increase of ninety-five hundred jobs. “Today,” the statement read, “we learned that over 50 percent of U.S. job growth in June came from our state.” The talking point was picked up and distributed by GOP politicians, like Representative Jim Sensenbrenner, who told an audience in a Milwaukee suburb, “The labor report that came out last week had an anemic eighteen thousand created in this country, but half of them came here in Wisconsin. Something we are doing here must be working.”

In fact, what was going on is that job losses in other states almost exactly balanced out the jobs created in places like Wisconsin, Massachusetts, and Texas. That’s how Wisconsin’s governor could claim his state accounted for half the nation’s job growth, and Minnesota’s governor, if he’d cared to, could have said that his own state was responsible for 70% of it, and they could both, in this technically correct but fundamentally misleading way, be right.

The Washington Post graded the Romney campaign’s 92.3% figure as “true but false.” That classification drew mockery by Romney supporters, but I think it’s just right, and has something deep to say about the use of numbers in politics. There’s no question about the accuracy of the number. You divide the net jobs lost by women by the net jobs lost, and you get 92.3%. But that makes the claim “true” only in a very weak sense. It’s as if the Obama campaign had released a statement saying, “Mitt Romney has never denied allegations that for years he’s operated a bicontinental cocaine-trafficking ring in Colombia and Salt Lake City.” That statement is also 100% true! But it’s designed to create a false impression. So “true but false” is a pretty fair assessment. It’s the right answer to the wrong question.

But real-world questions aren’t like word problems. A real-world problem is something like “Has the recession and its aftermath been especially bad for women in the workforce, and if so, to what extent is this the result of Obama administration policies?” Your calculator doesn’t have a button for this. Because in order to give a sensible answer, you need to know more than just numbers. What shape do the job-loss curves for men and women have in a typical recession? Was this recession notably different in that respect? What kind of jobs are disproportionately held by women, and what decisions has Obama made that affect that sector of the economy?

Dividing one number by another is mere computation; figuring out what you should divide by what is mathematics.

there’s a realm of questions out beyond cosmic, questions about The Meaning and Origin of It All, questions you might think mathematics could have no purchase on. Never underestimate the territorial ambitions of mathematics! You want to know about God? There are mathematicians on the case.

At this point, the rabbi determined that the codes were to be rejected; for while there is indeed a Jewish tradition, particularly among rabbis with mystical leanings, of carrying out numerical analysis of the letters of the Torah, the process is meant only to aid in understanding and appreciating the holy book. If the method could be used, even in principle, to induce doubt as to the basic laws of the faith, it was about as authentically Jewish as a bacon cheeseburger.

The Baltimore stockbroker con works because, like all good magic tricks, it doesn’t try to fool you outright. That is, it doesn’t try to tell you something false—rather, it tells you something true from which you’re likely to draw incorrect conclusions.

The universe is big, and if you’re sufficiently attuned to amazingly improbable occurrences, you’ll find them. Improbable things happen a lot.

Most coincidences lose their snap when viewed from the appropriate distance. On July 9, 2007, the North Carolina Cash 5 lottery numbers came up 4, 21, 23, 34, 39. Two days later, the same five numbers came up again. That seems highly unlikely, and it seems that way because it is.

That doesn’t make each individual coincidence any less improbable. But here comes the chorus again: improbable things happen a lot.

Aristotle, as usual, was here first: despite lacking any formal notion of probability, he was able to understand that “it is probable that improbable things will happen. Granted this, one might argue that what is improbable is probable.”

In the British statistician R. A. Fisher’s famous formulation, “the ‘one chance in a million’ will undoubtedly occur, with no less and no more than its appropriate frequency, however surprised we may be that it should occur to us.”

When you’re trying to draw reliable inferences from improbable events, wiggle room is the enemy.

Drosnin said of code skeptics, “When my critics find a message about the assassination of a prime minister encrypted in Moby-Dick, I’ll believe them.” McKay quickly found equidistant letter sequences in Moby-Dick referring to the assassination of John F. Kennedy, Indira Gandhi, Leon Trotsky, and, for good measure, Drosnin himself.

But the paper, of course, is a deadpan gag. (And a well-executed one: I especially like the “Methods” section, which starts “One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning. . . . Foam padding was placed within the head coil as a method of limiting salmon movement during the scan, but proved to be largely unnecessary as subject motion was exceptionally low.”)

We need to think very carefully about whether our standards for evidence are strict enough, if the empathetic salmon makes the cut.

If a random Internet stranger who eliminated all North American grains from his food intake reports that he dropped fifteen pounds and his eczema went away, you shouldn’t take that as powerful evidence in favor of the maize-free plan. Somebody’s selling a book about that plan, and thousands of people bought that book and tried it, and the odds are very good that, by chance alone, one among them will experience some weight loss and clear skin the next week. And that’s the guy who’s going to log in as saygoodbye2corn452 and post his excited testimonial, while the people for whom the diet failed stay silent.

There are two moments in the course of education where a lot of kids fall off the math train. The first comes in the elementary grades, when fractions are introduced. Until that moment, a number is a natural number, one of the figures 0, 1, 2, 3 . . . It is the answer to a question of the form “how many.”* To go from this notion, so primitive that many animals are said to understand it, to the radically broader idea that a number can mean “what portion of,” is a drastic philosophical shift. (“God made the natural numbers,” the nineteenth-century algebraist Leopold Kronecker famously said, “and all the rest is the work of man.”)

Perhaps it doesn’t seem so troubling, especially if you’re used to the quadratic formula, to get two answers to the same question. But when you’re twelve it represents a real philosophical shift. You’ve spent six long years in grade school figuring out what the answer is, and now, suddenly, there is no such thing.

When you try to think carefully about what probability means, you get a little woozy. When we say, “The probability that a flipped coin will land heads is 1/2,” we’re invoking the Law of Large Numbers from chapter 4, which says that if you flip the coin many, many times, the proportion of heads will almost inevitably approach 1/2, as if constrained by a narrowing channel. This is what’s called the frequentist view of probability. But what can we mean when we say, “The probability that it will rain tomorrow is 20%”? Tomorrow only happens once; it’s not an experiment we can repeat like a coin flip again and again. With some effort, we can shoehorn the weather into the frequentist model; maybe we mean that among some large population of days with conditions similar to this one, the following day was rainy 20% of the time. But then you’re stuck when asked, “What’s the probability that the human race will go extinct in the next thousand years?” This is, almost by definition, an experiment you can’t repeat.

we find ourselves able to say, of questions like this, “It seems improbable” or “It seems likely.” Once we’ve done so, how can we resist the temptation to ask, “How likely?” It’s one thing to ask, another to answer.

Improbability, as described here, is a relative notion, not an absolute one; when we say an outcome is improbable, we are always saying, explicitly or not, that it is improbable under some set of hypotheses we’ve made about the underlying mechanisms of the world.

The “does nothing” scenario is called the null hypothesis. That is, the null hypothesis is the hypothesis that the intervention you’re studying has no effect.

The standard framework, called the null hypothesis significance test, was developed in its most commonly used form by R. A. Fisher, the founder of the modern practice of statistics,* in the early twentieth century.

It’s not enough that the data be consistent with your theory; they have to be inconsistent with the negation of your theory, the dreaded null hypothesis. I may assert that I possess telekinetic abilities so powerful that I can drag the sun out from beneath the horizon—if you want proof, just go outside at about five in the morning and see the results of my work! But this kind of evidence is no evidence at all, because, under the null hypothesis that I lack psychic gifts, the sun would come up just the same.

In the same way, it’s not very likely that exactly the same number of drug patients and placebo patients expire during the course of the trial. I computed: 13.3% chance equally many drug and placebo patients die 43.3% chance fewer placebo patients than drug patients die 43.3% chance fewer drug patients than placebo patients die. Seeing better results among the drug patients than the placebo patients says very little, since this isn’t at all unlikely, even under the null hypothesis that your drug doesn’t work.

If I claim I can make the sun come up with my mind, and it does, you shouldn’t be impressed by my powers; but if I claim I can make the sun not come up, and it doesn’t, then I’ve demonstrated an outcome very unlikely under the null hypothesis, and you’d best take notice.

So here’s the procedure for ruling out the null hypothesis, in executive bullet-point form: Run an experiment. Suppose the null hypothesis is true, and let p be the probability (under that hypothesis) of getting results as extreme as those observed. The number p is called the p-value. If it is very small, rejoice; you get to say your results are statistically significant. If it is large, concede that the null hypothesis has not been ruled out.

There’s no principled way to choose a sharp dividing line between what is significant and what is not; but there’s a tradition, which started with Fisher himself and is now widely adhered to, of taking p= 0.05, or 1/20, to be the threshold.

But why should the chance be equal? Nicholas Bernoulli proposed a different null hypothesis: that the sex of a child is determined by chance, with an 18/35 chance of being a boy and 17/35 of being a girl. Bernoulli’s null hypothesis is just as atheistic as Arbuthnot’s, and it fits the data perfectly. If you flip a coin 82 times and get 82 heads, you ought to be thinking, “Something is biased about this coin,” not “God loves heads.”*

Arbuthnot is intellectual father not only to the Bible coders but to the “creation scientists,” who argue, even today, that mathematics demands there must be a god, on the grounds that a godless world would be highly unlikely to look like the one we have.*

I once had an uneasy moment with a colleague in an airport when he made the remark, unexceptional in a mathematical context, that it might be necessary to blow up the plane at one point.

one scientist, a supporter of the CSM decision to issue the warning, said the total number of embolism deaths prevented was “possibly one.” The added risk posed by third-generation birth control pills, while significant in Fisher’s statistical sense, was not so significant in the sense of public health.

Among women of childbearing age using first- and second-generation oral contraceptives, 1 in 7,000 could expect to suffer a thrombosis; users of the new pill indeed had twice as much risk, 2 in 7,000. But that’s still a very small risk, because of this certified math fact: twice a tiny number is a tiny number.

Every part of the body speaks to every other, in a complex feedback loop of influence and control. Everything you do either gives you cancer or prevents it.

Frost wrote Skinner a very satisfactory letter praising his stories and counseling: “All that makes a writer is the ability to write strongly and directly from some unaccountable and almost invincible personal prejudice. . . . I take it that everybody has the prejudice and spends some time feeling for it to speak and write from. But most people end as they begin by acting out the prejudices of other people.”

Skinner had been much struck by an experience of spontaneous verbal production he’d experienced in his lab; a machine in the background was making a repetitive, rhythmic sound, and Skinner found himself talking along with it, following the beat, silently repeating the phrase “You’ll never get out, you’ll never get out, you’ll never get out.” What seemed like speech, or even, in a small way, like poetry, was actually the result of a kind of autonomous verbal process, requiring nothing like a conscious author.* This provided just the idea Skinner needed to settle his score with literature. What if language, even the language of the great poets, was just another behavior, trained by exposure to stimuli, and manipulable in the lab?