Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

A model, after all, is nothing more than an abstract representation of some process, be it a baseball game, an oil company’s supply chain, a foreign government’s actions, or a movie theater’s attendance. Whether it’s running in a computer program or in our head, the model takes what we know and uses it to predict responses in various situations. All of us carry thousands of models in our heads. They tell us what to expect, and they guide our decisions.

To create a model, then, we make choices about what’s important enough to include, simplifying the world into a toy version that can be easily understood and from which we can infer important facts and actions. We expect it to handle only one job and accept that it will occasionally act like a clueless machine, one with enormous blind spots.

A model’s blind spots reflect the judgments and priorities of its creators.

Here we see that models, despite their reputation for impartiality, reflect goals and ideology. When I removed the possibility of eating Pop-Tarts at every meal, I was imposing my ideology on the meals model. It’s something we do without a second thought. Our own values and desires influence our choices, from the data we choose to collect to the questions we ask. Models are opinions embedded in mathematics.

So to sum up, these are the three elements of a WMD: Opacity, Scale, and Damage. All of them will be present, to one degree or another, in the examples we’ll be covering.

Now, if this were a tiny anomaly in financial markets, and not a commuter train, a quant at a hedge fund—someone like me—could zero in on it. It would involve going through years of data, even decades, and then training an algorithm to predict this one recurring error—a fifty-cent swing in price—and to place bets on it. Even the smallest patterns can bring in millions to the first investor who unearths them. And they’ll keep churning out profits until one of two things happens: either the phenomenon comes to an end or the rest of the market catches on to it, and the opportunity vanishes. By that point, a good quant will be hot on the trail of dozens of other tiny wrinkles. The quest for what quants call market inefficiencies is like a treasure hunt. It can be fun.

We had about fifty quants in total. In the early days, it was entirely men, except for me. Most of them were foreign born. Many of them had come from abstract math or physics; a few, like me, had come from number theory. I didn’t get much of a chance to talk shop with them, though. Since our ideas and algorithms were the foundation of the hedge fund’s business, it was clear that we quants also represented a risk: if we walked away, we could quickly use our knowledge to fuel a fierce competitor. To keep this from happening on a large, firm-threatening scale, Shaw mostly prohibited us from talking to colleagues in other groups—or sometimes even our own office mates—about what we were doing. In a sense, information was cloistered in a networked cell structure, not unlike that of Al Qaeda. That way, if one cell collapsed—if one of us hightailed it to Bridgewater or J.P. Morgan, or set off on our own—we’d take with us only our own knowledge. The rest of Shaw’s business would carry on unaffected. As you can imagine, this wasn’t terrific for camaraderie.

The troubles had actually started a year earlier. In July of 2007, “interbank” interest rates spiked. After the recession that followed the terrorist attacks in 2001, low interest rates had fueled a housing boom. Anyone, it seemed, could get a mortgage, builders were turning exurbs, desert, and prairie into vast new housing developments, and banks gambled billions on all kinds of financial instruments tied to the building bonanza. But these rising interest rates signaled trouble. Banks were losing trust in each other to pay back overnight loans. They were slowly coming to grips with the dangerous junk they held in their own portfolios and judged, wisely, that others were sitting on just as much risk, if not more. Looking back, you could say the interest rate spikes were actually a sign of sanity, although they obviously came too late.

To understand how hedge funds operate at the margins, picture a World Series game at Chicago’s Wrigley Field. With a dramatic home run in the bottom of the ninth inning, the Cubs win their first championship since 1908, back when Teddy Roosevelt was president. The stadium explodes in celebration. But a single row of fans stays seated, quietly analyzing a slew of results. These gamblers don’t hold the traditional win-or-lose bets. Instead they may have bet that Yankees relievers would give up more walks than strikeouts, that the game would feature at least one bunt but no more than two, or that the Cubs’ starter would last at least six innings. They even hold bets that other gamblers will win or lose their own bets. These people wager on many movements associated with the game, but not as much on the game itself. In this, they behave like hedge funds.

And so in the 1980s, investment bankers started to buy thousands of mortgages and package them into securities—a kind of bond, which is to say an instrument that pays regular dividends, often at quarterly intervals. A few of the home owners would default, of course. But most people would stay afloat and keep paying their mortgages, generating a smooth and predictable flow of revenue. In time, these bonds grew into an entire industry, a pillar of the capital markets. Experts grouped the mortgages into different classes, or tranches. Some were considered rock solid. Others carried more risk—and higher interest rates. Investors had reason to feel confident because the credit-rating agencies, Standard & Poor’s, Moody’s, and Fitch, had studied the securities and scored them for risk. They considered them sensible investments. But consider the opacity. Investors remained blind to the quality of the mortgages in the securities. Their only glimpse of what lurked inside came from analyst ratings. And these analysts collected fees from the very companies whose products they were rating. Mortgage-backed securities, needless to say, were an ideal platform for fraud.

As the world later learned, mortgage companies were making rich profits during the boom by loaning money to people for homes they couldn’t afford. The strategy was simply to write unsustainable mortgages, snarf up the fees, and then unload the resulting securities—the sausages—into the booming mortgage security market. In one notorious case, a strawberry picker named Alberto Ramirez, who made $14,000 a year, managed to finance a $720,000 house in Rancho Grande, California. His broker apparently told him that he could refinance in a few months and later flip the house and make a tidy profit. Months later, he defaulted on the loan.

To be clear, the subprime mortgages that piled up during the housing boom, whether held by strawberry pickers in California or struggling black congregants in Baltimore, were not WMDs. They were financial instruments, not models, and they had little to do with math. (In fact, the brokers went to great lengths to ignore inconvenient numbers.) But when banks started loading mortgages like Alberto Ramirez’s into classes of securities and selling them, they were relying on flawed mathematical models to do it. The risk model attached to mortgage-backed securities was a WMD.

The first false assumption was that crack mathematicians in all of these companies were crunching the numbers and ever so carefully balancing the risk. The bonds were marketed as products whose risk was assessed by specialists using cutting-edge algorithms. Unfortunately, this just wasn’t the case. As with so many WMDs, the math was directed against the consumer as a smoke screen. Its purpose was only to optimize short-term profits for the sellers. And those sellers trusted that they’d manage to unload the securities before they exploded. Smart people would win. And dumber people, the providers of dumb money, would wind up holding billions (or trillions) of unpayable IOUs. Even rigorous mathematicians—and there were a few—were working with numbers provided by people carrying out wide-scale fraud. Very few people had the expertise and the information required to know what was actually going on statistically, and most of the people who did lacked the integrity to speak up. The risk ratings on the securities were designed to be opaque and mathematically intimidating, in part so that buyers wouldn’t perceive the true level of risk associated with the contracts they owned. The second false assumption was that not many people would default at the same time. This was based on the theory, soon to be disproven, that defaults were largely random and unrelated events. This led to a belief that solid mortgages would offset the losers in each tranche. The risk models were assuming that...

when you create a model from proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.

As the rankings grow, so do efforts to game them. In a 2014 U.S. News ranking of global universities, the mathematics department at Saudi Arabia’s King Abdulaziz University landed in seventh place, right behind Harvard. The department had been around for only two years but had somehow leapfrogged ahead of several giants of mathematics, including Cambridge and MIT. At first blush, this might look like a positive development. Perhaps MIT and Cambridge were coasting on their fame while a hardworking insurgent powered its way into the elite. With a pure reputational ranking, such a turnaround would take decades. But data can bring surprises to the surface in a hurry. Algorithms, though, can also be gamed. Lior Pachter, a computational biologist at Berkeley, looked into it. He found that the Saudi university had contacted a host of mathematicians whose work was highly cited and had offered them $72,000 to serve as adjunct faculty. The deal, according to a recruiting letter Pachter posted on his blog, stipulated that the mathematicians had to work three weeks a year in Saudi Arabia. The university would fly them there in business class and put them up at a five-star hotel. Conceivably, their work in Saudi Arabia added value locally. But the university also required them to change their affiliation on the Thomson Reuters academic citation website, a key reference for the U.S. News rankings. That meant the Saudi university could claim the publications of their new adjunct faculty ...

Much of the scheduling technology has its roots in a powerful discipline of applied mathematics called “operations research,” or OR. For centuries, mathematicians used the rudiments of OR to help farmers plan crop plantings and help civil engineers map highways to move people and goods efficiently. But the discipline didn’t really take off until World War II, when the US and British military enlisted teams of mathematicians to optimize their use of resources. The Allies kept track of various forms of an “exchange ratio,” which compared Allied resources spent versus enemy resources destroyed. During Operation Starvation, which took place between March and August 1945, the Twenty-first Bomber Command was tasked with destroying Japanese merchant ships in order to prevent food and other goods from arriving safely on Japanese shores. OR teams worked to minimize the number of mine-laying aircraft for each Japanese merchant ship that was sunk. They managed an “exchange ratio” of over 40 to 1—only 15 aircraft were lost in sinking 606 Japanese ships. This was considered highly efficient, and was due, in part, to the work of the OR team. Following World War II, major companies (as well as the Pentagon) poured enormous resources into OR. The science of logistics radically transformed the way we produce goods and bring them to market. In the 1960s, Japanese auto companies made another major leap, devising a manufacturing system called Just in Time. The idea was that instead of storing...

Seven years after A Nation at Risk was published with such fanfare, researchers at Sandia National Laboratories took a second look at the data gathered for the report. These people were no amateurs when it came to statistics—they build and maintain nuclear weapons—and they quickly found the error. Yes, it was true that SAT scores had gone down on average. However, the number of students taking the test had ballooned over the course of those seventeen years. Universities were opening their doors to more poor students and minorities. Opportunities were expanding. This signaled social success. But naturally, this influx of newcomers dragged down the average scores. However, when statisticians broke down the population into income groups, scores for every single group were rising, from the poor to the rich. In statistics, this phenomenon is known as Simpson’s Paradox: when a whole body of data displays one trend, yet when broken into subgroups, the opposite trend comes into view for each of those subgroups. The damning conclusion in the Nation at Risk report, the one that spurred the entire teacher evaluation movement, was drawn from a grievous misinterpretation of the data.

With the relentless growth of e-scores, we’re batched and bucketed according to secret formulas, some of them fed by portfolios loaded with errors. We’re viewed not as individuals but as members of tribes, and we’re stuck with that designation. As e-scores pollute the sphere of finance, opportunities dim for the have-nots.

Mathematicians didn’t pretend to foresee the fate of each individual. That was unknowable. But they could predict the prevalence of accidents, fires, and deaths within large groups of people. Over the following three centuries, a vast insurance industry grew around these predictions. The new industry gave people, for the first time, the chance to pool their collective risk, protecting individuals when misfortune struck. Now, with the evolution of data science and networked computers, insurance is facing fundamental change. With ever more information available—including the data from our genomes, the patterns of our sleep, exercise, and diet, and the proficiency of our driving—insurers will increasingly calculate risk for the individual and free themselves from the generalities of the larger pool. For many, this is a welcome change. A health enthusiast today can demonstrate, with data, that she sleeps eight hours a night, walks ten miles a day, and eats little but green vegetables, nuts, and fish oil. Why shouldn’t she get a break on her health insurance?

As insurance companies learn more about us, they’ll be able to pinpoint those who appear to be the riskiest customers and then either drive their rates to the stratosphere or, where legal, deny them coverage. This is a far cry from insurance’s original purpose, which is to help society balance its risk. In a targeted world, we no longer pay the average. Instead, we’re saddled with anticipated costs. Instead of smoothing out life’s bumps, insurance companies will demand payment for those bumps in advance. This undermines the point of insurance, and the hits will fall especially hard on those who can least afford them.

Employers, which have long been nickel and diming workers to lower their costs, now have a new tactic to combat these growing costs. They call it “wellness.” It involves growing surveillance, including lots of data pouring in from the Internet of Things—the Fitbits, Apple Watches, and other sensors that relay updates on how our bodies are functioning. The idea, as we’ve seen so many times, springs from good intentions. In fact, it is encouraged by the government. The Affordable Care Act, or Obamacare, invites companies to engage workers in wellness programs, and even to “incentivize” health. By law, employers can now offer rewards and assess penalties reaching as high as 50 percent of the cost of coverage. Now, according to a study by the Rand Corporation, more than half of all organizations employing fifty people or more have wellness programs up and running, and more are joining the trend every week.

Trouble is, the intrusions cannot be ignored or wished away. Nor can the coercion. Take the case of Aaron Abrams. He’s a math professor at Washington and Lee University in Virginia. He is covered by Anthem Insurance, which administers a wellness program. To comply with the program, he must accrue 3,250 “HealthPoints.” He gets one point for each “daily log-in” and 1,000 points each for an annual doctor’s visit and an on-campus health screening. He also gets points for filling out a “Health Survey” in which he assigns himself monthly goals, getting more points if he achieves them. If he chooses not to participate in the program, Abrams must pay an extra $50 per month toward his premium. Abrams was hired to teach math. And now, like millions of other Americans, part of his job is to follow a host of health dictates and to share that data not only with his employer but also with the third-party company that administers the program. He resents it, and he foresees the day when the college will be able to extend its surveillance. “It is beyond creepy,” he says, “to think of anyone reconstructing my daily movements based on my own ‘self-tracking’ of my walking.” My fear goes a step further. Once companies amass troves of data on employees’ health, what will stop them from developing health scores and wielding them to sift through job candidates? Much of the proxy data collected, whether step counts or sleeping patterns, is not protected by law, so it would theoretically be perfect...

It also turns out that wellness programs, despite well-publicized individual successes, often don’t lead to lower health care spending. A 2013 study headed by Jill Horwitz, a law professor at UCLA, rips away the movement’s economic underpinning. Randomized studies, according to the report, “raise doubts” that smokers and obese workers chalk up higher medical bills than others. While it is true that they are more likely to suffer from health problems, these tend to come later in life, when they’re off the corporate health plan and on Medicare. In fact, the greatest savings from wellness programs come from the penalties assessed on the workers. In other words, like scheduling algorithms, they provide corporations with yet another tool to raid their employees’ paychecks.

The campaigns use similar analysis to identify potential donors and to optimize each one. Here it gets complicated, because many of the donors themselves are carrying out their own calculations. They want the biggest bang for their buck. They know that if they immediately hand over the maximum contribution the campaign will view them as “fully tapped” and therefore irrelevant. But refusing to give any money will also render them irrelevant. So many give a drip-feed of money based on whether the messages they hear are ones they agree with. For them, managing a politician is like training a dog with treats. This training effect is all the more powerful for contributors to Super PACS, which do not limit political contributions. The campaigns, of course, are well aware of this tactic. With microtargeting, they can send each of those donors the information most likely to pry more dollars from their bank accounts. And these messages will vary from one donor to the next.

Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.

Like doctors, data scientists should pledge a Hippocratic Oath, one that focuses on the possible misuses and misinterpretations of their models. Following the market crash of 2008, two financial engineers, Emanuel Derman and Paul Wilmott, drew up such an oath. It reads: ~ I will remember that I didn’t make the world, and it doesn’t satisfy my equations. ~ Though I will use models boldly to estimate value, I will not be overly impressed by mathematics. ~ I will never sacrifice reality for elegance without explaining why I have done so. ~ Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights. ~ I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.

Movements toward auditing algorithms are already afoot. At Princeton, for example, researchers have launched the Web Transparency and Accountability Project. They create software robots that masquerade online as people of all stripes—rich, poor, male, female, or suffering from mental health issues. By studying the treatment these robots receive, the academics can detect biases in automated systems from search engines to job placement sites. Similar initiatives are taking root at universities like Carnegie Mellon and MIT.

On a summer day in 2013, I took the subway to the southern tip of Manhattan and walked to a large administrative building across from New York’s City Hall. I was interested in building mathematical models to help society—the opposite of WMDs. So I’d signed on as an unpaid intern in a data analysis group within the city’s Housing and Human Services Departments. The number of homeless people in the city had grown to sixty-four thousand, including twenty-two thousand children. My job was to help create a model that would predict how long a homeless family would stay in the shelter system and to pair each family with the appropriate services. The idea was to give people what they needed to take care of themselves and their families and to find a permanent home. My job, in many ways, was to help come up with a recidivism model. Much like the analysts building the LSI–R model, I was interested in the forces that pushed people back to shelters and also those that led them to stable housing. Unlike the sentencing WMD, though, our small group was concentrating on using these findings to help the victims and to reduce homelessness and despair. The goal was to create a model for the common good.

As I survey the data economy, I see loads of emerging mathematical models that might be used for good and an equal number that have the potential to be great—if they’re not abused. Consider the work of Mira Bernstein, a slavery sleuth. A Harvard PhD in math, she created a model to scan vast industrial supply chains, like the ones that put together cell phones, sneakers, or SUVs, to find signs of forced labor. She built her slavery model for a nonprofit company called Made in a Free World. Its goal is to use the model to help companies root out the slave-built components in their products. The idea is that companies will be eager to free themselves from this scourge, presumably because they oppose slavery, but also because association with it could devastate their brand. Bernstein collected data from a number of sources, including trade data from the United Nations, statistics about the regions where slavery was most prevalent, and detailed information about the components going into thousands of industrial products, and incorporated it all into a model that could score a given product from a certain region for the likelihood that it was made using slave labor. “The idea is that the user would contact his supplier and say, ‘Tell me more about where you’re getting the following parts of your computers,’ ” Bernstein told Wired magazine. Like many responsible models, the slavery detector does not overreach. It merely points to suspicious places and leaves the last part of the ...

Data is not going away. Nor are computers—much less mathematics. Predictive models are, increasingly, the tools we will be relying on to run our institutions, deploy our resources, and manage our lives. But as I’ve tried to show throughout this book, these models are constructed not just from data but from the choices we make about which data to pay attention to—and which to leave out. Those choices are not just about logistics, profits, and efficiency. They are fundamentally moral. If we back away from them and treat mathematical models as a neutral and inevitable force, like the weather or the tides, we abdicate our responsibility. And the result, as we’ve seen, is WMDs that treat us like machine parts in the workplace, that blackball employees and feast on inequities. We must come together to police these WMDs, to tame and disarm them. My hope is that they’ll be remembered, like the deadly coal mines of a century ago, as relics of the early days of this new revolution, before we learned how to bring fairness and accountability to the age of data. Math deserves much better than WMDs, and democracy does too.

ABOUT THE AUTHOR Cathy O’Neil is a data scientist and the author of the blog mathbabe.​org. She earned a PhD in mathematics from Harvard and taught at Barnard College before moving to the private sector, where she worked for the hedge fund D. E. Shaw. She then worked as a data scientist at various start-ups, building models that predict people’s purchases and clicks. O’Neil started the Lede Program in Data Journalism at Columbia and is the author of Doing Data Science. She is currently a columnist for Bloomberg View.