Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

Nevertheless, many of these models encoded human prejudice, misunderstanding, and bias into the software systems that increasingly managed our lives. Like gods, these mathematical models were opaque, their workings invisible to all but the highest priests in their domain: mathematicians and computer scientists. Their verdicts, even when wrong or harmful, were beyond dispute or appeal. And they tended to punish the poor and the oppressed in our society, while making the rich richer.

I came up with a name for these harmful kinds of models: Weapons of Math Destruction, or WMDs for short.

Statisticians count on large numbers to balance out exceptions and anomalies. (And WMDs, as we’ll see, often punish individuals who happen to be the exception.)

Equally important, statistical systems require feedback—something to tell them when they’re off track.

Statisticians use errors to train their models and m...

Without feedback, however, a statistical engine can continue spinning out faulty and damaging analysis while never learning from its mistakes.

Many of the WMDs I’ll be discussing in this book, including the Washington school district’s value-added model, behave like that. They define their own reality and use it to justify their results. This type of model is self-perpetuating, highly destructive—and very common.

In WMDs, many poisonous assumptions are camouflaged by math and go largely untested and unquestioned.

The privileged, we’ll see time and again, are processed more by people, the masses by machines.

The model itself is a black box, its contents a fiercely guarded corporate secret. This allows consultants like Mathematica to charge more, but it serves another purpose as well: if the people being evaluated are kept in the dark, the thinking goes, they’ll be less likely to attempt to game the system. Instead, they’ll simply have to work hard, follow the rules, and pray that the model registers and appreciates their efforts. But if the details are hidden, it’s also harder to question the score or to protest against it.

But you cannot appeal to a WMD. That’s part of their fearsome power. They do not listen. Nor do they bend. They’re deaf not only to charm, threats, and cajoling but also to logic—even when there is good reason to question the data that feeds their conclusions.

The human victims of WMDs, we’ll see time and again, are held to a far higher standard of evidence than the algorithms themselves.

Ill-conceived mathematical models now micromanage the economy, from advertising to prisons.

The trouble is that profits end up serving as a stand-in, or proxy, for truth. We’ll see this dangerous confusion crop up again and again.

This may sound obvious, but as we’ll see throughout this book, the folks building WMDs routinely lack data for the behaviors they’re most interested in. So they substitute stand-in data, or proxies.

Whatever they learn, they can feed back into the model, refining it. That’s how trustworthy models operate. They maintain a constant back-and-forth with whatever in the world they’re trying to understand or predict. Conditions change, and so must the model.

model, after all, is nothing more than an abstract representation of some process, be it a baseball game, an oil company’s supply chain, a foreign government’s actions, or a movie theater’s attendance. Whether it’s running in a computer program or in our head, the model takes what we know and uses it to predict responses in various situations. All of us carry thousands of models in our heads. They tell us what to expect, and they guide our decisions.

The updates and adjustments make it what statisticians call a “dynamic model.”

There would always be mistakes, however, because models are, by their very nature, simplifications. No model can include all of the real world’s complexity or the nuance of human communication.

To create a model, then, we make choices about what’s important enough to include, simplifying the world into a toy version that can be easily understood and from which we can infer important facts and actions. We expect it to handle only one job and accept that it will occasionally act like a clueless machine, one with enormous blind spots.

model’s blind spots reflect the judgments and priorities of its creators.

Here we see that models, despite their reputation for impartiality, reflect goals and ideology. When I removed the possibility of eating Pop-Tarts at every meal, I was imposing my ideology on the meals model. It’s something we do without a second thought. Our own values and desires influence our choices, from the data we choose to collect to the questions we ask. Models are opinions embedded in mathematics.

In each case, we must ask not only who designed the model but also what that person or company is trying to accomplish.

And a model built for today will work a bit worse tomorrow. It will grow stale if it’s not constantly updated.

The first question: Even if the participant is aware of being modeled, or what the model is used for, is the model opaque, or even invisible?

Opaque and invisible models are the rule, and clear ones very much the exception.

One common justification is that the algorithm constitutes a “secret sauce” crucial to their business.

the second question: Does the model work against the subject’s interest? In short, is it unfair? Does it damage or destroy lives?

The third question is whether a model has the capacity to grow exponentially.

can it scale?

So to sum up, these are the three elements of a WMD: Opacity, Scale, and Damage.

was forced to confront the ugly truth: people had deliberately wielded formulas to impress rather than clarify.

In both cultures, wealth is no longer a means to get by. It becomes directly tied to personal worth.

I worried about the separation between technical models and real people, and about the moral repercussions of that separation. In fact, I saw the same pattern emerging that I’d witnessed in finance: a false sense of security was leading to widespread use of imperfect models, self-serving definitions of success, and growing feedback loops. Those who objected were regarded as nostalgic Luddites. I

formula, whether it’s a diet or a tax code, might be perfectly innocuous in theory. But if it grows to become a national or global standard, it creates its own distorted and dystopian economy. This is what has happened in higher education.

The ranking, in short, was destiny.

However, when you create a model from proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.

As people game the system, the proxy loses its effectiveness. Cheaters wind up as false positives.

The proxies the journalists chose for educational excellence make sense, after all. Their spectacular failure comes, instead, from what they chose not to count: tuition and fees. Student financing was left out of the model.

By analyzing the virtues of the name-brand universities, the ratings team created an elite yardstick to measure excellence.

Now, if they incorporated the cost of education into the formula, strange things might happen to the results. Cheap universities could barge into the excellence hierarchy. This could create surprises and sow doubts. The public might receive the U.S. News rankings as something less than the word of God. It was much safer to start with the venerable champions on top. Of course they cost a lot. But maybe that was the price of excellence.

In fact, if they raised prices, they’d have more resources for addressing the areas where they were being measured.

As colleges position themselves to move up the U.S. News charts, they manage their student populations almost like an investment portfolio.

each prospective student represents a series of assets and usually a liability or two.

If the editors rejigger the weightings on the model, paying less attention to SAT scores, for example, or more to graduation rates, the entire ecosystem of education must adapt.

The only way to win in such a scenario is to gain an advantage and to make sure that others aren’t getting a bigger one. This is the case not only in China but also in the United States, where high school admissions officers, parents, and students find themselves caught in a frantic effort to game the system spawned by the U.S. News model.

All of those sound like worthy goals, to be sure, but every ranking system can be gamed. And when that happens, it creates new and different feedback loops and a host of unintended consequences.