Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

Math provided a neat refuge from the messiness of the real world.

It marched forward, its field of knowledge expanding relentlessly, proof by proof.

And I could ad...

Mathematicians and statisticians were studying our desires, movements, and spending power.

They were predicting our trustworthiness and calculating our potential as students, workers, lovers, criminals.

This was the Big Data economy, and it promised spectacular gains.

A computer program could speed through thousands of résumés or loan applications in a second or two and sort them into neat lists, with the most promising candidates on top.

This not only saved time but also was marketed as fa...

After all, it didn’t involve prejudiced humans digging through reams of paper, just machin...

By 2010 or so, mathematics was asserting itself as never before in human affairs, and the ...

Yet I saw t...

The math-powered applications powering the data economy were based on choices made ...

Nevertheless, many of these models encoded human prejudice, misunderstanding, and bias into the software systems tha...

Like gods, these mathematical models were opaque, their workings invisible to all but the highest priests in their domain: mat...

Their verdicts, even when wrong or harmful, were beyond ...

And they tended to punish the poor and the oppressed in our society, while ...

I came up with a name for these harmful kinds of models: Weapons of Math Destruc...

attempting to reduce human behavior, performance, and potential to algorithms is no easy job.

attempting to score a teacher’s effectiveness by analyzing the test results of only twenty-five or thirty students is statistically unsound, even laughable.

Employers, for example, are increasingly using credit scores to evaluate potential hires.

Those who pay their bills promptly, the thinking goes, are more likely to show up to work on time and follow the rules.

In fact, there are plenty of responsible people and good workers who suffer misfortune and se...

But the belief that bad credit correlates with bad job performance leaves those with low scor...

Joblessness pushes them toward poverty, which further worsens their scores, making it even ha...

It’s a downward...

And employers never learn how many good employees they’ve missed out on by fo...

In WMDs, many poisonous assumptions are camouflaged by math and go largely un...

The privileged, we’ll see time and again, are processed more by people, the masses by machines.

“How do you justify evaluating people by a measure for which you are unable to provide explanation?”

But that’s the nature of WMDs.

The analysis is outsourced to coders and statisticians. And as a rule, they let the...

teacher evaluation algorithms are a powerful tool for behavioral modification.

That’s their purpose, and in the Washington schools they featured both a stick and a carrot.

Teachers knew that if their students stumbled on the test their o...

This gave teachers a strong motivation to ensure their students passed, especially as the Great Recess...

At the same time, if their students outperformed their peers, teachers and administrators could rec...

If you add those powerful incentives to the evidence in the case—the high number of erasures and the abnormally high test scores—there were grounds for suspicion that fourth-grade teachers, bowing either t...

It is conceivable, then, that Sarah Wysocki’s fifth-grade students started the school year with artificially inflated scores.

If so, their results the following year would make it appear that they’d lost ground in fifth grade—and that their teacher was an underperformer.

Yes, if it becomes clear that automated systems are screwing up on an embarrassing and systematic basis, programmers will go back in and tweak the algorithms.

But for the most part, the programs deliver unflinching verdicts, and the human beings employing them can only shrug, as if to say, “Hey, what can you do?”

Do you see the paradox?

An algorithm processes a slew of statistics and comes up with a probability that a certain person might be a bad hire, a risky borrower, a terrorist, or a miserable teacher.

The human victims of WMDs, we’ll see time and again, are held to a far higher standard of evidence than the algorithms themselves.

So thanks to a highly questionable model, a poor school lost a good teacher, and a rich school, which didn’t fire people on the basis of their students’ scores, gained one.

Ill-conceived mathematical models now micromanage the economy, from advertising to prisons.

They’re opaque, unquestioned, and unaccountable,

and they operate at a scale to sort, target, or “optimize” millions of people.

By confusing their findings with on-the-ground reality, most of them create perni...

A model’s blind spots reflect the judgments and priorities of its creators.