Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

The value-added model in Washington, D.C., schools, to return to that example,

evaluates teachers largely on the basis of students’ test scores,

while ignoring how much the teachers engage the students, work on specific skills, deal with classroom management, or help studen...

It’s overly simple, sacrificing accuracy and insigh...

Yet from the administrators’ perspective it provides an effective tool to ferret out hundreds of apparently underperforming teachers, even ...

Here we see that models, despite their reputation for impartiality, refle...

Our own values and desires influence our choices, from the data we choose to collect to the questions we ask. Models are opinions embedded in mathematics.

Regardless of whether the issue of race comes up explicitly at trial, it has long been a major factor in sentencing.

A University of Maryland study showed that in Harris County, which includes Houston, prosecutors were three times more likely to seek the death penalty for African Americans, and four times more likely for Hispanics, than for whites convicted of the same charges.

That pattern isn’t unique...

According to the American Civil Liberties Union, sentences imposed on black men in the federal system are nearly 20 percent longer than those...

And though they make up only 13 percent of the population, blacks fill up 40 percent ...

But as the questions continue, delving deeper into the person’s life, it’s easy to imagine how inmates from a privileged background would answer one way and those from tough inner-city streets another.

Ask a criminal who grew up in comfortable suburbs about “the first time you were ever involved with the police,” and he might not have a single incident to report other than the one that brought him to prison.

Young black males, by contrast, are likely to have been stopped by police dozens of times, even whe...

A 2013 study by the New York Civil Liberties Union found that while black and Latino males between the ages of fourteen and twenty-four made up only 4.7 percent of the city’s population, they accounted f...

More than 90 percent of those stopped ...

If you were told by an usher, upon entering an open-air concert, that you couldn’t sit in the first ten rows of seats, you might find it unreasonable.

it were explained to you that the first ten rows were being reserved for people in wheelchairs, then it might well make a difference.

Transparency ...

And yet many companies go out of their way to hide the results of their models ...

I was forced to confront the ugly truth: people had deliberately wielded formulas to impress rather than clarify.

It was the first time I had been directly confronted with this toxic concept, and it made me want to escape, to go back in time to the world of proofs and Rubik’s Cubes.

The statistical work, as it turned out, was highly transferable from the hedge fund to e-commerce—the

the biggest difference was that, rather than the movement of markets, I was now predicting people’s clicks.

In fact, I saw all kinds of parallels between fina...

Both industries gobble up the same pool of talent, much of it from elite universities like M...

These new hires are ravenous for success and have been focused on external metrics—like SAT scores and colleg...

Whether in finance or tech, the message they’ve received is that they will be rich, tha...

Their productivity indicates that they’re on the right track, and it tr...

This leads to the fallacious conclusion that whatever they’re doing to bring...

It “adds value.” Otherwise, why would the ma...

In both cultures, wealth is no longer a m...

It becomes directly tied to per...

A young suburbanite with every advantage—the prep school education, the exhaustive coaching for college admissions tests, the oversea...

still flatters himself that it is his skill, hard work, and prodigious problem-solving abilities that have lifte...

Money vindicates al...

And the rest of his circle plays along, forming a mutual ...

They’re eager to convince us all that Darwinism is at work, when it looks very much to the outside like a combination...

In both of these industries, the real world, with all of its me...

The inclination is to replace people with data trails, turning them into more effective shoppers, voters, or wor...

This is easy to do, and to justify, when success comes back as an anonymous score and when the people affected remain every bit as abstract a...

More and more, I worried about the separation between technical models and real people, and about the moral repercussions of that separation.

In fact, I saw the same pattern emerging that I’d witnessed in finance:

a false sense of security was leading to widespread use of imperfect models, self-serving definitions of succ...

Instead of a bust, I saw a growing dystopia, with inequality rising.

The algorithms would make sure that those deemed losers would remain that way.

A lucky minority would gain ever more control over the data economy, raking in outrageous fortunes and convincing themselves ...

A formula, whether it’s a diet or a tax code, might be perfectly innocuous in theory.

But if it grows to become a national or global standard, it creates its own distorted and dystopian economy.