Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

As stop and frisk grew, the venerable legal concept of probable cause was rendered virtually meaningless, because police were hunting not only people who might have already committed a crim...

But was the policy constitutional?

In August of 2013, federal judge Shira A. Scheindlin ruled that it was not.

She said officers routinely “stopped blacks and Hispanics who would not have been sto...

Stop and frisk, she wrote, ran afoul of the Fourth Amendment, which protects against unreasonable searches and seizures by the government, and it also failed to provide the equal p...

She called for broad reforms to the practice, including increased use of body cameras...

This would help establish probable cause—or the lack of it—and remove some of the opacity fro...

But it would do nothing to address the issue of ...

While looking at WMDs, we’re often faced with a choice between fa...

Our legal traditions lean strongly to...

The Constitution, for example, presumes innocence and is engi...

From a modeler’s perspective, the presumption of innocence is a constraint, and the result is that some guilty people go free, especia...

The Constitution’s implicit judgment is that freeing someone who may well have committed a crime, for lack of evidence, poses less of a danger to our society than jailing or executing an innocent person.

WMDs, by contrast, tend to favor efficiency.

By their very nature, they feed on data that can be mea...

But fairness is squishy and hard ...

It is a c...

And computers, for all of their advances in language and logic, still struggle...

They “understand” beauty only as a word associated with the Grand Canyon, ocean sunsets, and groo...

They try in vain to measure “friendship” by counting likes and con...

And the concept of fairness utterly ...

Programmers don’t know how to code for it, and few of their ...

So fairness isn’t calculated into WMDs. And the result is massive, industrial p...

If you think of a WMD as a factory, unfairness is the black stuff belching out of the smoke stacks. I...

The question is whether we as a society are willing to sacrifice a bit of efficiency in...

Should we handicap the models, leaving ce...

It’s a tough case to make, similar in many ways to the battles over wiretapping by the National Security Agency.

Advocates of the snooping argue that it’s important for our safety.

And those running our vast national security apparatus will keep pushing for more informatio...

They’ll continue to encroach on people’s privacy until they get the message that they must find a way to do their job wit...

It might be harder, but it’s...

The other issue is ...

Would society be so willing to sacrifice the concept of probable cause if everyone had to endure the harassment a...

a crucial part of justice is equality.

And that means, among many other things, experiencing criminal justice equally.

People who favor policies like stop and frisk should experi...

Justice cannot just be something that one part of society infl...

And why are nonwhite prisoners from poor neighborhoods more likely to commit crimes?

According to the data inputs for the recidivism models, it’s because they’re more likely to be jobless, lack a high school diploma, and have had previous run-ins with the law. And their friends have, too.

Another way of looking at the same data, though, is that these prisoners live in poor neighborhoods with terrible...

And they’re highly...

So the chance that an ex-convict returning to that neighborhood will have another brush with the law is no doubt larger than that of a tax frauds...

In this system, the poor and nonwhite are punished more for being who they are an...

If I had a chance to be a data scientist for the justice system, I would do my best to dig deeply to learn what goes on inside those prisons and what impact those experiences might have on prisoners’ behavior.

I’d first look into solitary confinement. Hundreds of thousands of prisoners are kept for twenty-three hours a day in these prisons within prisons, most of them no bigger than a horse stall.

Researchers have found that time in solitary produces deep feelings of hopelessness and despair. Could that...

It’s another important topic for analysis that anyone with the relevant data and expertise could work out, but prison systems have thus far been uninterested in cataloging the long-term effects of this abuse.

A serious scientist would also search for positive signals from the prison experience.

What’s the impact of more sunlight, more sports, better food,...

Maybe these factors will improve convicts’ behavior a...