Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

but in many cases the real culprit is the poverty that leads workers to take jobs wit...

and the scheduling models that squeeze struggling fam...

The root of the trouble, as with so many other WMDs, is the modelers’...

The model is optimized for efficiency and profitability, not for justice or ...

This is, of course, the nature of...

For companies, revenue is like oxygen. It ke...

From their perspective, it would be profoundly stupid, even unnatural, to turn awa...

That’s why society needs countervailing forces, such as vigorous press coverage that highlights the abuses of efficiency and shames ...

And when they come up short, as Starbucks did, it must expose t...

It also needs regulators to keep them in line, strong unions to organize workers and amplify ...

Following the New York Times report in 2014, Democrats in Congress promptly drew up bills to ...

But facing a Republican majority fiercely opposed to government regulations, the chances that their bill would become l...

Very soon, the roles of the employees appeared to come into focus.

Some people were idea generators, the system concluded.

On its chart of employees, Cataphora marked idea generators with circles, which were bigger and darker i...

Other people were co...

Like neurons in a distributed network, they transmi...

The most effective connectors made snippets of...

The system painted those people in dark c...

Now, whether or not this system effectively measured the flow of ideas, the concept ...

A few years ago, MIT researchers analyzed the behavior of call center employees for Bank of America to find out why some teams were more productive than others.

The researchers found, to their surprise, that the fastest and most efficient call center team was also the most social.

These employees pooh-poohed the rules and gabbed much more than the others.

And when all of the employees were encouraged to socialize more, call cent...

But data studies that track employees’ behavior can also be used ...

They saw that some workers were represented as big dark circles, while others were smaller and dimmer.

If they had to lay off workers, and most companies did, it made sense to start with the small and dim ones on the chart.

Were those workers really ...

If a system designates a worker as a low idea generator or weak connector, that verdict becomes its...

Perhaps someone can come in with counterva...

The worker with the dim circle might generate fabulous ideas but not shar...

Or perhaps she proffers priceless advice over lunch or breaks up the tension in...

Maybe everybody l...

That has great value in the...

But computing systems have trouble finding digital proxies for these ...

The relevant data simply isn’t collected, and anyway it’s hard to...

They’re usually easier to leave ou...

So the system identifies appa...

And a good number of them lost their jobs during the recession. T...

But what’s worse is that systems like Cataphora’s receive min...

Someone identified as a loser, and subsequently fired, may have found another job and gene...

That data usually isn’t collected. The system has no inkling that it got one person, or even a tho...

That’s a problem, because scientists need this error feedback—in this case the presence of false negatives—to delve into forensic analysis and figure out what went w...

It’s how systems learn and g...

Yet as we’ve seen, loads of WMDs, from recidivism models to teacher scores, blithely g...

Managers assume that the scores are true enough to be useful, and the algorithm ma...

They can fire employees and cut costs and blame their decisions on an objective number, wh...

Bogus is the word for it. In fact, misinterpreted statistics run through the history of teacher evaluation.

Yes, it was true that SAT scores had gone down on average.

However, the number of students taking the test had ballooned over the course of those seventeen years.