Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

nefarious

The playbook for predatory advertisers is similar, but they carry it out at massive scale, targeting millions of people every day. The customers’ ignorance, of course, is a crucial piece of the puzzle.

unwittingly

To optimize recruiting—and revenue—they need to know whom their messages reached and, if possible, what impact they had. Only with this data can they go on to optimize the operation.

That’s why much of the advertising money at for-profit universities goes to Google and Facebook. Each of these platforms allows advertisers to segment their target populations in meticulous detail.

Once these campaigns move online, the learning accelerates. The Internet provides advertisers with the greatest laboratory ever for consumer research and lead generation. Feedback from each promotion arrives within seconds—a lot faster than the mail.

throbbing

teem

underwhelmed

An advertising program might start out with the usual demographic and geographic details. But over the course of weeks and months it begins to learn the patterns of the people it’s targeting and to make predictions about their next moves. It gets to know them. And if the program is predatory, it gauges their weaknesses and vulnerabilities and pursues the most efficient path to exploit them.

beavering away.

scrape together

They found that diplomas from for-profit colleges were worth less in the workplace than those from community colleges and about the same as a high school diploma. And yet these colleges cost on average 20 percent more than flagship public universities.

What these people need is money. And the key to earning more money, they hear again and again, is education.

languished.

all the rage

Like those in the rest of the Big Data industry, the developers of crime prediction software are hurrying to incorporate any information that can boost the accuracy of their models.

straying

chiseling

frisk.

What we found, to no great surprise, was that an overwhelming majority of these encounters—about 85 percent—involved young African American or Latino men.

ensnared

The Constitution’s implicit judgment is that freeing someone who may well have committed a crime, for lack of evidence, poses less of a danger to our society than jailing or executing an innocent person.

racketeering.

In this system, the poor and nonwhite are punished more for being who they are and living where they live.

The goal, if data were used constructively, would be to optimize prisons—much the way companies like Amazon optimize websites or supply chains—for the benefit of both the prisoners and society at large.

upend

Concern for privacy, on that occasion, trumped efficiency. But this won’t always be the case.

From a mathematical point of view, however, trust is hard to quantify. That’s a challenge for people building models. Sadly, it’s far simpler to keep counting arrests, to build models that assume we’re birds of a feather and treat us as such.

vouch

bewildered.

burgeoning

Naturally, these hiring programs can’t incorporate information about how the candidate would actually perform at the company. That’s in the future, and therefore unknown. So like many other Big Data programs, they settle for proxies.

galling

cinch.

And we do not know what those patterns are. We’re not told what the tests are looking for. The process is entirely opaque.

So in this sense, the unequal paths to opportunity are nothing new. They have simply returned in a new incarnation, this time to guide society’s winners past electronic gatekeepers.

The job was to teach the computerized system how to replicate the same procedures that human beings had been following. As I’m sure you can guess, these inputs were the problem. The computer learned from the humans how to discriminate, and it carried out this work with breathtaking efficiency.

echelons

cogs

grist

clerks

muttering

tow.

harried

scrambling

hinges

write-up.

And because they need money so desperately, the companies can bend their lives to the dictates of a mathematical model.

frazzled