Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

sloshing

mathematics, once my refuge, was not only deeply entangled in the world’s problems but also fueling many of them.

abetted

wielding

many of these models encoded human prejudice, misunderstanding, and bias into the software systems that increasingly managed our lives.

Without feedback, however, a statistical engine can continue spinning out faulty and damaging analysis while never learning from its mistakes.

The privileged, we’ll see time and again, are processed more by people, the masses by machines.

as we’ll see throughout this book, the folks building WMDs routinely lack data for the behaviors they’re most interested in. So they substitute stand-in data, or proxies. They draw statistical correlations between a person’s zip code or language patterns and her potential to pay back a loan or handle a job. These correlations are discriminatory, and some of them are illegal.

Our own values and desires influence our choices, from the data we choose to collect to the questions we ask. Models are opinions embedded in mathematics.

And while Walter Quijano’s words were transcribed for the record, which could later be read and challenged in court, the workings of a recidivism model are tucked away in algorithms, intelligible only to a tiny elite.

But in fact the model itself contributes to a toxic cycle and helps to sustain it. That’s a signature quality of a WMD.

A key component of this suffering is the pernicious feedback loop. As we’ve seen, sentencing models that profile a person by his or her circumstances help to create the environment that justifies their assumptions. This destructive loop goes round and round, and in the process the model becomes more and more unfair.

So to sum up, these are the three elements of a WMD: Opacity, Scale, and Damage. All of them will be present, to one degree or another, in the examples we’ll be covering.

The risk ratings on the securities were designed to be opaque and mathematically intimidating, in part so that buyers wouldn’t perceive the true level of risk associated with the contracts they owned.

chatter

afloat.

liaison

tattered

prowess,

The statistical work, as it turned out, was highly transferable from the hedge fund to e-commerce—the biggest difference was that, rather than the movement of markets, I was now predicting people’s clicks.

Their productivity indicates that they’re on the right track, and it translates into dollars. This leads to the fallacious conclusion that whatever they’re doing to bring in more money is good. It “adds value.” Otherwise, why would the market reward it?

In both of these industries, the real world, with all of its messiness, sits apart. The inclination is to replace people with data trails, turning them into more effective shoppers, voters, or workers to optimize some objective.

More and more, I worried about the separation between technical models and real people, and about the moral repercussions of that separation. In fact, I saw the same pattern emerging that I’d witnessed in finance: a false sense of security was leading to widespread use of imperfect models, self-serving definitions of success, and growing feedback loops.

earnest.

wringer.

shrivel.

hunches.

scant

ideal for higher education—“a way to deeper personal fulfillment, greater personal productivity and increased personal reward”—didn’t

surmising

egregious

However, when you create a model from proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.

maven.

As people game the system, the proxy loses its effectiveness. Cheaters wind up as false positives.

tumbling

shortcoming

sow

As colleges position themselves to move up the U.S. News charts, they manage their student populations almost like an investment portfolio. We’ll see this often in the world of data, from advertising to politics.

acing

dismayed

surrendered

In a system in which cheating is the norm, following the rules amounts to a handicap.

sap

cinch

rejiggered

stint

deemed

scant

If it was true during the early dot-com days that “nobody knows you’re a dog,” it’s the exact opposite today. We are ranked, categorized, and scored in hundreds of models, on the basis of our revealed preferences and patterns.

Anywhere you find the combination of great need and ignorance, you’ll likely see predatory ads.