Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

The question we’re left with is this: How many Wanda Taylors are out there clearing up false identities and other errors in our data? The answer: not nearly enough. Humans in the data economy are outliers and throwbacks.

flummoxed

troves

meddling

For big banks, the new platforms provide a convenient alternative to the tightly regulated banking economy.

compared to the slew of WMDs running amok, the prejudiced loan officer of yesteryear doesn’t look all that bad. At the very least, a borrower could attempt to read his eyes and appeal to his humanity.

uninsurable.

Like Hoffman, the creators of these new models confuse correlation with causation. They punish the poor, and especially racial and ethnic minorities. And they back up their analysis with reams of statistics, which give them the studied air of evenhanded science.

draper

embryonic.

The trouble is, it’s not individual. The models place us into groups we cannot see, whose behavior appears to resemble ours. Regardless of the quality of the analysis, its opacity can lead to gouging.

What’s different here is the focus on the proxy when far more relevant data is available. I cannot imagine a more meaningful piece of data for auto insurers than a drunk driving record. It is evidence of risk in precisely the domain they’re attempting to predict.

In short, while an e-score might not correlate with safe driving, it does create a lucrative pool of vulnerable drivers.

pristine

forlorn

Yet in the age of Big Data, urging insurers to judge us by how we drive means something entirely new.

At some point, the trackers will likely become the norm. And consumers who want to handle insurance the old-fashioned way, withholding all but the essential from their insurers, will have to pay a premium, and probably a steep one. In the world of WMDs, privacy is increasingly a luxury that only the wealthy can afford.

sleuthing

These automatic programs will increasingly determine how we are treated by the other machines, the ones that choose the ads we see, set prices for us, line us up for a dermatologist appointment, or map our routes. They will be highly efficient, seemingly arbitrary, and utterly unaccountable. No one will understand their logic or be able to explain it.

And it would make sense. As we’ve seen, they routinely reject applicants on the basis of credit scores and personality tests. Health scores represent a natural—and frightening—next step.

A research report from the California Health Benefits Review Program concludes that corporate wellness programs fail to lower the average blood pressure, blood sugar, or cholesterol of those who participate in them. Even when people succeed in losing weight on one of these programs, they tend to gain it back. (The one area where wellness programs do show positive results is in quitting smoking.)

hone

stoking

wherewithal

grump

But what they have demonstrated is Facebook’s enormous power to affect what we learn, how we feel, and whether we vote. Its platform is massive, powerful, and opaque. The algorithms are hidden from us, and we see only the results of the experiments researchers choose to publish.

nook.

catered.

ruffle

Basking in

let loose

lar...

Direct mail was microtargeting on training wheels. The convergence of Big Data and consumer marketing now provides politicians with far more powerful tools. They can target microgroups of citizens for both votes and money and appeal to each of them with a meticulously honed message, one that no one else is likely to see.

bugged

In this sense, we can think of the voting public very much as we think of financial markets. With the flow of information, values rise and fall, as do investments. In these new political markets, each one of us represents a stock with its own fluctuating price.

While that campaign launched into public view, hundreds of others continue to hover below the surface, addressing individual voters. These quieter campaigns are equally deceptive and even less accountable.

As television and the rest of the media move toward profiling their viewers, the potential for political microtargeting grows.

The scoring of individual voters also undermines democracy, making a minority of voters important and the rest little more than a supporting cast. Indeed, looking at the models used in presidential elections, we seem to inhabit a shrunken country.

I might point out here that while many of the WMDs we’ve been looking at, from predatory ads to policing models, deliver most of their punishment to the struggling classes, political microtargeting harms voters of every economic class.

With political messaging, as with most WMDs, the heart of the problem is almost always the objective. Change that objective from leeching off people to helping them, and a WMD is disarmed—and can even become a force for good.

Promising efficiency and fairness, they distort higher education, drive up debt, spur mass incarceration, pummel the poor at nearly every juncture, and undermine democracy.

The quiet and personal nature of this targeting keeps society’s winners from seeing how the very same models are destroying lives, sometimes just a few blocks away.

scourge

grappling with,

putatively

While inclusiveness no doubt caused grumbling in some pockets of intolerance, it also paid rich dividends.

disenfranchised

scurry

sowing

But human decision making, while often flawed, has one chief virtue. It can evolve. As human beings learn and adapt, we change, and so do our processes. Automated systems, by contrast, stay stuck in time until engineers dive in to change them.