Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

Yet here they were at the summit of their careers dedicating enormous energy toward boosting performance in fifteen areas defined by a group of journalists at a second-tier newsmagazine. They were almost like students again, angling for good grades from a taskmaster. In fact, they were trapped by a rigid model, a WMD.

The great majority of college administrators looked for less egregious ways to improve their rankings. Instead of cheating, they worked hard to improve each of the metrics that went into their score.

However, when you create a model from proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.

This brings us to the crucial question we’ll confront time and again. What is the objective of the modeler?

By leaving cost out of the formula, it was as if U.S. News had handed college presidents a gilded checkbook. They had a commandment to maximize performance in fifteen areas, and keeping costs low wasn’t one of them. In fact, if they raised prices, they’d have more resources for addressing the areas where they were being measured. Tuition has skyrocketed ever since. Between 1985 and 2013, the cost of higher education rose by more than 500 percent, nearly four times the rate of inflation.

While Facebook may feel like a modern town square, the company determines, according to its own interests, what we see and learn on its social network.

Nearly half of them, according to a Pew Research Center report, count on Facebook to deliver at least some of their news, which leads to the question: By tweaking its algorithm and molding the news we see, can Facebook game the political system?

At the same time, Facebook researchers were studying how different types of updates influenced people’s voting behavior.

Other publicly held corporations, including Google, Apple, Microsoft, Amazon, and cell phone providers like Verizon and AT&T, have vast information on much of humanity—and the means to steer us in any way they choose.

Usually, as we’ve seen, they’re focused on making money. However, their profits are tightly linked to government policies.

The Facebook campaign started out with a constructive and seemingly innocent goal: to encourage people to vote. And it succeeded. After comparing voting records, researchers estimated that their campaign had increased turnout by 340,000 people. That’s a big enough crowd to swing entire states, and even national elections.

Two years later, Facebook took a step further. For three months leading up to the election between President Obama and Mitt Romney, a researcher at the company, Solomon Messing, altered the news feed algorithm for about two million people, all of them politically engaged. These people got a higher proportion of hard news, as opposed to the usual cat videos, graduation announcements, or photos from Disney World. If their friends shared a news story, it showed up high on their feed.

Of course, it wasn’t really the friends delivering the newspaper, but Facebook itself.

decision was the right one. Facebook is more like the Wizard of Oz: we do not see the human beings involved.

62 percent of the people were unaware that the company tinkered with the news feed.

In 2012, researchers experimented on 680,000 Facebook users to see if the updates in their news feeds could affect their mood.

Using linguistic software, Facebook sorted positive (stoked!) and negative (bummed!) updates.

When they studied the users’ subsequent posting behavior, they found evidence that the doctored new feeds had indeed altered their moods.

people’s emotions on Election Day? I have no reason to believe that the social scientists at Facebook are actively gaming the political system. Most of them are serious academics carrying out research on a platform that they could only have dreamed about two decades ago.

Much the same is true of Google. Its search algorithm appears to be focused on raising revenue. But search results, if Google so chose, could have a dramatic effect on what people learn and how they vote.

Two researchers, Robert Epstein and Ronald E. Robertson, recently asked undecided voters in both the United States and India to use a search engine to learn about upcoming elections. The engines they used were programmed to skew the search results, favoring one party over another. Those results, they said, shifted voting preferences by 20 percent.

Then again, how would anyone know? What we learn about these Internet giants comes mostly from the tiny proportion of their research that they share.

wouldn’t yet call Facebook or Google’s algorithms political WMDs, because I have no evidence that the companies are using their networks to cause harm. Still, the potential for abuse is vast.

Basking in the company of people he believed to be supportive and like-minded, Romney let loose with his observation that 47 percent of the population were “takers,” living off the largesse of big government. These people would never vote for him, the governor said—which made it especially important to reach out to the other 53 percent. But Romney’s targeting, it turned out, was inexact. The caterers circulating among the donors, serving drinks and canapés, were outsiders.

Success for Romney at that Boca Raton gathering required both accurate targeting and secrecy.

Modern consumer marketing, however, provides politicians with new pathways to specific voters so that they can tell them what they know they want to hear.

Direct mail was microtargeting on training wheels. The convergence of Big Data and consumer marketing now provides politicians with far more powerful tools.

Those fickle shoppers who switched brands to save a few cents, for example, behaved very much like swing voters.

Would similar calculations work for swing voters? Armed with massive troves of consumer, demographic, and voting data, Ghani and his team set out to investigate.

Throughout this process, each campaign developed profiles of American voters. Each profile contained numerous scores, which not only gauged their value as a potential voter, volunteer, and donor but also reflected their stances on different issues.

Four years later, Hillary Clinton’s campaign built upon the methodology established by Obama’s team. It contracted a microtargeting start-up, the Groundwork, financed by Google chairman Eric Schmidt and run by Michael Slaby, the chief technology officer of Obama’s 2012 campaign.

In late 2015, the Guardian reported that a political data firm, Cambridge Analytica, had paid academics in the United Kingdom to amass Facebook profiles of US voters, with demographic details and records of each user’s “likes.” They used this information to develop psychographic analyses of more than forty million voters, ranking each on the scale of the “big five” personality traits: openness, conscientiousness, extroversion, agreeableness, and neuroticism.

In this sense, we can think of the voting public very much as we think of financial markets. With the flow of information, values rise and fall, as do investments. In these new political markets, each one of us represents a stock with its own fluctuating price. And each campaign must decide if and how to invest in us.

But with microtargeting, the focus shifts from the region to the individual.

The campaigns use similar analysis to identify potential donors and to optimize each one. Here it gets complicated, because many of the donors themselves are carrying out their own calculations. They want the biggest bang for their buck. They know that if they immediately hand over the maximum contribution the campaign will view them as “fully tapped” and therefore irrelevant. But refusing to give any money will also render them irrelevant.

According to Zeynep Tufekci, a techno-sociologist and professor at the University of North Carolina, these groups pinpoint vulnerable voters and then target them with fear-mongering campaigns, scaring them about their children’s safety or the rise of illegal immigration. At the same time, they can keep those ads from the eyes of voters likely to be turned off (or even disgusted) by such messaging.

Successful microtargeting, in part, explains why in 2015 more than 43 percent of Republicans, according to a survey, still believed the lie that President Obama is a Muslim. And 20 percent of Americans believed he was born outside the United States and, consequently, an illegitimate president. (Democrats may well spread their own disinformation in microtargeting, but nothing that has surfaced matches the scale of the anti-Obama campaigns.)

New advertising companies like Simulmedia, in New York, assemble TV viewers into behavioral buckets, so that advertisers can target audiences of like-minded people, whether hunters, pacifists, or buyers of tank-sized SUVs.

As this happens, it will become harder to access the political messages our neighbors are seeing—and as a result, to understand why they believe what they do, often passionately.

The result of these subterranean campaigns is a dangerous imbalance. The political marketers maintain deep dossiers on us, feed us a trickle of information, and measure how we respond to it. But we’re kept in the dark about what our neighbors are being fed. This resembles a common tactic used by business negotiators. They deal with different parties separately so that none of them knows what the other is hearing. This asymmetry of information prevents the various parties from joining forces—which is precisely the point of a democratic government.

The scoring of individual voters also undermines democracy, making a minority of voters important and the rest little more than a supporting cast. Indeed, looking at the models used in presidential elections, we seem to inhabit a shrunken country. As I write this, the entire voting population that matters lives in a handful of counties in Florida, Ohio, Nevada, and a few other swing states.

I might point out here that while many of the WMDs we’ve been looking at, from predatory ads to policing models, deliver most of their punishment to the struggling classes, political microtargeting harms voters of every economic class.

In any case, the entire political system—the money, the attention, the fawning—turns to targeted voters like a flower following the sun. The rest of us are virtually ignored (except for fund-raising come-ons). The programs have already predicted our voting behavior, and any attempt to change it is not worth the investment.*2

They still seem to brim with high voting potential. But those expected not to vote are largely ignored. The systems are searching for the cheapest votes to convert, with the highest return for each dollar spent. And nonvoters often look expensive. This dynamic prods a certain class of people to stay active and lets the rest lie fallow forever.

With political messaging, as with most WMDs, the heart of the problem is almost always the objective. Change that objective from leeching off people to helping them, and a WMD is disarmed—and can even become a force for good.

The problem is that they’re feeding on each other. Poor people are more likely to have bad credit and live in high-crime neighborhoods, surrounded by other poor people. Once the dark universe of WMDs digests that data, it showers them with predatory ads for subprime loans or for-profit schools. It sends more police to arrest them, and when they’re convicted it sentences them to longer terms. This data feeds into other WMDs, which score the same people as high risks or easy targets and proceed to block them from jobs, while jacking up their rates for mortgages, car loans, and every kind of insurance imaginable. This drives their credit rating down further, creating nothing less than a death spiral of modeling.

The same WMDs that abuse the poor also place the comfortable classes of society in their own marketing silos.

The quiet and personal nature of this targeting keeps society’s winners from seeing how the very same models are destroying lives, sometimes just a few blocks away.

But WMDs reverse the equation. Working in darkness, they carve one into many, while hiding us from the harms they inflict upon our neighbors near and far. And those harms are legion.

In September of 1996, two months before his reelection, President Bill Clinton signed the Defense of Marriage Act. This law, defining marriage as between one man and one woman, promised to firm up support for the president in conservative patches of battleground states, including Ohio and Florida.