Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

I could barely keep up with all the ways I was hearing of people being manipulated, controlled, and intimidated by algorithms.

The story starts in 1983. That was the year a struggling newsmagazine, U.S. News & World Report, decided to undertake an ambitious project. It would evaluate 1,800 colleges and universities throughout the United States and rank them for excellence.

President Lyndon Johnson’s ideal for higher education—“a way to deeper personal fulfillment, greater personal productivity and increased personal reward”—didn’t fit into their model.

you look at this development from the perspective of a university president, it’s actually quite sad. Most of these people no doubt cherished their own college experience—that’s part of what motivated them to climb the academic ladder. Yet here they were at the summit of their careers dedicating enormous energy toward boosting performance in fifteen areas defined by a group of journalists at a second-tier newsmagazine.

However, when you create a model from proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.

Even schools that improved their numbers would fall behind if others advanced faster.

red queen

With the objective of boosting its selectivity score, the safety school can now reject the excellent candidates that, according to its own algorithm, are most likely not to matriculate.

The proxies the journalists chose for educational excellence make sense, after all. Their spectacular failure comes, instead, from what they chose not to count: tuition and fees. Student financing was left out of the model.

By leaving cost out of the formula, it was as if U.S. News had handed college presidents a gilded checkbook. They had a commandment to maximize performance in fifteen areas, and keeping costs low wasn’t one of them. In fact, if they raised prices, they’d have more resources for addressing the areas where they were being measured.

As colleges position themselves to move up the U.S. News charts, they manage their student populations almost like an investment portfolio.

an entire consulting industry has risen up to “optimize recruitment.”

Lior Pachter, a computational biologist at Berkeley, looked into

In a system in which cheating is the norm, following the rules amounts to a handicap.

The victims, of course, are the vast majority of Americans, the poor and middle-class families who don’t have thousands of dollars to spent on courses and consultants.

also wouldn’t be too hard to lower costs. One approach already gaining popularity is to lower the percentage of tenured faculty, replacing these expensive professors, as they retire, with cheaper instructors, or adjuncts.

So the government capitulated. And the result might be better. Instead of a ranking, the Education Department released loads of data on a website. The result is that students can ask their own questions about the things that matter to them—including class size, graduation rates, and the average debt held by graduating students. They don’t need to know anything about statistics or the weighting of variables. The software itself, much like an online travel site, creates individual models for each person. Think of it: transparent, controlled by the user, and personal. You might call it the opposite of a WMD.

These so-called diploma mills were often underwritten by government-financed loans, and the diplomas they awarded had scant value in the workplace.

When it comes to WMDs, predatory ads practically define the genre. They zero in on the most desperate among us at enormous scale. In education, they promise what’s usually a false road to prosperity, while also calculating how to maximize the dollars they draw from each prospect. Their operations cause immense and nefarious feedback loops and leave their customers buried under mountains of debt. And the targets have little idea how they were scammed, because the campaigns are opaque.

Vulnerability is worth gold. It always has been.

Once the ignorance is established, the key for the recruiter, just as for the snake-oil merchant, is to locate the most vulnerable people and then use their private information against them. This involves finding where they suffer the most, which is known as the “pain point.”

Every time you discover another credit card offer in your mailbox, you’re participating in one of these tests. By throwing out the letter unopened, you’re providing the company with a valuable piece of data: that campaign didn’t work for you.

a 1 percent response rate is the stuff of dreams.

In addition to cutting-edge computer science, predatory advertisers often work with middlemen, who use much cruder methods to target prospects. In 2010, one effective ad featured a photo of President Obama and said: “Obama Asks Moms to Return to School: Finish Your Degree—Financial Aid Available to Those Who Qualify.” The ad suggested that the president had signed a new bill aimed at getting mothers back in school. This was a lie. But if it spurred people to click, it served its purpose.

This kind of online targeting is called “lead generation.” Its goal is to come up with lists of prospects,

between 20 and 30 percent of the promotional budgets at for-profit colleges go to lead generation. For the most promising leads, colleges will pay as much as $150 each.

For-profit colleges also provide free services in exchange for face time with students. Cassie Magesis, another readiness counselor at the Urban Assembly, told me that the colleges provide free workshops to guide students in writing their résumés. These sessions help the students. But impoverished students who provide their contact information are subsequently stalked. The for-profit colleges do not bother targeting rich students. They and their parents know too much.

A Senate report on thirty for-profit systems found that they employed one recruiter for every forty-eight students.

amost better than their student teacher ratio

That came out to $2,225 per student on marketing and only $892 per student on instruction.

Compare that to Portland Community College in Oregon, which spends $5,953 per student on instruction and about 1.2 percent of its budget, or $185 per student, on marketing.

The crucial metric is the so-called 90-10 rule, included in the Higher Education Act of 1965. It stipulates that colleges cannot get more than 90 percent of their funding from federal aid.

In 2014, investigators at CALDER/American Institutes for Research created nearly nine thousand fictitious résumés. Some of their fake job applicants held associate degrees from for-profit universities, others had similar diplomas from community colleges, while a third group had no college education at all. The researchers sent their résumés to job postings in seven major cities and then measured the response rate. They found that diplomas from for-profit colleges were worth less in the workplace than those from community colleges and about the same as a high school diploma. And yet these colleges cost on average 20 percent more than flagship public universities.

hoy crap! tis is an intetesting wmd for good though. a white hat wmd

The presidents of the leading for-profit universities make millions of dollars every year. For example, Gregory W. Cappelli, CEO of Apollo Education Group, the parent company of the University of Phoenix, took home $25.1 million in total compensation in 2011. At public universities, which have their own distortions, only football and basketball coaches can hope to make that much.

To date, a couple of federal laws, such as the Fair Credit Reporting Act and the Health Insurance Portability and Accountability Act, or HIPAA, establish some limits on health and credit data. Maybe, with an eye on lead generators, they’ll add more.

hot-spot predictors are similar to the shifting defensive models in baseball that we discussed earlier. Those systems look at the history of each player’s hits and then position fielders where the ball is most likely to travel.

When police set up their PredPol system, they have a choice. They can focus exclusively on so-called Part 1 crimes. These are the violent crimes, including homicide, arson, and assault, which are usually reported to them. But they can also broaden the focus by including Part 2 crimes, including vagrancy, aggressive panhandling, and selling and consuming small quantities of drugs. Many of these “nuisance” crimes would go unrecorded if a cop weren’t there to see them.

This creates a pernicious feedback loop. The policing itself spawns new data, which justifies more policing. And our prisons fill up with hundreds of thousands of people found guilty of victimless crimes.

geography is a highly effective proxy for race.

and why? because of an older wmd called redlining

1982, when a criminologist named George Kelling teamed up with a public policy expert, James Q. Wilson, to write a seminal article in the Atlantic Monthly on so-called broken-windows policing.

the criminal justice system sent millions of mostly young minority men to prison, many of them for minor offenses.

The idea was only to make sure the standards didn’t fall. The cops, in this scheme, were helping a neighborhood maintain its own order but not imposing their own.

PredPol,

we can understand why police departments would choose to include nuisance data. Raised on the orthodoxy of zero tolerance, many have little more reason to doubt the link between small crimes and big ones than the correlation between smoke and fire.

More data, it’s easy to believe, is better data.

here is an great ccounter example

But how about crimes far removed from the boxes on the PredPol maps, the ones carried out by the rich?

finance is underpoliced.

Clamping down on white-collar crime would require people with different tools and skills.

why not train for this especially vwhen its a data driven business?

The result is that we criminalize poverty, believing all the while that our tools are not only scientific but fair.

Stop and frisk isn’t exactly a WMD, because it relies on human judgment and is not formalized into an algorithm. But it is built upon a simple and destructive calculation. If police stop one thousand people in certain neighborhoods, they’ll uncover, on average, one significant suspect and lots of smaller ones. This isn’t so different from the long-shot calculations used by predatory advertisers or spammers. Even when the hit ratio is miniscule, if you give yourself enough chances you’ll reach your target.

So the system sacrifices enormous efficiencies for the promise of fairness.

WMDs, by contrast, tend to favor efficiency.