Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

Universities were opening their doors to more poor students and minorities. Oppor...

This signaled social...

But naturally, this influx of newcomers dragged down t...

However, when statisticians broke down the population into income groups, scores for every single group were ri...

In statistics, this phenomenon is known as Simpson’s Paradox: when a whole body of data displays one trend, yet when broken into subgroups, the opposite trend ...

instead of measuring teachers on an absolute scale, they tried to adjust for social inequalities in the model.

The students each had a predicted score. If they surpassed this prediction, the teacher got the credit. If they came up short, the teacher got the blame.

If that sounds primitive to you, believe me, it is.

Statistically speaking, in these attempts to free the tests from class and color, the administrators moved from...

Instead of basing scores on direct measurement of the students, they based them on the so-called error term—the gap ...

Mathematically, this is a much sketchie...

Since the expectations themselves are derived from statistics, these amount to guesses on top of guesses. The result is a model with loads of random ...

And I think serving either the neediest or the top students—or both—creates problems.

Needy students’ scores are hard to move because they have learning problems, and top students’ scores are hard to move because they have already scored high so there’s little room for improvement.”

That suggests that the evaluation data is practically random.

It wasn’t the teachers’ performance that was bouncing all over the place. It was the scoring generated by a bogus WMD.

“I’ve seen some great teachers convince themselves that they were mediocre at best based on those scores,”

“It moved them away from the great lessons they used to teach, toward increasing test prep.

To a young teacher, a poor value-added score is punishing, and a good one may lead to a false sense of accomplis...

In other words, education officials can attempt to study what’s happening at each individual school—and pay less attention to WMDs like value-added models.

Or better yet, jettison them entirely.

(Reduce debt, pay bills on time, and stop ordering new credit cards.)

If you have questions about your score, you have the legal right to ask for your credit report, which includes all the information that goes into the score, including your record of mortgage and utility payments, your total debt, and the percentage of available credit you’re using.

Though the process can be slow to the point of torturous, if you find mistakes, ...

Credit card companies such as Capital One carry out similar rapid-fire calculations as soon as someone shows up on their website.

They can often access data on web browsing and purchasing patterns, which provide loads of insights about the potential customer.

Chances are, the person clicking for new Jaguars is richer than the one checking out a 2...

Most scoring systems also pick up the location of the visitor’s computer. When this is matched with real estate data, the...

But consider the nasty feedback loop that e-scores create.

There’s a very high chance that the e-scoring system will give the borrower from the rough section of East Oakland a low score. A lot of people default there.

So the credit card offer popping up on her screen will be targeted to ...

That means less available credit and higher interest rates for those who ...

Fair and Isaac’s great advance was to ditch the proxies in favor of the relevant financial data, like past behavior with respect to paying bills.

They focused their analysis on the individual in question—and not on other people with similar attributes.

From time to time, people ask me how to teach ethics to a class of data scientists. I usually begin with a discussion of how to build an e-score model and ask them whether it makes sense to use “race” as an input in the model.

They inevitably respond that such a question would be unfair and probably illegal.

The next question is whether to use “zip code.” This seems fa...

But it doesn’t take long for the students to see that they are codifying past inj...

When they include an attribute such as “zip code,” they are expressing the opinion that the history of human behavior in that patch of real estate should determine, at least in part, wha...

In other words, the modelers for e-scores have to make do with trying to answer the question “How have people like you behaved in the past?” when ideally they wo...

The difference between these two ques...

Imagine if a highly motivated and responsible person with modest immigrant beginnings is trying to start a business and needs to rely ...

Who would take a chance on such a person? Probably not a model trained on such demogr...

If people pay their bills on time and avoid debt, employers ask, wouldn’t that signal trustworthiness and dependability?

It’s not exactly the same thing, they know.

But wouldn’t there be a signific...

But if you’re looking for a job, there’s an excellent chance that a missed credit card payment or late fees on student loans could be working against you.

According to a survey by the Society for Human Resource Management, nearly half of America’s employers screen potential hires by looking at their credit reports.

Before companies carry out these checks, they must first ask for permission.

But that’s usually little more than a formality; at many companies, those refusing to surrender their credit data won’t even be considered for jobs.