Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

And if their credit record is poor, there’s a good chance the...

One in ten participants reported hearing from employers that blemished credit histories had sunk their chances, and it’s anybody’s guess how many were disqualified by their credit reports but left in the dark.

While the law stipulates that employers must alert job seekers when credit issues disqualify them, it’s hardly a stretch to believe that some of them simply tell candidates that they weren’t a good fit or that others were more qualified.

The practice of using credit scores in hirings and promotions creates a d...

After all, if you can’t get a job because of your credit record, that record will likely get worse, makin...

It’s not unlike the problem young people face when they look for their first job—and are disquali...

Or the plight of the longtime unemployed, who find that few will hire them because they’ve been...

It’s a spiraling and defeating feedback loop for the unlucky peo...

Employers, naturally, have little sympathy fo...

Good credit, they argue, is an attribute of a responsible person, the k...

But framing debt as a moral issue is a mistake. Plenty of hardworking and trustworthy people lose jobs every day as companies fai...

These numbers climb during ...

And many of the newly unemployed find themselves without...

At that point, all it takes is an accident or an illness for them to mis...

Even with the Affordable Care Act, which reduced the ranks of the uninsured, medical expenses remain the single biggest...

People with savings, of course, can keep their credit intact ...

Those living from paycheck to paycheck are far ...

Consequently, a sterling credit rating is not just a proxy for responsibili...

It is also a proxy for wealth. And wealth is highly cor...

As I write this, ten states have passed legislation to outlaw the use of credit scores in hiring.

In banning them, the New York City government declared that using credit checks “disproportionately affects low-income applicants and applicants of color.” Still, the practice remains legal in forty states.

This is not to say that personnel departments across America are intentionally building a poverty ...

They no doubt believe that credit reports hold relevant facts that help them m...

After all, “The more data, the better” is the guiding principle of...

Yet in the name of fairness, some of this data should ...

in the mainstream and, especially, in the lower echelons of the economy, much of the work, as we’ve seen, is automated.

When mistakes appear in a dossier—and they often do—even the best-designed algorithms will make the wrong decision.

As data hounds have long said: garbage in...

A person at the receiving end of this automated process can suffer the c...

Computer-generated terrorism no-fly lists, for example, are famou...

An innocent person whose name resembles that of a suspected terrorist faces a hellish ordeal every t...

(Wealthy travelers, by contrast, are often able to pay to acquire “trusted traveler” status, which permits them to waltz through security. In effect, they’re spe...

Nearly a half century later, however, redlining is still with us, though in far more subtle forms.

It’s coded into the latest generation of WMDs.

Like Hoffman, the creators of these new models confuse correla...

They punish the poor, and especially racial and e...

And they back up their analysis with reams of statistics, which give them the studied ...

On this algorithmic voyage through life, we’ve clawed our way through education and we’ve landed a job (even if it is one t...

We’ve taken out loans and seen how our creditworthiness is a stand-in for ...

Now it’s time to protect our most treasured assets—our home and car and our family’s health—and make arrangements f...

With ever more information available—including the data from our genomes, the patterns of our sleep, exercise, and diet, and the proficiency of our driving—insurers will increasingly calculate risk for the individual and free themselves from the generalities of the larger pool.

For many, this is a welcome change.

A health enthusiast today can demonstrate, with data, that she sleeps eight hours a night, walks ten miles a day, and eats little but...

Why shouldn’t she get a break on her hea...

But already insurers are using data to divide us into smaller tribes, to offer us different products and services at varying prices.

Some might call this customized service.

The trouble is, it’s not ...

The models place us into groups we cannot see, whose behavior appears to resemble ours. Regardless of the quality of the analys...

Insurers draw these scores from credit reports, and then, using the insurer’s proprietary algorithm, create their own ratings, or e-scores.

In other words, how you manage money can matter more than how you drive a car.