Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

by Cathy O'Neil

And a model built for today will work a bit worse tomorrow. It will grow stale if it’s not constantly updated.

This is not to say that good models cannot be primitive. Some very effective ones hinge on a single variable.

But modelers run into problems—or subject us to problems—when they focus models as simple as a smoke alarm on their fellow humans.

So to sum up, these are the three elements of a WMD: Opacity, Scale, and Damage.

And finally, you might note that not all of these WMDs are universally damaging. After all, they send some people to Harvard, line others up for cheap loans or good jobs, and reduce jail sentences for certain lucky felons. But the point is not whether some people benefit. It’s that so many suffer.

To keep this from happening on a large, firm-threatening scale, Shaw mostly prohibited us from talking to colleagues in other groups—or sometimes even our own office mates—about what we were doing. In a sense, information was cloistered in a networked cell structure, not unlike that of Al Qaeda.

saw that if it didn’t happen soon someone theoretically would have to show up in Tokyo with $50 million in yen. Ironing out that problem added a few frantic hours to the holiday.

All of those issues might fit into the category of occupational hazard. But the real problem came from a nasty feeling I started to have in my stomach.

But unlike the numbers in my academic models, the figures in my models at the hedge fund stood for something. They were people’s retirement funds and mortgages. In retrospect, this seems blindingly obvious.

It was when the markets collapsed in 2008 that the ugly truth struck home in a big way. Even worse than filching dumb money from people’s accounts, the finance industry was in the business of creating WMDs, and I was playing a small part.

But these rising interest rates signaled trouble. Banks were losing trust in each other to pay back overnight loans.

I remember a gala event to celebrate the architects of the system that would soon crash. The firm welcomed Alan Greenspan, the former Fed chairman, and Robert Rubin, the former Treasury secretary and Goldman Sachs executive. Rubin had pushed for a 1999 revision of the Depression-era Glass-Steagall Act. This removed the glass wall between banking and investment operations, which facilitated the orgy of speculation over the following decade. Banks were free to originate loans (many of them fraudulent) and sell them to their customers in the form of securities.

At the D. E. Shaw event, Greenspan warned us about problems in mortgage-backed securities. That memory nagged me when I realized a couple of years later that Rubin, who at the time worked at Citigroup, had been instrumental in collecting a massive portfolio of these exact toxic contracts—a major reason Citigroup later had to be bailed out at taxpayer expense.

But what if the frightening tomorrow on the horizon didn’t resemble any of the yesterdays? What if it was something entirely new and different? That was a concern, because mathematical models, by their nature, are based on the past, and on the assumption that patterns will repeat.

For decades, mortgage securities had been the opposite of scary. They were boring financial instruments that individuals and investment funds alike used to diversify their portfolios.

And so in the 1980s, investment bankers started to buy thousands of mortgages and package them into securities—a kind of bond, which is to say an instrument that pays regular dividends, often at quarterly intervals.

As the world later learned, mortgage companies were making rich profits during the boom by loaning money to people for homes they couldn’t afford. The strategy was simply to write unsustainable mortgages, snarf up the fees, and then unload the resulting securities—the sausages—into the booming mortgage security market.

In the run-up to the housing collapse, mortgage banks were not only offering unsustainable deals but actively prospecting for victims in poor and minority neighborhoods. In a federal lawsuit, Baltimore officials charged Wells Fargo with targeting black neighborhoods for so-called ghetto loans. The bank’s “emerging markets” unit, according to a former bank loan officer, Beth Jacobson, focused on black churches. The idea was that trusted pastors would steer their congregants toward loans. These turned out to be subprime loans carrying the highest interest rates.

To be clear, the subprime mortgages that piled up during the housing boom, whether held by strawberry pickers in California or struggling black congregants in Baltimore, were not WMDs. They were financial instruments, not models, and they had little to do with math. (In fact, the brokers went to great lengths to ignore inconvenient numbers.)

But when banks started loading mortgages like Alberto Ramirez’s into classes of securities and selling them, they were relying on flawed mathematical models to do it. The risk model attached to mortgage-backed securities was a WMD.

The first false assumption was that crack mathematicians in all of these companies were crunching the numbers and ever so carefully balancing the risk.

And those sellers trusted that they’d manage to unload the securities before they exploded. Smart people would win. And dumber people, the providers of dumb money, would wind up holding billions (or trillions) of unpayable IOUs. Even rigorous mathematicians—and there were a few—were working with numbers provided by people carrying out wide-scale fraud.

The second false assumption was that not many people would default at the same time. This was based on the theory, soon to be disproven, that defaults were largely random and unrelated events. This led to a belief that solid mortgages would offset the losers in each tranche. The risk models were assuming that the future would be no different from the past.

In order to sell these mortgage-backed

backed bonds, the banks needed AAA ratings. For this, they looked to the three credit-rating agencies. As the market expanded, rating the growing billion-dollar market in mortgage bonds turned into a big business for the agencies, bringing in lucrative fees. They grew addicted to those fees. And they understood all too clearly that if they provided anything less than AAA ratings, the banks would take the work to their competitors. So the agencies played ball. They paid more attention to customer satisfaction than to the accuracy of their models.

Of all the WMD qualities, the one that turned these risk models into a monstrous force of global dimension was scale.

But this time the power of modern computing fueled fraud at a scale unequaled in history.

Credit default swaps were small insurance policies that transferred the risk on a bond.

The overheated (and then collapsing) market featured $3 trillion of subprime mortgages by 2007, and the market around it—including the credit default swaps and synthetic CDOs, which magnified the risks—was twenty times as big. No national economy could compare.

Paradoxically, the supposedly powerful algorithms that created the market, the ones that analyzed the risk in tranches of debt and sorted them into securities, turned out to be useless when it came time to clean up the mess and calculate what all the paper was actually worth.

math could multiply the horseshit, but it could ...

Over the following months, disaster finally hit the mainstream. That’s when everyone finally saw the people on the other side of the algorithms. They were desperate home owners losing their homes and millions of Americans losing their jobs. Credit card defaults leapt to record highs. The human suffering, which had been hidden from view behind numbers, spreadsheets, and risk scores, became palpable.

By 2009, it was clear that the lessons of the market collapse had brought no new direction to the world of finance and had instilled no new values. The lobbyists succeeded, for the most part, and the game remained the same: to rope in dumb money. Except for a few regulations that added a few hoops to jump through, life went on.

was forced to confront the ugly truth: people had deliberately wielded formulas to impress rather than clarify.

To calculate risk, our team employed the Monte Carlo method. To picture it, just imagine spinning the roulette wheel at a casino ten thousand times, taking careful notes all the while.

The fact was, the hedge funds always considered themselves the smartest of the smart, and since understanding risk was fundamental to their existence, they would never rely entirely on outsiders like us.

Throughout my time at the hotline, I got the sense that the people warning about risk were viewed as party poopers or, worse, a threat to the bank’s bottom line. This was true even after the cataclysmic crash of 2008, and it’s not hard to understand why. If they survived that one—because they were too big to fail—why were they going to fret over risk in their portfolio now?

The refusal to acknowledge risk runs deep in finance. The culture of Wall Street is defined by its traders, and risk is something they actively seek to underestimate.

This is a result of the way we define a trader’s prowess, namely by his “Sharpe ratio,” which is calculated as the profits he generates divided by the risks in his portfolio. This ratio is crucial to a trader’s career, his annual bonus, his very sense of being. If you disembody those traders and consider them as a set of algorithms, those algorithms are relentlessly...

In both cultures, wealth is no longer a means to get by. It becomes directly tied to personal worth. A young suburbanite with every advantage—the prep school education, the exhaustive coaching for college admissions tests, the overseas semester in Paris or Shanghai—still flatters himself that it is his skill, hard work, and prodigious problem-solving abilities that have lifted him into a world of privilege. Money vindicates all doubts. And the rest of his circle plays along, forming a mutual admiration society. They’re eager to convince us all that Darwinism is at work, when it looks very much to the outside like a combination of gaming a system and dumb luck.

In both of these industries, the real world, with all of its messiness, sits apart. The inclination is to replace people with data trails, turning them into more effective shoppers, voters, or workers to optimize some objective.

Instead of a bust, I saw a growing dystopia, with inequality rising. The algorithms would make sure that those deemed losers would remain that way. A lucky minority would gain ever more control over the data economy, raking in outrageous fortunes and convincing themselves all the while that they deserved it.

What does a single national diet have to do with WMDs? Scale. A formula, whether it’s a diet or a tax code, might be perfectly innocuous in theory. But if it grows to become a national or global standard, it creates its own distorted and dystopian economy. This is what has happened in higher education.

The story starts in 1983. That was the year a struggling newsmagazine, U.S. News & World Report, decided to undertake an ambitious project. It would evaluate 1,800 colleges and universities throughout the United States and rank them for excellence.

In the following years, editors at U.S. News tried to figure out what they could measure. This is how many models start out, with a series of hunches. The process is not scientific and has scant grounding in statistical analysis.

The journalists at U.S. News, though, were grappling with “educational excellence,” a much squishier value than the cost of corn or the micrograms of protein in each kernel.

Instead they picked proxies that seemed to correlate with success. They looked at SAT scores, student-teacher ratios, and acceptance rates. They analyzed the percentage of incoming freshmen who made it to sophomore year and the percentage of those who graduated. They calculated the percentage of living alumni who contributed money to their alma mater, surmising that if they gave a college money there was a good chance they appreciated the education there. Three-quarters of the ranking would be produced by an algorithm—an opinion formalized in code—that incorporated these proxies. In the other quarter, they would factor in the subjective views of college officials throughout the country.

The trouble was that the rankings were self-reinforcing. If a college fared badly in U.S. News, its reputation would suffer, and conditions would deteriorate.

The ranking would tumble further. The ranking, in short, was destiny.

But was Macalester better than Reed, or Iowa better than Illinois? It was hard to say. Colleges were like different types of music, or different diets. There was room for varying opinions, with good arguments on both sides.