Mindf*ck: Cambridge Analytica and the Plot to Break America

by Christopher Wylie

I spent time talking to voters and seeing up close what American culture was really like. This was both fun and eye-opening; coming from Canada, I was struck by how different our sensibilities were. The first time an American told me he was dead set against “socialized medicine,” the same kind of public healthcare I accessed almost every month back home, I was shocked that someone could even think this way. The hundredth time, not so much.

What we played with as the questioner was how she was weighting that information, which in turn affected her judgment of that information. We biased her mental model of her life. So which is true? Is she happy or not happy? The answer depends on which information is being pulled to the front of her mind. In psychology, this is called priming. And this is, in essence, how you weaponize data: You figure out which bits of salient information to pull to the fore to affect how a person feels, what she believes, and how she behaves.

Unless someone’s parents are secretly Vulcan, no one on earth is a purely rational thinker. We are all affected with cognitive biases, which are the commonly occurring errors in our thinking that generate flawed subjective interpretations of information. It is completely normal for people to process information with bias—in fact, everyone does—and oftentimes these biases are harmless in day-to-day life. These biases are not random in each person. Rather, they are systematic errors, meaning they create patterns in common forms of irrational thinking. In fact, thousands of cognitive biases have been identified in the field of psychology. Some biases are so common and seemingly intuitive that it can be hard for people to even recognize that they are actually irrational. For example, the psychologists Amos Tversky and Daniel Kahneman conducted a study that asked participants a very simple question: “Suppose you sample a word at random from an English text. Is it more likely that the word starts with a k, or that k is the third letter?” Most people responded with the former, that words that start with k (e.g., kitchen, kite, or kilometer) are more likely. However, the opposite is true, and one is actually twice as likely in a typical English text to encounter words where the third letter is a k, such as ask, like, make, joke, or take. They tested for five letters (k, l, n, r, and v) like this. It is easier for people to think of words by first letter because we are taught to or...

By profiling every citizen in a country, imputing their personalities and unique behaviors, and placing those profiles in an in silico simulation of that society (one created inside a computer), we would be building the first prototype of the artificial society. If we could play with an economy or culture in a simulation of artificial agents with the same traits as the actual people they represented, we could just possibly create the most powerful market intelligence tool yet imagined. And by adding quantified cultural signals, we were verging on a new area of something akin to “cultural finance.” We thought that if we got it right, we could run simulations of different futures of whole societies. Forget shorting companies; think about entire economies.

They told me, essentially, that Facebook simply let them take it, through apps the professors had created. Facebook wants people to do research on its platform. The more it learns about its users, the more it can monetize them. It became clear when they explained how they collected data that Facebook’s permissions and controls were incredibly lax. When a person used their app, Stillwell and Kosinski could receive not only that person’s Facebook data, but the data of all of their friends as well. Facebook did not require express consent for apps to collect data from an app user’s friends, as it viewed being a user of Facebook as enough consent to take their data—even if the friends had no idea the app was harvesting their private data. The average Facebook user has somewhere between 150 and 300 friends. My mind turned to Bannon and Mercer, as I knew they would love this idea—and Nix would simply love that they loved it. “Let me get this straight,” I said. “If I create a Facebook app, and a thousand people use it, I’ll get…like 150,000 profiles? Really? Facebook actually lets you do that?” That’s right, they said. And if a couple million people downloaded the app, then we’d get 300 million profiles, minus the overlapping mutual friends. This would be an astonishingly huge data set. Up to that point, the largest data set I had worked on was Trinidad, which I thought was quite large, with profiles of one million people. But this set was on an entirely different level. In other...

In fact, a 2015 study by Youyou, Kosinski, and Stillwell showed that, using Facebook likes, a computer model reigned supreme in predicting human behavior. With ten likes, the model predicted a person’s behavior more accurately than one of their co-workers. With 150 likes, better than a family member. And with 300 likes, the model knew the person better than their own spouse. This is in part because friends, colleagues, spouses, and parents typically see only part of your life, where your behavior is moderated by the context of that relationship. Your parents may never see how wild you can get at a 3 A.M. rave after dropping two hits of MDMA, and your friends may never see how reserved and deferential you are in the office with your boss. They all have slightly different impressions of who you are. But Facebook peers into your relationships, follows you around in your phone, and tracks what you click and buy on the Internet. This is how data from the site becomes more reflective of who you “really are” than the judgments of friends or family. In some respects, a computer model can know a person’s habits better than they even know themselves—a finding that compelled the researchers to add a warning. “Computers outpacing humans in personality judgment,” they wrote, “presents significant opportunities and challenges in the areas of psychological assessment, marketing, and privacy.”

Using Kogan’s app, we would not only get a training set that gave us the ability to create a really good algorithm—because the data was so rich, dense, and meaningful—but we also got the extra benefit of hundreds of additional friend profiles. All for $1 to $2 per app install. We finished the first round of harvesting with money left over. In management, they always say there is a golden rule for running any project: You can get a project done cheap, fast, or well. But the catch is you can choose only two, because you’ll never get all three. For the first time in my life, I saw that rule totally broken—because the Facebook app Kogan created was faster, better, and cheaper than anything I could have imagined.

And not only did we have all her Facebook data, but we were merging it with all the commercial and state bureau data we’d bought as well. And imputations made from the U.S. Census. We had data about her mortgage applications, we knew how much money she made, whether she owned a gun. We had information from her airline mileage programs, so we knew how often she flew. We could see if she was married (she wasn’t). We had a sense of her physical health. And we had a satellite photo of her house, easily obtained from Google Earth. We had re-created her life in our computer. She had no idea. “Give me another,” said Jucikas. And he did it again. And again. And by the third profile, Nix—who’d hardly been paying attention at all—suddenly sat up very straight. “Wait,” he said, his eyes widening behind his black-rimmed glasses. “How many of these do we have?” “What the fuck?” Bannon interjected with a look of annoyance at Nix’s disengagement with the project. “We’re in the tens of millions now,” said Jucikas. “At this pace, we could get to 200 million by the end of the year with enough funding.” “And we know literally everything about these people?” asked Nix. “Yes,” I told him. “That’s the whole point.” The light went on: This was the first time Nix truly understood what we were doing. He could not have been less interested in things like “data” and “algorithms,” but seeing actual people onscreen, knowing everything about them, had seized his imagination. “Do we have their phone num...

It was the first time I was allowed to explore ideas without the constraints of petty internal politics or people snubbing an idea just because it had never been tried before. As much as Nix was a dick, he did give me a lot of leeway to try out new ideas. After Kogan joined, I had professors at the University of Cambridge constantly fawning over the groundbreaking potential that the project could have for advancing psychology and sociology, which made me feel like I was on a mission. And if their colleagues at universities like Harvard or Stanford were also getting interested in our work, I thought that surely we must be onto something. The institute that Kogan proposed really was an inspiring idea to me, and I saw how unlocking this data for researchers around the world could contribute so much value to so many fields. As corny as this might sound, it really felt like I was working on something important—not just for Mercer or the company, but for science. However, I let this feeling distract me to the point of allowing myself to excuse the inexcusable. I told myself that truly learning about society includes delving into uncomfortable questions about our darker sides. How could we understand racial bias, authoritarianism, or misogyny if we did not explore them? What I did not appreciate is the fine line between exploring something and actually creating it. Bannon had assumed control of the company, and he was an ambitious and surprisingly sophisticated cultural warrior. ...

I told Bannon that the most striking thing CA had noticed was how many Americans felt closeted—and not just gay people. This first came up in focus groups and later was confirmed in quantitative research done via online panels. Straight white men, particularly ones who were older, had grown up with a value set that granted them certain social privileges. Straight white men did not have to moderate their speech around women or people of color, because casual racism and misogyny were normalized behaviors. As social norms in America evolved, these privileges began to erode and many of these men were experiencing challenges to their behavior for the first time. At the workplace, “casual flirting” with female secretaries now imperiled your job, and talking about the “thugs” in the African American part of town could get you shunned by peers. These encounters were often uncomfortable and threatening to their identity as “regular men.” Men who were not used to moderating their impulses, body language, and speech began to resent what they saw as the unfair mental and emotional labor it took to change and constantly correct how they presented in public. What I found interesting was how similar the discourse that emerged from these groups of angry straight men was to liberation discourse from gay communities. These men began to experience the burden of the closet, and they did not like the feeling of having to change who they felt they were in order to “pass” in society. Although th...

Social media platforms also use designs that activate “ludic loops” and “variable reinforcement schedules” in our brains. These are patterns of frequent but irregular rewards that create anticipation, but where the end reward is too unpredictable and fleeting to plan around. This establishes a self-reinforcing cycle of uncertainty, anticipation, and feedback. The randomness of a slot machine prevents the player from being able to strategize or plan, so the only way to get a reward is to keep playing. The rewards are designed to be just frequent enough to reengage you after a losing streak and keep you going. In gambling, a casino makes money from the number of turns a player takes. On social media, a platform makes money from the number of clicks a user performs. This is why there are infinite scrolls on newsfeeds—there is very little difference between a user endlessly swiping for more content and a gambler pulling the slot machine lever over and over.

Cambridge Analytica began to use this content to touch on an implied belief about racial competition for attention and resources—that race relations were a zero-sum game. The more they take, the less you have, and they use political correctness so you cannot speak out. This framing of political correctness as an identity threat catalyzed a “boomerang” effect in people where counternarratives would actually strengthen, not weaken, the prior bias or belief. This means that when targets would see clips containing criticism of racist statements by candidates or celebrities, this exposure would have the effect of further entrenching the target’s racialized views, rather than causing them to question those beliefs. In this way, if you could frame racialized views through the lens of identity prior to exposure to a counternarrative, that counternarrative would be interpreted as an attack on identity instead. What was so useful for Bannon was that it in effect inoculated target groups from counternarratives criticizing ethno-nationalism. It created a wicked reinforcement cycle in which the cohort would strengthen their racialized views when they were exposed to criticism. This may be in part because the area of the brain that is most highly activated when we process strongly held beliefs is the same area that is involved when we think about who we are and our identity. Later, when Donald Trump was aggressively criticized in the media for racist or misogynist statements, these crit...

What CA observed was that when respondents were angry, their need for complete and rational explanations was also significantly reduced. In particular, anger put people in a frame of mind in which they were more indiscriminately punitive, particularly to out-groups. They would also underestimate the risk of negative outcomes. This led CA to discover that even if a hypothetical trade war with China or Mexico meant the loss of American jobs and profits, people primed with anger would tolerate that domestic economic damage if it meant they could use a trade war to punish immigrant groups and urban liberals.

This approach proved effective against Remain’s “Project Fear” messaging, which tried to focus voters on the potentially catastrophic economic risks of exiting the European Union. In short, it is far harder to make angry people fearful. The “affect bias” arising out of anger mediates people’s estimation of negative outcomes, which is why angry people are more inclined to engage in risky behavior—the same is true whether they are voting or starting a bar fight. If you have ever been in a bar fight, you know that literally the worst way imaginable to make your opponent think twice about a rash move is to yell threats at him. It only eggs him on. Remain’s focus on the economy also neglected to stop and ask people what they thought the economy was in the first place. Cambridge Analytica identified that many people in non-urban regions or in lower socioeconomic strata often externalized the notion of “the economy” to something that only the wealthy and metropolitan participated in. “The economy” was not their job in a local store; it was something that bankers did. This is also what made certain groups comfortable with economic risks and even trade wars, since, in their minds, that chaos would be unleashed upon the people who worked in “the economy.” And the more forceful the economic argument they heard, the more confident they would become that what they were “actually” hearing was the fears of a cowering elite worried about losing its wealth. This made them feel powerful, an...

Canadians have a hard time understanding populism because, unlike America or Britain, it has never had any Rupert Murdoch–owned media. There is no Fox News or The Sun in Canada. Because of its more risk-conscious banking system, the country did not experience a housing crisis or financial crash. And unlike the rest of the OECD, Canada is the outlier where patriotism and support for immigration actually correlate positively with each other. So I would find myself repeating the same conversation over and over to baffled Canadians who simply could not understand how Brexit or Trump were even possible.

But this is not merely about privacy or consent. This is about who gets to influence our truths and the truths of those around us. This is about the architectures of manipulation we are constructing around our society. And herein lies the lesson of Cambridge Analytica. To understand the harms of social media, we have to first understand what it is. Facebook may call itself a “community” to its users, or a “platform” to regulators, but it is not a service, in the same way a building is not a service. Even if you don’t understand exactly how cyberspace works, it is important to understand that it now surrounds you. Every connected device and computer is part of an interconnected information architecture—and shapes your experience of the world. The most common job titles in most Silicon Valley companies are engineer and architect, not service manager or client relations. But unlike engineering in other sectors, tech companies do not have to perform safety tests to conform to any building codes before releasing their products. Instead, platforms are allowed to adopt dark pattern designs that deliberately mislead users into continual use and giving up more data. Tech engineers intentionally design confounding mazes on their platforms that keep people moving deeper and deeper into these architectures, without any clear exit. And when people keep clicking their way through their maze, these architects delight in the increase in “engagement.” Social media and Internet platforms ar...

If we are to prevent another Cambridge Analytica from attacking our civil institutions, we must seek to address the flawed environment in which it incubated. For too long the congresses and parliaments of the world have fallen for a mistaken view that somehow “the law cannot keep up with technology.” The technology sector loves to parrot this idea, as it tends to make legislators feel too stupid or out of touch to challenge their power. But the law can keep up with technology, just as it has with medicines, civil engineering, food standards, energy, and countless other highly technical fields. Legislators do not need to understand the chemistry of molecular isomers inside a new cancer drug to create effective drug review processes, nor do they need to know about the conductivity of copper in high-voltage wiring to create effective insulation safety standards. We do not expect our legislators to have expert technical knowledge in any other sector because we devolve technical oversight responsibility to regulators. Regulation works because we trust people who know better than we do to investigate industries and innovations as the guardians of public safety. “Regulation” may be one of the least sexy words, evoking an image of faceless jobsworths with their treasured checklists, and we will always argue about the details of their imperfect rules, but nonetheless safety regulation generally works. When you buy food in the grocery store or visit your doctor or step onto an airpl...

The history of building codes stretches back to the year 64 C.E., when Nero restricted housing height, street width, and public water supplies after a devastating fire ravaged Rome for nine days. Though a fire in 1631 prompted Boston to ban wooden chimneys and thatched roofs, the first modern building code emerged out of the devastating carnage of the Great Fire of London, in 1666. As in Boston, London houses had been densely constructed from timber and thatch, which allowed the fire to spread rapidly over four days. It destroyed 13,200 homes, eighty-four churches, and nearly all of the city’s government buildings. Afterward, King Charles II declared that no one shall “erect any House or Building, great or small, but of Brick, or Stone.” His declaration also widened thoroughfares to stop future fires from spreading from one side of the street to the other. After other historic fires in the nineteenth century, many cities followed suit, and eventually public surveyors were tasked with inspections and ensuring that the construction of private property was safe for the inhabitants and for the public at large. New rules emerged, and eventually the notion of public safety became an overarching principle that could override unsafe or unproven building designs, regardless of the desires of property owners or even the consent of inhabitants. A platform like Facebook has been burning for years with its own disasters—Cambridge Analytica, Russian interference, Myanmar’s ethnic cleans...

Software, AI, and digital ecosystems now permeate our lives, and yet those who make the devices and programs we use every single day are not obligated by any federal statute or enforceable code to give due consideration to the ethical impacts to users or society at large. As a profession, software engineering has a serious ethics problem that needs to be addressed. Tech companies do not magically create problematic or dangerous platforms out of thin air—there are people inside these companies who build these technologies. But there is an obvious problem: Software engineers and data scientists do not have any skin in the game. If an engineer’s employer instructs her or him to create systems that are manipulative, ethically dubious, or recklessly implemented, without consideration for user safety, there are no requirements to refuse. Currently, such a refusal to act unethically creates a risk to the employed engineer of repercussions or termination. Even if the unethical design later is found to run afoul of a regulation, the company can absorb liability and pay fines, and there are no professional consequences for the engineers who built the technology, as there would be in the case of a doctor or lawyer who commits a serious breach of professional ethics. This is a perverse incentive that does not exist in other professions. If an employer asked a lawyer or nurse to act unethically, they would be obligated to refuse or face losing their professional license. In other words...