The Age of Surveillance Capitalism

by Shoshana Zuboff

The first question is “Who knows?” This is a question about the distribution of knowledge and whether one is included or excluded from the opportunity to learn. The second question is “Who decides?” This is a question about authority: which people, institutions, or processes determine who is included in learning, what they are able to learn, and how they are able to act on their knowledge. What is the legitimate basis of that authority? The third question is “Who decides who decides?” This is a question about power. What is the source of power that undergirds the authority to share or withhold knowledge?

How different might our society be if US businesses had chosen to invest in people as well as in machines?

This results in what economists call “job polarization,” which features some high-skill jobs and other low-skill jobs, with automation displacing most of the jobs that were once “in the middle.”

that the principles of capitalism initially aimed at production eventually shape the wider social and moral milieu. “Whatever opinion one has about the division of labor,” Durkheim wrote, “everyone knows that it exists, and is more and more becoming one of the fundamental bases of the social order.”

The most remarkable effect of the division of labor is not that it increases output of functions divided, but that it renders them solidary. Its role… is not simply to embellish or ameliorate existing societies, but to render societies possible which, without it, would not exist.… It passes far beyond purely economic interests, for it consists in the establishment of a social and moral order sui generis.

Durkheim’s vision was neither sterile nor naive. He recognized that things can take a dark turn and often do, resulting in what he called an “abnormal” (sometimes translated as “pathological”) division of labor that produces social distance, injustice, and discord in place of reciprocity and interdependency. In this context, Durkheim singled out the destructive effects of social inequality on the division of labor in society, especially what he viewed as the most dangerous form of inequality: extreme asymmetries of power that make “conflict itself impossible” by “refusing to admit the right of combat.” Such pathologies can be cured only by a politics that asserts the people’s right to contest, confront, and prevail in the face of unequal and illegitimate power over society. In the late nineteenth century and most of the twentieth century, that contest was led by labor and other social movements that asserted social equality through institutions such as collective bargaining and public education.

The transformation that we witness in our time echoes these historical observations as the division of learning follows the same migratory path from the economic to the social domain once traveled by the division of labor. Now the division of learning “passes far beyond purely economic interests,” for it establishes the basis for our social order and its moral content.

“Personal information is increasingly used to enforce standards of behavior. Information processing is developing, therefore, into an essential element of long-term strategies of manipulation intended to mold and adjust individual conduct.”34 Simitis argued that these trends were incompatible not only with privacy but with the very possibility of democracy, which depends upon a reservoir of individual capabilities associated with autonomous moral judgment and self-determination.

“The danger that the computer poses is to human autonomy. The more that is known about a person, the easier it is to control him. Insuring the liberty that nourishes democracy requires a structuring of societal use of information and even permitting some concealment of information.”

So here is what is at stake: surveillance capitalism is profoundly antidemocratic, but its remarkable power does not originate in the state, as has historically been the case. Its effects cannot be reduced to or explained by technology or the bad intentions of bad people; they are the consistent and predictable consequences of an internally consistent and successful logic of accumulation. Surveillance capitalism rose to dominance in the US under conditions of relative lawlessness. From there it spread to Europe, and it continues to make inroads in every region of the world. Surveillance capitalist firms, beginning with Google, dominate the accumulation and processing of information, especially information about human behavior. They know a great deal about us, but our access to their knowledge is sparse: hidden in the shadow text and read only by the new priests, their bosses, and their machines.

Between 2012 and 2015, I interviewed 52 data scientists and specialists in the “internet of things.” They came from 19 different companies with a combined 586 years of experience in high-technology corporations and startups, primarily in Silicon Valley. I spoke with them about the prominence of inevitability rhetoric among the purveyors of the new apparatus, and I posed the same question to each one: why do so many people say that ubiquitous computing is inevitable? The agreement among their responses was striking. Although they did not have the language of surveillance capitalism, nearly every interviewee regarded inevitability rhetoric as a Trojan horse for powerful economic imperatives, and each one of them lamented the lack of any critical discussion of these assumptions.

Inevitability rhetoric is a cunning fraud designed to render us helpless and passive in the face of implacable forces that are and must always be indifferent to the merely human.

“The changes and disruptions that an evolving technology repeatedly caused in modern life were accepted as given or inevitable simply because no one bothered to ask whether there were other possibilities.”

Surveillance capitalist leaders assume that we will succumb to the naturalistic fallacy as Steinbeck’s farmers were meant to do. Because Google is successful—because surveillance capitalism is successful—its rules must obviously be right and good. Like the bank agents, Google wants us to accept that its rules simply reflect the requirements of autonomous processes, something that people cannot control. However, our grasp of the inner logic of surveillance capitalism suggests otherwise. Men and women made it, and they can control it. They merely choose not to do so.

During a three-week period, twenty-three participants were continuously informed of the number of apps accessing their location information and the total number of accesses in a given period. They were flabbergasted by the sheer volume of the onslaught as they each variously learned that their locations were accessed 4,182 times, 5,398 times, 356 times, and so on, over a 14-day period—all for the sake of advertisers, insurers, retailers, marketing firms, mortgage companies, and anyone else who pays to play in these behavioral markets.28 As one participant summed it up, “It felt like I’m being followed by my own phone. It’s scary.”29 Fifty-eight percent of participants subsequently restricted the permissions granted to their mobile apps.

The agencies’ well-meaning guidelines overlook the inconvenient truth that transparency and privacy represent friction for surveillance capitalists in much the same way that improving working conditions, rejecting child labor, or shortening the working day represented friction for the early industrial capitalists. It took targeted laws to change working conditions back then, not suggestions. Then as now, the problems to which these pleas for self-restraint are addressed cannot be understood as excesses, mistakes, oversights, or lapses of judgment. They are necessitated by the reigning logic of accumulation and its relentless economic imperatives.

In 2015 privacy advocates discovered that the corporation’s smart TVs were actually too smart, recording everything said in the vicinity of the TV—please pass the salt; we’re out of laundry detergent; I’m pregnant; let’s buy a new car; we’re going to the movies now; I have a rare disease; she wants a divorce; he needs a new lunch box; do you love me?—and sending all that talk to be transcribed by another market leader in voice-recognition systems, Nuance Communications.

When Mattel hired a new CEO in January 2017, it’s no surprise that the company looked to Google, selecting its president for the Americas with responsibility for Google’s commercial and advertising sales operations.36 Most analysts agreed that the appointment heralded Mattel’s commitment to its innovations in internet-enabled toys and virtual reality, but the appointment underscores the shift in focus from making great products for you to collecting great data about you. The doll that was once a beloved mirror of a child’s unfettered imagination, along with all the other toys in the toy box—and the box, and the room that hosts the box, and the house that hosts the room—are all earmarked for rendition, calculation, connection, and profit. No longer mere things, they are reinvented as vehicles for a horde of commercial opportunities fabricated from our dialogue chunks and assorted gold dust.

By 2018, Amazon had inked deals with home builders, installing its Dot speakers directly into ceilings throughout the house as well as Echo devices and Alexa-powered door locks, light switches, security systems, door bells, and thermostats. As one report put it, “Amazon can acquire more comprehensive data on people’s living habits.…” The company wants to sell real-world services such as house cleaning, plumbing, and restaurant delivery, but according to some insiders, the vision is more far-reaching: an omniscient voice that knows all experience and anticipates all action.41 Already, forward-looking Amazon patents include the development of a “voice-sniffer algorithm” integrated into any device and able to respond to hot words such as “bought,” “dislike,” or “love” with product and service offers.

It was probably no coincidence that the leaked Facebook presentation appeared around the same time that a young Cambridge Analytica mastermind-turned-whistleblower, Chris Wylie, unleashed a torrent of information on that company’s secret efforts to predict and influence individual voting behavior, quickly riveting the world on the small political analytics firm and the giant source of its data: Facebook. There are many unanswered questions about the legality of Cambridge Analytica’s complex subterfuge, its actual political impact, and its relationship with Facebook. Our interest here is restricted to how its machinations shine a bright light on the power of surveillance capitalism’s mechanisms, especially the determination to render data from the depth dimension.

“We exploited Facebook to harvest millions of people’s profiles,” Wylie admitted, “and built models to exploit what we knew about them and target their inner demons.” His summary of Cambridge Analytica’s accomplishments is a précis of the surveillance capitalist project and a rationale for its determination to render from the depths. These are the very capabilities that have gathered force over the nearly two decades of surveillance capitalism’s incubation in lawless space. These practices produced outrage around the world, when in fact they are routine elements in the daily elaboration of surveillance capitalism’s methods and goals, both at Facebook and within other surveillance capitalist companies. Cambridge Analytica merely reoriented the surveillance capitalist machinery from commercial markets in behavioral futures toward guaranteed outcomes in the political sphere. It was Eric Schmidt, not Wylie, who first pried open this Pandora’s box, paving the way for the transfer of surveillance capitalism’s core mechanisms to the electoral process as he cemented the mutual affinities that produced surveillance exceptionalism. In fact, Wylie enjoyed his early training under Obama’s “director of targeting.”83 Schmidt’s now-weaponized innovations have become the envy of every political campaign and, more dangerously, of every enemy of democracy.84

I don’t quite know whether it is especially computer science or its sub-discipline Artificial Intelligence that has such an enormous affection for euphemism. We speak so spectacularly and so readily of computer systems that understand, that see, decide, make judgments… without ourselves recognizing our own superficiality and immeasurable naiveté with respect to these concepts. And, in the process of so speaking, we anesthetize our ability to… become conscious of its end use.… One can’t escape this state without asking, again and again: “What do I actually do? What is the final application and use of the products of my work?” and ultimately, “Am I content or ashamed to have contributed to this use?”

Behavioral economists argue a worldview based on the notion that human mentation is frail and flawed, leading to irrational choices that fail to adequately consider the wider structure of alternatives. Thaler and Sunstein have encouraged governments to actively design nudges that adequately shepherd individual choice making toward outcomes that align with their interests, as perceived by experts. One classic example favored by Thaler and Sunstein is the cafeteria manager who nudges students to healthier food choices by prominently displaying the fruit salad in front of the pudding; another is the automatic renewal of health insurance policies as a way of protecting individuals who overlook the need for new approvals at the end of each year.

“Conditioning at scale is essential to the new science of massively engineered human behavior.”

What Facebook researchers failed to acknowledge in either experiment is that a person’s susceptibility to subliminal cues and his or her vulnerability to a “contagion” effect is largely dependent upon empathy: the ability to understand and share in the mental and emotional state of another person, including feeling another’s feelings and being able to take another’s point of view—sometimes characterized as “affective” or “cognitive” empathy. Psychologists have found that the more a person can project himself or herself into the feelings of another and take the other’s perspective, the more likely he or she is to be influenced by subliminal cues, including hypnosis. Empathy orients people toward other people. It allows one to get absorbed in emotional experience and to resonate with others’ experiences, including unconsciously mimicking another’s facial expressions or body language. Contagious laughing and even contagious yawning are examples of such resonance.

Once again, public outcry was substantial. “If Facebook can tweak emotions and make us vote, what else can it do?” the Guardian asked. The Atlantic quoted the study’s editor, who had processed the article for publication despite her apparent misgivings.13 She told the magazine that as a private company, Facebook did not have to adhere to the legal standards for experimentation required of academic and government researchers.

A document acquired by the Australian press in May 2017 would eventually reveal this fact. Three years after the publication of the contagion study, the Australian broke the story on a confidential twenty-three-page Facebook document written by two Facebook executives in 2017 and aimed at the company’s Australian and New Zealand advertisers. The report depicted the corporation’s systems for gathering “psychological insights” on 6.4 million high school and tertiary students as well as young Australians and New Zealanders already in the workforce. The Facebook document detailed the many ways in which the corporation uses its stores of behavioral surplus to pinpoint the exact moment at which a young person needs a “confidence boost” and is therefore most vulnerable to a specific configuration of advertising cues and nudges: “By monitoring posts, pictures, interactions, and Internet activity, Facebook can work out when young people feel ‘stressed,’ ‘defeated,’ ‘overwhelmed,’ ‘anxious,’ ‘nervous,’ ‘stupid,’ ‘silly,’ ‘useless,’ and a ‘failure.’”

“The hard reality is that Facebook will never try to limit such use of their data unless the public uproar reaches such a crescendo as to be un-mutable.”

In one sense there is nothing remarkable in observing that capitalists would prefer individuals who agree to work and consume in ways that most advantage capital. We need only to consider the ravages of the subprime mortgage industry that helped trigger the great financial crisis of 2008 or the daily insults to human autonomy at the hands of countless industries from airlines to insurance for plentiful examples of this plain fact. However, it would be dangerous to nurse the notion that today’s surveillance capitalists simply represent more of the same. This structural requirement of economies of action turns the means of behavioral modification into an engine of growth. At no other time in history have private corporations of unprecedented wealth and power enjoyed the free exercise of economies of action supported by a pervasive global architecture of ubiquitous computational knowledge and control constructed and maintained by all the advanced scientific know-how that money can buy.

the power of games to change behavior is shamelessly instrumentalized as gamification spreads to thousands of situations in which a company merely wants to tune, herd, and condition the behavior of its customers or employees toward its own objectives. Typically, this involves importing a few components, such as reward points and levels of advancement, in order to engineer behaviors that serve the company’s immediate interests, with programs such as customer loyalty schemes or internal sales competitions. One analyst compiled a survey of more than ninety such “gamification cases,” complete with return-on-investment statistics.35 Ian Bogost, a professor of interactive computing at Georgia Tech and a digital culture observer, insists that these systems should be called “exploitationware” rather than games because their sole aim is behavior manipulation and modification.

1. Unprecedented: Most of us did not resist the early incursions of Google, Facebook, and other surveillance capitalist operations because it was impossible to recognize the ways in which they differed from anything that had gone before. The basic operational mechanisms and business practices were so new and strange, so utterly sui generis, that all we could see was a gaggle of “innovative” horseless carriages. Most significantly, anxiety and vigilance have been fixed on the known threats of surveillance and control associated with state power. Earlier incursions of behavior modification at scale were understood as an extension of the state, and we were not prepared for the onslaught from private firms. 2. Declaration as invasion: The lack of precedence left us disarmed and charmed. Meanwhile, Google learned the art of invasion by declaration, taking what it wanted and calling it theirs. The corporation asserted its rights to bypass our awareness, to take our experience and transform it into data, to claim ownership of and decisions over the uses of those data, to produce strategies and tactics that keep us ignorant of its practices, and to insist on the conditions of lawlessness required for these operations. These declarations institutionalized surveillance capitalism as a market form. 3. Historical context: Surveillance capitalism found shelter in the neoliberal zeitgeist that equated government regulation of business with tyranny. This “paranoid style” favored self-man...

If industrial capitalism dangerously disrupted nature, what havoc might surveillance capitalism wreak on human nature?

Under surveillance capitalism, the “means of production” serves the “means of behavioral modification.”

The remainder of this chapter prepares the way. The first task is to develop our understanding of what instrumentarian power is not, so in the section that follows we briefly consider key elements of twentieth-century totalitarianism. Most important is the fact that, like instrumentarianism, totalitarian power was also unprecedented. It literally defied human comprehension. There is much that we can learn from the struggles and missteps of scholars, journalists, and citizens as they found themselves overwhelmed by a force that they could neither fathom nor resist. Once we have tackled these questions, we will be equipped to bear down on an exploration of instrumentarianism’s origins in a field of intellectual endeavor that came to be known as “radical behaviorism,” most notably championed by B. F. Skinner and his dream of a “technology of behavior.” In Chapter 13 we integrate our insights to consider the unique aims and strategies of instrumentarian power.

Instrumentarian power has no interest in our souls or any principle to instruct. There is no training or transformation for spiritual salvation, no ideology against which to judge our actions. It does not demand possession of each person from the inside out. It has no interest in exterminating or disfiguring our bodies and minds in the name of pure devotion. It welcomes data on the behavior of our blood and shit, but it has no interest in soiling itself with our excretions. It has no appetite for our grief, pain, or terror, although it eagerly welcomes the behavioral surplus that leaches from our anguish. It is profoundly and infinitely indifferent to our meanings and motives. Trained on measurable action, it only cares that whatever we do is accessible to its ever-evolving operations of rendition, calculation, modification, monetization, and control.

The Sesame Credit system produces a “holistic” rating of “character” with algorithmic learning that goes far beyond the timely payment of bills and loans. Algorithms evaluate and rank purchases (video games versus children’s books), education degrees, and the quantity and “quality” of friends. One reporter’s account of her Sesame Credit experience warns that the algorithm veers into “voodoo,” considering the credit scores of her social contacts, the car she drives, her job, school, and a host of unspecified behavioral variables that supposedly “correlate with good credit.” The shadow text remains out of reach, and users are left to guess how to improve their scores, including shedding friends with low scores and bulking up on high-scoring individuals who, some believe, can boost one’s own rank.

Pentland’s theory of instrumentarian society came to full flower in his 2014 book Social Physics, in which his tools and methods are integrated into an expansive vision of our futures in a data-driven instrumentarian society governed by computation.

In a lecture at Google that garnered enthusiastic applause, Pentland flattered the audience by signaling that the digital cognoscenti will easily accept the obsolescence of the individual as a necessary fate. “What about free will?” he asked the audience in Mountain View. “That may not have occurred to you, but that’s a traditional thing to ask.” He went on to explain that most human behavior—from political views to spending choices to the music that people listen to—is predicted by “what’s cool to do… exposure to what other people do.” Many people reject this idea, he noted, because “it’s not the rhetoric in our society.” Then he assured the Googlers, “You guys are the last people to be saying this to, because you guys are like the best and smartest in the world.” For such people as these, Pentland appeared to say, the death of the individual is yesterday’s news: So you’ve heard about rational individuals. And everybody rags on the rational part. I’m not going to do that. I’m going to rag on the individual part, OK? Because I don’t think we are individuals. What we desire, the ways we learn to go about doing it, what’s valuable, are consensual things… individual incentives… that’s part of this mindset that comes from the 1700s… the action is not between our ears. The action is in our social networks, OK? We are a social species.71

Instrumentarianism reimagines society as a hive to be monitored and tuned for guaranteed outcomes, but this tells us nothing of the lived experience of its members. What are the consequences of life lived in the hive, where one is perceived as an “other” to the surveillance capitalists, designers, and tuners who impose their instruments and methods? How and when do we each become an organism among organisms to ourselves and to one another, and with what result? The answers to these questions are not all guesswork. We can begin by asking our children. Without knowing it, we sent the least formed and most vulnerable among us to scout the hive and settle its wilderness. Now their messages are filtering in from the frontier.

“I felt so lonely… I could not sleep well without sharing or connecting to others,” a Chinese girl recalled. “Emptiness,” an Argentine boy moaned. “Emptiness overwhelms me.” A Ugandan teenager muttered, “I felt like there was a problem with me,” and an American college student whimpered, “I went into absolute panic mode.” These are but a few of the lamentations plucked from one thousand student participants in an international study of media use that spanned ten countries and five continents. They had been asked to abstain from all digital media for a mere twenty-four hours, and the experience released a planet-wide gnashing of teeth and tearing of flesh that even the study’s directors found disquieting.1 Capping the collective cri de coeur, a Slovakian university student reflected, “Maybe it is unhealthy that I can’t be without knowing what people are saying and feeling, where they are, and what’s happening.”

By 2018 Pew Research reported that nearly 40 percent of young people ages 18–29 report being online “almost constantly,” as do 36 percent of those ages 30–49. Generation Z intensifies the trend: 95 percent use smartphones, and 45 percent of teens say they are online “on a near-constant basis.”5 If that is how you spend your days and nights, then the findings of a 2016 study are all too logical, as 42 percent of teenage respondents said that social media affects how people see them, having adopted what the researchers call an outside-looking-in approach to how they express themselves. Their dependency penetrates deeply into their sense of well-being, affecting how they feel about themselves (42 percent) and their happiness (37 percent).

Social media marks a new era in the intensity, density, and pervasiveness of social comparison processes, especially for the youngest among us, who are “almost constantly online” at a time of life when one’s own identity, voice, and moral agency are a work in progress. In fact, the psychological tsunami of social comparison triggered by the social media experience is considered unprecedented. If television created more life dissatisfaction, what happens in the infinite spaces of social media?

This is not a rehearsal. This is the show. Facebook is a prototype of instrumentarian society, not a prophecy. It is the first frontier of a new societal territory, and the youngest among us are its vanguard. The frontier experience is an epidemic of the viewpoint of the Other-One, a hyper-objectification of one’s own personhood shaped by the relentless amplification of life lived from the “outside looking in.” The consequence is a pattern of overwhelming anxiety and disorientation in the simple act of digital disconnection, while connection itself is haunted by fresh anxieties that paradoxically leave too many feeling isolated, diminished, and depressed. One wants to say that the struggles of youth can be painful in any era and that it is simply the destiny of today’s young people to encounter the work of self-construction in this milieu of digital connection and illumination, with its truly marvelous opportunities for voice, community, information, and exploration. One wants to say they will get through it, just as other generations survived the adolescent trials of their time and place.

They crave the hive, just as Hall’s teenagers did in 1904, but the hive they encounter is not the unadulterated product of their natures and their culture of mutuality. It is a zone of asymmetrical power, constructed by surveillance capital as it operates in secrecy beyond confrontation or accountability. It is an artificial creation designed in the service of surveillance capital’s greater good.

It is already possible to see a new awakening to empowering collective action, at least in the privacy domain. One example is None of Your Business (NOYB), a nonprofit organization led by privacy activist Max Schrems. After many years of legal contest, Schrems made history in 2015 when his challenge to Facebook’s data-collection and data-retention practices—which he asserted were in violation of EU privacy law—led the Court of Justice of the European Union to invalidate the Safe Harbor agreement that governed data transfers between the US and the EU. In 2018 Schrems launched NOYB as a vehicle for “professional privacy enforcement.” The idea is to push regulators to close the gap between written regulations and corporate privacy practices, leveraging the threat of significant fines to change a company’s actual procedures. NOYB wants to become “a stable European enforcement platform” that unites groups of users and assists them through the litigation process while building coalitions and advancing “targeted and strategic litigation to maximize the impact ‘on the right to privacy.’”26 However this undertaking progresses, the key point for us is the way in which it points to a social void that must be filled with creative new forms of collective action, if the life of the law is to move against surveillance capitalism.

Between 2009 and the first quarter of 2018, Google’s share of the search market in Europe declined by about 2 percent while increasing in the US by about 9 percent. (Google’s European market share remained high, at 91.5 percent in 2018, compared to 88 percent in the US.) In the case of its Android mobile phones, however, Google’s market share increased by 69 percent in Europe compared to 44 percent in the US. Google’s Chrome browser increased its market share by 55 percent in Europe and 51 percent in the US.

With the application of radical indifference, content is judged by its volume, range, and depth of surplus as measured by the “anonymous” equivalence of clicks, likes, and dwell times, despite the obvious fact that its profoundly dissimilar meanings originate in distinct human situations.

Similarly, reporters at BuzzFeed discovered that Google enables advertisers to target ads to people who type racist terms into the search bar and even suggests ad placements next to searches for “evil jew” and “Jewish control of banks.”

Radical indifference means that it doesn’t matter what is in the pipelines as long as they are full and flowing.

Surveillance capitalism arrived on the scene with democracy already on the ropes, its early life sheltered and nourished by neoliberalism’s claims to freedom that set it at a distance from the lives of people. Surveillance capitalists quickly learned to exploit the gathering momentum aimed at hollowing out democracy’s meaning and muscle. Despite the democratic promise of its rhetoric and capabilities, it contributed to a new Gilded Age of extreme wealth inequality, as well as to once-unimaginable new forms of economic exclusivity and new sources of social inequality that separate the tuners from the tuned.