More on this book
Community
Kindle Notes & Highlights
Recently, however, technology has begun to develop new methods of behavior control capable of altering not just an individual’s actions but his very personality and manner of thinking . . . the behavioral technology being developed in the United States today touches upon the most basic sources of individuality and the very core of personal freedom . . . the most serious threat . . . is the power this technology gives one man to impose his views and values on another.
The National Research Act, passed in 1974, stipulated the creation of institutional review boards and laid the foundation for the evolution and institutionalization of the Common Rule for the ethical treatment of human subjects, from which Facebook famously held itself exempt.
The corporation was to enjoy the rights of personhood but be free of democratic obligations, legal constraints, moral calculations, and social considerations.
Varian’s uncontract disposes of several millennia of societal evolution during which Western civilization institutionalized the contract as a grand achievement of shared will.
Jim Ford reminded us of the most cherished requirements of a civilized life: our shared assertion of rights to the future tense and its expression in the joining of wills in mutual commitment to dialogue, problem solving, and empathy.
Surveillance capitalists’ interests have shifted from using automated machine processes to know about your behavior to using machine processes to shape your behavior according to their interests. In other words, this decade-and-a-half trajectory has taken us from automating information flows about you to automating you.
If industrial capitalism dangerously disrupted nature, what havoc might surveillance capitalism wreak on human nature?
if industrial civilization flourished at the expense of nature and now threatens to cost us the Earth, an information civilization shaped by surveillance capitalism will thrive at the expense of human nature and threatens to cost us our humanity.
Stalin took the floor for a toast as the room fell silent. “Our tanks are worthless if the souls who must steer them are made of clay. This is why I say: The production of souls is more important than that of tanks.
Although it is not murderous, instrumentarianism is as startling, incomprehensible, and new to the human story as totalitarianism was to its witnesses and victims.
Skinner argues that knowledge does not make us free but rather releases us from the illusion of freedom.
For Meyer and for Skinner, our attachments to notions such as freedom, will, autonomy, purpose, and agency are defense mechanisms that protect us from the uncomfortable facts of human ignorance.
freedom is ignorance. The felt experience of free will is but a bit of undigested denial, produced by a lack of information about the actual determinants of behavior.
The belief in “autonomous man” is a regressive source of resistance to a rational future, an “alternative explanation of behavior” that obstructs the advancement of society.
“The boundary shifts with every discovery of a technique for making private events public. . . . The problem of privacy may, therefore, eventually be solved by technical advances.”67
When finally you surrender to us, it must be of your own free will. We do not destroy the heretic because he resists us: so long as he resists us we never destroy him. We convert him, we capture his inner mind, we reshape him. . . . We bring him over to our side, not in appearance, but genuinely, heart and soul. We make him one of ourselves before we kill him.
Walden Two’s fictional format provided the cover that Skinner needed to extrapolate from Meyer’s methodological principles of otherness and his own research on animal behavior to a utopian community in which behavior has replaced the human spirit as the locus of control.
Big Other’s capabilities, instrumentarian power reduces human experience to measurable observable behavior while remaining steadfastly indifferent to the meaning of that experience. I call this new way of knowing radical indifference. It is a form of observation without witness
Instrumentarianism’s radical indifference is operationalized in Big Other’s dehumanized methods of evaluation that produce equivalence without equality. These methods reduce individuals to the lowest common denominator of sameness—an organism among organisms—despite all the vital ways in which we are not the same. From Big Other’s point of view we are strictly Other-Ones: organisms that behave.
the percentage of Americans who “think that most people can be trusted” remained relatively steady between 1972 and 1985. Despite some fluctuations, 46 percent of Americans registered high levels of interpersonal trust in 1972 and nearly 50 percent in 1985. As the neoliberal disciplines began to bite, that percentage steadily declined to 34 percent in 1995, just as the public internet went live. The late 1990s through 2014 saw another period of steady and decisive decline to only 30 percent.
Societies that display low levels of interpersonal trust also tend to display low levels of trust toward legitimate authority; indeed, levels of trust toward the government have also declined substantially in the US, especially during the decade and a half of growing connectivity and the spread of surveillance capitalism.
In the age of surveillance capitalism it is instrumentarian power that fills the void, substituting machines for social relations, which amounts to the substitution of certainty for society.
In 2015 the Chinese central bank announced a pilot project in which the top e-commerce companies would pioneer the data integration and software development for personal credit scoring.
People on the list can be prevented from buying aeroplane, bullet-train or first- or business-class rail tickets; selling, buying or building a house; or enrolling their children in expensive fee-paying schools. There are restrictions on offenders joining or being promoted in the party and army,
“Progress now requires humanity coming together not just as cities or nations,” Zuckerberg urged, “but also as a global community . . . the most important thing we at Facebook can do is develop the social infrastructure . . . to build a global community. .
The most “profound shift,” Nadella explained, is that “people and their relationship with other people is now a first-class thing in the cloud. It’s not just people but it’s their relationships, it’s their relationships to all of the work artifacts, their schedules, their project plans, their documents; all of that now is manifest in this Microsoft Graph.” These streams of total information are key to optimizing “the future of productivity,” Nadella exulted.28 In
Microsoft scientists have been working for years on how to take the same logic of automated preemptive control at the network’s edge and transpose it to social relations.
2013 Microsoft patent application updated and republished in 2016 and titled “User Behavior Monitoring on a Computerized Device.”
The scientists propose an application that can sit in an operating system, server, browser, phone, or wearable device continuously monitoring a person’s behavioral data: interactions with other people or computers, social media posts, search queries, and online activities. The app may activate sensors to record voice and speech, videos and images, and movement, such as detecting “when the user engages in excessive shouting by examining the user’s phone calls and comparing related features with the predication model.”
When one of the self-driving cars makes an error, all of the self-driving cars learn from it. In fact, new self-driving cars are “born” with the complete skill set of their ancestors and peers. So collectively, these cars can learn faster than people.
Sophisticated AI-powered tools will empower us to better learn from the experiences of others.
In a 2009 collaboration with several graduate students, Pentland presented results on the design and deployment of a “wearable computing platform” based on the sociometric badge and its machine analytics. The goal, the authors said, was to make machines that can “monitor social communication and provide real-time intervention.”
They warned that organizations will become “truly sensible” only when they employ “hundreds or thousands of wireless environmental and wearable sensors capable of monitoring human behavior, extracting meaningful information, and providing managers with group performance metrics and employees with self-performance evaluations and recommendation.”
Pent-land says that “continuous streams of data about human behavior” mean that everything from traffic, to energy use, to disease, to street crime will be accurately forecast, enabling a “world without war or financial crashes, in which infectious disease is quickly detected and stopped, in which energy, water, and other resources are no longer wasted, and in which governments are part of the solution rather than part of the problem.”48 This new “collective intelligence” operates to serve the greater good as we learn to act “in a coordinated manner” based on “social universals.”
“The main barriers are privacy concerns and the fact that we don’t yet have any consensus around the trade-offs between personal and social values.”
The greater good is someone’s, but it may not be ours.
Walden Two Frazier insists that “I don’t like the despotism of ignorance. I don’t like the despotism of neglect, of irresponsibility, the despotism of accident, even. And I don’t like the despotism of democracy!”
The velocity of instrumentarian society leaves us no time to get our bearings, and that speed is repurposed here as a moral imperative demanding that we relinquish individual agency to the automated systems that can keep up the pace in order to quickly perceive and impose correct answers for the greater good.
Human behavior must be herded and penned within the parameters of the plan, just as behavior at Nadella’s construction site was continuously and automatically molded to policy parameters. Pentland calls this “tuning the network.”
“Tuners” fill the role of Pentland’s “we.” He says, for example, that cities can be understood as “idea engines” and that “we can use the equations of social physics to begin to tune them to perform better.”
Pentland subscribes to the label Homo imitans to convey that it is mimicry, not empathy, and certainly not politics, which defines human existence.
A new social class of tuners exercises perpetual vigilance to cure human nature of its weaknesses by ensuring that populations are tuned, herded, and conditioned to produce the most-efficient behaviors.
Individuality is a threat to instrumentarian society, troublesome friction that sucks energy from “collaboration,” “harmony,” and “integration.” In an article titled “The Death of Individuality,” Pentland
It is time that we dropped the fiction of individuals as the unit of rationality and recognised that our rationality is largely determined by the surrounding social fabric. . . .”
The surrender of the individual to manipulation by the planners clears the way for a safe and prosperous future built on the forfeit of freedom for knowledge. Skinner was unrelenting on this point:
The international “unplug” study helps to set the stage, for it reveals a range of emotional anguish summarized in six categories: addiction, failure to unplug, boredom, confusion, distress, and isolation. The students’ sudden disconnection from the network produced the kinds of cravings, depression, and anxiety that are characteristic of clinically diagnosed addictions.
“Increasingly no young person who wants a social life can afford not to be active on the site, and being active on the site means living one’s life on the site.”
42 percent of teenage respondents said that social media affects how people see them, having adopted what the researchers call an outside-looking-in approach to how they express themselves. Their dependency penetrates deeply into their sense of well-being, affecting how they feel about themselves (42 percent) and their happiness (37 percent).6
Social media is designed to engage and hold people of all ages, but it is principally molded to the psychological structure of adolescence and emerging adulthood, when one is naturally oriented toward the “others,” especially toward the rewards of group recognition, acceptance, belonging, and inclusion.
Facebook relies on specific practices that feed the inclinations of people, especially young people, to know themselves from “the outside looking in.” Most critical is that the more the need for the “others” is fed, the less able one is to engage the work of self-construction. So devastating is the failure to attain that positive equilibrium between inner and outer life that Lapsley and Woodbury say it is “at the heart” of most adult personality disorders.32
