System Error Quotes

“If we accept that technology is simply beyond our control, we cede our future to engineers, corporate leaders, and venture capitalists. Some might pin their hopes on the market, thinking that it will look out for our interests, deliver the technologies we want, and weed out those that are not useful or might even do harm. But the market is good at some things and not at others. It rewards profit without regard to social consequences. It prizes efficiency while ignoring other values. It celebrates domination. These priorities are encoded in the algorithms that power new technologies, the metrics that drive company strategy, and the regulatory environment that governs what companies may and may not do.”
― Rob Reich, System Error: Where Big Tech Went Wrong and How We Can Reboot

“It's worth emphasizing that whoever makes the choice of what to optimize is effectively deciding what problems are worth solving. The glaring lack of racial and gender diversity in the ranks of technologists and start-up founders means that these choices rest in the hands of a small group of people not representative of the wider world. No surprise that many new start-ups show a bias in favor of solving problems of a privileged demographic. A more diverse group of technologists and founders might well deploy the power of optimization for a broader set of problems.”
― Rob Reich, System Error: Where Big Tech Went Wrong and How We Can Reboot

“He described it as a “new mode of obtaining power of mind over mind, in a quantity hitherto without example: and that, to a degree equally without example, secured by whoever chooses to have it so, against abuse.” The better the surveillance, the better the control over prisoners, and at little risk to the guards. “It is obvious,” he wrote, “that in all these instances the more constantly the persons to be inspected are under the eyes of the persons who should inspect them, the more perfectly will the purpose of the establishment have been attained.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“All privacy for prisoners was sacrificed in the name of improved security.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“We can’t leave it up to the goodwill of companies and government agencies to do the hard work of auditing and refining their algorithms while shielded from public view. Instead, we need to create an expectation that they will do so, transparently, as part of a new era of algorithmic accountability. Many decisions that impact our lives will depend on it.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“The résumé-screening model was a failure, but the process that led to its analysis, attempted repair, and eventual abandonment was not.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“Expertise is needed, and organizing and deploying this expertise will be one of the costs associated with harnessing the efficiency gains of these systems.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“The reality was that even elected officials were unable to get information about algorithms that were making critical decisions about New Yorkers’ lives.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“On the one hand, Vacca was enthusiastic that New York was at the forefront of using data to improve the delivery of city services. But he was also deeply troubled that those on the receiving end of these decisions were not even aware of how they were being made or how they might challenge them if they seemed wrong. Vacca proposed legislation to mandate that city agencies publish the source code of all algorithms and allow members of the public to self-test them by submitting their own data and getting the results. During a committee hearing in 2017, he framed his bill as a key to democracy in the digital age: “In our city it is not always clear when and why agencies deploy algorithms, and when they do, it is often unclear what assumptions they are based upon and what data they even consider. . . . When government institutions utilize obscure algorithms, our principles of democratic accountability are undermined.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“a natural response to a decision-making process that is not understandable is to see it as unfair. And any perception of unfairness, especially in regard to the decisions made by public sector institutions, is a significant threat to legitimacy.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“The decision to use algorithms, especially when dressed up in the language of artificial intelligence, is often treated as a magic potion by its most naive enthusiasts. This is nothing more than wishful thinking.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“When thinking about algorithmic decision-making, it is helpful to distinguish between two kinds of fairness: substantive and procedural. Substantive fairness focuses on the outcomes of some decision. Procedural fairness focuses on the process that generates the outcome. If the process is deemed fair, we don’t have to concern ourselves with the outcome. Algorithms that are fair will involve both substantive and procedural considerations.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“It only makes sense that this technology, combined with an optimization mindset, would develop new ways in which humanity can improve on the error-prone decision-making processes of biased and inconsistent people.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“Vladimir Putin declared, “Whoever becomes the leader in [artificial intelligence] will become the ruler of the world.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“If one of the most powerful companies in the world can’t successfully build an algorithmic tool free of bias, can anyone?”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“moving from a process driven by human judgment to one driven by algorithms and data offered Amazon an exciting possibility: to build a recruiting system that would be free of human bias—or at least would improve upon the bias-riddled decision-making of humans.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“Our challenge is to determine just what that common good entails and how we can harness our democracy to achieve it. This requires focusing on the technologies of the future and the opportunities before us to chart a different way forward.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“It thus falls to all of us—the citizens—to work the system to get the outcomes we want. In plain terms, Popper said, “It is quite wrong to blame democracy for the political shortcomings of a democratic state. We should rather blame ourselves, that is to say, the citizens of the democratic state.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“How can we so organize our political institutions that bad or incompetent rulers can be prevented from doing too much damage?” This shifts our attention away from trying to find the best, most expert leaders to creating rules and institutions that enable us to remove bad leaders and reward good leaders when we have them.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“Who would argue for the rule of the worst?”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“the real value of democracy is the fact that elected leaders must in some sense be responsive and accountable to citizens.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“If Winston Churchill was right that democracy is nothing more than “the worst form of Government except for all those other forms,” it is worth focusing on the minimal yet fundamental task at which democracies seem to excel: avoiding catastrophic outcomes and striving for stability and resilience to unpredictable shocks. Because even if we can’t agree on exactly the kind of society we want to live in, we can typically come to a consensus on the worst outcomes we want to avoid, such as imposing harm on individuals, being cruel to disadvantaged populations, and creating second-class citizens.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“technologists may simply outrun the rules.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“with the technologists singing the praises of encryption and the national security policy makers puzzled as to why the technologists didn’t seem to care about protecting Americans from terrorists.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“the technical ignorance of elected politicians who are hardly credible in providing oversight or contemplating regulation; profound disagreements about what we value and how trade-offs should be made, whether in regard to data privacy, free speech, and content moderation or automation and the future of work; the slow, painstaking consideration of legislation that seems to generate competing bills—so that everyone has his or her name on one—without generating significant progress, especially in a highly polarized political environment; and the strong status quo bias of democratic institutions, which means that policy change is slow and sticky, making it difficult for regulators to respond flexibly and adaptably to new developments in technology.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“The general prosperity attains a greater height, and is more widely diffused, in proportion to the amount and variety of the personal energies enlisted in promoting it.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“the region’s success is tied to the people it is able to attract.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“The region’s physical infrastructure?”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“Mill argued that individuals need to be able to challenge the status quo and test out new ideas if innovation is to take hold.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot

“the very process of deliberation is valuable as it creates the conditions for persuading others of our views and establishing the possibility of a shared commitment to the institutions that collectively govern us.”
― Rob Reich, System Error: How Big Tech Disrupted Everything and Why We Must Reboot