More on this book
Community
Kindle Notes & Highlights
Read between
January 5 - January 12, 2024
On one trajectory, some liberal democratic states will continue to be eroded from within, becoming a kind of zombie government. Trappings of liberal democracy and the traditional nation-state remain, but functionally they are hollowed out, the core services increasingly threadbare, the polity unstable and fractious. Lurching on in the absence of anything else, they become ever more degraded and dysfunctional. On another, unthinking adoption of some aspects of the coming wave opens pathways to domineering state control, creating supercharged Leviathans whose power goes beyond even history’s
...more
fundamentally responsible for controlling powerful new technologies. Neither direction can or will contain the coming wave.
Put simply, gain-of-function experiments deliberately engineer pathogens to be more lethal or infectious, or both. In nature, viruses usually trade off lethality for transmissibility. The more transmissible a virus, the less lethal it often is.
But there is no absolute reason this must be so. One way of understanding how it might happen—that is, how viruses might become more lethal and transmissible at the same time—and how we might combat that is to, well, make it happen.
This is not about bad actors weaponizing technology; this is about unintended consequences from good people who want to improve health outcomes. It’s about what goes wrong when powerful tools proliferate, what mistakes get made, what “revenge effects” unfurl, what random, unforeseen mess results from
technology’s collision with reality.
As the power and spread of any technology grows, so its failure modes escalate.
Who would have thought that “influencer” would become a highly sought-after role? Or imagined that in 2023 people would be working as “prompt
engineers”—nontechnical programmers of large language models who become adept at coaxing out specific responses?
The number of people who can get a PhD in machine learning will remain tiny in comparison to the scale of layoffs. And, sure, new demand will create new work, but that doesn’t mean it all gets done by human beings.
The Private Sector Job Quality Index, a measure of how many jobs provide above-average income, has plunged since 1990; it suggests that well-paying jobs as a proportion of
the total have already started to fall.
Countries like India and the Philippines have seen a huge boom from business process outsourcing, creating comparatively high-paying jobs in places like call centers. It’s precisely this kind of work that will be targeted by automation. New jobs might be created in the long te...
This highlight has been truncated due to consecutive passage length restrictions.
Labor market disruptions are, like social media, fragility amplifiers. They damage and undermine the nation-state. The first signs of this are coming into view, but like social media late in the first decade of the twenty-first century, it’s not quite clear what the exact shape and extent of the implications will be. In any case, just because the consequences aren’t yet evident doesn’t mean they can be wished away.
new forms of attack and vulnerability, the industrialization of misinformation, lethal autonomous weapons, accidents like lab leaks, and the consequences of automation—are all familiar to people in tech, policy, and security circles. Yet they are too often viewed in isolation.
The full amplification of fragility is missed because it often appears as if these impacts were happening incrementally and in convenient silos. They are not. They stem from a single coherent and interrelated phenomenon manifesting itself in different ways. The reality is much more enmeshed, entwined, emergent, and chaotic than any sequential presentation can convey. Fragility, amplified. The nation-state, weakened.
We are not quite heading for a neocolonial East India
Company 2.0. But I do think we have to confront the sheer scale and influence that some boardrooms have not just over the subtle nudges and choice architectures that shape culture and politics today but, more importantly, over where this could lead in decades to come. They are empires of a sort, and with the coming wave their scale, influence, and capability are set to radically expand.
To get a sense of these concentrations, consider that the combined revenues of companies in Fortune’s Global 500 are already at 44 percent of world GDP. Their total profits are larger than all but the top six countries’ annual GDPs. Companies already control the largest clusters of AI processors, the best models, the most advanced quantum computers, and the overwhelming majority of robotics capacity and IP. Unlike with rockets, satellites, and the internet, the frontier of this wave is found in corporations, not in
government organizations or academic labs. Accelerate this process with the next generation of technology, and a future of corporate concentration doesn’t seem so extraordinary.
The world’s top fifty cities have the lion’s share of wealth and corporate power (45 percent of big company HQs; 21 percent of world GDP) despite having only 8 percent of the
world’s population. The top 10 percent of global firms take 80 percent of the total profits. Expect the coming wave to feed into this picture, producing ever-richer and more successful superstars—whether regions, business sectors, companies, or research groups.
In a few decades, I predict most physical products will look like services. Zero marginal cost production and distribution will make it possible. The migration to the cloud will become all-encompassing, and the trend will be spurred by the ascendancy of low-code and no-code software, the rise of bio-manufacturing, and the boom in 3-D printing. When you combine all the facets of the coming wave, from the design, management, and logistical capabilities of AI to the modeling of chemical reactions enabled by quantum computing to the fine-grained assembly capabilities of robotics, you get a
...more
In 2019, the U.S. government banned federal agencies and their contractors from buying telecommunications and surveillance equipment from a number of Chinese providers including Huawei, ZTE, and Hikvision. Yet, just a year later, three federal agencies were found to have bought such equipment from prohibited vendors. More than one hundred U.S. towns have even acquired technology developed for use on the Uighurs in Xinjiang. A textbook failure of containment.
In short, key parts of modern society and social organization that today rely on scale and centralization could be radically devolved by capabilities unlocked with the coming wave. Mass rebellion, secessionism, and state formation of any kind look very different in this world. Redistributing real power means communities of all kinds can live as they wish, whether they are ISIS, FARC, Anonymous, secessionists from Biafra to Catalonia, or a major corporation building luxury theme parks on a remote island in the Pacific.
Being blinkered about what’s happening is, in
my view, more dangerous than being overly speculative.
Governance works by consent; it is a collective fiction resting on the belief...
This highlight has been truncated due to consecutive passage length restrictions.
The techno-libertarian movement takes Ronald
Reagan’s 1981 dictum “Government is the problem” to its logical extreme, seeing government’s many flaws but not its immense benefits, believing that its regulatory and tax functions are destructive rate limiters with few upsides—for them at least. I find it deeply depressing that some of the most powerful and privileged take such a narrow and destructive view, but it adds a further impetus to fragmentation.
The internet does precisely this: centralizes in a few key hubs while also empowering billions of people. It creates behemoths and yet
gives everyone the opportunity to join in.
Social media created a few giants and a million tribes. Everyone can build a website, but there’s only one Google. Everyone can sell their own niche products, but there’s only one Amazon. And on and on. The disruption of the internet era is largely explained by t...
This highlight has been truncated due to consecutive passage length restrictions.
Like breathless techno-optimism, breathless techno-catastrophism is easy to dismiss as a twisted, misguided form of hype unsupported by the historical record.
just because a warning has dramatic implications isn’t good grounds to automatically reject it. The pessimism-averse
averse complacency greeting the prospect of disaster is itself a ...
This highlight has been truncated due to consecutive passage length restrictions.
AI is both valuable and dangerous precisely because it’s an extension of our best and worst selves. And as a technology premised on learning, it can keep adapting, probing, producing novel strategies and ideas potentially far removed from anything before considered, even by other AIs.
Truth is, China is completely unsustainable without new technology.
Demographic trends take decades to shift. Generational cohorts do not change size. This slow, inexorable decline is already locked in, a looming iceberg we can do nothing to avoid—except find ways of replacing those workers.
Given the population and resource constraints, just standing still would probably require a global two- to threefold productivity improvement, and standing still is not acceptable for the world’s vast majority, among whom, for example, child mortality
is twelve times higher than in developed countries.
am, however, confident that the coming decades will see complex, painful
trade-offs between prosperity, surveillance, and the threat of catastrophe growing ever more acute. Even a system of states in the best possible health would struggle.
And yet 1.35 million people a year still die in traffic accidents. Regulation may lessen the negative effects, but it can’t erase bad outcomes like crashes, pollution, or sprawl. We have decided that this is an acceptable human cost, given the benefits.
In effect, Chinese AI policy has two tracks: a regulated civilian path and a freewheeling military-industrial one.
One challenge in even beginning to have this conversation is that technology, in the popular imagination, has become associated with a narrow band of often superfluous applications. “Technology” now mostly means social media platforms and wearable gadgets to measure our steps and heart rate. It’s easy to forget that technology includes the irrigation systems essential to feeding the planet and newborn life-support machines. Technology isn’t just a way to store your selfies; it represents access to the world’s accumulated culture and wisdom. Technology is not a niche; it is a hyper-object
...more
climate change. It too deals with risks that are often diffuse, uncertain, temporally distant, happening elsewhere, lacking the salience, adrenaline, and immediacy of an ambush on the savanna—the kind of risk we are well primed to respond to. Psychologically, none of this feels present. Our prehistoric brains are generally hopeless at dealing with amorphous threats like these. However, over the last decade or so, the challenge of climate change has come into better
focus. Although the world still spews out increasing amounts of CO2, scientists everywhere can measure CO2 parts per m...
This highlight has been truncated due to consecutive passage length restrictions.
Nudge forward the status quo so containment has a chance. We should do all this with the knowledge that it might fail but that it is our best shot at building a world where containment—and human flourishing—are possible. There are no guarantees here, no rabbits pulled out of hats. Anyone hoping for a quick fix, a smart answer, is going to be disappointed. Approaching the dilemma, we are left in the same all-too-human position as always: giving it everything and hoping it works out.
Technical safety, up close, in the code, in the lab, is the first item on any containment agenda.
