More on this book
Community
Kindle Notes & Highlights
The discovery of fire, the invention of the wheel, the harnessing of electricity—all of these were moments that transformed human civilization, altering the course of history forever.
With AI, we could unlock the secrets of the universe, cure diseases that have long eluded us, and create new forms of art and culture that stretch the bounds of imagination. With biotechnology, we could engineer life to tackle diseases and transform agriculture, creating a world that is healthier and more sustainable.
With AI, we could create systems that are beyond our control and find ourselves at the mercy of algorithms that we don’t understand. With biotechnology, we could manipulate the very building blocks of life, potentially creating unintended consequences for both individuals and entire ecosystems.
we stand at this turning point, we are faced with a choice—a choice between a future of unparalleled possibility
and a future of unimaginable peril. The fate of humanity ha...
This highlight has been truncated due to consecutive passage length restrictions.
the age of advanced technology is upon us, and we must be ready to face its challenges head-on.
foundational technology ever invented, from pickaxes to plows, pottery to photography, phones to planes, and everything in between, follows a single, seemingly immutable law: it gets cheaper and easier to use, and ultimately it proliferates, far and wide.
Homo technologicus—of the technological animal. Humanity’s quest to improve—ourselves, our lot, our abilities, and our influence over our environment—has powered a relentless evolution of ideas and creation.
The coming wave is defined by two core technologies: artificial intelligence
(AI) and synthetic biology.
What if we could distill the essence of what makes us humans so productive and
capable into software, into an algorithm? Finding the answer might unlock unimaginably powerful tools to help tackle our most intractable problems.
AI has been climbing the ladder of cognitive abilities for decades, and it now looks set to reach human-level performance across a very wide range of tasks within the next three years.
have long worried about not just the consequences of advancing AI but where the entire technological ecosystem was heading. Beyond AI, a wider revolution was underway, with AI feeding a powerful, emerging generation of genetic technologies and robotics.
What if the wave is actually a tsunami?
Both pursuing and not pursuing new technologies is, from here,
fraught with risk. The chances of muddling through a “narrow path” and avoiding one or the other outcome—techno-authoritarian dystopia on the one hand, openness-induced catastrophe on the other—grow smaller over time as the technology becomes cheaper, more powerful, and more pervasive and the risks accumulate. And yet stepping away is no option either.
coming wave more than ever before. This is the core dilemma: that, sooner or later, a powerful generation of technology leads humanity toward either catastrophic or dystopian outcomes. I believe this ...
This highlight has been truncated due to consecutive passage length restrictions.
Somehow we need to get the best out of technology, something essential to facing a daunting set of global challenges, and also get out of the dilemma.
see this as an interlocking set of technical, social, and legal mechanisms constraining and controlling technology working at every possible level: a means, in theory, of evading the dilemma. Yet even technology’s harshest critics tend to dodge this language of hard containment.
The presenter showed how the price of DNA synthesizers, which can print bespoke strands of DNA, was falling rapidly. Costing a few tens of thousands of dollars, they are small enough to sit on a bench in your garage and let people synthesize—that is, manufacture—DNA.
Someone could soon create novel pathogens far more transmissible and lethal than anything found in nature. These synthetic pathogens could evade
They finished with an alarming thought: a single person today likely “has the capacity to kill a billion people.” All it takes is motivation.
No one wanted to confront the implications of the hard facts and cold probabilities they’d heard.
Why wasn’t I, why weren’t we all, taking it more seriously? Why do we awkwardly sidestep further discussion?
pessimism-aversion trap: the misguided analysis that arises when you are overwhelmed by a fear of confronting potentially dark realities, and the resulting tendency to look the other way.
overcoming pessimism aversion. It means facing head-on the reality of what’s coming.
We must begin to suggest what to do if it looks like there is a real risk that technology fails us. What’s required is a societal and political response, not merely individual efforts, but it needs to begin with my peers and me.
Pessimism aversion is an emotional response, an ingrained gut refusal to accept the possibility of seriously destabilizing outcomes. It tends to come from those in secure and powerful positions with entrenched worldviews, people who can superficially cope with change
but struggle to accept any real challenge to their world order. Many of those whom I accuse of being stuck in the pessimism-aversion trap fully embrace the growing critiques of technology.
Spend time in tech or policy circles, and it quickly becomes obvious that head-in-the-sand is the default ideology. To believe and act otherwise risks becoming so crippled by fear of and outrage against enormous, inexorable forces that everything
feels futile.
A “techlash” of sorts emerged, with critics railing against tech and tech companies in op-eds and books, in the regulatory
capitals of Washington, Brussels, and Beijing. Previously niche fears around technology exploded into the mainstream, public skepticism of technology increased, and criticisms from academia, civil society, and politics sharpened.
Waves are everywhere in human life. This one is just the latest. Often people seem to think it’s still far off, so futuristic and absurd-sounding that it’s just the province of a few nerds and fringe thinkers, more hyperbole, more technobabble, more boosterism. That’s a mistake.
But without containment, every other aspect of technology, every discussion of its ethical shortcomings, or the benefits it could bring, is inconsequential. We urgently need watertight answers for how the coming
wave can be controlled and contained, how the safeguards and affordances of the democratic nation-state can be maintained, but right now no one has such a plan. This is a future that none of us want, but it’s one I fear is increasingly likely, and I will explain why in the chapters that follow.
“the containment problem.” How do we keep a grip on the most valuable technologies ever invented as they get cheaper and spread faster than any in history?
this isn’t business as usual: they are inherently general and therefore omni-use, they hyper-evolve, they have asymmetric impacts, and, in some respects, they are increasingly autonomous. Their creation is driven by powerful incentives: geopolitical competition, massive financial rewards, and an open, distributed culture of research. Scores of state and non-state actors will race ahead to develop them regardless of efforts to regulate and control what’s coming, taking
most of history, for most people, personal transportation meant one thing: walking.
Or if you were lucky, two: being carried or pulled by horses, oxen, elephants, or other beasts of burden. Just moving between neighboring settlements—forget about continents—was hard and slow.
early nineteenth century, the railway revolutionized transport, its biggest innovation in thousands of years, but most journeys could never be taken by rail, and...
This highlight has been truncated due to consecutive passage length restrictions.
Railways did make one thing clear: engines w...
This highlight has been truncated due to consecutive passage length restrictions.
German engineer called Nicolaus August Otto spent years working on a gas engine, much smaller than a steam engine. By
1876, in a Deutz AG factory in Cologne, Otto produced the first functional internal combustion engine, the “four-stroke” model.
It was ready for mass production, but not before Otto fell out with his business partners, Gottlieb...
This highlight has been truncated due to consecutive passage length restrictions.
By 1893, Benz had sold a measly 69 vehicles; by 1900, just 1,709.
The turning point was Henry Ford’s 1908 Model T. His simple but effective vehicle was built using a revolutionary approach:
assembly line. An efficient, linear, and repetitive process enabled him to slash the price of personal vehicles, and the buyers followed. Most cars at the time cost around $2,000. Ford priced his at $850.
Once there was momentum, the spread of the internal combustion engine became unstoppable.
