More on this book
Community
Kindle Notes & Highlights
the age of advanced technology is upon us, and we must be ready to face its challenges head-on.
Almost every foundational technology ever invented, from pickaxes to plows, pottery to photography, phones to planes, and everything in between, follows a single, seemingly immutable law: it gets cheaper and easier to use, and ultimately it proliferates, far and wide.
Invention is an unfolding, sprawling, emergent process driven by self-organizing and highly competitive inventors, academics, entrepreneurs, and leaders, each surging forward with their own motivations.
The coming wave is defined by two core technologies: artificial intelligence (AI) and synthetic biology.
What if we could distill the essence of what makes us humans so productive and capable into software, into an algorithm?
AI has been climbing the ladder of cognitive abilities for decades, and it now looks set to reach human-level performance across a very wide range of tasks within the next three years.
Moreover, attempting to ban development of new technologies is itself a risk: technologically stagnant societies are historically unstable and prone to collapse. Eventually, they lose the capacity to solve problems, to progress.
This is the core dilemma: that, sooner or later, a powerful generation of technology leads humanity toward either catastrophic or dystopian outcomes. I believe this is the great meta-problem of the twenty-first century.
They finished with an alarming thought: a single person today likely “has the capacity to kill a billion people.” All it takes is motivation.
pessimism-aversion trap: the misguided analysis that arises when you are overwhelmed by a fear of confronting potentially dark realities, and the resulting tendency to look the other way.
Properly addressing this wave, containing technology, and ensuring that it always serves humanity means overcoming pessimism aversion. It means facing head-on the reality of what’s coming.
I also believe that those of us driving technology’s creation must have the courage to predict—and take responsibility for—where it might take us in decades to come. We must begin to suggest what to do if it looks like there is a real risk that technology fails us.
Pessimism aversion is an emotional response, an ingrained gut refusal to accept the possibility of seriously destabilizing outcomes.
Spend time in tech or policy circles, and it quickly becomes obvious that head-in-the-sand is the default ideology. To believe and act otherwise risks becoming so crippled by fear of and outrage against enormous, inexorable forces that everything feels futile.
The various technologies I’m speaking of share four key features that explain why this isn’t business as usual: they are inherently general and therefore omni-use, they hyper-evolve, they have asymmetric impacts, and, in some respects, they are increasingly autonomous.
Over centuries, technology has dramatically increased the well-being of billions of people. We are immeasurably healthier thanks to modern medicine, the majority of the world lives in food abundance, people have never been more educated, more peaceful, or more materially comfortable. These are defining achievements produced in part by that great motor of humanity: science and the creation of technology.
Containment is not, on the face of it, possible. And yet for all our sakes, containment must be possible.
Technology has a clear, inevitable trajectory: mass diffusion in great roiling waves. This is true from the earliest flint and bone tools to the latest AI models. As science produces new discoveries, people apply these insights to make cheaper food, better goods, and more efficient transport. Over time demand for the best new products and services grows, driving competition to produce cheaper versions bursting with yet more features. This in turn drives yet more demand for the technologies that create them, and they also become easier and cheaper to use. Costs continue to fall. Capabilities
...more
More people whose lives do not revolve around subsistence means more possible inventors, and more possible reasons for having inventions, and those inventions mean more people in turn.
Waves—pulsating, emergent, successive, compounding, and cross-pollinating—define an era’s horizon of technological possibility. They are part of us. There is no such thing as a non-technological human being.
Proliferation is catalyzed by two forces: demand and the resulting cost decreases, each of which drives technology to become even better and cheaper.
Uber was impossible without the smartphone, which itself was enabled by GPS, which was enabled by satellites, which were enabled by rockets, which were enabled by combustion techniques, which were enabled by language and fire.
Technologists, innovators, and entrepreneurs get better by doing and, crucially, by copying.
Since the early 1970s the number of transistors per chip has increased ten-million-fold. Their power has increased by ten orders of magnitude—a seventeen-billion-fold improvement.
Back in 1983, only 562 computers total were connected to the primordial internet.
Technology’s unavoidable challenge is that its makers quickly lose control over the path their inventions take once introduced to the world.
What people actually do with your invention, however well intentioned, can never be guaranteed.
Gutenberg just wanted to make money printing Bibles. Yet his press catalyzed the Scientific Revolution and the Reformation, and so became the greatest threat to the Catholic Church since its establishment.
Understanding technology is, in part, about trying to understand its unintended consequences, to predict not just positive spillovers but “revenge effects.”
any technology is capable of going wrong, often in ways that directly contradict its original purpose.
As the power of our tools grows exponentially and as access to them rapidly increases, so do the potential harms, an unfolding labyrinth of consequences that no one can fully predict or forestall.
Containing technology needs to be a much more fundamental program, a balance of power not between competing actors but between humans and our tools. It’s a necessary prerequisite for the survival of our species over the next century.
In AI, for example, it means air gaps, sandboxes, simulations, off switches, hard built-in safety and security measures—protocols for verifying the safety or integrity or uncompromised nature of a system and taking it offline if needed.
Fear and suspicion of anything new and different are endemic.
Where there is demand, technology always breaks out, finds traction, builds users.
Inventions cannot be uninvented or blocked indefinitely, knowledge unlearned or stopped from spreading.
Technology is an eternally dangling carrot, constantly promising more, better, easier, cheaper.
Nuclear weapons are among the most contained technologies in history, and yet the containment problem—in its hardest, most literal sense—even here remains acutely unsolved.
As long as a technology is useful, desirable, affordable, accessible, and unsurpassed, it survives and spreads and those features compound.
In the space of around a hundred years, successive waves took humanity from an era of candles and horse carts to one of power stations and space stations.
Do we want to edit our genomes so that some of us can have children with immunity to certain diseases, or with more intelligence, or with the potential to live longer?
with just a day’s training, AlphaZero was capable of learning more about the game than the entirety of human experience could teach it.
Until recently, the history of technology could be encapsulated in a single phrase: humanity’s quest to manipulate atoms.
The coming wave of technology is built primarily on two general-purpose technologies capable of operating at the grandest and most granular levels alike: artificial intelligence and synthetic biology.
Our ability to manipulate atoms with precision enabled the invention of silicon wafers, which enabled the computation of trillions of operations per second, which in turn enabled us to decipher the code of life.
AI, synthetic biology, robotics, and quantum computing can sound like a parade of overhyped buzzwords.
Keep doing this, modifying the weights again and again, and you gradually improve the performance of the neural network so that eventually it’s able to go all the way from taking in single pixels to learning the existence of lines, edges, shapes, and then ultimately entire objects in scenes. This, in a nutshell, is deep learning.
Following the AlexNet breakthrough, AI suddenly became a major priority in academia, government, and corporate life.
a key ingredient of the LLM revolution is that for the first time very large models could be trained directly on raw, messy, real-world data, without the need for carefully curated and human-labeled data sets.
The human brain is said to contain around 100 billion neurons with 100 trillion connections between them—it is often said to be the most complex known object in the universe.
