More on this book
Community
Kindle Notes & Highlights
let’s instead define life very broadly, simply as a process that can retain its complexity and replicate.
What’s replicated isn’t matter (made of atoms) but information (made of bits) specifying how the atoms are arranged. When a bacterium makes a copy of its DNA, no new atoms are created, but a new set of atoms are arranged in the same pattern as the original, thereby copying the information.
In short, computation is a pattern in the spacetime arrangement of particles, and it’s not the particles but the pattern that really matters! Matter doesn’t matter.
verification: ensuring that software fully satisfies all the expected requirements. The more lives and resources are at stake, the higher confidence we want that the software will work as intended.
“People ask what is the relationship between humans and machines, and my answer is that it’s very obvious: Machines are our slaves.”3
Will life in our Universe fulfill its potential or squander it? This depends to a great extent on what we humans alive today do during our lifetime,
There are two mathematically equivalent ways of describing each physical law: either as the past causing the future, or as nature optimizing something.
One famous quantity that nature strives to maximize is entropy,
a bunch of molecules exposed to sunlight would over time tend to arrange themselves to get better and better at absorbing sunlight. In other words, nature appears to have a built-in goal of producing self-organizing systems that are increasingly complex and lifelike, and this goal is hardwired into the very laws of physics.
that a hallmark of a living system is that it maintains or reduces its entropy by increasing the entropy around it.
At some point, a particular arrangement of particles got so good at copying itself that it could do so almost indefinitely by extracting energy and raw materials from its environment. We call such a particle arrangement life.
if you start with one and double just three hundred times, you get a quantity exceeding the number of particles in our Universe.
before long any random life form you look at will be highly optimized for the goal of replication.
All machines are agents with bounded rationality, and even today’s most sophisticated machines have a poorer understanding of the world than we do, so the rules they use to figure out what to do are often too simplistic.
the real risk with AGI isn’t malice but competence. A superintelligent AI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble.
We face, in Nick Bostrom’s words, philosophy with a deadline.
the conscious system needs to be integrated into a unified whole, because if it instead consisted of two independent parts, then they’d feel like two separate conscious entities rather than one. In other words, if a conscious part of a brain or computer can’t communicate with the rest, then the rest can’t be part of its subjective experience.
If the information processing itself obeys certain principles, it can give rise to the higher-level emergent phenomenon that we call consciousness. This places your conscious experience not one but two levels up from the matter. No wonder your mind feels non-physical!
I think that consciousness is the way information feels when being processed in certain ways.
when people ask about the meaning of life as if it were the job of our cosmos to give meaning to our existence, they’re getting it backward: It’s not our Universe giving meaning to conscious beings, but conscious beings giving meaning to our Universe.
the very first goal on our wish list for the future should be retaining (and hopefully expanding) biological and/or artificial consciousness in our cosmos, rather than driving it extinct.
although we’ve focused on the future of intelligence in this book, the future of consciousness is even more important, since that’s what enables meaning.
Philosophers like to go Latin on this distinction, by contrasting sapience (the ability to think intelligently) with sentience (the ability to subjectively experience qualia). We humans have built our identity on being Homo sapiens, the smartest entities around. As we prepare to be humbled by ever smarter machines, I suggest that we rebrand ourselves as Homo sentiens!
What do you want it to mean to be human in the age of AI? Please discuss all this with those around you—it’s not only an important conversation, but a fascinating one.
