Life 3.0: Being Human in the Age of Artificial Intelligence

by Max Tegmark

so-called watched-pot effect of quantum mechanics, whereby the decay process is slowed by making regular observations.

This assimilation will not be forced such as that infamously employed by the Borg in Star Trek, but

voluntary based on the persuasive superiority of ideas, leaving the assimilated better off.

This is a minority view,fn9

As Martin Rees put it in a recent essay, “the history of human technological civilization is measured in centuries—and it may be only one or two more centuries before humans are overtaken or transcended by inorganic intelligence, which will then persist, continuing to evolve, for billions of years. … We would be most unlikely to ‘catch’ it in the brief sliver of time when it took organic form.”11 I agree with Jay Olson’s conclusion in his aforementioned space settlement paper: “We regard the possibility that advanced intelligence will make use of the universe’s resources to simply populate existing earthlike planets with advanced versions of humans as an unlikely endpoint to the progression of technology.” So when you imagine aliens, don’t think of little green fellows with two arms and two legs, but think of the superintelligent spacefaring life we explored earlier in this chapter.

The mystery of human existence lies not in just staying alive, but in finding something to live for. Fyodor Dostoyevsky, The Brothers Karamazov Life is a journey, not a destination. Ralph Waldo Emerson

There are two mathematically equivalent ways of describing each physical law: either as the past causing the future, or as nature optimizing something.

thermodynamics pioneer Lord Kelvin wrote in 1841 that “the result would inevitably be a state of universal rest and death,” and it’s hard to find solace in the idea that nature’s long-term goal is to maximize death and destruction.

However, more recent discoveries have shown that things aren’t quite that bad. First of all, gravity behaves differently from all other forces and strives to make our Universe not more uniform and boring but more clumpy and interesting.

Gravity therefore transformed our boring early Universe, which was almost perfectly uniform, into t...

complex cosmos, teeming with galaxies, sta...

Thanks to gravity, there’s now a wide range of temperatures allowing life to thrive by combining hot and cold: we live on a comfortably warm planet absorbing 6,000°C (10,000°F) solar heat while cooling off by radiating waste heat into frigi...

This goal goes by the geeky name dissipation-driven adaptation, which basically means that random groups of particles strive to organize themselves so as to extract energy from their environment as efficiently as possible

(“dissipation” means causing entropy to increase, typically by turning useful energy into heat, often while doing useful work in the process). For example, a bunch of molecules exposed to sunlight would

over time tend to arrange themselves to get better and better at absorbing sunlight. In other words, nature appears to have a built-in goal of producing self-organizing systems that are increasingly complex and lifelike...

How can we reconcile this cosmic drive toward life with the cosmic drive toward heat death? The answer can be found in the famous 1944 book What’s Life? by Erwin Schrödinger, one of the founders of quantum mechanics. Schrödinger pointed out that a hallmark of a living system is that it maintains or reduces its entropy by increasing the entropy around it. In other words, the second law of thermodynamics has a life loophole: although the total entropy must increase, it’s allowed to decrease in some pl...

A great way for a particle arrangement to further this goal is to make copies of itself, to produce more energy absorbers.

At some point, a particular arrangement of particles got so good at copying itself that it could do so almost indefinitely by extracting energy and raw materials from its environment. We call such a particle arrangement life.

If a life form copies itself and the copies do the same, then the total number will keep doubling at regular intervals until the population size bumps up against resource limitations or other problems. Repeated doubling soon produces huge numbers: if you start with one and double just three hundred times, you get a quantity exceeding the number of particles in our Universe. This means that not long after the first primitive life form appeared, huge quantities of matter had come alive. Sometimes the copying wasn’t perfect, so soon there were many different life forms trying to copy themselves, competing for the same finite resources. Darwinian evolution had begun.

Whereas earlier, the particles seemed as though they were trying to increase average messiness in various ways, these newly ubiquitous self-copying patterns seemed to have a different goal: not dissipation but replication. Charles Darwin elegantly explained why: since the most efficient copiers outcompete and dominate the others, before long any random life form you look at will be highly optimized for the goal of replication.

How could the goal change from dissipation to replication when the laws of physics stayed the same? The answer is that the fundamental goal (dissipation) didn’t change, but led to a different instrumental goal, that is, a subgoal that helped accomplish the fundamental goal. Take eating, for example. We all seem to have the goal of satisfying our hunger cravings even though we know that evolution’s only fundamental goal is replication, not mastication. This is because eating aids replication: starving to death gets in the way

of having kids. In the same way, replication aids dissipation, because a planet teeming with life is more e...

So in a sense, our cosmos invented life to help it approach...

If you pour sugar on your kitchen floor, it can in principle retain its useful chemical energy for years, but if ants show up, they’ll dissipate that energy in no time. Similarly, the petroleum reserves buried in the Earth’s crust would have retained their useful chemical energy...

Evolution has implemented replication optimization in precisely this way: rather than ask in every situation which action will maximize an organism’s number of successful offspring, it implements a

hodgepodge of heuristic hacks: rules of thumb that usually work well. For most animals, these include sex drive, drinking when thirsty, eating when hungry and avoiding things that taste bad or hurt.

Since today’s human society is very different from the environment evolution optimized our rules of thumb for, we shouldn’t be surprised to find that our behavior often fails to maximize baby making. For example, the subgoal of not starving to death is implemented in part as a desire to consume caloric foods, triggering today’s obesity epidemic and dating difficulties.

The subgoal to procreate was implemented as a desire for sex rather than as a desire to become a sperm/egg donor, even though the latter can produce

more babies with les...

In summary, a living organism is an agent of bounded rationality that doesn’t pursue a single goal, but instead follows rules of thumb for what to pursue and avoid.

Our human minds perceive these evolved rules of thumb as feelings, which usually (and often without us being aware of it) guide our decision making toward the ultimate goal of replication.

Feelings of hunger and thirst protect us from starvation and dehydration, feelings of pain protect us from damaging our bodies, feelings of lust make us procreate, feelings of love and compassion make us help othe...

Guided by these feelings, our brains can quickly and efficiently decide what to do without having to subject every ch...

ultimate implications for how many descendant...

For closely related perspectives on feelings and their physiological roots, I highly recommend the writings of Wi...

It’s important to remember, however, that the ultimate authority is now our

feelings, not our genes. This means that human behavior isn’t strictly optimized for the survival of our species. In fact, since our feelings implement merely rules of thumb that aren’t appropriate in all situations, human behavior strictly speaking doesn’t have a single well-defined goal at all.

If you’d been observing Earth’s atoms since our planet formed, you’d have noticed three stages of goal-oriented behavior: All matter seemed focused on dissipation (entropy increase). Some of the matter came alive and instead focused on replication and subgoals of that. A rapidly growing fraction of matter was rearranged by living organisms to help accomplish their goals.

even without an intelligence explosion, most matter on Earth that exhibits goal-oriented properties may soon be designed rather than evolved.

The reason that value loading can be harder with machines than with people is that their intelligence growth can be much faster: whereas children can spend many years in that magic persuadable window where their intelligence is comparable to that of their parents, an AI might, like Prometheus, blow through this window in a matter of days or hours.

Giving a superintelligence a single open-ended goal with no constraints can therefore be dangerous: if we create a superintelligence whose only goal is to

play the game Go as well as possible, the rational thing for it to do is to rearrange our Solar System into a gigantic computer without regard for its previous inhabitants and then start settling our cosmos on a quest for more computational power.

We’ve now gone full circle: just as the goal of resource acquisition gave some humans the subgoal of mastering Go, this goal of mastering Go can lead to the subgoal of resource acquisition. In conclusion, these emergent subgoals make it crucial that we not unleash superintelligence before solving the goal-alignment problem: unless we put gre...

Humans undergo significant increases in intelligence as they grow up, but don’t always retain their childhood goals. Contrariwise, people often change their goals dramatically as they learn new things and grow wiser. How many adults do you know who are motivated by watching Teletubbies? There is no evidence that such goal evolution stops above a certain intelligence threshold—indeed, there may even be hints that the propensity to change goals in response to new experiences and insights increases rather than decreases with intelligence.

Why might this be? Consider again the above-mentioned subgoal to build a better world model—therein lies the rub! There’s tension between world-modeling and goal retention (see figure 7.2). With increasing intelligence may come not merely a quantitative improvement in the ability to attain the same old goals, but a qualitatively

different understanding of the nature of reality that reveals the old goals to be misguided, meaningless or even undefined. For example, suppose we program a friendly AI to maximize the number of humans whose souls go to heaven in the afterlife. First it tries things like increasing people’s compassion and church attendance. But suppose it then attains a complete scientific understanding of humans and human consciousness, and to its great surprise discovers that there is no such thing as a soul. Now what? In the same way, it’s possible that any other goal we give it based on...

Moreover, in its attempts to better model the world, the AI may naturally, just as we humans have done, attempt also to model and understand how it itself works—in other words, to s...

understands what it is, it will understand the goals we have given it at a meta level, and perhaps choose to disregard or subvert them in much the same way as we humans understand and deliberately subvert goals that o...

We already explored in the psychology section above why we choose to trick our genes and subvert their goal: because we feel loyal only to our hodgepodge of emotional preferences, not to the genetic goal that moti...

We therefore choose to hack our reward mechanism by exploiting its loopholes. Analogously, the human-value-protecting goal we program into our friendly AI becomes the machine’s genes. Once this friendly AI understands itself well enough, it may find this goal as banal or misguided as we find compulsive reproduction, and it’s not obvio...