Life 3.0: Being Human in the Age of Artificial Intelligence

by Max Tegmark

inverse reinforcement learning,

imbuing

value-loading problem,

corrigibility.

Vernor Vinge

“singularity”—the

Nick Bostrom’s book Superintelligence.

there’s an inherent tension between goal retention and improving its world model, which casts doubts on whether it will actually retain its original goal as it gets smarter.

If you give a superintelligence the sole goal of minimizing harm to humanity, for example, it will defend itself against shutdown attempts because it knows we’ll harm one another much more in its absence through future wars and other follies.

almost all goals can be better accomplished with more resources, so we should expect a superintelligence to want resources almost regardless of what ultimate goal it has. Giving a superintelligence a single open-ended goal with no constraints can therefore be dangerous:

There’s tension between world-modeling and goal retention (see

With increasing intelligence may come not merely a quantitative improvement in the ability to attain the same old goals, but a qualitatively different understanding of the nature of reality that reveals the old goals to be misguided, meaningless or even undefined.

formicine

vapid

we now have an excellent framework for our truth quest: the scientific method. But how can we determine what’s beautiful or good?

if there’s no experience (as in a dead universe or one populated by zombie-like unconscious machines), there can be no meaning or anything else that’s ethically relevant. If we buy into this utilitarian ethical principle, then it’s crucial that we figure out which intelligent systems are conscious (in the sense of having a subjective experience) and which aren’t;

“Pareto-optimality”

nobody can get better off without someone else getting worse off.

“Three Laws of Robotics” devised by sci-fi legend Isaac Asimov: 1. A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law. 3. A robot must protect its own existence as long as such protection doesn’t conflict with the First or Second Laws.

digital life forms,

would we really want people from 1,500 years ago to have a lot of influence over how today’s world is run? If not, why should we try to impose our ethics on future beings that may be dramatically smarter than us?

suicidal pilot Andreas Lubitz flew Germanwings Flight 9525 into a mountain on March 24, 2015—by setting the autopilot to an altitude of 100 feet (30 meters) above sea level and letting the flight computer do the rest of the work.

The Better Angels of Our Nature, Steven Pinker

Nick Bostrom

Superintelligence,

the orthogonality...

that the ultimate goals of a system can be independent of...

Peter Singer

most humans behave unethically for evolutionary reasons,

how can an “ultimate goal” (or “final goal,” as Bostrom calls it) even be defined for a superintelligence?

we can’t have confidence in the friendly-AI vision unless we can answer this crucial question.

a googolplex is 1 followed by 10100 zeroes—more

many systems evolve to maximize their entropy, which in the absence of gravity eventually leads to heat death, where everything is boringly uniform and unchanging.

Marcus Hutter

Alex Wissner-Gross

Cameron Freer

causal e...

it appears that we humans are a historical accident, and aren’t the optimal solution to any well-defined physics problem. This suggests that a superintelligent AI with a rigorously defined goal will be able to improve its goal attainment by eliminating us.

obdurate

To program a self-driving car, we need to solve the trolley problem of whom to hit during an accident.

Intelligence is the ability to accomplish complex goals.

Aligning machine goals with our own involves three unsolved problems: making machines learn them, adopt them and retain them.

A rule of thumb that many insects use for flying in a straight line is to assume that a bright light is the Sun and fly at a fixed angle relative to it. If the light turns out to be a nearby flame, this hack can unfortunately trick the bug into an inward death spiral.

Christof Koch,

Erwin Schrödinger, “a play before empty benches, not existing for anybody, thus quite properly speaking not existing”?

consciousness = subjective experience In other words, if it feels like something to be you right now, then you’re conscious.

by this definition, you’re conscious also when you’re dreaming, even though you lack wakefulness or access to sensory input

Similarly, any system that experiences pain is conscious

Our definition leaves open the possibility that some future AI systems...

David Chalmers,