Rationality: From AI to Zombies

by Eliezer Yudkowsky

“If the future were not determined by reality, it could not be determined by you,” or “If the future were not determined by something, it could not be determined by you.”

If the mind were not embodied in the brain, it would be embodied in something else; there would be some real thing that was a mind. If the future were not determined by physics, it would be determined by something, some law, some order, some grand reality that included you within it.

But if the laws of physics control us, then how can we be said to control ourselves? Turn it around: If the laws of physics did not control us, how could we possibly control ourselves?

Interesting

If we were not in reality, where could we be?

The thoughts of your decision process are all real, they are all something. But a thought is too big and complicated to be an atom. So thoughts are made of smaller things, and our name for the stuff that stuff is made of is “physics.”

In the Schrödinger’s Cat setup, an unstable atom goes into a superposition of disintegrating, and not-disintegrating. A sensor, tuned to the atom, goes into a superposition of triggering and not-triggering. (Actually, the superposition is now a joint state of [atom-disintegrated × sensor-triggered] + [atom-stable × sensor-not-triggered].) A charge of explosives, hooked up to the sensor, goes into a superposition of exploding and not exploding; a cat in the box goes into a superposition of being dead and alive; and a human, looking inside the box, goes into a superposition of throwing up and being calm. The same law at all levels.

The existence of other versions of ourselves, and indeed other Earths, is not supposed additionally. We are simply supposing that the same laws govern at all levels, having no reason to suppose differently, and all experimental tests having succeeded so far. The existence of other decoherent Earths is a logical consequence of the simplest generalization that fits all known facts.

Instead, we try to harness the individual scientist’s stubborn desire to prove their personal theory, by saying: “Make a new experimental prediction, and do the experiment. If you’re right, and the experiment is replicated, you win.” So long as scientists believe this is true, they have a motive to do experiments that can falsify their own theories.

right now you’ve got people dismissing cryonics out of hand as “not scientific,” like it was some kind of pharmaceutical you could easily administer to 1,000 patients and see what happened. “Call me when cryonicists actually revive someone,” they say; which, as Mike Li observes, is like saying “I refuse to get into this ambulance; call me when it’s actually at the hospital.”

If 1% of women presenting for a routine screening have breast cancer, and 80% of women with breast cancer get positive mammographies, and 10% of women without breast cancer get false positives, what is the probability that a routinely screened woman with a positive mammography has breast cancer? It is 7.5%. You cannot say, “I believe she doesn’t have breast cancer, because the experiment isn’t definite enough.” You cannot say, “I believe she has breast cancer, because it is wise to be pessimistic and that is what the only experiment so far seems to indicate.” Seven point five percent is the rational estimate given this evidence, not 7.4% or 7.6%. The laws of probability are laws.

you cannot make a true map of a city by sitting in your bedroom with your eyes shut and drawing lines upon paper according to impulse. You must walk through the city and draw lines on paper that correspond to what you see. If, seeing the city unclearly, you think that you can shift a line just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.

Carl Schurz said: Ideals are like stars. You will not succeed in touching them with your hands. But, like the seafaring man on the desert of waters, you choose them as your guides and following them you will reach your destiny.

For some ideals are like dreams: they come from within us, not from outside.

You have to guess where your ideals are, and if you guess wrong, you go astray.

But do not limit your ideals to mere stars, to mere humans who actually existed, especially if they were born more than fifty years before you and are dead. Each succeeding generation has a chance to do better. To let your ideals be composed only of humans, especially dead ones, is to limit yourself to what has already been accomplished

The truly important problems are often the ones you’re not even considering, because they appear to be impossible, or, um, actually difficult, or worst of all, not clear how to solve. If you worked on them for years, they might not seem so impossible . . . but this is an extra and unusual insight; naive realism will tell you that solvable problems look solvable, and impossible-looking problems are impossible.

The point being, the problem is not that you need an aura of destiny and the aura of destiny is missing. If you’d met Albert before he published his papers, you would have perceived no aura of destiny about him to match his future high status. He would seem like just another Jewish genius.

There is no separate magisterium for people who do important things.

I say this, because I want to do important things with my life, and I have a genuinely important problem, and an angle of attack, and I’ve been banging my head on it for years, and I’ve managed to set up a support structure for it; and I very frequently meet people who, in one way or another, say: “Yeah? Let’s see your aura of destiny, buddy.”

Yet the way that you acquire magical powers is not by being born with them, but by seeing, with a sudden shock, that they really are perfectly normal. This is a general principle in life.

It is an undeniable fact that we tend to do things that make us happy, but this doesn’t mean we should regard the happiness as the only reason for so acting.

The best way I can put it is that my moral intuition appears to require both the objective and subjective component to grant full value.

my decision system has a lot of terminal values, none of them strictly reducible to anything else. Art, science, love, lust, freedom, friendship . . .

After spending a decade or two living inside a mind, you might think you knew a bit about how minds work, right? That’s what quite a few AGI wannabes (people who think they’ve got what it takes to program an Artificial General Intelligence) seem to have concluded. This, unfortunately, is wrong. Artificial Intelligence is fundamentally about reducing the mental to the non-mental.

Living inside a human mind doesn’t teach you the art of reductionism, because nearly all of the work is carried out beneath your sight, by the opaque black boxes of the brain. So far beneath your sight that there is no introspective sense that the black box is there—no internal sensory event marking that the work has been delegated.

Asking what “AIs” will do is a trick question because it implies that all AIs form a natural class. Humans do form a natural class because we all share the same brain architecture. But when you say “Artificial Intelligence,” you are referring to a vastly larger space of possibilities than when you say “human.”

But wait—natural selection designs complex artifacts and selects among complex strategies. So where is natural selection on this map? So this entire map really floats in a still vaster space, the space of optimization processes. At the bottom of this vaster space, below even humans, is natural selection as it first began in some tidal pool: mutate, replicate, and sometimes die, no sex.

“Science is based on faith too!”

To those who say “Nothing is real,” I once replied, “That’s great, but how does the nothing work?”

if you believe that “beavers” live in deserts, are pure white in color, and weigh 300 pounds when adult, then you do not have any beliefs about beavers, true or false.

So in the future, we’ll have programs that help you play the game—taking over if you get stuck on the game, or just bored; or so that you can play games that would otherwise be too advanced for you.

The function of your organs

if we’re looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.

Timothy Ferris is worth quoting: To find happiness, “the question you should be asking isn’t ‘What do I want?’ or ‘What are my goals?’ but ‘What would excite me?’”

A world in which nothing ever goes wrong, or no one ever experiences any pain or sorrow, is a world containing no stories worth reading about.

A world that you wouldn’t want to read about is a world where you wouldn’t want to live.

Ordinarily, we prefer pleasure to pain, joy to sadness, and life to death. Yet it seems we prefer to empathize with hurting, sad, dead characters.

stories about happier people aren’t serious, aren’t artistically great enough to be worthy of praise—but then why selectively praise stories containing unhappy people? Is there some hidden benefit to us in it?

Orson Scott Card: Oh, I said to myself, that’s what I’ve been doing wrong, my characters aren’t hurting.

I once read that in the BDSM community, “intense sensation” is a euphemism for pain. Upon reading this, it occurred to me that, the way humans are constructed now, it is just easier to produce pain than pleasure.

Maybe all the Great Stories are tragedies because happiness can’t shout loud enough—to a human reader.

Someone who can remember starving will appreciate a loaf of bread more than someone who’s never known anything but cake. This was George Orwell’s hypothesis for why Utopia is impossible in literature and reality:

It would seem that human beings are not able to describe, nor perhaps to imagine, happiness except in terms of contrast . . . The inability of mankind to imagine happiness except in the form of relief, either from effort or pain, presents Socialists with a serious problem.

Wow

Dickens can describe a poverty-stricken family tucking into a roast goose, and can make them appear happy; on the other hand, the inhabitants of perfect universes seem to have no spontaneous gaiety and are usually somewhat repulsive into the bargain.

Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals will contain almost nothing of worth.

when you really do understand evolutionary psychology, you can see how parental love and romance and honor, and even true altruism and moral arguments, bear the specific design signature of natural selection in particular adaptive contexts of the hunter-gatherer savanna.

Evolutionary psychology

You evolved with a psychology alien to evolution: Evolution has nothing like the intelligence or the precision required to exactly quine its goal system.

You evolved with a psychology that attaches utility to things which evolution does not care about—human life, human happiness.

healthy flowers are a sign of fertile land, likely to bear fruits and other treasures, and probably prey animals as well; so is it any wonder that humans evolved to be attracted to flowers?

For love to first enter Time, it must come of something that is not love; if this were not possible, then love could not be.

“And their parents will say: Once upon a time, long ago and far away, ever so long ago, there were intelligent beings who were not themselves intelligently designed. Once upon a time, there were lovers created by something that did not love.