Rationality: From AI to Zombies

by Eliezer Yudkowsky

Your brain isn’t hardwired with a specific, prerecorded statement that “Blowing up a burning building containing my mother is a bad idea.” And yet you’re trying to prerecord that exact specific statement in the Outcome Pump’s future function. So the wish is exploding, turning into a giant lookup table that records your judgment of every possible path through time. You failed to ask for what you really wanted. You wanted your mother to go on living, but you wished for her to become more distant from the center of the building.

AI

To be a safe fulfiller of a wish, a genie must share the same values that led you to make the wish. Otherwise the genie may not choose a path through time that leads to the destination you had in mind, or it may fail to exclude horrible side effects that would lead you to not even consider a plan in the first place. Wishes are leaky generalizations, derived from the huge but finite structure that is your entire morality; only by including this entire structure can you plug all the leaks. With a safe genie, wishing is superfluous. Just run the genie.

AI

Miyamoto Musashi said:1 The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him. You must thoroughly research this.

The essential thing in the art of epistemic rationality is to understand how every rule is cutting through to the truth in the same movement. The corresponding essential of pragmatic rationality—decision theory, versus probability theory—is to always see how every expected utility cuts through to utility. You must thoroughly research this. C. J. Cherryh said:2 Your sword has no blade. It has only your intention. When that goes astray you have no weapon.

The fundamental problem with arguing that things are true “by definition” is that you can’t make reality go a different way by choosing a different definition.

Whenever you feel tempted to say the words “by definition” in an argument that is not literally about pure mathematics, remember that anything which is true “by definition” is true in all possible worlds, and so observing its truth can never constrain which world you live in.

we are telepathic, in fact; but magic isn’t exciting when it’s merely real, and all your friends can do it too. Think telepathy is simple? Try building a computer that will be telepathic with you. Telepathy, or “language,” or whatever you want to call our partial thought transfer ability, is more complicated than it looks.

as soon as you see a blue egg-shaped thing and the central blegg unit fires, you holler “Blegg!” to Susan. And what that algorithm feels like from inside is that the label, and the concept, are very nearly identified; the meaning feels like an intrinsic property of the word itself. The cognoscenti will recognize this as a case of E. T. Jaynes’s “Mind Projection Fallacy.” It feels like a word has a meaning, as a property of the word itself; just like how redness is a property of a red apple, or mysteriousness is a property of a mysterious phenomenon. Indeed, on most occasions, the brain will not distinguish at all between the word and the meaning—only bothering to separate the two while learning a new language, perhaps. And even then, you’ll see Susan pointing to a blue egg-shaped thing and saying “Blegg!,” and you’ll think, I wonder what “blegg” means, and not, I wonder what mental category Susan associates to the auditory label “blegg.” Consider, in this light, the part of the Standard Dispute of Definitions where the two parties argue about what the word “sound” really means—the same way they might argue whether a particular apple is really red or green: ALBERT: “My computer’s microphone can record a sound without anyone being around to hear it, store it as a file, and it’s called a ‘sound file.’ And what’s stored in the file is the pattern of vibrations in air, not the pattern of neural firings in anyone’s brain. ‘Sound’ means a pattern of vibrations.” BARRY: “Oh, yeah?...

We have no obvious mutual interest in using the word “oto” to mean sound, or “sound” to mean oto; but we have a mutual interest in using the same word, whichever word it happens to be. (Preferably, words we use frequently should be short, but let’s not get into information theory just yet.) But, while we have a mutual interest, it is not strictly necessary that you and I use the similar labels internally; it is only convenient. If I know that, to you, “oto” means sound—that is, you associate “oto” to a concept very similar to the one I associate to “sound”—then I can say “Paper crumpling makes a crackling oto.” It requires extra thought, but I can do it if I want. Similarly, if you say “What is the walking-stick of a bowling ball dropping on the floor?” and I know which concept you associate with the syllables “walking-stick,” then I can figure out what you mean. It may require some thought, and give me pause, because I ordinarily associate “walking-stick” with a different concept. But I can do it just fine.

Cf. Donald Davidson's "A Nice Derangement of Epitaphs"

When both sides know what the other side wants to say, and both sides accuse the other side of defecting from “common usage,” then whatever it is they are about, it is clearly not working out a way to communicate with each other. But this is the whole benefit that common usage provides in the first place. Why would you argue about the meaning of a word, two sides trying to wrest it back and forth? If it’s just a namespace conflict that has gotten blown out of proportion, and nothing more is at stake, then the two sides need merely generate two new words and use them consistently.

The illusion of unity across religions can be dispelled by making the term “God” taboo, and asking them to say what it is they believe in; or making the word “faith” taboo, and asking them why they believe it. Though mostly they won’t be able to answer at all, because it is mostly profession in the first place, and you cannot cognitively zoom in on an audio recording.

The chief obstacle to performing an original seeing is that your mind already has a nice neat summary, a nice little easy-to-use concept handle. Like the word “baseball,” or “bat,” or “base.” It takes an effort to stop your mind from sliding down the familiar path, the easy path, the path of least resistance, where the small featureless word rushes in and obliterates the details you’re trying to see. A word itself can have the destructive force of cliché; a word itself can carry the poison of a cached thought.

Could complex networks of cache lookups be the origin of language itself?

To categorize is to throw away information. If you’re told that a falling tree makes a “sound,” you don’t know what the actual sound is; you haven’t actually heard the tree falling. If a coin lands “heads,” you don’t know its radial orientation. A blue egg-shaped thing may be a “blegg,” but what if the exact egg shape varies, or the exact shade of blue? You want to use categories to throw away irrelevant information, to sift gold from dust, but often the standard categorization ends up throwing out relevant information too. And when you end up in that sort of mental trouble, the first and most obvious solution is to play Taboo.

Just because there’s a word “art” doesn’t mean that it has a meaning, floating out there in the void, which you can discover by finding the right definition. It feels that way, but it is not so.

You ask whether something “is” or “is not” a category member but can’t name the question you really want answered. What is a “man”? Is Barney the Baby Boy a “man”? The “correct” answer may depend considerably on whether the query you really want answered is “Would hemlock be a good thing to feed Barney?” or “Will Barney make a good husband?”

You argue about a category membership even after screening off all questions that could possibly depend on a category-based inference. After you observe that an object is blue, egg-shaped, furred, flexible, opaque, luminescent, and palladium-containing, what’s left to ask by arguing, “Is it a blegg?” But if your brain’s categorizing neural network contains a (metaphorical) central unit corresponding to the inference of blegg-ness, it may still feel like there’s a leftover question.

You pull out a dictionary in the middle of any argument ever. Seriously, what the heck makes you think that dictionary editors are an authority on whether “atheism” is a “religion” or whatever? If you have any substantive issue whatsoever at stake, do you really think dictionary editors have access to ultimate wisdom that settles the argument?

You use complex renamings to create the illusion of inference. Is a “human” defined as a “mortal featherless biped”? Then write: “All [mortal featherless bipeds] are mortal; Socrates is a [mortal featherless biped]; therefore, Socrates is mortal.” Looks less impressive that way, doesn’t it?

You get into arguments that you could avoid if you just didn’t use the word. If Albert and Barry aren’t allowed to use the word “sound,” then Albert will have to say “A tree falling in a deserted forest generates acoustic vibrations,” and Barry will say “A tree falling in a deserted forest generates no auditory experiences.” When a word poses a problem, the simplest solution is to eliminate the word and its synonyms.

people do not take prior frequencies sufficiently into account, meaning that when people approach a problem where there’s some evidence X indicating that condition A might hold true, they tend to judge A’s likelihood solely by how well the evidence X seems to match A, without taking into account the prior frequency of A. If you think, for example, that under the mammography example, the woman’s chance of having breast cancer is in the range of 70%–80%, then this kind of reasoning is insensitive to the prior frequency given in the problem; it doesn’t notice whether 1% of women or 10% of women start out having breast cancer. “Pay more attention to the prior frequency!” is one of the many things that humans need to bear in mind to partially compensate for our built-in inadequacies.

A related error is to pay too much attention to P(X|A) and not enough to P(X|¬A) when determining how much evidence X is for A. The degree to which a result X is evidence for A depends not only on the strength of the statement we’d expect to see result X if A were true, but also on the strength of the statement we wouldn’t expect to see result X if A weren’t true. For example, if it is raining, this very strongly implies the grass is wet—P(wetgrass|rain) ≈ 1—but seeing that the grass is wet doesn’t necessarily mean that it has just rained; perhaps the sprinkler was turned on, or you’re looking at the early morning dew. Since P(wetgrass|¬rain) is substantially greater than zero, P(rain|wetgrass) is substantially less than one. On the other hand, if the grass was never wet when it wasn’t raining, then knowing that the grass was wet would always show that it was raining, P(rain|wetgrass) ≈ 1, even if P(wetgrass|rain) = 50%; that is, even if the grass only got wet 50% of the times it rained. Evidence is always the result of the differential between the two conditional probabilities. Strong evidence is not the product of a very high probability that A leads to X, but the product of a very low probability that not-A could have led to X.

Equivalent to modus tollens vs. denying the antecedent

The Bayesian revolution in the sciences is fueled, not only by more and more cognitive scientists suddenly noticing that mental phenomena have Bayesian structure in them; not only by scientists in every field learning to judge their statistical methods by comparison with the Bayesian method; but also by the idea that science itself is a special case of Bayes’s Theorem; experimental evidence is Bayesian evidence. The Bayesian revolutionaries hold that when you perform an experiment and get evidence that “confirms” or “disconfirms” your theory, this confirmation and disconfirmation is governed by the Bayesian rules. For ...

The likelihood ratio for X, the quantity P(X|A)/P(X|¬A), determines how much observing X slides the probability for A; the likelihood ratio is what says how strong X is as evidence. Well, in your theory A, you can predict X with probability 1, if you like; but you can’t control the denominator of the likelihood ratio, P(X|¬A)—there will always be some alternative theories that also predict X, and while we go with the simplest theory that fits the current evidence, you may someday encounter some evidence that an alternative theory predicts but your theory does not. That’s the hidden gotcha that toppled Newton’s theory of gravity. So there’s a limit on how much mileage you can get from successful predictions; there’s a limit on how high the likelihood ratio goes for confirmatory evidence.

If your map breaks into four pieces for easy storage, it doesn’t mean the territory is also broken into disconnected parts. Our minds store different surface-level rules in different compartments, but this does not reflect any division in the laws that govern Nature.

Reality is laced together a lot more tightly than humans might like to believe.

If you would learn to think like reality, then here is the Tao: Since the beginning not one unusual thing has ever happened.

Reality, we have learned to our shock, is not a collection of separate magisteria, but a single unified process governed by mathematically simple low-level rules. Different buildings on a university campus do not belong to different universes, though it may sometimes seem that way. The universe is not divided into mind and matter, or life and nonlife; the atoms in our heads interact seamlessly with the atoms of the surrounding air. Nor is Bayes’s Theorem different from one place to another.

The exact state of a glass of boiling-hot water may be unknown to you—indeed, your ignorance of its exact state is what makes the molecules’ kinetic energy “heat,” rather than work waiting to be extracted like the momentum of a spinning flywheel. So the water might cool down your hand instead of heating it up, with probability ~0. Decide to ignore the laws of thermodynamics and stick your hand in anyway, and you’ll get burned. “But you don’t know that!” I don’t know it with certainty, but it is mandatory that I expect it to happen. Probabilities are not logical truths, but the laws of probability are. “But what if I guess the state of the boiling water, and I happen to guess correctly?” Your chance of guessing correctly by luck, is even less than the chance of the boiling water cooling your hand by luck.

if I see a mirage of a lake in a desert, the correct causal explanation of my vision does not involve the fact of any actual lake in the desert. In this case, my belief in the lake is not just explained, but explained away. But either way, the belief itself is a real phenomenon taking place in the real universe—psychological events are events—and its causal history can be traced back. “Why is there a lake in the middle of the desert?” may fail if there is no lake to be explained. But “Why do I perceive a lake in the middle of the desert?” always has a causal explanation, one way or the other.

Probabilities express uncertainty, and it is only agents who can be uncertain. A blank map does not correspond to a blank territory. Ignorance is in the mind.

the way physics really works, as far as we can tell, is that there is only the most basic level—the elementary particle fields and fundamental forces. You can’t handle the raw truth, but reality can handle it without the slightest simplification. (I wish I knew where Reality got its computing power.) The laws of physics do not contain distinct additional causal entities that correspond to lift or airplane wings, the way that the mind of an engineer contains distinct additional cognitive entities that correspond to lift or airplane wings. This, as I see it, is the thesis of reductionism. Reductionism is not a positive belief, but rather, a disbelief that the higher levels of simplified multilevel models are out there in the territory. Understanding this on a gut level dissolves the question of “How can you say the airplane doesn’t really have wings, when I can see the wings right there?” The critical words are really and see.

Emergentism or thoughts of layers are due to scientific realism. With empirical constructivism, you can understand the "levels" of theory with their own approximations that we find convenient. It's in us.

The rainbow was explained. The haunts in the air, and gnomes in the mine, were explained away. I think this is the key distinction that anti-reductionists don’t get about reductionism.

You have to interpret the anti-reductionists’ experience of “reductionism,” not in terms of their actually seeing how rainbows work, not in terms of their having the critical “Aha!,” but in terms of their being told that the password is “Science.” The effect is just to move rainbows to a different literary genre—a literary genre they have been taught to regard as boring. For them, the effect of hearing “Science has explained rainbows!” is to hang up a sign over rainbows saying, “This phenomenon has been labeled BORING by order of the Council of Sophisticated Literary Critics. Move along.” And that’s all the sign says: only that, and nothing more.

I have already remarked that nothing is inherently mysterious—nothing that actually exists, that is. If I am ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon; to worship a phenomenon because it seems so wonderfully mysterious is to worship your own ignorance; a blank map does not correspond to a blank territory, it is just somewhere we haven’t visited yet, etc., etc. . . . Which is to say that everything—everything that actually exists—is liable to end up in “the dull catalogue of common things,” sooner or later. Your choice is either: Decide that things are allowed to be unmagical, knowable, scientifically explicable—in a word, real—and yet still worth caring about; Or go about the rest of your life suffering from existential ennui that is unresolvable. (Self-deception might be an option for others, but not for you.)

A friend once told me that I look at the world as if I’ve never seen it before. I thought, that’s a nice compliment . . . Wait! I never have seen it before! What—did everyone else get a preview? —Ran Prieur

It is a Hollywood Rationality meme that “Science takes the fun out of life.” Science puts the fun back into life. Rationality directs your emotional energies into the universe, rather than somewhere else.

The worst catastrophe you could visit upon the New Age community would be for their rituals to start working reliably, and for UFOs to actually appear in the skies. What would be the point of believing in aliens, if they were just there, and everyone else could see them too? In a world where psychic powers were merely real, New Agers wouldn’t believe in psychic powers, any more than anyone cares enough about gravity to believe in it. (Except for scientists, of course.)

if you only care about scientific issues that are controversial, you will end up with a head stuffed full of garbage. The media thinks that only the cutting edge of science is worth reporting on. How often do you see headlines like “General Relativity Still Governing Planetary Orbits” or “Phlogiston Theory Remains False”? So, by the time anything is solid science, it is no longer a breaking headline. “Newsworthy” science is often based on the thinnest of evidence and wrong half the time—if it were not on the uttermost fringes of the scientific frontier, it would not be breaking news. Scientific controversies are problems so difficult that even people who’ve spent years mastering the field can still fool themselves. That’s what makes for the heated arguments that attract all the media attention.

A steady diet of science news is bad for you: You are what you eat, and if you eat only science reporting on fluid situations, without a solid textbook now and then, your brain will turn to liquid.

A good “atheistic hymn” is simply a song about anything worth singing about that doesn’t happen to be religious.

If God did speak plainly, and answer prayers reliably, God would just become one more boringly real thing, no more worth believing in than the postman. If God were real, it would destroy the inner uncertainty that brings forth outward fervor in compensation. And if everyone else believed God were real, it would destroy the specialness of being one of the elect.

SO TRUE

The choice between God and humanity is not just a choice of drugs. Above all, humanity actually exists.

When a lie has been defended for ages upon ages, the true origin of the inherited habits lost in the mists, with layer after layer of undocumented sickness; then the wise, I think, will start over from scratch, rather than trying to selectively discard the original lie while keeping the habits of thought that protected it. Just admit you were wrong, give up entirely on the mistake, stop defending it at all, stop trying to say you were even a little right, stop trying to save face, just say “Oops!” and throw out the whole thing and begin again. That capacity—to really, really, without defense, admit you were entirely wrong—is why religious experience will never be like scientific experience. No religion can absorb that capacity without losing itself entirely and becoming simple humanity . . .

if you can see how (not just know that) a higher level reduces to a lower one, they will not seem like separate things within your map; you will be able to see how silly it is to think that your fingers could be in one place, and your hand somewhere else; you will be able to see how silly it is to argue about whether it is your hand that picks up the cup, or your fingers. The operative word is “see,” as in concrete visualization. Imagining your hand causes you to imagine the fingers and thumb and palm; conversely, imagining fingers and thumb and palm causes you to identify a hand in the mental picture. Thus the high level of your map and the low level of your map will be tightly bound together in your mind. In reality, of course, the levels are bound together even tighter than that—bound together by the tightest possible binding: physical identity. You can see this: You can see that saying (1) “hand” or (2) “fingers and thumb and palm,” does not refer to different things, but different points of view.

1) XYZ is not conceivable if you can see why - and not just know that - H2O is water. 2) Composition as identity?

you shouldn’t draw too many conclusions from how it seems conceptually possible, in the mind of some specific conceiver, to separate the hand from its constituent elements of fingers, thumb, and palm. Conceptual possibility is not the same as logical possibility or physical possibility. It is conceptually possible to you that 235,757 is prime, because you don’t know any better. But it isn’t logically possible that 235,757 is prime; if you were logically omniscient, 235,757 would be obviously composite (and you would know the factors). That’s why we have the notion of impossible possible worlds, so that we can put probability distributions on propositions that may or may not be in fact logically impossible.

Your inability to imagine something is just a computational fact about what your brain can or can’t imagine. Another brain might work differently.

Explanatory gaps can be crossed, if you accept help from science, and don’t trust the view from the interior of your own mind.

you can say, “My beliefs about heat and motion are not the same beliefs, but it’s possible that actual heat and actual motion are the same thing.” It’s just like being able to acknowledge that “the morning star” and “the evening star” might be the same planet, while also understanding that you can’t determine this just by examining your beliefs—you’ve got to haul out the telescope.

“Snow” doesn’t have a logical definition known to us—it’s more like an empirically determined pointer to a logical definition. This is true even if you believe that snow is ice crystals is low-temperature tiled water molecules. The water molecules are made of quarks. What if quarks turn out to be made of something else? What is a snowflake, then? You don’t know—but it’s still a snowflake, not a fire hydrant. And of course, these very paragraphs I have just written are likewise far above the level of quarks. “Sensing white stuff, visually categorizing it, and thinking ‘snow’ or ‘not snow’”—this is also talking very far above the quarks. So my meta-beliefs are also promissory notes, for things that an Ideal Omniscient Science Interpreter might know about which configurations of the quarks (or whatever) making up my brain correspond to “believing ‘snow is white.’” But then, the entire grasp that we have upon reality is made up of promissory notes of this kind. So, rather than calling it circular, I prefer to call it self-consistent. This can be a bit unnerving—maintaining a precarious epistemic perch, in both object-level beliefs and reflection, far above a huge unknown underlying fundamental reality, and hoping one doesn’t fall off. On reflection, though, it’s hard to see how things could be any other way.

Anomalous monism + coherentism (Donald Davidson)

There are patterns that exist in reality where we see “hands,” and these patterns have something in common, but they are not fundamental. If I really had no hands—if reality suddenly transitioned to be in a state that we would describe as “Eliezer has no hands”—reality would shortly thereafter correspond to a state we would describe as “Eliezer screams as blood jets out of his wrist stumps.” And this is true, even though the above paragraph hasn’t specified any quark positions.