Rationality: From AI to Zombies

by Eliezer Yudkowsky

The map is multilevel, the territory is single-level. This doesn’t mean that the higher levels “don’t exist,” like looking in your garage for a dragon and finding nothing there, or like seeing a mirage in the desert and forming an expectation of drinkable water when there is nothing to drink. The higher levels of your map are not false, without referent; they have referents in the single level of physics. It’s not that the wings of an airplane unexist—then the airplane would drop out of the sky. The “wings of an airplane” exist explicitly in an engineer’s multilevel model of an airplane, and the wings of an airplane exist implicitly in the quantum physics of the real airplane. Implicit existence is not the same as nonexistence. The exact description of this implicitness is not known to us—is not explicitly represented in our map. But this does not prevent our map from working, or even prevent it from being true.

Against eliminativism

Why postulate an extramaterial soul, and then postulate that the soul has no effect on the physical world, and then postulate a mysterious unknown material process that causes your internal narrative to talk about conscious experience? Why not postulate the true stuff of consciousness which no amount of mere mechanical atoms can add up to, and then, having gone that far already, let this true stuff of consciousness have causal effects like making philosophers talk about consciousness? I am not endorsing Descartes’s view. But at least I can understand where Descartes is coming from. Consciousness seems mysterious, so you postulate a mysterious stuff of consciousness. Fine. But now the zombie-ists postulate that this mysterious stuff doesn’t do anything, so you need a whole new explanation for why you say you’re conscious. That isn’t vitalism. That’s something so bizarre that vitalists would spit out their coffee. “When fires burn, they release phlogiston. But phlogiston doesn’t have any experimentally detectable impact on our universe, so you’ll have to go looking for a separate explanation of why a fire can melt snow.” What? Are property dualists under the impression that if they postulate a new active force, something that has a causal impact on observables, they will be sticking their necks out too far? Me, I’d say that if you postulate a mysterious, separate, additional, inherently mental property of consciousness, above and beyond positions and velocities, then, at tha...

If we do something to you, where you don’t see how it could possibly change your internal narrative—the little voice in your head that sometimes says things like “I think therefore I am,” whose words you can choose to say aloud—then it shouldn’t make you cease to be conscious.

Ordinarily, when we’re talking to a person, we tend to think that whatever is inside the skull must be “where the consciousness is.” It’s only by playing Follow-The-Improbability that we can realize that the real source of the conversation we’re having is that-which-is-responsible-for the improbability of the conversation—however distant in time or space, as the Sun moves a wind-up toy. “No, no!” says the philosopher. “In the thought experiment, they aren’t randomly generating lots of GLUTs, and then using a conscious algorithm to pick out one GLUT that seems humanlike! I am specifying that, in this thought experiment, they reach into the inconceivably vast GLUT bin, and by pure chance pull out a GLUT that is identical to a human brain’s inputs and outputs! There! I’ve got you cornered now! You can’t play Follow-The-Improbability any further!” Oh. So your specification is the source of the improbability here. When we play Follow-The-Improbability again, we end up outside the thought experiment, looking at the philosopher.

A “supernatural” explanation appeals to ontologically basic mental things, mental entities that cannot be reduced to nonmental entities.

Suppose that you discover what seems to be a spirit, inhabiting a tree—a dryad who can materialize outside or inside the tree, who speaks in English about the need to protect her tree, et cetera. And then suppose that we turn a microscope on this tree spirit, and she turns out to be made of parts—not inherently spiritual and ineffable parts, like fabric of desireness and cloth of belief, but rather the same sort of parts as quarks and electrons, parts whose behavior is defined in motions rather than minds. Wouldn’t the dryad immediately be demoted to the dull catalogue of common things?

I previously defined the reductionist thesis as follows: human minds create multi-level models of reality in which high-level patterns and low-level patterns are separately and explicitly represented. A physicist knows Newton’s equation for gravity, Einstein’s equation for gravity, and the derivation of the former as a low-speed approximation of the latter. But these three separate mental representations are only a convenience of human cognition. It is not that reality itself has an Einstein equation that governs at high speeds, a Newton equation that governs at low speeds, and a “bridging law” that smooths the interface. Reality itself has only a single level, Einsteinian gravity. It is only the Mind Projection Fallacy that makes some people talk as if the higher levels could have a separate existence—different levels of organization can have separate representations in human maps, but the territory itself is a single unified low-level mathematical object.

The basic error of anthropomorphism, and the reason why supernatural explanations sound much simpler than they really are, is your brain using itself as an opaque black box to predict other things labeled “mindful.” Because you already have big, complicated webs of neural circuitry that implement your “wanting” things, it seems like you can easily describe water that “wants” to flow downhill—the one word “want” acts as a lever to set your own complicated wanting-machinery in motion. Or you imagine that God likes beautiful things, and therefore made the flowers. Your own “beauty” circuitry determines what is “beautiful” and “not beautiful.” But you don’t know the diagram of your own synapses. You can’t describe a nonmental system that computes the same label for what is “beautiful” or “not beautiful”—can’t write a computer program that predicts your own labelings. But this is just a defect of knowledge on your part; it doesn’t mean that the brain has no explanation.

If we’re going over the archeological records to test the assertion that Jehovah parted the Red Sea out of an explicit desire to display its superhuman power, then it makes little difference whether Jehovah is ontologically basic, or an alien with nanotech, or a Dark Lord of the Matrix. You do some archeology, find no skeletal remnants or armor at the Red Sea site, and indeed find records that Egypt ruled much of Canaan at the time. So you stamp the historical record in the Bible “disproven” and carry on. The hypothesis is coherent, falsifiable and wrong.

If collapse actually worked the way its adherents say it does, it would be: The only non-linear evolution in all of quantum mechanics. The only non-unitary evolution in all of quantum mechanics. The only non-differentiable (in fact, discontinuous) phenomenon in all of quantum mechanics. The only phenomenon in all of quantum mechanics that is non-local in the configuration space. The only phenomenon in all of physics that violates CPT symmetry. The only phenomenon in all of physics that violates Liouville’s Theorem (has a many-to-one mapping from initial conditions to outcomes). The only phenomenon in all of physics that is acausal / non-deterministic / inherently random. The only phenomenon in all of physics that is non-local in spacetime and propagates an influence faster than light. WHAT DOES THE GOD-DAMNED COLLAPSE POSTULATE HAVE TO DO FOR PHYSICISTS TO REJECT IT? KILL A GOD-DAMNED PUPPY?

It is not enough to say whether one theory seems more simple, or seems more complex, than another—you have to assign a number; and the number has to be meaningful, you can’t just make it up. Crossing this gap is like the difference between being able to eyeball which things are moving “fast” or “slow,” and starting to measure and calculate velocities.

Think of probability as a conserved quantity: there’s only so much to go around. As the number of details in a story goes up, the number of possible stories increases exponentially, but the sum over their probabilities can never be greater than 1. For every story “X and Y,” there is a story “X and ¬Y.” When you just tell the story “X,” you get to sum over the possibilities Y and ¬Y. If you add ten details to X, each of which could potentially be true or false, then that story must compete with 210 - 1 other equally detailed stories for precious probability. If on the other hand it suffices to just say X, you can sum your probability over 210 stories ((X and Y and Z and . . .) or (X and ¬Y and Z and . . .) or . . .). The “entities” counted by Occam’s Razor should be individually costly in probability; this is why we prefer theories with fewer of them.

To talk about “intelligent design” whenever you point to a purported flaw or open problem in evolutionary theory is, again, privileging the hypothesis—you must have evidence already in hand that points to intelligent design specifically in order to justify raising that particular idea to our attention, rather than a thousand others.

Someone who spends all day thinking about whether the Trinity does or does not exist, rather than Allah or Thor or the Flying Spaghetti Monster, is more than halfway to Christianity. If leaving, they’re less than half departed; if arriving, they’re more than halfway there.

Decision and decoherence are entirely orthogonal concepts. If your brain never became decoherent, then that single cognitive process would still have to imagine different choices and their different outcomes. And a rock, which makes no decisions, obeys the same laws of quantum mechanics as anything else, and splits frantically as it lies in one place.

In no world does two plus two equal five. In no world can spaceships travel faster than light. All the quantum worlds obey our laws of physics; their existence is asserted in the first place by our laws of physics. Since the beginning, not one unusual thing has ever happened, in this or any other world. They are all lawful.

If you’re thinking about a world that could arise in a lawful way, but whose probability is a quadrillion to one, and something very pleasant or very awful is happening in this world . . . well, it does probably exist, if it is lawful. But you should try to release one quadrillionth as many neurotransmitters, in your reward centers or your aversive centers, so that you can weigh that world appropriately in your decisions. If you don’t think you can do that . . . don’t bother thinking about it. Otherwise you might as well go out and buy a lottery ticket using a quantum random number, a strategy that is guaranteed to result in a very tiny mega-win.

Tell people to “shut up and calculate” because you don’t know what the calculations mean, and inside of five years, “Shut up!” will be masquerading as a positive theory of quantum mechanics. I have the highest respect for any historical physicists who even came close to actually shutting up and calculating, who were genuinely conservative in assessing what they did and didn’t know. This is the best they could possibly do without actually being Hugh Everett, and I award them fifty rationality points. My scorn is reserved for those who interpreted “We don’t know why it works” as the positive knowledge that the equations were definitely not real.

If the laws of physics did not control us, how could we possibly control ourselves? How could thoughts judge other thoughts, how could emotions conflict with each other, how could one course of action appear best, how could we pass from uncertainty to certainty about our own plans, in the midst of utter chaos? If we were not in reality, where could we be? The future is determined by physics. What kind of physics? The kind of physics that includes the actions of human beings. People’s choices are determined by physics. What kind of physics? The kind of physics that includes weighing decisions, considering possible outcomes, judging them, being tempted, following morals, rationalizing transgressions, trying to do better . . . There is no point where a quark swoops in from Pluto and overrides all this. The thoughts of your decision process are all real, they are all something. But a thought is too big and complicated to be an atom. So thoughts are made of smaller things, and our name for the stuff that stuff is made of is “physics.” Physics underlies our decisions and includes our decisions. It does not explain them away.

But it is dangerous to focus too much on specific hypotheses that you have no specific reason to think about. This is the same root error of the Intelligent Design folk, who pick any random puzzle in modern genetics, and say, “See, God must have done it!” Why “God,” rather than a zillion other possible explanations?—which you would have thought of long before you postulated divine intervention, if not for the fact that you secretly started out already knowing the answer you wanted to find.

we assume and accept that each individual scientist may be crazily attached to their personal theories. Nor do we assume that anyone can be trained out of this tendency—we don’t try to choose Eminent Judges who are supposed to be impartial. Instead, we try to harness the individual scientist’s stubborn desire to prove their personal theory, by saying: “Make a new experimental prediction, and do the experiment. If you’re right, and the experiment is replicated, you win.” So long as scientists believe this is true, they have a motive to do experiments that can falsify their own theories. Only by accepting the possibility of defeat is it possible to win. And any great claim will require replication; this gives scientists a motive to be honest, on pain of great embarrassment. And so the stubbornness of individual scientists is harnessed to produce a steady stream of knowledge at the group level. The System is somewhat more trustworthy than its parts.

Evolutionary psychology is another example of a case where rationality has to take over from science. While theories of evolutionary psychology form a connected whole, only some of those theories are readily testable experimentally. But you still need the other parts of the theory, because they form a connected web that helps you to form the hypotheses that are actually testable—and then the helper hypotheses are supported in a Bayesian sense, but not supported experimentally. Science would render a verdict of “not proven” on individual parts of a connected theoretical mesh that is experimentally productive as a whole. We’d need a new kind of verdict for that, something like “indirectly supported.”

I sometimes say that the method of science is to amass such an enormous mountain of evidence that even scientists cannot ignore it; and that this is the distinguishing characteristic of a scientist. (A non-scientist will ignore it anyway.) Max Planck was even less optimistic:1 A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it. I am much tickled by this notion, because it implies that the power of science to distinguish truth from falsehood ultimately rests on the good taste of grad students.

To let your ideals be composed only of humans, especially dead ones, is to limit yourself to what has already been accomplished. You will ask yourself, “Do I dare to do this thing, which Einstein could not do? Is this not lèse majesté?” Well, if Einstein had sat around asking himself, “Am I allowed to do better than Newton?” he would not have gotten where he did.

The test of whether a model of reality “explains” my arm’s turning into a blue tentacle is whether the model concentrates significant probability mass into that particular outcome. Why that dream, in the hospital? Why would aliens do that particular thing to me, as opposed to the other billion things they might do? Why would my arm turn into a tentacle on that morning, after remaining an arm through every other morning of my life? And in all cases I must look for an argument compelling enough to make that particular prediction in advance, not mere compatibility. Once I already knew the outcome, it would become far more difficult to sift through hypotheses to find good explanations. Whatever hypothesis I tried, I would be hard-pressed not to allocate more probability mass to yesterday’s blue-tentacle outcome than if I extrapolated blindly, seeking the model’s most likely prediction for tomorrow.

To a Bayesian, probabilities are anticipations, not mere beliefs to proclaim from the rooftops. If I have a model that assigns probability mass to waking up with a blue tentacle, then I am nervous about waking up with a blue tentacle. What if the model is a fanciful one, like a witch casting a spell that transports me into a randomly selected webcomic? Then the prior probability of webcomic witchery is so low that my real-world understanding doesn’t assign any significant weight to that hypothesis.

If I was worried I might someday need a clever excuse for waking up with a tentacle, the reason I was nervous about the possibility would be my explanation.

The very fact that a religious person would be afraid of God withdrawing Its threat to punish them for committing murder shows that they have a revulsion of murder that is independent of whether God punishes murder or not. If they had no sense that murder was wrong independently of divine retribution, the prospect of God not punishing murder would be no more existentially horrifying than the prospect of God not punishing sneezing.

YAAAAAS

You don’t hear religious fundamentalists using the argument: “If we did not fear hell and yearn for heaven, then what would stop people from eating pork?” Yet by their assumptions—that we have no moral compass but divine reward and retribution—this argument should sound just as forceful as the other.

If you have something that just maximizes the number of paperclips in its future light cone, and you raise it with loving parents, it’s still going to come out as a paperclip maximizer. There is not that within it that would call forth the conditional response of a human child. Kindness is not sneezed into an AI by miraculous contagion from its programmers.

It is a general principle that the world is deeper by far than it appears. As with the many levels of physics, so too with cognitive science. Every word you see in print, and everything you teach your children, are only surface levers controlling the vast hidden machinery of the mind. These levers are the whole world of ordinary discourse: they are all that varies, so they seem to be all that exists; perception is the perception of differences.

it’s important to distinguish between reflecting on your mind using your mind (it’s not like you can use anything else) and having an unquestionable assumption that you can’t reflect on.

If, for example, someone believes the authority of “Thou Shalt Not Kill” derives from God, then there are several and well-known things to say that can help set up a line of retreat—as opposed to immediately attacking the plausibility of God. You can say, “Take personal responsibility! Even if you got orders from God, it would be your own decision to obey those orders. Even if God didn’t order you to be moral, you could just be moral anyway.” The above argument actually generalizes to quite a number of metaethics—you just substitute Their-Favorite-Source-Of-Morality, or even the word “morality,” for “God.” Even if your particular source of moral authority failed, couldn’t you just drag the child off the train tracks anyway? And indeed, who is it but you that ever decided to follow this source of moral authority in the first place? What responsibility are you really passing on?

Maybe that which you would do even if there were no morality, is your morality.

If one knows nothing about morality, what does the word “should” mean, at all? If you don’t know whether death is right or wrong—and don’t know how you can discover whether death is right or wrong—and don’t know whether any given procedure might output the procedure for saying whether death is right or wrong—then what do these words, “right” and “wrong,” even mean? If the words “right” and “wrong” have nothing baked into them—no starting point—if everything about morality is up for grabs, not just the content but the structure and the starting point and the determination procedure—then what is their meaning? What distinguishes, “I don’t know what is right” from “I don’t know what is wakalixes”?

For an expected utility maximizer, rescaling the utility function to add a trillion to all outcomes is meaningless—it’s literally the same utility function, as a mathematical object. A utility function describes the relative intervals between outcomes; that’s what it is, mathematically speaking.

I endorse “the end doesn’t justify the means” as a principle to guide humans running on corrupted hardware, but I wouldn’t endorse it as a principle for a society of AIs that make well-calibrated estimates.

Self-effacing consequentialism

Historically speaking, science won because it displayed greater raw strength in the form of technology, not because science sounded more reasonable. To this very day, magic and scripture still sound more reasonable to untrained ears than science. That is why there is continuous social tension between the belief systems. If science not only worked better than magic, but also sounded more intuitively reasonable, it would have won entirely by now.

You shouldn’t claim to be more rational than someone and simultaneously envy them their choice—only their choice. Just do the act you envy.

You shouldn’t find yourself distinguishing the winning choice from the reasonable choice. Nor should you find yourself distinguishing the reasonable belief from the belief that is most likely to be true.

To feel the burning itch of curiosity requires both that you be ignorant, and that you desire to relinquish your ignorance. If in your heart you believe you already know, or if in your heart you do not wish to know, then your questioning will be purposeless and your skills without direction. Curiosity seeks to annihilate itself; there is no curiosity that does not want an answer. The glory of glorious mystery is to be solved, after which it ceases to be mystery.

“That which can be destroyed by the truth should be.”1 Do not flinch from experiences that might destroy your beliefs. The thought you cannot think controls you more than thoughts you speak aloud. Submit yourself to ordeals and test yourself in fire. Relinquish the emotion which rests upon a mistaken belief, and seek to feel fully that emotion which fits the facts.

Surrender to the truth as quickly as you can. Do this the instant you realize what you are resisting, the instant you can see from which quarter the winds of evidence are blowing against you. Be faithless to your cause and betray it to a stronger enemy. If you regard evidence as a constraint and seek to free yourself, you sell yourself into the chains of your whims.

Beware lest you place huge burdens of proof only on propositions you dislike, and then defend yourself by saying: “But it is good to be skeptical.” If you attend only to favorable evidence, picking and choosing from your gathered data, then the more data you gather, the less you know. If you are selective about which arguments you inspect for flaws, or how hard you inspect for flaws, then every flaw you learn how to detect makes you that much stupider.

In argument strive for exact honesty, for the sake of others and also yourself: the part of yourself that distorts what you say to others also distorts your own thoughts. Do not believe you do others a favor if you accept their arguments; the favor is to you. Do not think that fairness to all sides means balancing yourself evenly between positions; truth is not handed out in equal portions before the start of a debate.

Do not ask which beliefs to profess, but which experiences to anticipate.

When you profess a huge belief with many details, each additional detail is another chance for the belief to be wrong. Each specification adds to your burden; if you can lighten your burden you must do so. There is no straw that lacks the power to break your back.

To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty. Who are most humble? Those who most skillfully prepare for the deepest and most catastrophic errors in their own beliefs and plans.

In every art, if you do not seek perfection you will halt before taking your first steps. If perfection is impossible that is no excuse for not trying.

The narrowest statements slice deepest, the cutting edge of the blade. As with the map, so too with the art of mapmaking: The Way is a precise Art. Do not walk to the truth, but dance. On each and every step of that dance your foot comes down in exactly the right spot. Each piece of evidence shifts your beliefs by exactly the right amount, neither more nor less.