Rationality: From AI to Zombies

by Eliezer Yudkowsky

There’s a lot of things I could be curious about in the falling-tree scenario. I could go into the forest and look at trees, or learn how to derive the wave equation for changes of air pressure, or examine the anatomy of an ear, or study the neuroanatomy of the auditory cortex. Instead of doing any of these things, I am to consult a dictionary, apparently. Why? Are the editors of the dictionary expert botanists, expert physicists, expert neuroscientists? Looking in an encyclopedia might make sense, but why a dictionary?

“Personally I’d say that if the issue arises, both sides should switch to describing the event in unambiguous lower-level constituents, like acoustic vibrations or auditory experiences. Or each side could designate a new word, like ‘alberzle’ and ‘bargulum,’ to use for what they respectively used to call ‘sound’; and then both sides could use the new words consistently. That way neither side has to back down or lose face, but they can still communicate.

Main

Why would you argue about the meaning of a word, two sides trying to wrest it back and forth? If it’s just a namespace conflict that has gotten blown out of proportion, and nothing more is at stake, then the two sides need merely generate two new words and use them consistently.

Exactly

Albert says that people have “free will.” Barry says that people don’t have “free will.” Well, that will certainly generate an apparent conflict. Most philosophers would advise Albert and Barry to try to define exactly what they mean by “free will,” on which topic they will certainly be able to discourse at great length. I would advise Albert and Barry to describe what it is that they think people do, or do not have, without using the phrase “free will” at all. (If you want to try this at home, you should also avoid the words “choose,” “act,” “decide,” “determined,” “responsible,” or any of their synonyms.)

Interesting

Why are you going to “school”? To get an “education” ending in a “degree.” Blank out the forbidden words and all their obvious synonyms, visualize the actual details, and you’re much more likely to notice that “school” currently seems to consist of sitting next to bored teenagers listening to material you already know, that a “degree” is a piece of paper with some writing on it, and that “education” is forgetting the material as soon as you’re tested on it.

If you see your activities and situation originally, you will be able to originally see your goals as well. If you can look with fresh eyes, as though for the first time, you will see yourself doing things that you would never dream of doing if they were not habits. Purpose is lost whenever the substance (learning, knowledge, health) is displaced by the symbol (a degree, a test score, medical care). To heal a lost purpose, or a lossy categorization, you must do the reverse:

Wow

Bayes’s Rule itself describes “evidence” in pure math, without using words like “implies,” “means,” “supports,” “proves,” or “justifies.” Set out to define such philosophical terms, and you’ll just go in circles.

compression fallacy: things given the same name end up dumped into the same mental bucket, blurring them together into the same point on the map.

“Art is that which is designed for the purpose of creating a reaction in an audience.”

People have a tendency to talk, and presumably think, at the basic level of categorization—to draw the boundary around “chairs,” rather than around the more specific category “recliner,” or the more general category “furniture.” People are more likely to say “You can sit in that chair” than “You can sit in that recliner” or “You can sit in that furniture.”

And it is no coincidence that the word for “chair” contains fewer syllables than either “recliner” or “furniture.” Basic-level categories, in general, tend to have short names; and nouns with short names tend to refer to basic-level categories. Not a perfect rule, of course, but a definite tendency. Frequent use goes along with short words; short words go along with frequent use.

Tegmark’s Our Mathematical Universe discusses a number of relevant ideas in philosophy and physics.5 Among Tegmark’s more novel ideas is his argument that all consistent mathematical structures exist, including worlds with physical laws and boundary conditions entirely unlike our own.

Antoine-Laurent de Lavoisier discovered that breathing (respiration) and fire (combustion) operated on the same principle. It was one of the most startling unifications in the history of science, for it brought together the mundane realm of matter and the sacred realm of life, which humans had divided into separate magisteria.

The idea of a rule with literally no exceptions seems insanely rigid, the product of closed-minded thinking by fanatics so in the grip of their one big idea that they can’t see the richness and complexity of the real universe.

when you actually look at the universe, it turns out to be, by human standards, insanely rigid in applying its rules. As far as we know, there has been not one single violation of Conservation of Momentum from the uttermost dawn of time up until now.

Since the beginning not one unusual thing has ever happened.

The best current knowledge says that the “real world” is a perfectly regular, deterministic, and very large mathematical object which is highly expensive to simulate.

Bayesian calculation is computationally intractable

Reality, we have learned to our shock, is not a collection of separate magisteria, but a single unified process governed by mathematically simple low-level rules. Different buildings on a university campus do not belong to different universes, though it may sometimes seem that way. The universe is not divided into mind and matter, or life and nonlife; the atoms in our heads interact seamlessly with the atoms of the surrounding air. Nor is Bayes’s Theorem different from one place to another.

“A witty saying proves nothing,” as Voltaire said.

The First Law of Thermodynamics, better known as Conservation of Energy, says that you can’t create energy from nothing: it prohibits perpetual motion machines of the first type, which run and run indefinitely without consuming fuel or any other resource.

Probabilities are not logical truths, but the laws of probability are.

Philosophy may lead you to reject the concept, but rejecting a concept is not the same as understanding the cognitive algorithms behind it.

Those who dream do not know they dream, but when you wake you know you are awake. Which is to say: When you’re done, you’ll know you’re done, but unfortunately the reverse implication does not hold.

Mystery exists in the mind, not in reality. If I am ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon itself. All the more so if it seems like no possible answer can exist: Confusion exists in the map, not in the territory.

Interesting

If your brain uses that data structure, or something metaphorically similar to it, then from the inside it feels like sexiness is an inherent property of the woman, not a property of the alien looking at the woman. Since the woman is attractive, the alien monster will be attracted to her—isn’t that logical?

Probabilities express uncertainty, and it is only agents who can be uncertain. A blank map does not correspond to a blank territory. Ignorance is in the mind.

the notion of truth is quite different from the notion of reality. Saying “true” compares a belief to reality. Reality itself does not need to be compared to any beliefs in order to be real. Remember this the next time someone claims that nothing is true.

Different societies have different truths! No, different societies have different beliefs. Belief is of a different type than truth;

From the inside, our beliefs about the world look like the world, and our beliefs about our beliefs look like beliefs. When you see the world, you are experiencing a belief from the inside. When you notice yourself believing something, you are experiencing a belief about belief from the inside. So if your internal representations of belief, and belief about belief, are dissimilar, then you are less likely to mix them up and commit the Mind Projection Fallacy—I hope.

Reality has been around since long before you showed up. Don’t go calling it nasty names like “bizarre” or “incredible.” The universe was propagating complex amplitudes through configuration space for ten billion years before life ever emerged on Earth. Quantum physics is not “weird.” You are weird.

“I just don’t understand how a PhD physicist can believe in astrology?” Well, if you literally don’t understand, this indicates a problem with your model of human psychology. Perhaps you are indignant—you wish to express strong moral disapproval. But if you literally don’t understand, then your indignation is stopping you from coming to terms with reality.

People compartmentalize, enough said.

If I genuinely don’t understand how, then my model is being surprised by the facts, and I should discard it and find a better model.

There are no surprising facts, only models that are surprised by facts.

And the reason we don’t think of the alternative explanation “I’m stupid,” is not—I suspect—that we think so highly of ourselves. It’s just that we don’t think of ourselves at all. We just see a chaotic feature of the environment. So now it’s occurred to me that my productivity problem may not be chaos, but my own stupidity.

If I’m going to be happy anywhere, Or achieve greatness anywhere, Or learn true secrets anywhere, Or save the world anywhere, Or feel strongly anywhere, Or help people anywhere, I may as well do it in reality.

“Newsworthy” science is often based on the thinnest of evidence and wrong half the time—if it were not on the uttermost fringes of the scientific frontier, it would not be breaking news.

If you just want to have fun, remember that simplicity is at the core of scientific beauty.

But the really fundamental problem with desiring the unattainable is that as soon as you actually get it, it stops being unattainable. If we cannot take joy in the merely available, our lives will always be frustrated

Sacredness is something intensely private and individual.

the price of shielding yourself from criticism is that you are cast into solitude—the solitude that William James admired as the core of religious experience, as if loneliness were a good thing.

There are questions that must not be asked, and answers that must not be acknowledged, to defend the lie. Thus unanswerability comes to be associated with sacredness. And the price of shielding yourself from criticism is giving up the true curiosity that truly wishes to find answers.

belief without good evidence came to be associated with the experience of the sacred. And the price of shielding yourself from criticism is that you sacrifice your ability to think clearly about that which is sacred, and to progress in your understanding of the sacred, and relinquish mistakes.

Is it obvious that trees can’t think? Trees, let us not forget, are in fact our distant cousins. Go far enough back, and you have a common ancestor with your fern. If lumps of flesh can think, why not lumps of wood?

“You are not the one who speaks your thoughts—you are the one who hears your thoughts.”

Each problem that I solved became a rule which served afterwards to solve other problems. —René Descartes, Discours de la Méthode

Explanations are supposed to make you less confused.

There are no surprising facts, only models that are surprised by facts; and if a model is surprised by the facts, it is no credit to that model.

it’s not you who chooses, it’s your brain.