Rationality: From AI to Zombies

by Eliezer Yudkowsky

Truth can’t be evaluated just by looking inside your own head—if you want to know, for example, whether “the morning star = the evening star,” you need a telescope; it’s not enough just to look at the beliefs themselves.

because we understand that “truth” involves comparing an internal belief to an external fact; so we use an instrument, the telescope, whose perceived behavior we believe to depend on the external fact of the planet.

Not-quite-faith-based reductionism: That-which-we-name “consciousness” happens within physics, in a way not yet understood, just like what happened the last three thousand times humanity ran into something mysterious.

There are times when, as a rationalist, you have to believe things that seem weird to you. Relativity seems weird, quantum mechanics seems weird, natural selection seems weird. But these weirdnesses are pinned down by massive evidence. There’s a difference between believing something weird because science has confirmed it overwhelmingly— —versus believing a proposition that seems downright deranged, because of a great big complicated philosophical argument centered around unspecified miracles and giant blank spots not even claimed to be understood— —in a case where even if you accept everything that has been told to you so far, afterward the phenomenon will still seem like a mystery and still have the same quality of wondrous impenetrability that it had at the start

The rule of the rationalist’s game is that every improbable-seeming belief needs an equivalent amount of evidence to justify it.

The one comes to you and says: “I believe with firm and abiding faith that there’s an object in the asteroid belt, one foot across and composed entirely of chocolate cake; you can’t prove that this is impossible.” But, unless the one had access to some kind of evidence for this belief, it would be highly improbable for a correct belief to form spontaneously.

So either the one can point to evidence, or the belief won’t turn out to be true.

In Follow-The-Improbability, it’s highly suspicious to even talk about a specific hypothesis without having had enough evidence to narrow down the space of possible hypotheses.

Lex parsimoniae: Entia non sunt multiplicanda praeter necessitatem. That was Occam’s original formulation, the law of parsimony: Entities should not be multiplied beyond necessity.

parts whose behavior is defined in motions rather than minds.

If the “boring view” of reality is correct, then you can never predict anything irreducible because you are reducible. You can never get Bayesian confirmation for a hypothesis of irreducibility, because any prediction you can make is, therefore, something that could also be predicted by a reducible thing, namely your brain. Some boxes you really can’t think outside. If our universe really is Turing computable, we will never be able to concretely envision anything that isn’t Turing-computable—no matter how many levels of halting oracle hierarchy our mathematicians can talk about, we won’t be able to predict what a halting oracle would actually say, in such fashion as to experimentally discriminate it from merely computable reasoning.

Of course, that’s all assuming the “boring view” is correct. To the extent that you believe evolution is true, you should not expect to encounter strong evidence against evolution. To the extent you believe reductionism is true, you should expect non-reductionist hypotheses to be incoherent as well as wrong. To the extent you believe supernaturalism is false, you should expect it to be inconceivable as well.

answer: “Of course not.” The irreducibility of the intelligent designer is not an indispensable part of the ID hypothesis. For every irreducible God that can be proposed by the IDers, there exists a corresponding reducible alien that behaves in accordance with the same predictions—since the IDers themselves are reducible. To the extent I believe reductionism is in fact correct, which is a rather strong extent, I must expect to discover reducible formulations of all supposedly supernatural predictive models.

Ultimately, reductionism is just disbelief in fundamentally complicated things. If “fundamentally complicated” sounds like an oxymoron . . . well, that’s why I think that the doctrine of non-reductionism is a confusion, rather than a way that things could be, but aren’t. You would be wise to be wary, if you find yourself supposing such things.

Which just goes to say: The existence of psychic powers is a privileged probabilistic assertion of non-reductionist worldviews—they own that advance prediction; they devised it and put it forth, in defiance of reductionist expectations. So by the laws of science, if psychic powers are discovered, non-reductionism wins. I am therefore confident in dismissing psychic powers as a priori implausible, despite all the claimed experimental evidence in favor of them.

Besides, as a Bayesian, I don’t believe in phenomena that are inherently confusing. Confusion exists in our models of the world, not in the world itself. If a subject is widely known as confusing, not just difficult . . . you shouldn’t leave it at that.

There are no surprising facts, only models that are surprised by facts; and if a model is surprised by the facts, it is no credit to that model.

Now it must be said, in all fairness, that those who say this will usually also confess: But this is not a universally accepted application of Occam’s Razor; some say that Occam’s Razor should apply to the laws governing the model, not the number of objects inside the model.

The original formulation of William of Ockham stated: Lex parsimoniae: Entia non sunt multiplicanda praeter necessitatem. “The law of parsimony: Entities should not be multiplied beyond necessity.”

Robert Heinlein once claimed (tongue-in-cheek, I hope) that the “simplest explanation” is always: “The woman down the street is a witch; she did it.” Eleven words—not many physics papers can beat that.

Imagine a lottery which sells up to a million tickets, where each possible ticket is sold only once, and the lottery has sold every ticket at the time of the drawing. A friend of yours has bought one ticket for $1—which seems to you like a poor investment, because the payoff is only $500,000. Yet your friend says, “Ah, but consider the alternative hypotheses, ‘Tomorrow, someone will win the lottery’ and ‘Tomorrow, I will win the lottery.’ Clearly, the latter hypothesis is simpler by Occam’s Razor; it only makes mention of one person and one ticket, while the former hypothesis is more complicated: it mentions a million people and a million tickets!” To say that Occam’s Razor only counts laws, and not objects, is not quite correct: what counts against a theory are the entities it must mention explicitly, because these are the entities

In large answer spaces, attention without evidence is more than halfway to belief without evidence.

Then go get a 10-sided die from your local gaming store, and, before you begin thinking about that strange world, start rolling the die. If the die comes up 9 twelve times in a row, then you can think about that world. Otherwise don’t waste your time; thought-time is a resource to be expended wisely. You can roll the dice as many times as you like, but you can’t think about the world until 9 comes up twelve times in a row. Then you can think about it for a minute. After that you have to start rolling the die again. This may help you to appreciate the concept of “trillion to one” on a more visceral level.

As you may recall from Semantic Stopsigns, there are words and phrases which are not so much answers to questions, as cognitive traffic signals which indicate you should stop asking questions. “Why does anything exist in the first place? God!” is the classical example, but there are others, such as “Élan vital!”

If there is a moral to the whole story, it is the moral of how very hard it is to stay in a state of confessed confusion, without making up a story that gives you closure—how

The future is always absurd and never unlawful.

Likewise that old chestnut, “Reductionism undermines rationality itself. Because then, every time you said something, it wouldn’t be the result of reasoning about the evidence—it would be merely quarks bopping around.” Of course the actual diagram should be: Or better yet: Why is this not obvious? Because there are many levels of organization that separate our models of our thoughts—our emotions, our beliefs, our agonizing indecisions, and our final choices—from our models of electrons and quarks. We can intuitively visualize that a hand is made of fingers (and thumb and palm). To ask whether it’s really our hand that picks something up, or merely our fingers, thumb, and palm, is transparently a wrong question. But the gap between physics and cognition cannot be crossed by direct visualization. No one can visualize atoms making up a person, the way they can see fingers making up a hand. And so it requires constant vigilance to maintain your perception of yourself as an entity within physics.

But it is dangerous to focus too much on specific hypotheses that you have no specific reason to think about. This is the same root error of the Intelligent Design folk, who pick any random puzzle in modern genetics, and say, “See, God must have done it!” Why “God,” rather than a zillion other possible explanations?—which you would have thought of long before you postulated divine intervention, if not for the fact that you secretly started out already knowing the answer you wanted to find.

But what Special Relativity really says is that human intuitions about space and time are simply wrong. There is no global “now,” there is no “before” or “after” across spacelike gaps. The ability to visualize a single global world, even in principle, comes from not getting Special Relativity on a gut level. Otherwise it would be obvious that physics proceeds locally with invariant states of distant entanglement, and the requisite information is simply not locally present to support a globally single world.

rational guesses in states of partial knowledge,

“It wouldn’t be just one mistake,” Taji corrected her. “As the saying goes: Mistakes don’t travel alone; they hunt in packs.”

Bend not the truth to make your points! I believe your Conspiracy has a phrase: ‘Comparative advantage.’

They were confused, but they had no desperate need for an answer.

Work expands to fill the time allotted, as the saying goes.

The question is not whether you get there eventually! Anyone can find the truth in five thousand years! You need to move faster!”

“Assume, Brennan, that it takes five whole minutes to think an original thought, rather than learning it from someone else. Does even a major scientific problem require 5,760 distinct insights?”

“A woman of wisdom,” Brennan said, “once told me that it is wisest to regard our past selves as fools beyond redemption—to see the people we once were as idiots entire.

Max Planck was even less optimistic:1 A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.

Yet the way that you acquire magical powers is not by being born with them, but by seeing, with a sudden shock, that they really are perfectly normal. This is a general principle in life. *

A theory that can explain any experience corresponds to a hypothesis of complete ignorance—a uniform distribution with probability density spread evenly over every possible outcome.

Imagine that you wake up one morning and your left arm has been replaced by a blue tentacle. The blue tentacle obeys your motor commands—you can use it to pick up glasses, drive a car, etc. How would you explain this hypothetical scenario? Take a moment to ponder this puzzle before continuing. (Spoiler space . . .) How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn’t. It isn’t going to happen. It would be easy enough to produce a verbal explanation that “fit” the hypothetical. There are many explanations that can “fit” anything, including (as a special case of “anything”) my arm’s being replaced by a blue tentacle. Divine intervention is a good all-purpose explanation. Or aliens with arbitrary motives and capabilities. Or I could be mad, hallucinating, dreaming my life away in a hospital. Such explanations “fit” all outcomes equally well, and equally poorly, equating to hypotheses of complete ignorance.

Since the beginning Not one unusual thing Has ever happened.