Rationality: From AI to Zombies

by Eliezer Yudkowsky

shut up and multiply!

When you see people insisting that no amount of money whatsoever is worth a single human life, and then driving an extra mile to save $10; or when you see people insisting that no amount of money is worth a decrement of health, and then choosing the cheapest health insurance available; then you don’t think that their protestations reveal some deep truth about incommensurable utilities.

If the ends don’t justify the means, what does? —variously attributed

Would you kill babies if it was the right thing to do? If no, under what circumstances would you not do the right thing to do? If yes, how right would it have to be, for how many babies? —horrible job interview question

Hobbes said, “I don’t know what’s worse, the fact that everyone’s got a price, or the fact that their price is so low.”

In Western comics, the magic comes first, then the purpose: Acquire amazing powers, decide to protect the innocent. In Japanese fiction, often, it works the other way around.

Historically speaking, science won because it displayed greater raw strength in the form of technology, not because science sounded more reasonable. To this very day, magic and scripture still sound more reasonable to untrained ears than science. That is why there is continuous social tension between the belief systems. If science not only worked better than magic, but also sounded more intuitively reasonable, it would have won entirely by now.

I like the complexity of simultaneously having to love True-seeming ideas, and also being ready to drop them out the window at a moment’s notice.

morality and aesthetics alone, believing that one ought to be “rational” or that certain ways of thinking are “beautiful,” will not lead you to the center of the Way. It wouldn’t have gotten humanity out of the authority-hole.

But if your precious daughter is one of the 500, and you don’t know which one, then, perhaps, you may feel more impelled to shut up and multiply—to notice that you have an 80% chance of saving her in the first case, and a 90% chance of saving her in the second.

Lots of people think that “rationality” is about choosing only methods that are certain to work, and rejecting all uncertainty. But, hopefully, you care more about your daughter’s life than about “rationality.”

Will pride in your own virtue as a rationalist save you? Not if you believe that it is virtuous to choose certainty. You will only be able to learn something about rationality if your daughter’s life matters more to you than your pride as a rationalist.

Only when you become more wedded to success than to any of your beloved techniques of rationality do you begin to appreciate these words of Miyamoto Musashi:1 You can win with a long weapon, and yet you can also win with a short weapon. In short, the Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size. —Miyamoto Musashi, The Book of Five Rings

Don’t mistake this for a specific teaching of rationality. It describes how you learn the Way, beginning with a desperate need to succeed. No one masters the Way until more than their life is at stake. More than their comfort, more even than their pride.

You can’t just pick out a Cause like that because you feel you need a hobby. Go looking for a “good cause,” and your mind w...

But if you have a cause like that, it is right and proper to wield your rationality in its service.

There is nothing privileged about uncertainty that is expressed in words, unless the verbal parts of your brain do, in fact, happen to work better on the problem.

The goal is to win, after all.

“If you’re so rational, why ain’cha rich?”

But at any rate, WIN. Don’t lose reasonably; WIN.

Unreasonable? I am a rationalist: what do I care about being unreasonable? I don’t have to conform to a particular ritual of cognition. I don’t have to take only box B because I believe my choice affects the box, even though Omega has already left. I can just . . . take only box B.

The point is not to have an elegant theory of winning—the point is to win; elegance is a side effect.

Rather than starting with a concept of what is the reasonable decision, and then asking whether “reasonable” agents leave with a lot of money, start by looking at the agents who leave with a lot of money, develop a theory of which agents tend to leave with the most money, and from this theory, try to figure out what is “reasonable.” “Reasonable” may just refer to decisions in conformance with our current ritual of cognition—what else would determine whether something seems “reasonable” or not?

Watch the replay. Study and learn.

You shouldn’t claim to be more rational than someone and simultaneously envy them their choice—only their choice. Just do the act you envy.

If it ever turns out that Bayes fails—receives systematically lower rewards on some problem, relative to a superior alternative, in virtue of its mere decisions—then Bayes has to go out the window.

The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him.

If you regard evidence as a constraint and seek to free yourself, you sell yourself into the chains of your whims. For you cannot make a true map of a city by sitting in your bedroom with your eyes shut and drawing lines upon paper according to impulse.

You must walk through the city and draw lines on paper that correspond to what you see. If, seeing the city unclearly, you think that you can shift a line just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.

If you first write at the bottom of a sheet of paper “And therefore, the sky is green!” it does not matter what arguments you write above it afterward; the conclusion is already written, and it is already correct or already wrong. To be clever in argument is not rationality but rationalization. Intelligence, to be useful, must be used for something other than defeating itself.

Do not think that fairness to all sides means balancing yourself evenly between positions; truth is not handed out in equal portions before the start of a debate. You cannot move forward on factual questions by fighting with fists or insults. Seek a test that lets reality judge between you.

The roots of knowledge are in observation and its fruit is prediction.

Who are most humble? Those who most skillfully prepare for the deepest and most catastrophic errors in their own beliefs and plans.

But it is useless to be superior: Life is not graded on a curve. The best physicist in ancient Greece could not calculate the path of a falling apple. There is no guarantee that adequacy is possible given your hardest effort; therefore spare no thought for whether others are doing worse. If you compare yourself to others you will not see the biases that all humans share.

wow

The more errors you correct in yourself, the more you notice. As your mind becomes more silent, you hear more noise. When you notice an error in yourself, this signals your readiness to seek advancement to the next level. If you tolerate the error rather than correcting it, you will not advance to the next level and you will not gain the skill to notice new errors.

Hold yourself to the highest standard you can imagine, and look for one still higher. Do not be content with the answer that is almost right; seek one that is exactly right.

If for many years you practice the techniques and submit yourself to strict constraints, it may be that you will glimpse the center. Then you will see how all techniques are one technique, and you will move correctly without feeling constrained.

Musashi wrote: “When you appreciate the power of nature, knowing the rhythm of any situation, you will be able to hit the enemy naturally and strike naturally. All this is the Way of the Void.”

intelligence wasn’t supposed to be simple, it wasn’t supposed to have an answer that fit on a T-shirt. It was supposed to be a big puzzle with lots of pieces; and when you found one piece, you didn’t run off holding it high in triumph, you kept on looking.

What you actually end up doing screens off the clever reason why you’re doing it.

Do nothing because it is righteous, or praiseworthy, or noble, to do so; do nothing because it seems good to do so; do only that which you must do, and which you cannot do in any other way.

Interesting

In Orthodox Judaism there is a saying: “The previous generation is to the next one as angels are to men; the next generation is to the previous one as donkeys are to men.”

The most important thing is that there should be progress. So long as you keep moving forward you will reach your destination; but if you stop moving you will never reach it.

You might say it when expressing your determination to become a professional Go player—or after you lose an important match, but you haven’t given up—or after you win an important match, but you’re not a ninth-dan player yet—or after you’ve become the greatest Go player of all time, but you still think you can do better. That is tsuyoku naritai, the will to transcendence.

By the same token, the Ashamnu does not end, “But that was this year, and next year I will do better.” The Ashamnu bears a remarkable resemblance to the notion that the way of rationality is to beat your fist against your heart and say, “We are all biased, we are all irrational, we are not fully informed, we are overconfident, we are poorly calibrated . . .” Fine. Now tell me how you plan to become less biased, less irrational, more informed, less overconfident, better calibrated.

I’ll always run as fast as I can, even if I pull ahead, I’ll keep on running; and someone, someday, will surpass me; but even though I fall behind, I’ll always run as fast as I can.

No! Try not! Do, or do not. There is no try. —Yoda

You have been asking what you could do in the great events that are now stirring, and have found that you could do nothing. But that is because your suffering has caused you to phrase the question in the wrong way . . . Instead of asking what you could do, you ought to have been asking what needs to be done. —Steven Brust, The Paths of the Dead1

When there’s a will to fail, obstacles can be found. —John McCarthy

Mark: “The audience isn’t going to buy it.” George: “Trust me, they will.” Mark: “They’re going to get up and walk out of the theater.” George: “They’re going to sit there and nod along and not notice anything out of the ordinary. Look, you don’t understand human nature. People wouldn’t try for five minutes before giving up if the fate of humanity were at stake.”

“Persevere.” It’s a piece of advice you’ll get from a whole lot of high achievers in a whole lot of disciplines. I didn’t understand it at all, at first.