Rationality: From AI to Zombies

by Eliezer Yudkowsky

Study many sciences and absorb their power as your own. Each field that you consume makes you larger. If you swallow enough sciences the gaps between them will diminish and your knowledge will become a unified whole. If you are gluttonous you will become vaster than mountains.

Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory.

Do not ask whether it is “the Way” to do this or that. Ask whether the sky is blue or green. If you speak overmuch of the Way you will not attain it.

“Parents do all the things they tell their children not to do, which is how they know not to do them.”

there really is a reason to be allergic to people who go around saying, “Ah, but technology has risks as well as benefits.” There’s a historical record showing over-conservativeness, the many silent deaths of regulation being outweighed by a few visible deaths of nonregulation. If you’re really playing the middle, why not say, “Ah, but technology has benefits as well as risks”?

Only knowledge can foretell the cost of ignorance. The ancient alchemists had no logical way of knowing the exact reasons why it was hard for them to turn lead into gold. So they poisoned themselves and died. Nature doesn’t care.

think inside the black box.

That’s the problem with converting one big “Oops!” into a gradient of shifting probability. It means there isn’t a single watershed moment—a visible huge impact—to hint that equally huge changes might be in order. Instead, there are all these little opinion shifts . . . that give you a chance to repair the arguments for your strategies; to shift the justification a little, but keep the “basic idea” in place. Small shocks that the system can absorb without cracking, because each time, it gets a chance to go back and repair itself. It’s just that in the domain of rationality, cracking = good, repair = bad. In the art of rationality it’s far more efficient to admit one huge mistake, than to admit lots of little mistakes.

When I finally saw the magnitude of my own folly, everything fell into place at once. The dam against realization cracked; and the unspoken doubts that had been accumulating behind it crashed through all together. There wasn’t a prolonged period, or even a single moment that I remember, of wondering how I could have been so stupid. I already knew how.

I saw that only my own ignorance of the rules had enabled me to argue for going ahead without complete knowledge of the rules; for if you do not know the rules, you cannot model the penalty of ignorance.

Conway’s Life has been proven Turing-complete, so it would be possible to build a sentient being in the Life universe, although it might be rather fragile and awkward.

!?

Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it. Afterward you will still have plenty of flaws left, but that’s not the point; the important thing is to do better, to keep moving ahead, to take one more step forward. Tsuyoku naritai!

Don’t try your best. Win, or fail. There is no best.

Never give up? Don’t be ridiculous. Doing the impossible should be reserved for very special occasions. Learning when to lose hope is an important skill in life. But if there’s something you can imagine that’s even worse than wasting your life, if there’s something you want that’s more important than thirty chips, or if there are scarier things than a life of inconvenience, then you may have cause to attempt the impossible.

To do things that are very difficult or “impossible,” First you have to not run away. That takes seconds. Then you have to work. That takes hours. Then you have to stick at it. That takes years.

Your goal is not to do better, to try desperately, or even to try extraordinarily. Your goal is to get out of the box. To accept this demand creates an awful tension in your mind, between the impossibility and the requirement to do it anyway. People will try to flee that awful tension.

One way to run from the awful tension is to seize on a solution, any solution, even if it’s not very good. Which is why it’s important to go forth with the true intent-to-solve—to have produced a solution, a good solution, at the end of the search, and then to implement that solution and win.

To have faith that you could solve the problem would just be another way of running from that awful tension.

(One of the key Rules For Doing The Impossible is that, if you can state exactly why something is impossible, you are often close to a solution.)

When someone can’t think of any possible argument that would convince them of something, that just means their brain is running a search that hasn’t yet turned up a path. It doesn’t mean they can’t be convinced.

Yes, a group that can’t tolerate disagreement is not rational. But if you tolerate only disagreement—if you tolerate disagreement but not agreement—then you also are not rational. You’re only willing to hear some honest thoughts, but not others. You are a dangerous half-a-rationalist. We are as uncomfortable together as flying-saucer cult members are uncomfortable apart. That can’t be right either. Reversed stupidity is not intelligence.

Even the fear of hell is not a perfect motivator. Human beings are not given so much slack on evolution’s leash; we can resist motivation for a short time, but then we run out of mental energy (hat tip: infotropism). Even believing that you’ll go to hell does not change this brute fact about brain circuitry. So the religious sin, and then are tormented by thoughts of going to hell, in much the same way that smokers reproach themselves for being unable to quit.

Economies of trade and professional specialization are not just vaguely good yet unnatural-sounding ideas, they are the only way that anything ever gets done in this world. Money is not pieces of paper, it is the common currency of caring.

If you know everything you need to know in order to know that you are better off deceiving yourself, it’s much too late to deceive yourself.

Let it be widely believed—and I do believe it, for exactly the same reason I one-box on Newcomb’s Problem—that if people decided as individuals not to be soldiers or if soldiers decided to run away, then that is the same as deciding for the Barbarians to win. By that same theory whereby, if an election is won by 100,000 votes to 99,998 votes, it does not make sense for every voter to say “my vote made no difference.” Let it be said (for it is true) that utility functions don’t need to be solipsistic, and that a rational agent can fight to the death if they care enough about what they’re protecting. Let them not be told that rationalists should expect to lose reasonably.