Rationality: From AI to Zombies

by Eliezer Yudkowsky

Science has heroes, but no gods. The great Names are not our superiors, or even our rivals; they are passed milestones on our road. And the most important milestone is the hero yet to come.

all-female groups (a female subject alongside female confederates) conform significantly more often than all-male groups. Around one-half the women conform more than half the time, versus a third of the men.

when subjects can respond in a way that will not be seen by the group, conformity also drops,

The point is that, for rationalists, disagreeing with the group is serious business. You can’t wave it off with “Everyone is entitled to their own opinion.” I think the most important lesson to take away from Asch’s experiments is to distinguish “expressing concern” from “disagreement.” Raising a point that others haven’t voiced is not a promise to disagree with the group at the end of its discussion.

Once you speak out, you’ve committed a socially irrevocable act; you’ve become the nail sticking up, the discord in the comfortable group harmony, and you can’t undo that. Anyone insulted by a concern you expressed about their competence to successfully complete task XYZ, will probably hold just as much of a grudge afterward if you say “No problem, I’ll go along with the group” at the end.

Asch’s experiment shows that the power of conformity is real. If everyone refrains from voicing their private doubts, that will indeed lead groups into madness. But history abounds with lessons on the price of being the first, or even the second, to say that the Emperor has no clothes.

If you perform the group service of being the one who gives voice to the obvious problems, don’t expect the group to thank you for it. These are the costs and the benefits of dissenting—whether you “disagree” or just “express concern”—and the decision is up to you.

What takes real courage is braving the outright incomprehension of the people around you, when you do something that isn’t Standard Rebellion #37, something for which they lack a ready-made script. They don’t hate you for a rebel, they just think you’re, like, weird, and turn away. This prospect generates a much deeper fear. It’s the difference between explaining vegetarianism and explaining cryonics. There are other cryonicists in the world, somewhere, but they aren’t there next to you. You have to explain it, alone, to people who just think it’s weird.

There are certain people who have no fear of departing the pack. Many fewer such people really exist, than imagine themselves rebels; but they do exist. And yet scientific revolutionaries are tremendously rarer. Ponder that.

But if you think you would totally wear that clown suit, then don’t be too proud of that either! It just means that you need to make an effort in the opposite direction to avoid dissenting too easily. That’s what I have to do, to correct for my own nature. Other people do have reasons for thinking what they do, and ignoring that completely is as bad as being afraid to contradict them. You wouldn’t want to end up as a free thinker. It’s not a virtue, you see—just a bias either way.

Point one: “Cults” and “non-cults” aren’t separated natural kinds like dogs and cats. If you look at any list of cult characteristics, you’ll see items that could easily describe political parties and corporations—“group members encouraged to distrust outside criticism as having hidden motives,” “hierarchical authoritative structure.”

If you look at online arguments over “X is a cult,” “X is not a cult,” then one side goes through an online list of cult characteristics and finds one that applies and says “Therefore it is a cult!” And the defender finds a characteristic that does not apply and says “Therefore it is not a cult!”

smart ideas can have stupid followers.

Say “oops,” and get on with your life.

What is true is already so. Owning up to it doesn’t make it worse. Not being open about it doesn’t make it go away.

And because it’s true, it is what is there to be interacted with. Anything untrue isn’t there to be lived.

Ursula K. LeGuin wrote: “In innocence there is no strength against evil. But there is strength in it for good.”

If in your heart you believe you already know, or if in your heart you do not wish to know, then your questioning will be purposeless and your skills without direction. Curiosity seeks to annihilate itself; there is no curiosity that does not want an answer.

When you surround the enemy Always allow them an escape route. They must see that there is An alternative to death. —Sun Tzu, The Art of War

“Make sure,” I suggested to her, “that you visualize what the world would be like if there are no souls, and what you would do about that. Don’t think about all the reasons that it can’t be that way, just accept it as a premise and then visualize the consequences. So that you’ll think, ‘Well, if there are no souls, I can just sign up for cryonics,’ or ‘If there is no God, I can just go on being moral anyway,’ rather than it being too horrifying to face.

The prospect of losing your job, say, may seem a lot more scary when you can’t even bear to think about it, than after you have calculated exactly how long your savings will last, and checked the job market in your area, and otherwise planned out exactly what to do next. Only then will you be ready to fairly assess the probability of keeping your job in the planned layoffs next month.

Interesting

Be a true coward, and plan out your retreat in detail—visualize every step—preferably before you first come to the battlefield.

Remember that Bayesianism is precise—even if a scary proposition really should seem unlikely, it’s still important to count up all the evidence, for and against, exactly fairly, to arrive at the rational quantitative probability.

Wow

There are ideas which scare me, yet I still believe to be false. There are ideas to which I know I am attached, yet I still believe to be true. But I still plan my retreats, not because I’m planning to retreat, but because planning my retreat in advance helps me think about the problem without attachment.

Main

You shouldn’t be afraid to just visualize a world you fear. If that world is already actual, visualizing it won’t make it worse; and if it is not actual, visualizing it will do no harm. And remember, as you visualize, that if the scary things you’re imagining really are true—which they may not be!—then you would, indeed, want to believe it, and you should visualize that too; not believing wouldn’t help you.

Main

It ain’t a true crisis of faith unless things could just as easily go either way. —Thor Shenkel

It means you can be a world-class scientist and conversant with Bayesian mathematics and still fail to reject a belief whose absurdity a fresh-eyed ten-year-old could see.

Different Things

The first sequence of The Machine in the Ghost, “The Simple Math of Evolution,” aims to communicate the dissonance and divergence between our hereditary history, our present-day biology, and our ultimate aspirations. This will require digging deeper than is common in introductions to evolution for non-biologists, which often restrict their attention to surface-level features of natural selection.

Book 3 main

The third sequence, “A Human’s Guide to Words,” discusses the basic relationship between cognition and concept formation. This is followed by a longer essay introducing Bayesian inference

Third sequence

Bridging the gap between these topics, “Fragile Purposes” abstracts from human cognition and evolution to the idea of minds and goal-directed systems at their most general. These essays serve the secondary purpose of explaining the author’s general approach to philosophy and the science of rationality, which is strongly informed by his work in AI.

Main

Aristotle thought the brain was an organ that cooled the blood. It doesn’t look dangerous.

Lol

Many a loser had been removed from the game, but there was no sign of a winner. Where one species had shells, another species would evolve to crack them; where one species became poisonous, another would evolve to tolerate the poison.

Now explain to me again why an Artificial Intelligence can’t do anything interesting over the Internet unless a human programmer builds it a robot body.

The squishy things - humanity

It’s a serious failure of imagination to think that intelligence is good for so little. Who could have imagined, ever so long ago, what minds would someday do? We may not even know what our real problems are.

If you are temporarily ignorant about a phenomenon, that is a fact about your current state of mind, not a fact about the phenomenon. A blank map does not correspond to a blank territory. If one does not quite understand that power which put footprints on the Moon, nonetheless, the footprints are still there—real footprints, on a real Moon, put there by a real power.

Intelligence is as real as electricity. It’s merely far more powerful, far more dangerous, has far deeper implications for the unfolding story of life in the universe—and it’s a tiny little bit harder to figure out how to build a generator.

Foxes seem well-designed to catch rabbits. Rabbits seem well-designed to evade foxes. Was the Creator having trouble making up Its mind? When I design a toaster oven, I don’t design one part that tries to get electricity to the coils and a second part that tries to prevent electricity from getting to the coils. It would be a waste of effort.

The ecosystem would make much more sense if it wasn’t designed by a unitary Who, but, rather, created by a horde of deities—say from the Hindu or Shinto religions. This handily explains both the ubiquitous purposefulnesses, and the ubiquitous conflicts: More than one deity acted, often at cross-purposes. The fox and rabbit were both designed, but by distinct competing deities.

Interesting

Similarly, the Judeo-Christian God is alleged to be benevolent—well, sort of. And yet much of nature’s purposefulness seems downright cruel.

Exactly

The strength of a theory is not what it allows, but what it prohibits; if you can invent an equally persuasive explanation for any outcome, you have zero knowledge.

Wow

The key realization is that there is no Evolution Fairy. There’s no outside force deciding which genes ought to be promoted. Whatever happens, happens because of the genes themselves. Genes for constructing (incrementally better) rattles must have somehow ended up more frequent in the rattlesnake gene pool, because of the rattle. In this case it’s probably because rattlesnakes with better rattles survive more often—rather than mating more successfully, or having brothers that reproduce more successfully, etc.

The main point is that the gene’s effect must cause copies of that gene to become more frequent in the next generation. There’s no Evolution Fairy that reaches in from outside. There’s nothing which decides that some genes are “helpful” and should, therefore, increase in frequency. It’s just cause and effect, starting from the genes themselves.

Main

Fox genes are becoming more or less frequent in fox populations. Fox genes which construct foxes that catch rabbits, insert more copies of themselves in the next generation. Rabbit genes which construct rabbits that evade foxes are naturally more common in the next generation of rabbits. Hence the phrase “natural selection.”

Main

We are simply the embodied history of which organisms did in fact survive and reproduce, not which organisms ought prudentially to have survived and reproduced.

If you turn around the retina’s cells without also reprogramming the nerves and optic cable, the system as a whole won’t work. It doesn’t matter that, to a Fairy or a human engineer, this is one step forward in redesigning the retina. The organism is blind. Evolution has no foresight, it is simply the frozen history of which organisms did in fact reproduce. Evolution is as blind as a halfway-redesigned retina.

Wow

evolution is not God, but it is closer to God than it is to pure random entropy. Mutation is random, but selection is non-random. This doesn’t mean an intelligent Fairy is reaching in and selecting. It means there’s a non-zero statistical correlation between the gene and how often the organism reproduces. Over a few million years, that non-zero statistical correlation adds up to something very powerful. It’s not a god, but it’s more closely akin to a god than it is to snow on a television screen.

the Shaper of Life is not itself a creature. Evolution is bodiless,

In a way, Darwin discovered God—a God that failed to match the preconceptions of theology, and so passed unheralded. If Darwin had discovered that life was created by an intelligent agent—a bodiless mind that loves us, and will smite us with lightning if we dare say otherwise—people would have said “My gosh! That’s God!”