Rationality: From AI to Zombies

by Eliezer Yudkowsky

The diametric opposite of this advice, which sends the halo effect supercritical, is when it feels wrong to argue against any positive claim about the Great Idea. Politics is the mind-killer. Arguments are soldiers. Once you know which side you’re on, you must support all favorable claims, and argue against all unfavorable claims. Otherwise it’s like giving aid and comfort to the enemy, or stabbing your friends in the back.

There are things in this world that are worth praising greatly, and you can’t flatly say that praise beyond a certain point is forbidden. But there is never an Idea so true that it’s wrong to criticize any argument that supports it. Never. Never ever never for ever. That is flat. The vast majority of possible beliefs in a nontrivial answer space are false, and likewise, the vast majority of possible supporting arguments for a true belief are also false, and not even the happiest idea can change that.

And it is triple ultra forbidden to respond to criticism with violence. There are a very few injunctions in the human art of rationality that have no ifs, ands, buts, or escape clauses. This is one of them. Bad argument gets counterargument.

A mere factor of “ten times worse” turned out to be a vast understatement. Even I didn’t guess how badly things would go. That’s the challenge of pessimism; it’s really hard to aim low enough that you’re pleasantly surprised around as often and as much as you’re unpleasantly surprised.

Anyone who attacks the Enemy is a patriot; and whoever tries to dissect even a single negative claim about the Enemy is a traitor. But just as the vast majority of all complex statements are untrue, the vast majority of negative things you can say about anyone, even the worst person in the world, are untrue.

Any resemblance to modern politics is just your imagination. (Sometimes I think humanity’s second-greatest need is a supervillain. Maybe I’ll go into that line of work after I finish my current job.)

Yes, there are effectively certain truths of science. General Relativity may be overturned by some future physics—albeit not in any way that predicts the Sun will orbit Jupiter; the new theory must steal the successful predictions of the old theory, not contradict them.

one of the factors has to do with assuming a productive posture toward the truth, versus a defensive posture toward the truth. When you are the Guardian of the Truth, you’ve got nothing useful to contribute to the Truth but your guardianship of it.

When you are a Guardian of the Truth, all you can do is try to stave off the inevitable slide into entropy by zapping anything that departs from the Truth.

But this whole trick only works because the experimental method is a “criterion of goodness” which is not a mere “criterion of comparison.” Because experiments can recover the truth without need of authority, they can also override authority and create new true beliefs where none existed before.

I do mean to point out a deep psychological difference between seeing your grand cause in life as protecting, guarding, preserving, versus discovering, creating, improving. Does the “up” direction of time point to the past or the future? It’s a distinction that shades everything, casts tendrils everywhere.

It is a lesson in what happens when the truth becomes more important than the search for truth

Science moves forward by slaying its heroes, as Newton fell to Einstein. Every young physicist dreams of being the new champion that future physicists will dream of dethroning.

To me the thought of voluntarily embracing a system explicitly tied to the beliefs of one human being, who’s dead, falls somewhere between the silly and the suicidal. A computer isn’t five years old before it’s obsolete.

Rand grew up in Russia, and witnessed the Bolshevik revolution firsthand. She was granted a visa to visit American relatives at the age of 21, and she never returned. It’s easy to hate authoritarianism when you’re the victim. It’s easy to champion the freedom of the individual, when you are yourself the oppressed. It takes a much stronger constitution to fear authority when you have the power. When people are looking to you for answers, it’s harder to say “What the hell do I know about music? I’m a writer, not a composer,” or “It’s hard to see how liking a piece of music can be untrue.”

When you’re the one crushing those who dare offend you, the exercise of power somehow seems much more justifiable than when you’re the one being crushed. All sorts of excellent justifications somehow leap to mind.

Newton couldn’t realistically have discovered any of the ideas I’m lording over him—but progress isn’t fair! That’s the point!

Science has heroes, but no gods. The great Names are not our superiors, or even our rivals; they are passed milestones on our road.

To be one more milestone in humanity’s road is the best that can be said of anyone; but this seemed too lowly to please Ayn Rand. And that is...

The scary thing about Asch’s conformity experiments is that you can get many people to say black is white, if you put them in a room full of other people saying the same thing. The hopeful thing about Asch’s conformity experiments is that a single dissenter tremendously drove down the rate of conformity, even if the dissenter was only giving a different wrong answer.

Lonely dissent doesn’t feel like going to school dressed in black. It feels like going to school wearing a clown suit. That’s the difference between joining the rebellion and leaving the pack.

This is the true courage of lonely dissent, which every damn rock band out there tries to fake.

There are a few genuine packs of iconoclasts around. The Church of the SubGenius, for example, seems to genuinely aim at confusing the mundanes, not merely offending them. And there are islands of genuine tolerance in the world, such as science fiction conventions. There are certain people who have no fear of departing the pack. Many fewer such people really exist, than imagine themselves rebels; but they do exist.

the purpose of every doubt is to annihilate itself in success or failure, and a doubt that just hangs around accomplishes nothing.

Super Happy Agent (an agent defined primarily by agreeing with any nice thing said about it).

All curiosity seeks to annihilate itself; there is no curiosity that does not want an answer.

I now try to avoid using the English idiom “I just don’t understand how . . .” to express indignation. If I genuinely don’t understand how, then my model is being surprised by the facts, and I should discard it and find a better model.

Surprise exists in the map, not in the territory. There are no surprising facts, only models that are surprised by facts. Likewise for facts called such nasty names as “bizarre,” “incredible,” “unbelievable,” “unexpected,” “strange,” “anomalous,” or “weird.” When you find yourself tempted by such labels, it may be wise to check if the alleged fact is really factual. But if the fact checks out, then the problem isn’t the fact—it’s you.

As the saying goes, “The map is not the territory, but you can’t fold up the territory and put it in your glove compartment.” Sometimes you need a smaller map to fit in a more cramped glove compartment—but this does not change the territory. The scale of a map is not a fact about the territory, it’s a fact about the map.

“That which can be destroyed by the truth should be,”

I’ve written before on seeming knowledge that is not knowledge, and beliefs that are not about their supposed objects but only recordings to recite back in the classroom, and words that operate as stop signs for curiosity rather than answers, and technobabble that only conveys membership in the literary genre of “science”

For them, the effect of hearing “Science has explained rainbows!” is to hang up a sign over rainbows saying, “This phenomenon has been labeled BORING by order of the Council of Sophisticated Literary Critics. Move along.”

The vastness of the heavens stretches my imagination—stuck on this carousel my little eye can catch one-million-year-old light. A vast pattern—of which I am a part—perhaps my stuff was belched from some forgotten star, as one is belching there. Or see them with the greater eye of Palomar, rushing all apart from some common starting point when they were perhaps all together. What is the pattern, or the meaning, or the why? It does not do harm to the mystery to know a little about it.

Your choice is either: Decide that things are allowed to be unmagical, knowable, scientifically explicable—in a word, real—and yet still worth caring about; Or go about the rest of your life suffering from existential ennui that is unresolvable.

When that thought first occurred to me, I actually found it quite uplifting. Once I realized that someone, somewhere in the expanses of space and time, already knows the answer to any answerable question—even biology questions and history questions; there are other decoherent Earths—then I realized how silly it was to think as if the joy of discovery ought to be limited to one person. It becomes a fully inescapable source of unresolvable existential angst, and I regard that as a reductio.

The consistent solution which maintains the possibility of fun is to stop worrying about what other people know. If you don’t know the answer, it’s a mystery to you.

Sure, someone else knows the answer—but back in the hunter-gatherer days, someone else in an alternate Earth, or for that matter, someone else in the future, knew what the answer was. Mystery, and the joy of finding out, is either a personal thing, or it doesn’t exist at all—and I prefer to say it’s personal.

It’s not dreams I object to, only impossible dreams. The lottery isn’t impossible, but it is an un-actionable near-impossibility. It’s not that winning the lottery is extremely difficult—requires a desperate effort—but that work isn’t the issue.

The worst catastrophe you could visit upon the New Age community would be for their rituals to start working reliably, and for UFOs to actually appear in the skies. What would be the point of believing in aliens, if they were just there, and everyone else could see them too? In a world where psychic powers were merely real, New Agers wouldn’t believe in psychic powers, any more than anyone cares enough about gravity to believe in it.

So remember the Litany Against Being Transported Into An Alternate Universe: If I’m going to be happy anywhere, Or achieve greatness anywhere, Or learn true secrets anywhere, Or save the world anywhere, Or feel strongly anywhere, Or help people anywhere, I may as well do it in reality.

Don’t be embarrassed to read elementary science textbooks, either. If you want to pretend to be sophisticated, go find a play to sneer at. If you just want to have fun, remember that simplicity is at the core of scientific beauty.

Also, reversed stupidity is not intelligence. The world’s greatest idiot may say the Sun is shining, but that doesn’t make it dark out.

(Timothy Ferriss offers similar advice on planning your life: ask which ongoing experiences would make you happy, rather than which possessions or status-changes.)

And the price of shielding yourself from criticism is that you sacrifice your ability to think clearly about that which is sacred, and to progress in your understanding of the sacred, and relinquish mistakes.

Such distortions are why we had best not to try to salvage religion. No, not even in the form of “spirituality.” Take away the institutions and the factual mistakes, subtract the churches and the scriptures, and you’re left with . . . all this nonsense about mysteriousness, faith, solipsistic experience, private solitude, and discontinuity.

to just look up at the distant stars. Believable without strain, without a constant distracting struggle to fend off your awareness of the counterevidence.

With science, I think, people assume that if the information is freely available, it must not be important. So instead people join cults that have the sense to keep their Great Truths secret. The Great Truth may actually be gibberish, but it’s more satisfying than coherent science, because it’s secret.

And if you tried to start a cult around oh, say, Scientology, you’d get some degree of public interest, at first. But people would very quickly start asking uncomfortable questions like “Why haven’t you given a public demonstration of your Eighth Level powers, like the Physicists?” and “How come none of the Master Mathematicians seem to want to join your cult?” and “Why should I follow your Founder when they aren’t an Eighth Level anything outside their own cult?” and “Why should I study your cult first, when the Dentists of Doom can do things that are so much more impressive?”

The physics supposedly isn’t secret (even though you don’t know), and there’s a one-paragraph explanation in the newspaper that sounds vaguely authoritative and convincing—essentially, no one treats the lightbulb as a sacred mystery, so neither do you. Even the simplest little things, completely inert objects like crucifixes, can become magical if everyone looks at them like they’re magic. But since you’re theoretically allowed to know why the light bulb works without climbing the mountain to find the remote Monastery of Electricians, there’s no need to actually bother to learn.

in their full-fledged human form, our moral adaptations are the result of selection pressures over linguistic arguments about tribal politics.”