Rationality: From AI to Zombies

by Eliezer Yudkowsky

Epistemic rationality: systematically improving the accuracy of your beliefs.

Instrumental rationality: systematically achieving your values.

Epistemic rationality is about building accurate maps

Instrumental rationality, on the other hand, is about steering reality—sending the future where you want it to go.

“Washing one’s hands of the conflict between the powerful and the powerless means to side with the powerful, not to be neutral.”1

when the rational conclusion is to suspend judgment about an issue, all too many people instead conclude that any judgment is as plausible as any other.

Rational thought produces beliefs which are themselves evidence.

Your strength as a rationalist is your ability to be more confused by fiction than by reality. If you are equally good at explaining any outcome, you have zero knowledge.

for every expectation of evidence, there is an equal and opposite expectation of counterevidence.

The measure of your strength as a rationalist is your ability to be more confused by fiction than by reality.

Hindsight will lead us to systematically undervalue the surprisingness of scientific findings, especially the discoveries we understand—the ones that seem real to us, the ones we can retrofit into our models of the world.

the substance of a model is the control it exerts on anticipation.

Words do not have intrinsic definitions.

how many people, especially in the media, understand science only as a literary genre.

You had best ask yourself which future experiences your belief prohibits from happening to you. That is the sum of what you have assimilated and made a true part of yourself. Anything else is probably passwords or attire.

To worship a phenomenon because it seems so wonderfully mysterious is to worship your own ignorance.

error of encapsulating the mystery as a substance.

If a phenomenon feels mysterious, that is a fact about our state of knowledge, not a fact about the phenomenon itself.

Mystery is a property of questions, not answers.

“How would I regenerate this knowledge if it were deleted from my mind?”

Some psychologists these days go so far as to hypothesize that our ability to come up with explicit justifications for our conclusions evolved specifically to help us win arguments.

Given some background knowledge (priors) and a new piece of evidence, probability theory uniquely defines the best set of new beliefs (posterior) I could adopt. Likewise, decision theory defines what action I should take based on my beliefs.

“Faced with the choice of changing one’s mind and proving that there is no need to do so, almost everyone gets busy on the proof.”

“To be humble is to take specific actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty.”

If science is based on “faith,” then science is of the same kind as religion—directly comparable.

When people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together.

If what you believe doesn’t depend on what you see, you’ve been blinded as effectively as by poking out your eyeballs.

Maybe I could be legitimately sure about something. But once I assign a probability of 1 to a proposition, I can never undo it. No matter what I see or learn, I have to reject everything that disagrees with the axiom. I don’t like the idea of not being able to change my mind, ever.

When you work in log odds, the distance between any two degrees of uncertainty equals the amount of evidence you would need to go from one to the other.

Politics is an extension of war by other means. Arguments are soldiers. Once you know which side you’re on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it’s like stabbing your soldiers in the back—providing aid and comfort to the enemy.

there is also a human tendency to deny all costs of a favored policy, or deny all benefits of a disfavored policy; and people will therefore tend to think policy tradeoffs are tilted much further than they actually are.

you have mixed up logically distinct questions—treated facts like human soldiers on different sides of a war, thinking that any soldier on one side can be used to fight any soldier on the other side.

To understand why people act the way they do, we must first realize that everyone sees themselves as behaving normally.

Everyone is the hero of their own story.

Passive voice removes the actor, leaving only the acted-upon.

Nonfiction conveys knowledge, fiction conveys experience.

In all human history, every great leap forward has been driven by a new clarity of thought.

Politically knowledgeable subjects, because they possess greater ammunition with which to counter-argue incongruent facts and arguments, will be more prone to the above biases.

The problem with using black-and-white, binary, qualitative reasoning is that any single observation either destroys the theory or it does not.

For every expectation of evidence, there is an equal and opposite expectation of counterevidence.

You cannot obtain more truth for a fixed proposition by arguing it; you can make more people believe it, but you cannot make it more true.

What is true is already so. Owning up to it doesn’t make it worse. Not being open about it doesn’t make it go away. And because it’s true, it is what is there to be interacted with. Anything untrue isn’t there to be lived. People can stand what is true, for they are already enduring it. —Eugene Gendlin

A motivated skeptic asks if the evidence compels them to accept the conclusion; a motivated credulist asks if the evidence allows them to accept the conclusion.

“Promoting less than maximally accurate beliefs is an act of sabotage. Don’t do it to anyone unless you’d also slash their tires.”

It’s a good guess that the actual majority of human cognition consists of cache lookups.

Fame, in particular, seems to combine additively with all other personality characteristics.

The police officer who puts their life on the line with no superpowers, no X-Ray vision, no super-strength, no ability to fly, and above all no invulnerability to bullets, reveals far greater virtue than Superman—who is a mere superhero.

New Agers culturally inherit the cached thought that faith is positive, but lack Christianity’s exclusionary scripture to keep out competing memes. New Agers end up in happy death spirals around stars, trees, magnets, diets, spells, unicorns . . .

Every group of people with an unusual goal—good, bad, or silly—will trend toward the cult attractor unless they make a constant effort to resist it.

Cultishness is a characteristic of groups more than hypotheses.