Rationality: From AI to Zombies

by Eliezer Yudkowsky

There’s a completely different notion of “rationality” studied by mathematicians, psychologists, and social scientists. Roughly, it’s the idea of doing the best you can with what you’ve got.

Now it’s a free country and no one should put you in jail for illegal reasoning,

the facts don’t know whose side they’re on.

A rationalist must understand the mind and how to operate it.

Orwell knew that muddled language is muddled thinking;

Not every change is an improvement, but every improvement is necessarily a change.

Curiosity is the first virtue,

“A science is any discipline in which the fool of this generation can go beyond the point reached by the genius of the last generation.”

Science has heroes, but no gods.

Robert Aumann’s Agreement Theorem shows that honest Bayesians cannot agree to disagree—if they have common knowledge of their probability estimates, they have the same probability estimate.

If you’re interested in the idea, then look at the idea, not the people.

A doubt that is not investigated might as well not exist.

The future cannot cause the past.

eating chocolate is pleasurable,

To categorize is to throw away information.

Reality is very large—just the part we can see is billions of lightyears across.

in order to map a territory, you have to go out and look at the territory.

Mystery exists in the mind, not in reality.

Reality itself does not need to be compared to any beliefs in order to be real.

The coin is not a belief. It is a coin. The territory is not the map.

Reality has been around since long before you showed up.

Quantum physics is not “weird.” You are weird.

to stop worrying about what other people know. If you don’t know the answer, it’s a mystery to you.

But if you do make it to the Future, what you find, when you get there, will be another Now.

to establish a definition is only to promise to use a word consistently; it doesn’t settle any empirical questions,

Confusion exists in our models of the world, not in the world itself.

whether there are any similar clues you’re ignoring now on current mysteries.

But the mind is not magic.

Conversely, if you know the math, you can be as silly as you like, and still technical.

Imperfect maps can conflict, but there is only one territory.

The value of scientific discovery requires both a genuine scientific discovery, and a person to take joy in that discovery.

Valuable things appear because a goal system that values them takes action to create them.

Numbers should come from numbers.

The point is not to have an elegant theory of winning—the point is to win; elegance is a side effect.

The first virtue is curiosity.

Confusion exists in the mind, not in the reality,

For it is only the action that matters, and not the reasons for doing anything.

Don’t try your best. Win, or fail. There is no best.

Doing the impossible should be reserved for very special occasions.

“Shut up and do the impossible!”

Reversed stupidity is not intelligence.

This has not yet been proven, because it is not, in fact, true.