Rationality: From AI to Zombies

by Eliezer Yudkowsky

If you can find within yourself the slightest shred of true uncertainty, then guard it like a forester nursing a campfire. If you can make it blaze up into a flame of curiosity, it will make you light and eager, and give purpose to your questioning and direction to your skills.

if you can invent an equally persuasive explanation for any outcome, you have zero knowledge.

To write a culture that isn’t just like your own culture, you have to be able to see your own culture as a special case—not as a norm which all other cultures must take as their point of departure.

as a result of skills passed on through language and writing).

also demonstration

because we can more persuasively argue for what we honestly believe, we have evolved an instinct to honestly believe that other people’s goals, and our tribe’s moral code, truly do imply that they should do things our way for their benefit.

Whenever you feel tempted to say the words “by definition” in an argument that is not literally about pure mathematics, remember that anything which is true “by definition” is true in all possible worlds, and so observing its truth can never constrain which world you live in.

Dictionary editors are historians of usage, not legislators of language.

If the question is how to cluster together similar things for purposes of inference, empirical predictions will depend on the answer; which means that definitions can be wrong. A conflict of predictions cannot be settled by an opinion poll.

Purpose is lost whenever the substance (learning, knowledge, health) is displaced by the symbol (a degree, a test score, medical care).

’Tis easier to question one’s facts than one’s ontology.

You can always be wrong. Even when it’s theoretically impossible to be wrong, you can still be wrong.

Dictionary editors are historians of usage, not legislators of language.

Q. Why did the Bayesian reasoner cross the road? A. You need more information to answer this question.

Strong evidence is not the product of a very high probability that A leads to X, but the product of a very low probability that not-A could have led to X.

If you wanted to put it poetically, you could say that Bayes’s Theorem binds reasoning into the physical universe.

We are not logically omniscient; we cannot see all the implications of our thoughts; we do not know what we believe.

At least one of the two Old Style methods must discard relevant information—or simply do the wrong calculation—for the two methods to arrive at different answers.

“I know that what you say can’t possibly be true, and I can prove it. But I cannot write out a flowchart which shows how your brain makes the mistake, so I’m not done yet, and will continue investigating.”

Unanswerable questions do not mark places where magic enters the universe. They mark places where your mind runs skew to reality.

Similarly, the notion of truth is quite different from the notion of reality. Saying “true” compares a belief to reality. Reality itself does not need to be compared to any beliefs in order to be real. Remember this the next time someone claims that nothing is true.

Surprise exists in the map, not in the territory. There are no surprising facts, only models that are surprised by facts.

Science did not actually—as Keats’s poem itself would have it—take real Angel’s wings, and destroy them with a cold touch of truth. In reality there never were any haunts in the air, or gnomes in the mine.

Scientists didn’t do anything to gnomes, only to “gnomes.” The quotation is not the referent.

our moral adaptations are the result of selection pressures over linguistic arguments about tribal politics.”

“The fact that one apple added to one apple invariably gives two apples helps in the teaching of arithmetic, but has no bearing on the truth of the proposition that 1 + 1 = 2.”

I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies “belief,” and the latter thingy “reality.”

The map is multilevel, the territory is single-level. This doesn’t mean that the higher levels “don’t exist,”

“You are not the one who speaks your thoughts—you are the one who hears your thoughts.”

A “supernatural” explanation appeals to ontologically basic mental things, mental entities that cannot be reduced to nonmental entities.

Supernaturalism is a special case of non-reductionism, where it is not 747s that are irreducible, but just (some) mental things. Religion is a special case of supernaturalism, where the irreducible mental things are God(s) and souls; and perhaps also sins, angels, karma, etc.

religions have ignored the discovery of that ancient bodiless thing: omnipresent in the working of Nature and immanent in every falling leaf; vast as a planet’s surface and billions of years old; itself unmade and arising from the structure of physics; designing without brain to shape all life on Earth and the minds of humanity.

Quantum mechanics is counterintuitive, but that is a problem with your intuitions, not a problem with quantum mechanics.

Quantum mechanics was here before you were, and if you have a problem with that, you are the one who needs to change.

There are no surprising facts, only models that are surprised by facts; and if a model is surprised by the facts, it is no credit to that model.

It is always best to think of reality as perfectly normal. Since the beginning, not one unusu...

WHAT DOES THE GOD-DAMNED COLLAPSE POSTULATE HAVE TO DO FOR PHYSICISTS TO REJECT IT? KILL A GOD-DAMNED PUPPY?

Mistakes don’t travel alone; they hunt in packs.”

A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.

Einstein wasn’t finding an equation that covered the motion of gravitational bodies. Einstein was finding a character-of-physical-law that covered previously observed equations, and that he could crank to predict the next equation that would be observed.

Trying to beat low-capitalization prediction markets might make for good training in this?—though that is only speculation.

In the art of rationality, to explain is to anticipate. To anticipate is to explain.

Value theory is the study of what people care about. It’s the study of our goals, our tastes, our pleasures and pains, our fears and our ambitions.

We find it useful to reflect upon and debate our values because how we act is not always how we wish we’d act. Our preferences can conflict with each other. We can desire to have a different set of desires. We can lack the will, the attention, or the insight needed to act the way we’d like to.

When you think about a two-place function as though it were a one-place function, you end up with a Variable Question Fallacy / Mind Projection Fallacy. Like trying to determine whether a building is intrinsically on the left or on the right side of the road, independent of anyone’s travel direction.

Who is the most formidable, among the human kind? The strongest? The smartest? More often than either of these, I think, it is the one who can call upon the most friends.

Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals will contain almost nothing of worth.

Perhaps, as you say, there is no surprise from a causal viewpoint—no disruption of the physical order of the universe. But it still seems to me that, in this creation of humans by evolution, something happened that is precious and marvelous and wonderful. If we cannot call it a physical miracle, then call it a moral miracle.

I see the project of morality as a project of renormalizing intuition. We have intuitions about things that seem desirable or undesirable, intuitions about actions that are right or wrong, intuitions about how to resolve conflicting intuitions, intuitions about how to systematize specific intuitions into general principles.

Delete all the intuitions, and you aren’t left with an ideal philosopher of perfect emptiness; you’re left with a rock.

“Intuition,” as a term of art, is not a curse word when it comes to morality—there is nothing else to argue from. Even modus ponens is an “intuition” in this sense—it’s just that modus ponens still seems like a good idea after being formalized, reflected on, extrapolated out to see if it has sensible consequences, et cetera.