Rationality: From AI to Zombies

by Eliezer Yudkowsky

Traditional Rationality is phrased in terms of social rules, with violations interpretable as cheating—as defections from cooperative norms. If you want me to accept a belief from you, you are obligated to provide me with a certain amount of evidence. If you try to get out of it, we all know you’re cheating on your obligation. A theory is obligated to make bold predictions for itself, not just steal predictions that other theories have labored to make. A theory is obligated to expose itself to falsification—if it tries to duck out, that’s like trying to duck out of a fearsome initiation ritual; you must pay your dues.

Social rules of rationality.

To Bayesians, the brain is an engine of accuracy: it processes and concentrates entangled evidence into a map that reflects the territory. The principles of rationality are laws in the same sense as the Second Law of Thermodynamics: obtaining a reliable belief requires a calculable amount of entangled evidence, just as reliably cooling the contents of a refrigerator requires a calculable minimum of free energy.

The answer to theists and science deniers who complain science is beliefs too.

Not every doubt calls for staging an all-out Crisis of Faith. But you should consider it when: A belief has long remained in your mind; It is surrounded by a cloud of known arguments and refutations; You have sunk costs in it (time, money, public declarations); The belief has emotional consequences (note this does not make it wrong); It has gotten mixed up in your personality generally.

Warning signs that a cherished belief .may be wrong.

But meanwhile, because it’s hard to see how one process could have such diverse powers, it’s hard to imagine that one fell swoop could solve even such prosaic problems as obesity and cancer and aging. Well, one trick cured smallpox and built airplanes and cultivated wheat and tamed fire. Our current science may not agree yet on how exactly the trick works, but it works anyway. If you are temporarily ignorant about a phenomenon, that is a fact about your current state of mind, not a fact about the phenomenon. A blank map does not correspond to a blank territory. If one does not quite understand that power which put footprints on the Moon, nonetheless, the footprints are still there—real footprints, on a real Moon, put there by a real power. If one were to understand deeply enough, one could create and shape that power. Intelligence is as real as electricity. It’s merely far more powerful, far more dangerous, has far deeper implications for the unfolding story of life in the universe—and it’s a tiny little bit harder to figure out how to build a generator.

One thing can solve every problem we currently face.

The strength of a theory is not what it allows, but what it prohibits; if you can invent an equally persuasive explanation for any outcome, you have zero knowledge.

The wonder of evolution is that it works at all. I mean that literally: If you want to marvel at evolution, that’s what’s marvel-worthy. How does optimization first arise in the universe? If an intelligent agent designed Nature, who designed the intelligent agent? Where is the first design that has no designer? The puzzle is not how the first stage of the bootstrap can be super-clever and super-efficient; the puzzle is how it can happen at all. Evolution resolves the infinite regression, not by being super-clever and super-efficient, but by being stupid and inefficient and working anyway. This is the marvel. For professional reasons, I often have to discuss the slowness, randomness, and blindness of evolution. Afterward someone says: “You just said that evolution can’t plan simultaneous changes, and that evolution is very inefficient because mutations are random. Isn’t that what the creationists say? That you couldn’t assemble a watch by randomly shaking the parts in a box?” But the reply to creationists is not that you can assemble a watch by shaking the parts in a box. The reply is that this is not how evolution works. If you think that evolution does work by whirlwinds assembling 747s, then the creationists have successfully misrepresented biology to you; they’ve sold the strawman. The real answer is that complex machinery evolves either incrementally, or by adapting previous complex machinery used for a new purpose. Squirrels jump from treetop to treetop using just the...

How evolution works vs. how creationiists' strawman works.

That’s how one complex adaptation can jump-start a new complex adaptation. Complexity can also accrete incrementally, starting from a single mutation. First comes some gene A which is simple, but at least a little useful on its own, so that A increases to universality in the gene pool. Now along comes gene B, which is only useful in the presence of A, but A is reliably present in the gene pool, so there’s a reliable selection pressure in favor of B. Now a modified version of A* arises, which depends on B, but doesn’t break B’s dependency on A∕A*. Then along comes C, which depends on A* and B, and B*, which depends on A* and C. Soon you’ve got “irreducibly complex” machinery that breaks if you take out any single piece.

Here’s an example: DNA stores information very nicely, in a durable format that allows for exact duplication. A ribosome turns that stored information into a sequence of amino acids, a protein, which folds up into a variety of chemically active shapes. The combined system, DNA and ribosome, can build all sorts of protein machinery. But what good is DNA, without a ribosome that turns DNA information into proteins? What good is a ribosome, without DNA to tell it which proteins to make? Organisms don’t always leave fossils, and evolutionary biology can’t always figure out the incremental pathway. But in this case we do know how it happened. RNA shares with DNA the property of being able to carry information and replicate itself, although RNA is less durable and copies less accurately. And RNA also shares the ability of proteins to fold up into chemically active shapes, though it’s not as versatile as the amino acid chains of proteins. Almost certainly, RNA is the single A which predates the mutually dependent A* and B. It’s just as important to note that RNA does the combined job of DNA and proteins poorly, as that it does the combined job at all. It’s amazing enough that a single molecule can both store information and manipulate chemistry. For it to do the job well would be a wholly unnecessary miracle. What was the very first replicator ever to exist? It may well have been an RNA strand, because by some strange coincidence, the chemical ingredients of RNA are chemicals tha...

Why DNA complexity is not irreducibley complex.

“Nature is the ultimate bigot, because it is obstinately and intolerantly devoted to its own prejudices and absolutely refuses to yield to the most persuasive rationalizations of humans.”

Quote

The fundamental problem with arguing that things are true “by definition” is that you can’t make reality go a different way by choosing a different definition.

The Bayesian definition of evidence favoring a hypothesis is evidence which we are more likely to see if the hypothesis is true than if it is false. Observing that a syllogism is logically valid can never be evidence favoring any empirical proposition, because the syllogism will be logically valid whether that proposition is true or false. Syllogisms are valid in all possible worlds, and therefore, observing their validity never tells us anything about which possible world we actually live in.

Just as you can’t usually convey a concept’s whole intension in words because it’s a big complicated neural membership test, you can’t control the concept’s entire intension because it’s applied sub-deliberately. This is why arguing that XYZ is true “by definition” is so popular. If definition changes behaved like the empirical null-ops they’re supposed to be, no one would bother arguing them. But abuse definitions just a little, and they turn into magic wands—in arguments, of course; not in reality.

The flaw in arguing something is true by definition when tweaking it.

People who argue that atheism is a religion “because it states beliefs about God” are really trying to argue (I think) that the reasoning methods used in atheism are on a par with the reasoning methods used in religion, or that atheism is no safer than religion in terms of the probability of causally engendering violence, etc. . . . What’s really at stake is an atheist’s claim of substantial difference and superiority relative to religion, which the religious person is trying to reject by denying the difference rather than the superiority(!).

Have you ever run across theists claiming atheism is a religion?

People cling to their intuitions, I think, not so much because they believe their cognitive algorithms are perfectly reliable, but because they can’t see their intuitions as the way their cognitive algorithms happen to look from the inside. And so everything you try to say about how the native cognitive algorithm goes astray, ends up being contrasted to their direct perception of the Way Things Really Are—and discarded as obviously wrong.

Why debating on the Internet is so damned frustrating at times.

The argument starts shifting to focus on definitions. Whenever you feel tempted to say the words “by definition” in an argument that is not literally about pure mathematics, remember that anything which is true “by definition” is true in all possible worlds, and so observing its truth can never constrain which world you live in.

There's a flaw in arguing a non-mathematical assertion "by definition."

If you want to know whether atheism should be clustered with supernaturalist religions for purposes of some particular empirical inference, the dictionary can’t answer you. If you want to know whether blacks are people, the dictionary can’t answer you. If everyone believes that the red light in the sky is Mars the God of War, the dictionary will define “Mars” as the God of War. If everyone believes that fire is the release of phlogiston, the dictionary will define “fire” as the release of phlogiston. There is an art to using words; even when definitions are not literally true or false, they are often wiser or more foolish. Dictionaries are mere histories of past usage; if you treat them as supreme arbiters of meaning, it binds you to the wisdom of the past, forbidding you to do better. Though do take care to ensure (if you must depart from the wisdom of the past) that people can figure out what you’re trying to swim.

The folly of arguing about the definition of a word.

In the previous essay we saw how replacing terms with definitions could reveal the empirical unproductivity of the classical Aristotelian syllogism. All humans are mortal (and also, apparently, featherless bipeds); Socrates is human; therefore Socrates is mortal. When we replace the word “human” by its apparent definition, the following underlying reasoning is revealed: All [mortal, ¬feathers, biped] are mortal; Socrates is a [mortal, ¬feathers, biped]; Therefore Socrates is mortal.

One potential problem with sylogistic logic.

When you find yourself in philosophical difficulties, the first line of defense is not to define your problematic terms, but to see whether you can think without using those terms at all. Or any of their short synonyms. And be careful not to let yourself invent a new word to use instead. Describe outward observables and interior mechanisms; don’t use a single handle, whatever that handle may be.

Belief in God is HARD to defend when God and synonyms for the word are taboo.

When would someone feel the need to strengthen the argument with the emphatic phrase “by definition”? (E.g. “Humans are vulnerable to hemlock by definition!”) Why, when the inferred characteristic has been called into doubt—Socrates has been seen consulting herbologists—and so the speaker feels the need to tighten the vise of logic. So when you see “by definition” used like this, it usually means: “Forget what you’ve heard about Socrates consulting herbologists—humans, by definition, are mortal!” People feel the need to squeeze the argument onto a single course by saying “Any P, by definition, has property Q!,” on exactly those occasions when they see, and prefer to dismiss out of hand, additional arguments that call into doubt the default inference based on clustering. So too with the argument “X, by definition, is a Y!” E.g., “Atheists believe that God doesn’t exist; therefore atheists have beliefs about God, because a negative belief is still a belief; therefore atheism asserts answers to theological questions; therefore atheism is, by definition, a religion.” You wouldn’t feel the need to say, “Hinduism, by definition, is a religion!” because, well, of course Hinduism is a religion. It’s not just a religion “by definition,” it’s, like, an actual religion.

What leads us to argue "by definition"?

Atheism does not resemble the central members of the “religion” cluster, so if it wasn’t for the fact that atheism is a religion by definition, you might go around thinking that atheism wasn’t a religion. That’s why you’ve got to crush all opposition by pointing out that “Atheism is a religion” is true by definition, because it isn’t true any other way. Which is to say: People insist that “X, by definition, is a Y!” on those occasions when they’re trying to sneak in a connotation of Y that isn’t directly in the definition, and X doesn’t look all that much like other members of the Y cluster.

Arguing by definition is usually a way to sneak in a connotation.

It’s like the experiment in which you ask a second-grader: “If eighteen people get on a bus, and then seven more people get on the bus, how old is the bus driver?” Many second-graders will respond: “Twenty-five.” They understand when they’re being prompted to carry out a particular mental procedure, but they haven’t quite connected the procedure to reality.

Math is more useful when one understands what they are trying to calculate.

Previously, the most popular philosophy of science was probably Karl Popper’s falsificationism—this is the old philosophy that the Bayesian revolution is currently dethroning. Karl Popper’s idea that theories can be definitely falsified, but never definitely confirmed, is yet another special case of the Bayesian rules; if P(X|A) ≈ 1—if the theory makes a definite prediction—then observing ¬X very strongly falsifies A. On the other hand, if P(X|A) ≈ 1, and we observe X, this doesn’t definitely confirm the theory; there might be some other condition B such that P(X|B) ≈ 1, in which case observing X doesn’t favor A over B. For observing X to definitely confirm A, we would have to know, not that P(X|A) ≈ 1, but that P(X|¬A) ≈ 0, which is something that we can’t know because we can’t range over all possible alternative explanations. For example, when Einstein’s theory of General Relativity toppled Newton’s incredibly well-confirmed theory of gravity, it turned out that all of Newton’s predictions were just a special case of Einstein’s predictions.

Why Bayesian Probability replaced Karl Popper's Falsification in science.