Rationality: From AI to Zombies

by Eliezer Yudkowsky

as soon as you see a blue egg-shaped thing and the central blegg unit fires, you holler “Blegg!” to Susan. And what that algorithm feels like from inside is that the label, and the concept, are very nearly identified; the meaning feels like an intrinsic property of the word itself. The cognoscenti will recognize this as a case of E. T. Jaynes’s “Mind Projection Fallacy.” It feels like a word has a meaning, as a property of the word itself; just like how redness is a property of a red apple, or mysteriousness is a property of a mysterious phenomenon. Indeed, on most occasions, the brain will not distinguish at all between the word and the meaning—only bothering to separate the two while learning a new language, perhaps. And even then, you’ll see Susan pointing to a blue egg-shaped thing and saying “Blegg!,” and you’ll think, I wonder what “blegg” means, and not, I wonder what mental category Susan associates to the auditory label “blegg.” Consider, in this light, the part of the Standard Dispute of Definitions where the two parties argue about what the word “sound” really means—the same way they might argue whether a particular apple is really red or green: ALBERT: “My computer’s microphone can record a sound without anyone being around to hear it, store it as a file, and it’s called a ‘sound file.’ And what’s stored in the file is the pattern of vibrations in air, not the pattern of neural firings in anyone’s brain. ‘Sound’ means a pattern of vibrations.” BARRY: “Oh, yeah?...