Rationality: From AI to Zombies

by Eliezer Yudkowsky

We have no obvious mutual interest in using the word “oto” to mean sound, or “sound” to mean oto; but we have a mutual interest in using the same word, whichever word it happens to be. (Preferably, words we use frequently should be short, but let’s not get into information theory just yet.) But, while we have a mutual interest, it is not strictly necessary that you and I use the similar labels internally; it is only convenient. If I know that, to you, “oto” means sound—that is, you associate “oto” to a concept very similar to the one I associate to “sound”—then I can say “Paper crumpling makes a crackling oto.” It requires extra thought, but I can do it if I want. Similarly, if you say “What is the walking-stick of a bowling ball dropping on the floor?” and I know which concept you associate with the syllables “walking-stick,” then I can figure out what you mean. It may require some thought, and give me pause, because I ordinarily associate “walking-stick” with a different concept. But I can do it just fine.

Cf. Donald Davidson's "A Nice Derangement of Epitaphs"