A “thought experiment” only works, only produces correct conclusions, when you actually do what the experiment asks of you. But when we do this, we do not arrive at these two conclusions after all. For example, for a Chinese man to have a conversation with the room so convincing he would be sure he was speaking to a real person who understood him, the rulebook in that room would have to be capable of some remarkable things. First, it would have to be able to be changed: the room would have to be able to remember things the Chinese man said, and what the room itself said, and adjust future
A “thought experiment” only works, only produces correct conclusions, when you actually do what the experiment asks of you. But when we do this, we do not arrive at these two conclusions after all. For example, for a Chinese man to have a conversation with the room so convincing he would be sure he was speaking to a real person who understood him, the rulebook in that room would have to be capable of some remarkable things. First, it would have to be able to be changed: the room would have to be able to remember things the Chinese man said, and what the room itself said, and adjust future responses accordingly. Thus, the rulebook would not be static, but constantly changing itself, and growing ever larger, even though doing so according to its own rules. Second, not only would it thus accumulate knowledge, it would be accumulating knowledge about itself, for future use in its conversation (to answer questions like “How do you feel?” or “What’s the difference between you and me?”). Third, the rulebook would have to contain something equivalent to desires, or else it would not exhibit curiosity, nor any capacity for active learning, and such omissions would be inhuman. So, built into the rules would have to be a hierarchy of simulated interests, for instance things the room would “want” to ask or say to the man outside in the course of any conversation, as well as things it would not want to. Fourth, and related to this, would be creativity: the rulebook would have to have s...
...more
This highlight has been truncated due to consecutive passage length restrictions.