Rationality: From AI to Zombies

by Eliezer Yudkowsky

The Parable of the Dagger (Adapted from Raymond Smullyan.1) Once upon a time, there was a court jester who dabbled in logic. The jester presented the king with two boxes. Upon the first box was inscribed: Either this box contains an angry frog, or the box with a false inscription contains an angry frog, but not both. On the second box was inscribed: Either this box contains gold and the box with a false inscription contains an angry frog, or this box contains an angry frog and the box with a true inscription contains gold. And the jester said to the king: “One box contains an angry frog, the other box gold; and one, and only one, of the inscriptions is true.” The king opened the wrong box, and was savaged by an angry frog. “You see,” the jester said, “let us hypothesize that the first inscription is the true one. Then suppose the first box contains gold. Then the other box would have an angry frog, while the box with a true inscription would contain gold, which would make the second statement true as well. Now hypothesize that the first inscription is false, and that the first box contains gold. Then the second inscription would be—” The king ordered the jester thrown in the dungeons. A day later, the jester was brought before the king in chains and shown two boxes. “One box contains a key,” said the king, “to unlock your chains; and if you find the key you are free. But the other box contains a dagger for your heart if you fail.” And the first box was inscribed: Either bo...