Rationality: From AI to Zombies

by Eliezer Yudkowsky

Can even a human be that overconfident? I would suspect that something simpler is going on—that the all-blue strategy just didn’t occur to the subjects. People see a mix of mostly blue cards with some red, and suppose that the optimal betting strategy must be a mix of mostly blue cards with some red. It is a counterintuitive idea that, given incomplete information, the optimal betting strategy does not resemble a typical sequence of cards. It is a counterintuitive idea that the optimal strategy is to behave lawfully, even in an environment that has random elements. It seems like your behavior ought to be unpredictable, just like the environment—but no! A random key does not open a random lock just because they are “both random.”