Rationality: From AI to Zombies

by Eliezer Yudkowsky

You are not a Bayesian homunculus whose reasoning is “corrupted” by cognitive biases. You just are cognitive biases. Confirmation bias, status quo bias, correspondence bias, and the like are not tacked on to our reasoning; they are its very substance. That doesn’t mean that debiasing is impossible. We aren’t perfect calculators underneath all our arithmetic errors, either. Many of our mathematical limitations result from very deep facts about how the human brain works. Yet we can train our mathematical abilities; we can learn when to trust and distrust our mathematical intuitions, and share our knowledge, and help one another; we can shape our environments to make things easier on us, and build tools to offload much of the work. Our biases are part of us. But there is a shadow of Bayesianism present in us as well, a flawed apparatus that really can bring us closer to truth. No homunculus—but still, some truth. Enough, perhaps, to get started.