Rationality: From AI to Zombies

by Eliezer Yudkowsky

But if you can’t say “Oops” and give up when it looks like something isn’t working, you have no choice but to keep shooting yourself in the foot. You have to keep reloading the shotgun and you have to keep pulling the trigger. You know people like this. And somewhere, someplace in your life you’d rather not think about, you are people like this.

When your method of learning about the world is biased, learning more may not help. Acquiring more data can even consistently worsen a biased prediction.