Rationality: From AI to Zombies

by Eliezer Yudkowsky

This is why I’ve always insisted, for example, that if you’re going to start talking about “AI ethics,” you had better be talking about how you are going to improve on the current situation using AI, rather than just keeping various things from going wrong. Once you adopt criteria of mere comparison, you start losing track of your ideals—lose sight of wrong and right, and start seeing simply “different” and “same.” I would also argue that this basic psychological difference is one of the reasons why an academic field that stops making active progress tends to turn mean. (At least by the refined standards of science. Reputational assassination is tame by historical standards; most defensive-posture belief systems went for the real thing.) If major shakeups don’t arrive often enough to regularly promote young scientists based on merit rather than conformity, the field stops resisting the standard degeneration into authority. When there’s not many discoveries being made, there’s nothing left to do all day but witch-hunt the heretics.