Rationality: From AI to Zombies

by Eliezer Yudkowsky

The “improbability” here is improbability relative to a random selection from the design space, not improbability in an absolute sense—if you have an optimization process around, then “improbably” good designs become probable