Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom

We have already encountered the idea of perverse instantiation: a superintelligence discovering some way of satisfying the criteria of its final goal that violates the intentions of the programmers who defined the goal.