Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom

What is essential is that the AI believes that the button will more likely remain unpressed if the AI continuously acts in the principal’s interest than if it rebels.