Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom

Final goal: “Maximize the time-discounted integral of your future reward signal” Perverse instantiation: Short-circuit the reward pathway and clamp the reward signal to its maximal