Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom

Others may also have final preferences about an agent’s goals. The agent could then have reason to modify its goals, either to satisfy or to frustrate those preferences.