Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom

Capability control Boxing methods The system is confined in such a way that it can affect the external world only through some restricted, pre-approved channel. Encompasses physical and informational containment methods. Incentive methods The system is placed within an environment that provides appropriate incentives. This could involve social integration into a world of similarly powerful entities. Another variation is the use of (cryptographic) reward tokens. “Anthropic capture” is also a very important possibility but one that involves esoteric considerations. Stunting Constraints are imposed on the cognitive capabilities of the system or its ability to affect key internal processes. Tripwires Diagnostic tests are performed on the system (possibly without its knowledge) and a mechanism shuts down the system if dangerous activity is detected. Motivation selection Direct specification The system is endowed with some directly specified motivation system, which might be consequentialist or involve following a set of rules. Domesticity A motivation system is designed to severely limit the scope of the agent’s ambitions and activities. Indirect normativity Indirect normativity could involve rule-based or consequentialist principles, but is distinguished by its reliance on an indirect approach to specifying the rules that are to be followed or the values that are to be pursued. Augmentation One starts with a system that already has substantially human or benevolent motivations...