Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom

Yet these topical applications of epistemic modesty are not enough; they must be supplemented here by a systemic admission of uncertainty and fallibility.

it is possible that some state would embark on a larger-scale eugenics program, perhaps by paying surrogate mothers.

Rather, each brain develops its own idiosyncratic representations of higher-level content.

On this basis alone, current levels of human collective intelligence could be regarded as approaching superintelligence relative to a Pleistocene baseline.

In terms of raw computational power, human brains are probably inferior to those of some large animals, including elephants and whales.

(Anything the brain does in under a second cannot use much more than a hundred sequential operations—perhaps only a few dozen.)

The sluggishness of neural signals limits how big a biological brain can be while functioning as a single processing unit.

Thus, the AI system could avoid a sizeable chunk of the inefficiencies arising from agency problems in human enterprises.

The ability of our genes to code for the construction of a goal-acquiring mechanism explains how we come to have final goals of great informational complexity, greater than could be contained in the genome itself.