Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom

Those inane Terminator pictures are taking a toll. It can’t be much fun to have aspersions cast on one’s academic discipline, one’s professional community, one’s life’s work. There is a possibility that, in reaction to misguided public alarm about evil robot armies, the AI community might close ranks around a position that denigrates any real concerns about advanced machine intelligence.

Should a norm arise among AI researchers that it is uncouth to talk about superintelligence or inquire into its possible risks, for fear of “giving ammunition” to critics, fear-mongers, crazies, or would-be regulators, then the recent gains of legitimation could quickly be reversed.

The best path toward the development of beneficial superintelligence is one in which AI developers and AI safety researchers are on the same side—one in which they are indeed, to a considerable extent, the same persons. So I call on all sides to practice patience and restraint, and broad-mindedness, and to engage in direct dialogue and collaboration whenever possible.