Could one have a general intelligence that is not an agent? Intuitively, it is not just the limited capability of ordinary software that makes it safe: it is also its lack of ambition. There is no subroutine in Excel that secretly wants to take over the world if only it were smart enough to find a way. The spreadsheet application does not “want” anything at all; it just blindly carries out the instructions in the program.
