The Ends of the World: Volcanic Apocalypses, Lethal Oceans, and Our Quest to Understand Earth's Past Mass Extinctions

by Peter Brannen

“If you focus completely on the climate stuff and ignore super-intelligence, we might find that yeah, the Paper Clipper gets us first.” The Paper Clipper? “My idea with the Paper Clipper is, you have this artificial intelligence and you give it the goal of making paper clips. So it tries to do the actions that maximize the number of paper clips, and it figures out how to make itself smarter, because if it’s smarter, it’s better at making paper clips. So it makes itself really smart, and it comes up with a foolproof plan to convert Earth into paper clips, and it implements that plan. That’s very bad news for us. The problem here is, of course, if I tried to pull the plug on it, it’s smart enough that it’s figured out a way of stopping me, because if I pull the plug, there will be fewer paper clips in the world and that’s bad. So it must overcome any attempt at stopping it or changing its mind. It might even be that the universe has something like Kantian ethics, so a sufficiently smart mind will recognize that it’s a moral truth that it shouldn’t turn people into paper clips. Unfortunately, if the architecture of AI is such that it just maximizes its utility, and utility is defined by paper clips, it’s going to think, ‘Hmm, being moral or paper clips? Paper clips!’”