The Technological Singularity

by Murray Shanahan

But the story has a coda. For one of Moople’s most senior execs, the tragic death of the security guard precipitates a period of profound soul-searching. Eventually this leads her to renounce her considerable material wealth and devote her life to reducing depression among those whose jobs have been taken away by AI technology and are forced into a life of pointless leisure. In due course, the foundation she endows will become a worldwide movement, bringing light into countless lives where before there was only darkness. In short, everything goes exactly as Moople’s other AI had planned all along. Ah, yes, I forgot to mention something. There is another system. Moople Corp.’s ethics AI is much consulted by the company’s employees. It was the ethics AI that advised deployment of the marketing AI in the first place. Based not only on its model of human behavior but also its model of the marketing AI, the ethical AI anticipated the death of the security guard (who was terminally ill, with no access to medical facilities in the developing country) and correctly predicted the effect it would have on the senior Moople exec. So the moral of the story is that unintended consequences can be good as well as bad. What matters is that the reward function of every powerful AI is designed right.