Life 3.0: Being Human in the Age of Artificial Intelligence

by Max Tegmark

We already know that the Omegas have programmed Prometheus to strive for certain goals. Suppose that they’ve given it the overarching goal of helping humanity flourish according to some reasonable criterion, and to try to attain this goal as fast as possible. Prometheus will then rapidly realize that it can attain this goal faster by breaking out and taking charge of the project itself. To see why, try to put yourself in Prometheus’ shoes by considering the following example. Suppose that a mysterious disease has killed everybody on Earth above age five except you, and that a group of kindergartners has locked you into a prison cell and tasked you with the goal of helping humanity flourish. What will you do? If you try to explain to them what to do, you’ll probably find this process frustratingly inefficient, especially if they fear your breaking out, and therefore veto any of your suggestions that they deem a breakout risk. For example, they won’t let you show them how to plant food for fear that you’ll overpower them and not return to your cell, so you’ll have to resort to giving them instructions. Before you can write to-do lists for them, you’ll need to teach them to read. Moreover, they won’t bring any power tools into your cell where you can teach them how to use them, because they don’t understand these tools well enough to feel confident that you can’t use them to break out. So what strategy would you devise? Even if you share the overarching goal of helping these ...