If AI builders specified the wrong objective in their designs, their system could accidentally cause some damage. Reward a household robot for carrying a box from one side of a room to another, and it might knock over a vase that was in its path because it was so focused on its objective, he wrote. People needed to look at the real-world accidents that AI could cause after being integrated into industrial control systems and healthcare, Amodei argued.
