One can easily imagine the problems that might arise by having AIs align themselves to historical or observed human values. Whose values should an AI mirror? A Somali man? A Singaporean woman? The average of the two, whatever that means? We humans hold contradictory values, and we’re also not consistent about living up to them. Any individual person’s values might be irrational, immoral, or based on false information. History, literature, and philosophy are full of irrationality, immorality, and error. Humans are often not very good examples of our ideals.
