Racial differences, which have only relatively recently emerged, are of little Us/Them significance. For the hunter-gatherers of our hominin history, the most different person you’d ever encounter in your life came from perhaps a couple of dozen miles away, while the nearest person of a different race lived thousands of miles away—there is no evolutionary legacy of humans encountering people of markedly different skin color.
