There is compelling reason to believe that it was not considered significant in earlier periods. The Bible, for example, written over two thousand years ago in the Mediterranean, where black, brown, and white people were to be found, is filled with moralistic tribalism, but makes almost no mention of skin color.