For most of history, the people doing the enslaving came from similar(ish) regions of the world to those being enslaved. The very word slave comes from Slav, meaning Slavic, because so many ‘white’ Eastern Europeans were enslaved by other ‘Europeans’ and even sold to Muslims by them for centuries.