While everyone of every race holds prejudice and can discriminate against someone of another race, in the US and other white/settler nations, only white people are in the position to oppress people of color collectively and throughout the whole of society.
