Anybody can have and act on their prejudices. However, it’s my understanding that the term racism is talking more about the way these prejudices have been used to create organizations and institutions that privilege whites at the expense of people of color.
