This neat binary is a great way of avoiding any real discussion at all. But without the structural violence of unequal treatment before the law and in education, and a history of racial exploitation by states, simple acts of personal prejudice would have significantly less meaning.
