If an ethnic group or a sex has been disadvantaged by oppression in the past, its members may be saddled with different average traits in the present. If those base rates are fed into predictive formulas that determine their fate going forward, they would lock in those disadvantages forever. The problem is becoming acute now that the formulas are buried in deep learning networks with their indecipherable hidden layers (chapter 3). A society might rationally want to halt this cycle of injustice even if it took a small hit in predictive accuracy at that moment.
This is a perfect description of structural racism or “racism without racists”
