Unlike algorithmic systems, I could put a face to the decision-maker, and I also had the connections to challenge his initial decision. With black-box decision-makers, we are no longer facing the sexist hiring manager or xenophobic guard but a seemingly neutral device that nonetheless reflects the biases of the society it’s embedded in. Unless we demand not only a choice in deciding whether and how these systems are used but also pathways to challenge decisions, we will change the form of the gatekeeper, but the prejudice will remain.
