Unmasking AI: My Mission to Protect What Is Human in a World of Machines

by Joy Buolamwini

WHEN MACHINE LEARNING IS USED to diagnose medical conditions, to inform hiring decisions, or even to detect hate speech, we must keep in mind that the past dwells in our data. In the case of hiring, Amazon learned this lesson when it created a model to screen résumés.[7] The model was trained on data of prior successful employees who had been selected by humans, so the prior choices of human decision-makers then became the basis on which the system was trained. Internal tests revealed that the model was screening out résumés that contained the word “women” or women-associated colleges. The system had learned that the prior candidates deemed successful were predominantly male. Past hiring practices and centuries of denying women the right to education coupled with the challenges faced once entering higher education made it especially difficult to penetrate male-dominated fields. Faithful to the data the model was trained on, it filtered out résumés indicating a candidate was a woman. This was the by-product of prior human decisions that favored men. At Amazon, the initial system was not adopted after the engineers were unable to take out the gender bias. The choice to stop is a viable and necessary option. The face datasets I examined revealed data that was not representative of society. The example of the Amazon hiring model illustrates what happens when data does indeed reflect the assumptions of society. Their model reflected power shadows. Power shadows are cast when th...