Unmasking AI: My Mission to Protect What Is Human in a World of Machines

by Joy Buolamwini

The results of the phenotypic analysis showed that overall, Microsoft, IBM, and Face++ did better on lighter-skinned faces than darker-skinned faces when it came to guessing the gender of a face. If we were to stop at this level of single-axis analysis—that is, looking at gender in isolation and skin type in isolation—the assumption would be made that when it came to gender, regardless of skin type, systems performed better on men than women. Conversely, without digging deeper we might also assume that when it came to skin type, performance would be better on lighter skin than darker skin regardless of gender. These assumptions, which you might attempt to justify with the single-axis analysis, would, however, miss a more complex story. We needed to go another step further in the analysis, because we cannot assume, if the data has not been collected and tested, that the system performs the same across all groups. Furthermore, the burden of proof of performance needs to be placed on the people developing systems, not those who are impacted by their use.