Unmasking AI: My Mission to Protect What Is Human in a World of Machines

by Joy Buolamwini

A major challenge of neural networks is that during the training process computer scientists do not always know exactly why some weights are strengthened and others are weakened. As a result, current methods do not allow us to explain in full detail how a neural network recognizes a pattern like a face or outputs a response to a prompt. You may hear the term “black box” used to describe AI systems because there are unexplainable components involved. While it is true that parts of the process evade exact explanations, we still have to make sure we closely examine the AI systems being developed. Access to the training data is crucial when we want to have a deeper understanding of the risks posed by an AI system. Unless we know where the data comes from, who collected it, and how it is organized, we cannot know if ethical processes were used. Was the data obtained with consent? What were the working conditions and compensation for the workers who processed the data? These questions go beyond the technical.