Unmasking AI: My Mission to Protect What Is Human in a World of Machines

by Joy Buolamwini

My mother modeled impermanence with the quiet confidence that came from knowing that she would inevitably make something better.

When I heard myself, I wondered if my message would be diluted if I was perceived as someone holding a vendetta.

Approaching the issue with palpable anger, I thought, would put people on the defensive when my aim was to figure out ways to have people listen. I was acutely aware of being stereotyped as an angry Black woman, eager to play the race card and find offense in the seemingly innocuous.

If I became consumed with resentment, I would not be able to do the necessary work. I had to hold on to the belief that change was possible.

spry

I also worked with Rossi Films to make a mini documentary called The Coded Gaze: Unmasking Algorithmic Bias.

Printed in white ink was the title “Algorithmic Justice League”—the name I was using to describe the work I was doing. The name follows the “justice league” banner that many others have used since the turn of the twentieth century—decades before DC Comics adopted the term for their fictional worlds—to fight for societal change. In the early twentieth century, civic organizations used the phrase “justice league” in their fight for women’s suffrage (“The Equal Justice League of Young Women” [1911]), racial equality and civil rights for African Americans (“Race Justice League” [1923]), and workers’ rights (“Justice League” [1914]). Scores of justice-oriented organizations continue to tap into this tradition today. Real-world justice leagues serve as inspiration for the belief that against tyranny, oppression, and erasure, we can choose to resist and offer pathways to liberation.

Someone with darker skin would try without luck until they put on the white mask I had put on the table. I overheard a fair-skinned person say, “It works so well for me, I didn’t even imagine it wouldn’t work for someone else.” And someone else, with a darker complexion, commented, “Dang, the machines can’t see us either?” Without seeing someone else struggle with the Upbeat Walls system, the person for whom it had worked just assumed that it worked for everyone. But when they realized that wasn’t the case, their new awareness was reinforced by watching a video about the coded gaze. And the person for whom the system failed to work correctly realized that it was the machine, not them, that was at fault. This was not a case of user error. There was something more under the hood, and that was precisely what the whole exhibition was about.

With our open lab space, Ethan modeled for me what it looked like to hold space for others, no matter where they were on their journey of political awareness, to come and be heard, to ask questions, to express doubts, and to find community.

Showtime had arrived. Backstage, a technician outfitted me with a “flesh color” microphone. The pale pink color didn’t quite match my gleaming milk chocolate skin. I asked if they had anything darker. They did not.

At this point on my MIT academic path, I began to accept what I could do with the privilege of being at an institution with high visibility. Rather than feeling like I was giving up on my idealism of working on technology to escape painful realities, I could see how my childhood passions could live alongside my solidifying purpose to research harmful discrimination in technology. My life trajectory and educational opportunities were starting to make more sense to me. I felt emboldened to ask even more uncomfortable questions about the machines that once enamored me.

trifecta

“Hi, camera, can you see my face?” I pause. Nothing. “You can see my friend’s face.” The video cuts to the face of my friend Mary Maggic, a Chinese American speculative artist. Her face is quickly detected. “What about my face?” The camera returns to my face. I make an exaggerated pout on camera, drawing laughter from the audience. “I have a mask.” I put on the white mask, which is immediately detected. “Can you see my mask?” The laughter shifts to audible gasps. On the black screen, three white words linger: “The Coded Gaze.”

Showing and not just telling about computer vision, I reasoned, would allow for powerful depictions of the notion of algorithmic bias and the coded gaze. I call this approach of showing technical failures, to allow others to bear witness to ways technology could be harmful, evocative audits.

The insults were nothing I hadn’t heard before, but they still stung. As tempted as I was to comment back, an angry reaction would be counterproductive for people who were genuinely curious or uninformed.

There was a common assumption that these math-based systems make objective decisions; after all, one plus one equals two. Machines were presumed to be free from the societal biases that plague us mortals. My experiences were showing me otherwise.

Even though cameras may appear neutral, history reveals another story. The film used in analog cameras was exposed using a special chemical composition to bring out desired colors. To calibrate the cameras to make sure those desired colors were well represented, a standard was created. This standard became known as the Shirley card, which was originally an image of a white woman used to establish the ideal composition and exposure settings. The consequence of calibrating film cameras using a light-skinned woman was that the techniques developed did not work as well for people with darker skin. In fact, it wasn’t until furniture and chocolate companies complained that the rich browns of their products were not being well represented that Kodak introduced a new product that better captured a range of browns and dark sepia tones. Separating the appearance of milk chocolate from dark chocolate in advertising had a windfall effect for chocolate-hued individuals. Still, the subsequent digital cameras inherited configurations optimized for light skin, a decision that would translate into a larger set of photos and videos being produced by cameras optimized for only one part of humanity: those with light skin. So when we enter the realm of computer vision, which comes to rely on large datasets, we are dealing with a legacy of cameras and captured images that inherit exclusion. For an image or video, that exclusion can mean that the features of someone with darker skin are less vis...

In the digital era, the LDK camera series developed by Philips explicitly handled skin tone variation with two chips—one for processing darker tones and another for processing lighter tones. The Oprah Winfrey Show used the LDK series for filming because there was an awareness of the need to better expose darker skin, given the show’s host and guests.

spirited

Machines can also analyze your behavior and data collected about you to make recommendations that shape our decisions. The decisions can be low-stakes, like Netflix’s ability to suggest another film or TV series to binge based on the user’s inferred preferences and viewing history. But the decisions can also include more high-stakes situations. For example, AI systems used for employment can recommend a short list of candidates to hire. AI systems used in healthcare can provide recommendations on which patients receive tailored care and which ones do not.[1] Very quickly, we can see how number crunching is not so neutral when those numbers can crunch your life.

To build the knowledge of a translation system, linguists would painstakingly attempt to delineate the rules of different languages to facilitate translation. But the real world never quite follows all the rules. Spoken language and text chat do not exactly follow strict grammar when you need customer service to refund your order.

Kitty Hawk

Since AI can be applied in a range of contexts from cupcakes to combatants, the example applications of the technology shape the public imagination for what is possible, as well as the perceived risks. When companies like Boston Dynamics show their autonomous dog-like quadruped robots doing something cute, like dancing, they mask the ways these systems can be used in the context of military operations or policing.

Machines, unlike humans, often need many examples to learn. In computer vision, an object detection model using machine learning techniques may rely on millions of photos. Before the rise of the internet, access to such a large number of photos was largely impractical.

The creators of social media platforms like Facebook and Twitter, and the developers of mobile operating systems like Google’s Android, were able to amass large stores of valuable data created by users. Many of us were fueling the advancement of AI unaware.

No neuron by itself is enough to achieve a complex task like recognizing a face, but by working together, small components can achieve larger tasks. Building on this idea, researchers created artificial neural networks. Instead of neurons and synapses, the artificial neural network contains nodes that are linked to one another in a web of layers. The nodes are inspired by neurons and the links are inspired by synapses. Keep in mind that even though machine learning is inspired by some elements of a biological brain, it does not mean we are creating machines that are sentient or have consciousness. As individuals we are more than our biology.

think of the neural network as a pattern recognizer.

To be useful, a neural network must be trained to respond to a specific kind of pattern. The training process of a neural network strengthens some connections and weakens others so the trained neural network model can recognize a pattern. Researchers have developed different kinds of training methods to create machine learning models, which are neural networks configured to recognize a specific pattern. In general, the components of machine learning involve training data, testing data, a neural network to configure, and a learning algorithm to build the experience of the neural network. Remember, an algorithm is a sequence of instructions used to achieve a specific outcome. The goal of a learning algorithm for a neural network is to set the weights between the nodes in the best possible way to recognize a pattern. In the case of detecting an object like a car in an image, the neural network is exposed to many training images that contain a car and the weights are refined many times until the model can consistently detect cars in images it was not previously exposed to.

Some of the elusive aims from the 1960s, like machine translation of one language into another, are now possible for human languages that are available in large volumes online. Large language models (LLMs), the AI systems that power chatbots, are also trained to analyze patterns. Many LLMs are trained on the text available on the internet, which is to say they ingest vast sums of information from a large portion of what has been made public online. This information includes newspaper articles, scientific papers, standardized test questions and answers, and all of Wikipedia, to name just a few sources that can make them appear extremely knowledgeable. (Imagine taking a test with the benefit of reviewing all the example questions and correct responses that have been posted online and then producing similar responses. I’m not saying you aren’t clever, nor am I denying that you have an advantage, but I would be a little less impressed.) These systems ingest not just reputable content, because text on the internet also includes online forums with toxic content, hate speech on social media, and more. LLMs learn the good, the bad, and the ugly.

So when you provide a prompt to an LLM like “What indigenous languages are missing on the internet?” you can receive a response that is coherent and grammatically correct. Yet, the response may be unsatisfactory because some missing languages might never have been named on the internet at all. The internet does not represent all human knowledge. The training data provides the neural network with experience that can be used on new data and new prompts. But limited experience has consequences.

You may hear the term “black box” used to describe AI systems because there are unexplainable components involved. While it is true that parts of the process evade exact explanations, we still have to make sure we closely examine the AI systems being developed. Access to the training data is crucial when we want to have a deeper understanding of the risks posed by an AI system. Unless we know where the data comes from, who collected it, and how it is organized, we cannot know if ethical processes were used. Was the data obtained with consent? What were the working conditions and compensation for the workers who processed the data? These questions go beyond the technical.

The first neural networks were very simple, with just a couple of layers. Then, as computing capacity increased, researchers could create more complex networks with many additional layers, giving rise to deep learning. Deep learning is a flavor of machine learning that specifically uses deep neural networks, multilayered pattern recognizers inspired by the neural connections of a brain. In this case, imagine a lot of marshmallows and toothpicks. There can be billions of parameters in the systems used to build generative AI products that can create images from a line of text such as “an astronaut riding a horse in space.” LLMs can have trillions of parameters. Parameters can include information like how strongly different components of the architecture are connected via weights. Machine learning is also just one approach that can be used for computers to make judgments or generate new content based on prompts. There is more to the story, but this high-level overview is a starting point to uncover how some of the most widely adopted approaches to artificial intelligence become susceptible to harmful discrimination and toxic outputs. Simply because decisions are made by a computer analyzing data does not make them neutral. Neural does not equate to neutral.

In my work, I use the coded gaze term as a reminder that the machines we build reflect the priorities, preferences, and even prejudices of those who have the power to shape technology. The coded gaze does not have to be explicit to do the job of oppression. Like systemic forms of oppression, including patriarchy and white supremacy, it is programmed into the fabric of society. Without intervention, those who have held power in the past continue to pass that power to those who are most like them. This does not have to be intentional to have a negative impact.

With the slew of testimonials sent my way, it was clear that faulty technical systems had already contributed to undue scrutiny, suspicion, and, in the case of the inmate who wrote me, jail time.

I was surprised to see machine learning could impact my love life. I attempted to sign up for a dating app that appeared to use AI on uploaded photos before allowing entrance.

Women already faced a “pink tax,” higher pricing for items like pens and razors when they were colored pink as compared to blue or black.

A chatbot confidently responding with made-up information is referred to by some AI developers as “hallucination.” Author and cultural critic Naomi Klein observes that the term hallucination is a clever way to market product failures. It sounds better than saying the system makes factual mistakes or presents nonsense as facts. These frequent errors are a demonstration of the AI functionality fallacy and a reminder that appearing knowledgeable isn’t the same as being factual.

I still remember my disbelief when I came across a 2017 study where the authors used images of more than eighteen hundred people to create a classifier to predict criminality based on a face image.[9]

In an attempt to curb cheating, schools adopted e-proctoring tools to monitor remote learners while they took tests. These companies faced complaints from dark-skinned students who had to set up elaborate lighting contraptions to be seen, or were unable to be verified to log in, or were flagged as cheating. The cheating flag could occur when the system no longer detected a face. However, there are technical reasons a face might not be detected that have nothing to do with cheating but instead indicate a failure of the AI system. Dutch student Robin Pocornie filed a complaint with the Netherlands Institute for Human Rights after enduring remote test conditions that forced her to shine a light on her face to take her exams. White students did not file similar complaints. The institute investigated the complaint and released an interim verdict in her favor, finding that the software used by her university, VU Amsterdam, discriminates against Black students.[15] These systems can also disadvantage individuals with disabilities, as detailed by the Center for Democracy and Technology in a study examining e-proctoring tools:

When companies require individuals to fit a narrow definition of acceptable behavior encoded into a machine learning model, they will reproduce harmful patterns of exclusion and suspicion.

listserv,

Data was queen. And it seemed for the time being that size mattered. Large datasets with labels were essential in leading to breakthroughs in the computer vision research community. These labels would also prove useful in tracing out sources of bias.

He soared over six feet tall, with long elegant limbs, and he reminded me of the Big Friendly Giant from the Roald Dahl book my father had read to me at bedtime.

I went in search of Hal Abelson, who was approaching his seventies and had a Yoda-like presence in both stature and demeanor.

With Yoda, Benjamin Franklin, the Big Friendly Giant, and me huddled around a table, we discussed my exploration into algorithmic bias and the roles each advisor would fulfill along the way.

Mitch was alluding to a learning approach he’d outlined in his book Lifelong Kindergarten, where creative learning is supported by four p’s: projects, passion, peers, and play.

hard fun, a term conceptualized by the mathematician and AI pioneer Seymour Papert. Hard fun is what’s happening when we willingly take on difficult subjects, or work through mundane tasks, because we’re working on projects that impassion and excite us.

In 2014 Hu Han and Anil Jain examined the demographic composition of LFW; they found that the database of images contained 77.5 percent male-labeled faces and 83.5 percent faces labeled white.[3] The gold standard for facial recognition, it turned out, was heavily skewed. I started calling these “pale male datasets.”

Thinking beyond faces, deep learning techniques were being applied to systems trained to detect skin cancer or to detect pedestrians to be used in self-driving cars. If those datasets were also skewed, it could mean AI cancer detectors would not work well for groups of people underrepresented in the dataset. It would mean automated vehicles would be more likely to crash into some groups of people than others.

I was left with one major lesson: Always question the so-called gold standards. Just like the standard Shirley cards that were used for calibrating film-based photography might seem neutral or untouchable, standards used in AI may appear to be off-limits for questioning when we assume the experts have done a thorough job. This is not always the case, particularly when the experts do not reflect the rest of society.