Unmasking AI: My Mission to Protect What Is Human in a World of Machines

by Joy Buolamwini

Coding in whiteface was the last thing I expected to do when I came to MIT, but—for better or for worse—I had encountered what I now call the “coded gaze.” You may have heard of the male gaze, a concept developed by media scholars to describe how, in a patriarchal society, art, media, and other forms of representation are created with a male viewer in mind. The male gaze decides which subjects are desirable and worthy of attention, and it determines how they are to be judged. You may also be familiar with the white gaze, which similarly privileges the representation and stories of white Europeans and their descendants. Inspired by these terms, the coded gaze describes the ways in which the priorities, preferences, and prejudices of those who have the power to shape technology can propagate harm, such as discrimination and erasure. We can encode prejudice into technology even if it is not intentional.

Generative AI products are only one manifestation of AI. Predictive AI systems are already used to determine who gets a mortgage, who gets hired, who gets admitted to college, and who gets medical treatment—but products like ChatGPT have brought AI to new levels of public engagement and awareness.

Given the real harms of AI, how can we center the lives of everyday people, and especially those at the margins, when we consider the design and deployment of AI? Can we make room for the best of what AI has to offer while also resisting its perils? None of us can escape the impact of the coded gaze. Instead, we must face it. You have a place in this conversation and in the decisions that impact your daily life, which are increasingly being shaped by advancing technology that sits under the wide—often opaque—umbrella of artificial intelligence. This book offers a path into urgent and growing conversations about the future of technology that need your voice, the voice of everyday people with lived experiences of what it means to be excluded—indeed, excoded—from systems not designed with you in mind. We need the voice of people like Robert Williams, who was wrongfully arrested in front of his children due to a false facial recognition match.[8] We need the voice of students, those struggling with e-proctoring software that flags them as cheaters.[9] We need the voice of migrants from Haiti and Africa who were caught in limbo when applying for asylum because the U.S. government required use of a mobile app that failed to verify their faces.[10] We also need the voice of the unseen faces that do the ghost work, the data cleaning, the human translation that supports AI products. We need the voice of the parents whose children had intimate moments recorded by listening devices m...

We swap fallible human gatekeepers for machines that are also flawed but assumed to be objective. And when machines fail, the people who often have the least resources and most limited access to power structures are those who have to experience the worst outcomes.

we need to be able to recognize that not building a tool or not collecting intrusive data is an option, and one that should be the first consideration. Do we need this AI system or this data in the first place, or does it allow us to direct money at inadequate technical Band-Aids without addressing much larger systemic societal issues?

As Dr. Rumman Chowdhury reminds us in her work on AI accountability, the moral outsourcing of hard decisions to machines does not solve the underlying social dilemmas.

I invite you into my journey from an eager computer scientist ready to solve the world’s problems with code to an advocate for algorithmic justice concerned with how technology can encode harmful discrimination and exclusionary practices. I critique AI from a place of having been enamored with its promise, as an engineer more eager to work with machines than with people at times, as an aspiring academic turned into an accidental advocate, and also as an artist awakened to the power of the personal when addressing the seemingly technical. I am a child of Ghana born to an artist and a scientist, and my background informs my sensibilities in how I learn about the world and share my evolving understanding. I aim to tell stories that make daughters of diasporas dream and sons of privilege pause.

Regardless of where you are positioned at the beginning of this book, I hope you come away with a deeper understanding of why each and every one of us has a role to play in reaching toward algorithmic justice. I hope when you feel despair you return to the stories of triumphs I share. I hope when you feel there is no place for creative expression in your work you revisit the poetry crafted for you in this book. I hope when you are afraid to speak up you read about the Brooklyn tenants who organized to resist a harmful AI system and are reminded of the value of your voice and experiences. I hope when you are tired of feelings and just want some answers, you remember this is not a book of easy answers, but I hope you walk away with questions that push us all to rethink, reframe, and recode the future of AI.

With safety in sight and security calling, Would you turn back for the forgotten ones? Would you risk your comfort or diminish your power to reach out to those left in the shadows? Would your lips testify of uncanny truths or instead would you swallow your conscience and cough up excuses?

Prestige and privilege masquerade as merit though much of what is achieved is a function of what we inherit.

Like my mother, he was working on experiments that required bold curiosity to ask unexplored questions. But while my mother asked questions of colors, my father asked questions of cells. In the midst of their explorations, I began to ask questions about computers.

He showed me the software to introduce me to chemistry, but I found myself more and more enamored with the machines themselves. I quickly found games like Doom and Cycle that came preloaded. I listened to the whirs and beeps of a dial-up connection. In that office, I opened Netscape, my first browser experience into a portal I would later learn was the internet. And so it was that, surrounded by art and science from a very young age, I was emboldened to explore, to ask questions, to dare to alter what seemed fixed, and also to view the artist’s and the scientist’s search for truth as common companions. My parents taught me that the unknown was an invitation to learn, not a menacing dimension to avoid. Ignorance was a starting place to enter deeper realms of understanding.

my parents wouldn’t let me watch commercials: They wanted to shield me from the materialism that appeared to be the backbone of American culture. “You will never find your worth in things,” they cautioned. However, they encouraged me to watch educational programs, so PBS became the television channel of choice in our home. I soon found myself anticipating shows like Nature, National Geographic, Nova, and Scientific American Frontier.

I decided I wanted to go to MIT and become a robotics engineer. I was blissfully unaware of any barriers or requirements. I had more questions to ask of computers, nurtured in the incubator of youthful possibilities by the belief that I could become anything I imagined.

I wanted to go deeper than websites, and I was curious about how to make games like the ones I played with my brother on his Nintendo 64 or Tony Hawk Pro Skater 2, which I enjoyed on my Sony PlayStation. So I learned another programming language called Java. Here, I was introduced to the concept of an algorithm. An algorithm, at its most basic definition, is a sequence of instructions used to achieve a specific goal. To make my character move around the screen, I would write code that followed a logical sequence. For instance, if the user hits the left arrow, move the character left on the screen. Algorithms like this, as I would eventually learn, would become the basis for more powerful and dynamic systems.

I had gotten into computer science in some ways to escape the messiness of the multiheaded -isms of racism, sexism, classism, and more. I was acutely aware of discrimination in my own life. I just wanted to embrace the joy of coding and build futuristic technologies, or even real-world applications that focused on health, without needing to be bothered with taking down -isms. I also did not want to become a nuisance. Though all signs indicated otherwise, I wanted to believe that technology could be apolitical. And I hoped that if I could keep viewing technology and my work as apolitical, I would not have to act or speak up in ways that could put me at risk. Plus, I thought focusing on these ever-present factors too much would make me bitter and rob me of the jovial spirit that had driven me to optimistically pursue male-dominated fields. Work hard enough, I believed, and you could transcend the -isms or at least minimize them to a point where your life trajectory wasn’t significantly altered by their gravity. There must be some truth to meritocracy. Hadn’t I made it to MIT? I wanted to hold on to that fiction instead of facing the harsher realities.

Exceptionalism also carried the danger of tokenism, which allowed systemic issues to be ignored by pointing to a few examples of supposed success while ignoring the more common story. I was often a poster child of progress, appearing in college marketing materials and conference brochures to show that change was possible. Still, I knew that given another set of cards, my life trajectory could have been very different. There are many hardworking, brilliant people who could be in my place if they’d had similar opportunities. I had to concede that many factors outside my control contributed to my ability to take advantage of educational pathways. In many ways, I was born lucky. This acknowledgment doesn’t take away from my hard work, but it does expand the conversation to be less about one individual and more about the societies we have created.

In high school, my favorite track event was pole vaulting. Not only did I enjoy the physical challenge, which required the speed of a sprinter, the strength of a thrower, and the body awareness of a gymnast, but it also gave me a metaphor for life. I learned early on with pole vaulting that where you fix your eyes is where you ultimately land. Staring straight at the bar often led to colliding into the bar or just barely making it over. To execute a beautiful vault, you had to look beyond the bar and rise above it to the sky. Switching your mindset from bar gazing to star gazing allows your body to follow a more expansive vision. The older graduate students at MIT who had offered me their advice were bar gazing in their attempt to finish. I still had time. I wanted to star gaze.

My favorite poster was the largest of them all. Printed in white ink was the title “Algorithmic Justice League”—the name I was using to describe the work I was doing. The name follows the “justice league” banner that many others have used since the turn of the twentieth century—decades before DC Comics adopted the term for their fictional worlds—to fight for societal change. In the early twentieth century, civic organizations used the phrase “justice league” in their fight for women’s suffrage (“The Equal Justice League of Young Women” [1911]), racial equality and civil rights for African Americans (“Race Justice League” [1923]), and workers’ rights (“Justice League” [1914]). Scores of justice-oriented organizations continue to tap into this tradition today. Real-world justice leagues serve as inspiration for the belief that against tyranny, oppression, and erasure, we can choose to resist and offer pathways to liberation. I positioned the emerging work I was doing with the Algorithmic Justice League to follow this banner.

I started to notice a repeating experience. A fair-skinned person would try the interactive Upbeat Walls and have their face detected and the music start to play. Someone with darker skin would try without luck until they put on the white mask I had put on the table. I overheard a fair-skinned person say, “It works so well for me, I didn’t even imagine it wouldn’t work for someone else.” And someone else, with a darker complexion, commented, “Dang, the machines can’t see us either?” Without seeing someone else struggle with the Upbeat Walls system, the person for whom it had worked just assumed that it worked for everyone. But when they realized that wasn’t the case, their new awareness was reinforced by watching a video about the coded gaze. And the person for whom the system failed to work correctly realized that it was the machine, not them, that was at fault. This was not a case of user error. There was something more under the hood, and that was precisely what the whole exhibition was about.

I learned there was a TEDxBeaconStreet event taking place in mid-November, and while speakers had already been selected, John Werner, the organizer, was a familiar face around the Media Lab. The main stage schedule was already full, but John offered a slot on the TEDxYouth@BeaconStreet program. It wasn’t what I’d hoped for, but it was enough of a crack of the door to get going. I worked on my talk and then I emailed John in an attempt to persuade him to give me a chance on the main stage. He paired me with a speaking coach and resources on how to give a compelling talk. By the time I had refined my talk and attended a few of the practice sessions, John had invited me to the main stage, as long as I felt comfortable given the short time frame. He cautioned that I would have just a few weeks to prepare instead of the normal three months. This was the invitation I was waiting for. I accepted the challenge. On the day of the event, John greeted me and two friends who had come as emotional support. The day before, they had served as my audience as I practiced. By the time I committed my talk to memory they could repeat some of the parts with me. To add some flair, I decided to repurpose some of the elements of the MFA exhibition by bringing one of the newly minted shields with AJL etched on the front on top of a sly smile. Showtime had arrived. Backstage, a technician outfitted me with a “flesh color” microphone. The pale pink color didn’t quite match my gleaming milk chocolate...

“An unseen force is rising…that I call the coded gaze. It is spreading like a virus.” Slide by slide, the audience leaned in as I explained that from who gets hired or fired to even how much you pay for a product, algorithmic bias is ever present. Algorithmic bias occurs when one group is better served than another by an AI system. If you are denied employment because an AI system screened out candidates that attended women’s colleges, you have experienced algorithmic bias. Glancing down, I saw that the timer had expired. Running into overtime, I rushed to my final slide. “So I invite you to join me in creating a world where technology works for all of us, not just some of us…Will you join me in the fight?” The standing ovation was a resounding yes. Momentum was rising. At this point on my MIT academic path, I began to accept what I could do with the privilege of being at an institution with high visibility. Rather than feeling like I was giving up on my idealism of working on technology to escape painful realities, I could see how my childhood passions could live alongside my solidifying purpose to research harmful discrimination in technology. My life trajectory and educational opportunities were starting to make more sense to me. I felt emboldened to ask even more uncomfortable questions about the machines that once enamored me.

I walked to the front of the crowded room with my shield in hand. The wood-paneled lectern stood like a pedestal directing all eyes to the speaker. I began my presentation with two videos. In the first one I stare into a camera and say, “Hi, camera, can you see my face?” I pause. Nothing. “You can see my friend’s face.” The video cuts to the face of my friend Mary Maggic, a Chinese American speculative artist. Her face is quickly detected. “What about my face?” The camera returns to my face. I make an exaggerated pout on camera, drawing laughter from the audience. “I have a mask.” I put on the white mask, which is immediately detected. “Can you see my mask?” The laughter shifts to audible gasps. On the black screen, three white words linger: “The Coded Gaze.” I then presented another video. In this one I show a person walking in front of a car that is presumed to be self-driving. The car does not slow down and instead collides with the person. I used these videos to tee up the idea that computer vision technology was further infiltrating our lives and that the consequences of not being detected by a computer vision system were not just for chuckles but could in fact be grave. The AI techniques used to detect a face were similar to the techniques used to detect a body. I wanted to show that my focus on faces was only a starting point, and the implications of the work would reach beyond just the face space.

I call this approach of showing technical failures, to allow others to bear witness to ways technology could be harmful, evocative audits. The focus of my Media Lab master’s work would be “Unmasking Algorithmic Bias.” I ended the presentation with a fist pump and raised the AJL shield. Ethan shouted, “The shield is backwards.” I turned it around, ready to field questions. When the presentations were finished, a woman who had been sitting near the front approached me with a question. It was Cynthia Breazeal. Years after having Cynthia’s robot Kismet spark my curiosity, I stood in front of a woman I had long admired, engaging in conversation that had once been but a distant dream.

I wanted to defend my view and show the intellectual case for algorithmic bias. The comments inspired the title “Algorithms aren’t racist, your face is just too dark” for an article I wrote shortly after the TED attention. My white mask experience gave me the context that computer vision systems may have some racial bias. My use of bias was based on the idea of disadvantaging or privileging one group or another on the basis of race. Of course people have biases, but as one commenter put it, “There is no bias on math algorithms [sic].” There was a common assumption that these math-based systems make objective decisions; after all, one plus one equals two. Machines were presumed to be free from the societal biases that plague us mortals. My experiences were showing me otherwise. But as I began to discuss algorithmic bias with more people, I was often faced with variations of this hushed and sometimes not so hushed question: Isn’t the reason your face was not detected due to a lack of contrast given your dark complexion? (In other words, algorithms aren’t racist—your skin is just too dark.)

Even though cameras may appear neutral, history reveals another story. The film used in analog cameras was exposed using a special chemical composition to bring out desired colors. To calibrate the cameras to make sure those desired colors were well represented, a standard was created. This standard became known as the Shirley card, which was originally an image of a white woman used to establish the ideal composition and exposure settings. The consequence of calibrating film cameras using a light-skinned woman was that the techniques developed did not work as well for people with darker skin. In fact, it wasn’t until furniture and chocolate companies complained that the rich browns of their products were not being well represented that Kodak introduced a new product that better captured a range of browns and dark sepia tones. Separating the appearance of milk chocolate from dark chocolate in advertising had a windfall effect for chocolate-hued individuals.

Default settings are not neutral. They often reflect the coded gaze—the preferences of those who have the power to choose what subjects to focus on. But history has also shown us that alternative systems can be made. In the digital era, the LDK camera series developed by Philips explicitly handled skin tone variation with two chips—one for processing darker tones and another for processing lighter tones. The Oprah Winfrey Show used the LDK series for filming because there was an awareness of the need to better expose darker skin, given the show’s host and guests.

Though this example focuses on a face, computer vision can also be applied to attempts to detect cancer or a pedestrian crossing the street. I am less concerned about optimizing computers to detect faces and more interested in understanding how we train machines to see. The white mask demonstration is an entry point to larger conversations about bias in artificial intelligence and the people who can be harmed by these systems.

I think of artificial intelligence as the ongoing quest to give computers the ability to perceive the world (that is, make meaning of visual, aural, and other sensory inputs), to make judgments, to generate creative work, and to give them the ability to communicate with humans.

Machines can also analyze your behavior and data collected about you to make recommendations that shape our decisions. The decisions can be low-stakes, like Netflix’s ability to suggest another film or TV series to binge based on the user’s inferred preferences and viewing history. But the decisions can also include more high-stakes situations. For example, AI systems used for employment can recommend a short list of candidates to hire. AI systems used in healthcare can provide recommendations on which patients receive tailored care and which ones do not.[1] Very quickly, we can see how number crunching is not so neutral when those numbers can crunch your life.

A major challenge of neural networks is that during the training process computer scientists do not always know exactly why some weights are strengthened and others are weakened. As a result, current methods do not allow us to explain in full detail how a neural network recognizes a pattern like a face or outputs a response to a prompt. You may hear the term “black box” used to describe AI systems because there are unexplainable components involved. While it is true that parts of the process evade exact explanations, we still have to make sure we closely examine the AI systems being developed. Access to the training data is crucial when we want to have a deeper understanding of the risks posed by an AI system. Unless we know where the data comes from, who collected it, and how it is organized, we cannot know if ethical processes were used. Was the data obtained with consent? What were the working conditions and compensation for the workers who processed the data? These questions go beyond the technical.

In my work, I use the coded gaze term as a reminder that the machines we build reflect the priorities, preferences, and even prejudices of those who have the power to shape technology. The coded gaze does not have to be explicit to do the job of oppression. Like systemic forms of oppression, including patriarchy and white supremacy, it is programmed into the fabric of society. Without intervention, those who have held power in the past continue to pass that power to those who are most like them.

When Google launched Bard, an answer to OpenAI’s ChatGPT, the company decided to show off the system’s capabilities. In a segment on the television show 60 Minutes, the Bard system recommended and summarized five books, dazzling the host.[7] After the 60 Minutes team looked up the books the system recommended, they found out the books did not exist. The titles were made up. A chatbot confidently responding with made-up information is referred to by some AI developers as “hallucination.” Author and cultural critic Naomi Klein observes that the term hallucination is a clever way to market product failures. It sounds better than saying the system makes factual mistakes or presents nonsense as facts. These frequent errors are a demonstration of the AI functionality fallacy and a reminder that appearing knowledgeable isn’t the same as being factual.

Some companies and researchers go as far as claiming that their systems can predict someone’s sexual orientation, political affiliation, intelligence, or likelihood of committing a crime based solely on their facial features.[8] I still remember my disbelief when I came across a 2017 study where the authors used images of more than eighteen hundred people to create a classifier to predict criminality based on a face image.[9] I was also alarmed when I read a September 2017 article in The Economist about Stanford researchers who made classifiers to categorize someone’s sexual orientation based solely on an image of their face. In the article, one of the researchers suggested that with the right dataset they could “spot other intimate traits, such as IQ or political views.”[10] Predicting someone’s internal state, identity, or future behavior based on facial features alone is unreliable, but the impact can still be consequential. Being labeled criminal or homosexual can lead to harmful discrimination and even death.

Labels matter, and so we must be extremely skeptical about claims any company or researcher makes about using external features to predict psychological states, innate capabilities, or future behaviors.

The civil liberties organization Big Brother Watch released a report in 2018 documenting that the United Kingdom’s Metropolitan Police Department had piloted facial recognition systems that wrongly matched innocent members of the public with criminal suspects more than 98 percent of the time. The South Wales Police did slightly better with 91 percent false matches. In the process, 2,451 individuals unknowingly had their faces scanned by the department and stored for twelve months.[13] The sooner I could run experiments, the sooner I could gather evidence to help organizations like Big Brother Watch stop harmful use of facial recognition technologies.

Now that you know the differences between facial verification (one-to-one matching) and facial identification (one-to-many matching), you can see why the meaning of the term “facial recognition” needs to be clearly defined when we talk about policy. If we passed a law about facial recognition and defined the term to mean only one-to-one matching, it would not cover instances of facial identification being used for mass surveillance, like during a protest or in a department store. If facial recognition is defined to mean only one-to-many matching, then the law would not cover cases when an asylum seeker or senior citizen is attempting to access government services using facial verification.

When companies require individuals to fit a narrow definition of acceptable behavior encoded into a machine learning model, they will reproduce harmful patterns of exclusion and suspicion.

it was Mitch’s turn to speak up. “Work on what excites you!” He wanted me to enjoy the process and pursue the ideas that really moved me. Mitch was alluding to a learning approach he’d outlined in his book Lifelong Kindergarten, where creative learning is supported by four p’s: projects, passion, peers, and play. “Play” was what he was invoking in this moment, the idea of keeping an open and curious spirit, allowing for happy accidents and unanticipated pathways to emerge. This goes hand in hand with the idea of hard fun, a term conceptualized by the mathematician and AI pioneer Seymour Papert. Hard fun is what’s happening when we willingly take on difficult subjects, or work through mundane tasks, because we’re working on projects that impassion and excite us. Hard fun is what my high school classmates and I were experiencing playing with Mitch’s LEGO Mindstorm kits during lunch period, and what we experienced in Jill Connell’s classroom at Cordova High, when we were wading into the technically challenging, yet alluring, world of computer science. Or in my science fabrication class where, ignoring the siren song of a beautiful New England autumn, I’d spend entire days beneath the fluorescent lights of a lab space building my Aspire Mirror. As Mitch suggests, a spirit of play and a commitment to hard fun—especially in computer science and engineering, which some perceive as dry and extremely challenging—can lead to big breakthroughs.

I imagined AJL becoming a network of individuals from different backgrounds working together to uncover what ailed artificial intelligence so we could create better systems that prevented harms instead of perpetuating them. While we were at it, I wanted to maintain a playful attitude that kept the work inviting to outsiders and helped me go through the grind of day-to-day research. The work ahead would be tedious, but the results could transform the trajectory of AI.

In 2014 Hu Han and Anil Jain examined the demographic composition of LFW; they found that the database of images contained 77.5 percent male-labeled faces and 83.5 percent faces labeled white.[3] The gold standard for facial recognition, it turned out, was heavily skewed. I started calling these “pale male datasets.”

My example of coding in a white mask raised the question of whether the skew away from darker skin was also compounded by the face data collection methods themselves. To overcome this challenge, IJB-A was collected without a face detector and was curated by humans to further increase the difficulty level. Given the aim of demographic diversity with this dataset, I decided to label it to see its composition. Even with IJB-A, developed in 2015, eight years after LFW, I found that the dataset was 75.4 percent male and also 79.6 percent lighter-skinned individuals. This concerned me: Not only did we have a research gold standard that was heavily skewed, but the government agency NIST, which had set out to collect intentionally diverse datasets, also had a dataset that underrepresented women and people of color.

WHEN MACHINE LEARNING IS USED to diagnose medical conditions, to inform hiring decisions, or even to detect hate speech, we must keep in mind that the past dwells in our data. In the case of hiring, Amazon learned this lesson when it created a model to screen résumés.[7] The model was trained on data of prior successful employees who had been selected by humans, so the prior choices of human decision-makers then became the basis on which the system was trained. Internal tests revealed that the model was screening out résumés that contained the word “women” or women-associated colleges. The system had learned that the prior candidates deemed successful were predominantly male. Past hiring practices and centuries of denying women the right to education coupled with the challenges faced once entering higher education made it especially difficult to penetrate male-dominated fields. Faithful to the data the model was trained on, it filtered out résumés indicating a candidate was a woman. This was the by-product of prior human decisions that favored men. At Amazon, the initial system was not adopted after the engineers were unable to take out the gender bias. The choice to stop is a viable and necessary option. The face datasets I examined revealed data that was not representative of society. The example of the Amazon hiring model illustrates what happens when data does indeed reflect the assumptions of society. Their model reflected power shadows. Power shadows are cast when th...

The work of Nina Jablonski on skin distribution around the world shows the majority of the world’s populations have skin that would be classified on the darker end of most skin classification scales. Returning to the government IJB-A dataset that was created to have the widest geographic diversity of any face dataset, how was it that the dataset still was more than 80 percent lighter-skinned individuals?[9]

Stepping beyond a colonial past does not decolonize the mind. White supremacy as a cultural instrument, like the white gaze, defines who is worthy of attention and what is considered beautiful or desirable. Colorism is a stepchild of white supremacy that is seldom discussed. Colorism operates by assigning high social value and economic status based literally on the color of someone’s skin so that even if two people are grouped in the same race, the person with lighter skin is treated more favorably. We can see this in Hollywood and Bollywood. India with its vast diversity of skin types has an entertainment and beauty industry that elevates light-skinned actors and actresses. Women are judged on their beauty, and the standard of beauty is predicated on proximity to fair skin. Beyond beauty, lighter skin is also associated with having more intelligence in societies touched by white supremacy. Hollywood has long favored white actors, and when it began to open up slightly, lead roles for diverse cast members also skewed to the lighter hue. This is not to say that at the time I was doing this research there were no dark-skinned individuals who had gained fame or were positioned as intelligent. But the fact that they were the exception and not the norm is the point.

Diving into my study of facial recognition technologies, I could now understand how, despite all the technical progress brought on by the success of deep learning, I found myself coding in whiteface at MIT. The existing gold standards did not represent the full sepia spectrum of humanity. Skewed gold standard benchmark datasets led to a false sense of universal progress based on assessing the performance of facial recognition technologies on only a small segment of humanity. Unaltered data collection methods that rely on public figures inherited power shadows that led to overrepresentation of men and lighter-skinned individuals. To overcome power shadows, we must be aware of them. We must also be intentional in our approach to developing technology that relies on data. The status quo fell far too short. I would need to show new ways of constructing benchmark datasets and more in-depth approaches to analyzing the performance of facial recognition technologies. By showing these limitations, could I push for a new normal?

I experimented with having workers on Amazon Mechanical Turk (a platform that allowed researchers to put out low-priced micro tasks for crowdsource workers to complete) assign age, gender, and race labels to images from an existing face dataset. The same faces would be shown to multiple workers known as turkers, and I would examine the labels. When it came to age, having turkers guess a defined range instead of a specific age produced more consistent results. When it came to guessing race, I first used categories from the U.S. Census and left open an “other” category. The results of that category provided the most insights. It became clear that the Census categories were ill-equipped to deal with people the turkers perceived as South Asian, Southeast Asian, Middle Eastern, or of mixed race.

Examining the different labels turkers gave to the same face made me see the extent of guesswork that went into attempting to categorize perceived race. After the turkers’ experiments, I started using “perceived race” instead of “race” when talking about classification. Setting up the micro tasks also gave me power and privileged my perspectives. It was my human choice to select categories for classification that the turkers were then boxed into attempting to fit. My own choice of classification categories was informed by how others had grouped people in the past. I looked to existing systems as a starting place, such as the U.S. Census race categories, which have evolved over the centuries and reflected the social, political, and economic context of the day. For example, the U.S. Census, which started in 1790, did not allow people to categorize themselves into multiple racial groups until 2000. It was in 1960 when census takers could first self-identify how they fit into the given options. The categories themselves evolved, with enslaved people becoming “colored,” then “Black,” then “Negro.” The term “African American” debuted in the 2020 census.[1] None of this labeling felt precise, and my choices, just like the census labels, were not neutral. They were subject to the time period and shaped by those who held decision-making power.

When I visited Cape Town, South Africa, in 2019 during a tech conference, I went to the “Classification Building,” where people could go to have their hair and even their most private parts examined to determine race. The example of Sandra Liang, born to white Afrikaans parents but presenting in a way classified as colored, is an example that reveals how race is constructed. It was thus possible for parents classified white to birth a daughter classified colored who was ostracized by the white community and eventually found refuge in a township. In places like Canada, the term “visible minority” is used, a term that acknowledges that outward appearance is what is used to make assumptions about race that hold real social consequences. In the Canadian case, and in places like the United Kingdom and the United States, racial categorization is used by the government to better understand where to allocate resources and support minority groups or those who might face discrimination. In the U.S. case, I found a contradiction in this stated goal and the existing census classification, which included Middle Easterners in the group of white. The treatment of and discrimination against individuals perceived as white and those perceived as Middle Eastern differed, particularly after the September 11th World Trade Center attack. This would hardly be the only contradiction to be uncovered.

some researchers nonetheless attempted to create machine learning models to guess race and/or ethnicity, oftentimes not distinguishing the two. Some studies were so crude it was almost comical—their labels included “white” and “non-white.” Others tried to borrow from existing classification systems and used labels like “caucasoid” and “negroid,” classifications that have roots in eugenics and scientific racism. I even found a website called Ethnic Celebrities and created a system that collected the images with a combination of race and ethnic description of all the celebrities. I went as far as finding one of the most comprehensive ethnic classification systems from the Australian Bureau of Statistics, which enumerates more than 270 different cultural and ethnic groups.[2] Despite my efforts to try to make some technical distinctions, my attempt to label celebrities by ethnicity broke down, particularly when dealing with people who were multiracial. Then there was the challenge of trying to tease out how people identified as Hispanic should be classified. On the U.S. Census, Hispanic is the only ethnicity group included that can be applied to different race groups (that is, “White, Hispanic” or “Black, Hispanic”). Knowing I wasn’t the first to attempt this work of defining racial classification for face datasets, I turned to NIST. They hadn’t fared much better. Some of their studies used “white” and “black” labels, while more comprehensive studies eschewed racial categorie...