Unmasking AI: My Mission to Protect What Is Human in a World of Machines

by Joy Buolamwini

Looking at how machine learning models are being developed, we see the impact of what I first heard Kate Crawford term the “politics of classification.” Having the ability to define classification systems is in itself a power. The choices that are made are influenced by cultural, political, and economic factors, and while these classifications don’t have to be based on definite distinctions, they still have an impact on individual lives and societal attitudes. Despite the power shadows inherent in classification systems, often those systems go unchallenged. Instead, they are used as shorthand. Morgan Klaus Scheuerman and their colleagues have done tremendous scholarship on how gender classification systems lead to erasure, reification of social constructs, and the reinforcement of gender stereotypes.[5] In my research, I had to contend with understanding the limitations of classification systems we use as well as acknowledging those systems’ utility in showing discrimination, unfairness, or inequitable treatment. Yes, the classification systems were problematic, but in using them I could show their limitations as well as the limitations of the other systems I examined. To contend with those limitations, it is important to call out the assumptions of the classifications that are being used so there isn’t a universal acceptance that these are the ones that must be used or that they do not have problems.

I found it interesting to see African countries that I’d been socialized to think of as being behind in the world leading on gender representation. Other explorations I had for potential datasets included looking at Olympic teams. Like the UN, the Olympics brought together a world of nations, yet I also had to attend to the fact that elite athletes were not exactly representative of the typical population in ability, physical form, or age. In some ways an Olympic dataset would bring the same concerns as a celebrity dataset.

I reached out to the Boston University Technology Law Clinic for help. They looked at the copyright laws for all the countries I targeted and assessed that as long as I did not redistribute the dataset for profit and kept its use for research purposes I should be clear and within the realm of fair use. Nevertheless, lawful use did not overcome the basic fact that I would be using images of people’s faces without their consent to do this research, unless I could somehow obtain the consent of the 1,270 individuals who would be included in the dataset. Despite my struggle with this question, other computer vision researchers I spoke to felt these questions were irrelevant. Their position was: The photos are out there, and they are photos of public officials. What is your issue? With some hesitation I proceeded. The lawful yet awful standard allows practices that have questionable ethics. Maybe we needed a higher standard than “What are my chances of getting sued?”

When we look at proclamations about the performance of AI systems, we must have insight into the types of tests that were used to reach the conclusion, the data collection methods, the labels used, and who was involved in decision-making. Yet as I went through each face in my Pilot Parliaments Benchmark to determine gender and skin type, the subjectivity of the exercise became more apparent. We cannot ignore this subjectivity as more AI systems are introduced into our lives, and we need to push for high standards for AI deployers to support the claims they make about what their systems can do. We also need redlines and guardrails that prevent harmful if unintended consequences from the use of data.

Shortly after Magic Avatars gained popularity, some women started noticing that the avatars generated for them included hypersexualized images. Some women received avatars with their likeness depicted on scantily clad bodies or completely topless. Melissa Heikkilä wrote for the MIT Technology Review, “My avatars were cartoonishly pornified, while my male colleagues got to be astronauts, explorers, and inventors.”

Storing and distributing images and videos of child pornography is a criminal offense in the United States. Who is culpable if an AI system produces sexualized images of an adult when they upload childhood photos? What stops a nefarious actor from intentionally uploading the image of a child to get sexualized images? At the time of writing there are no laws that ban the generation of sexualized images of children from AI systems. Governments and companies have a responsibility to do more.

Images uploaded to the internet for one purpose are often repurposed without explicit consent. Artists who already struggle to make a living based on their creative practice have expressed alarm at the use of their work to fuel generative AI systems.

By the time someone finds out their images have been used, the company has already trained the AI system, so deleting the image doesn’t delete its contribution to training the AI system to do a specific task. Even if the image is deleted, the many copies of the dataset already made still contain the images. This is why when Meta (then Facebook) announced the deletion of nearly 1 billion faceprints, there was pushback alongside the celebration. The celebration was around the fact that a major tech company deleting the faceprints was an acknowledgment of the risks associated with face-based technologies that many organizations, including the AJL, had been highlighting. But we do not know if someone at the organization made a secret backup copy, and we may never know. Facebook’s actions provided a counternarrative to the assumption that once a company has your data there is nothing that can be done about it. Public pressure makes a difference, and so too does legislation and litigation. While the company deleted the faceprints, they did not delete the facial recognition models created with those faceprints. The company had legal reasons to rid themselves of the faceprints. In 2021, Facebook reached a $650 million settlement for allegations of violating BIPA, the Biometric Information Privacy Act of Illinois.[13] Crucially, faceprints are derived from images of a person’s face, but they are not a copy of the uploaded images. Think of the faceprint as a digital representation o...

We need deep data deletion, which consists of fully deleting data derived from user uploads and the AI models trained with this data. Commercial AI products should be built only on explicitly consented data sources. We should mandate deep data deletions to prevent the development of AI harms that stem from the collection of unconsented data. Companies like Clearview AI, which scraped billions of photos of people’s faces uploaded to public social media platforms, was fined 20 million euros by the Italian Supervisory Authority and ordered to delete the biometrics data of “persons in the Italian territory.”[14] The Italian face purge is a start. Deep deletions are possible, but achieving them will require a transformation in the development, auditing, and regulation of AI systems.

many online systems are linked to tech giants like Google and Amazon. For the Lensa app, the face data is processed in part using Amazon Web Services (AWS). AWS powers a significant part of the internet and is part of critical internet infrastructure. Companies like Netflix and Zoom are reliant on AWS.[15] Thus you have to think not just about what one company like Prisma Labs can do with your data, but also what interlinked companies that run the computing systems to process your data can do. There is a daisy chain of companies involved with internet and app-based AI services.

The responsibility of preventing harms from AI lies not with individual users but with the companies that create these systems, the organizations that adopt them, and the elected officials tasked with the public interest. What we can do as individuals is share our stories, document harms, and demand that our dignity be a priority, not an afterthought. We can think twice before participating in AI trends like stylized profile images, and we can support organizations that put pressure on companies and policymakers to prevent AI harms.

The classification systems I or other machine learning practitioners select, modify, inherit, or expand to label a dataset are a reflection of subjective goals, observations, and understandings of the world. These systems of labeling circumscribe the world of possibilities and experience for a machine learning model, which is also limited by the data available. For example, if you decide to use binary gender labels—male and female—and use them on a dataset that includes only the faces of middle-aged white actors, the system is precluded from learning about intersex, trans, or nonbinary representations and will be less equipped to handle faces that fall outside its initial binary training set. The classification system erases the existence of those groups not included in it. It can also reify the groups so that if the most dominant classification of gender is presented in the binary male and female categorization, over time that binary categorization becomes accepted as “truth.” This “truth” ignores rich histories and observations from all over the world regarding gender that acknowledge third-gender individuals or more fluid gender relationships. Two-spirit people have long been recognized in Native American cultures. Hijra or Kinnar people in India can include transgender and intersex individuals.

I stopped saying I was labeling gender and started saying I was labeling perceived gender, which from my experience of doing the labeling was a more apt description. Perceived gender introduced the notion that someone was doing the perceiving, and this perception might not be the so-called truth of the matter.

In addition to applying binary gender labels, I also hand-labeled each face with what I considered to be the appropriate Fitzpatrick skin type on a six-point scale. Because there was already a precedent for applying race and ethnic labels, I felt if I used the familiar labels, I would be grappling with widely discussed topics about racial and ethnic discrimination. Instead, I was applying skin type labels from the Fitzpatrick scale, and I hesitated, because from personal experience I knew how skin color was often used for discrimination. While many talk about racism (discrimination and prejudice based on perceived race), few talk about colorism (discrimination and prejudice based on actual skin color), which can happen within a delineated racial group. Proximity to whiteness or lighter-colored skin comes with privileges even within a subordinated racial group. As I was growing up fascinated by Hollywood, the Black actresses, in particular, who got desirable roles for the most part would be considered light-skinned. Social clubs and sororities established by elite Black organizations in the United States even used the brown paper bag test as a form of social exclusion. Being too black, darker than the brown paper bag, meant rejection based on skin color.[1] Spending time with immigrant Indian friends, I learned of other forms of colorism around the world, which, as in the United States, influenced who was perceived as a suitable marriage partner, employee, or leader. For pe...

The subject of colorism can be taboo, as some see it as divisive in the push for racial justice. The cruelty of colorism is that it recapitulates social rejection and exclusion based on race into a hierarchy based on skin color. Just as a white scholar might shy away from talking about racism and the ways in which she benefits from systemic racism, not many Black scholars who have investigated race and technology have focused on colorism. I wondered if that lack was because some of the leading Black voices, on the privileged end of colorism, did not see it as a topic worthy of discussion, were uncomfortable addressing their own color privilege, or saw the topic as dirty laundry to be kept out of the spotlight.

The Fitzpatrick scale was based not on color alone, but on how skin responds to UV radiation from sunlight. This factor meant that skin type, while linked to skin color, had more elements at play, namely different kinds of melanin cells. Melanin comes in three flavors: eumelanin, pheomelanin, and neuromelanin. Eumelanin and pheomelanin affect skin color and hair color, while neuromelanin affects the color of the brain. Descriptions of skin type in the Fitzpatrick scale include these: “Type I skin—burns easily, light colored eyes, green/blue.” “Type VI, skin never burns.”

I noticed that as more people labeled the dataset, the skin itself was not the deciding factor. Instead, facial features, national origin, and perceived ethnicity also played into whether someone on the borderline of classification would be placed in a lighter or a darker category. If the person appeared as South Asian they were labeled as Type IV; if they had lighter perceived skin but were in an African parliament and not phenotypically white, they were placed in a darker category. Like gender, the cases that defied clear-cut classification were those that exposed our assumptions most clearly.

We cannot assume that just because something is data driven or processed by an algorithm it is immune to bias. Labels and categories we may take for granted need to be interrogated. The more we know about the histories of racial categorization, the more we learn about how a variety of cultures approach gender, the deeper we dive into the development of scientific scales like the Fitzpatrick scale, the easier it is to see the human touch that shapes AI systems. Instead of erasing our fingerprints from the creation of algorithmic systems, exposing them more clearly gives us a better understanding of what can go wrong, for whom, and why. AI reflects both our aspirations and our limitations. Our human limitations provide ample reasons for humility about the capabilities of the AI systems we create. Algorithmic justice necessitates questioning the arbiters of truth, because those with the power to build AI systems do not have a monopoly on truth.

In the land of these giants, I had to navigate carefully. Funding for AI research at leading universities was often sponsored by these companies. It was not lost on me that the Stata Center at MIT, which housed the Computer Science Artificial Intelligence Lab, had a Gates wing named after the founder of Microsoft. Some companies, like Google, IBM, and Microsoft, provided fellowships that would pay for or significantly subsidize the cost of completing graduate programs in computer science. After finishing a computer science degree, top students found competitive and compelling job offers from the tech giants, who were ensured a steady supply of promising research talent. Between financing specific areas of research, providing the means for earning expensive graduate degrees, and offering lucrative tech sector jobs, the shadow of the tech giants touched all aspects of the AI research ecosystem. Getting on the bad side of one of these companies could have serious career consequences. When I was a graduate student, however, I was not particularly interested in working for one of them. I had my tuition fully covered and received a stipend. The funding structure of the Media Lab, and particularly my affiliation with the Center for Civic Media, gave me more freedom than most graduate students researching AI systems would have had. My advisor, Ethan, was also not beholden to any of these tech companies to directly fund our research. Instead, Ethan sought support from organizations...

All companies overall performed better on male faces than female faces. Microsoft had the smallest accuracy gap with an 8.1 percent difference. Face++ had the largest gap with a difference of 20.6 percent.

Though the dataset was labeled by six skin types, to account for off-by-one errors, I grouped skin types I–III, which covered mainly the parliament members from Iceland, Finland, and Sweden, as the lighter-skinned group. I categorized skin types IV–VI as the darker-skinned group, which covered many of the parliament members in Senegal, Rwanda, and South Africa. Taking this step of phenotypic analysis at the time was a significant contribution to the field of computer vision. Phenotypic analysis demonstrated not just additional ways of understanding gender classifiers like the ones I was testing from these commercial products, but also how we might understand other areas of computer vision. For example, subsequent studies showed that pedestrian tracking systems were more likely to miss detecting people with darker skin than those with lighter skin. When applied to self-driving cars, the result would be that people with darker skin were at increased risk of being hit by a car with automated driving capabilities fully engaged.

The results of the phenotypic analysis showed that overall, Microsoft, IBM, and Face++ did better on lighter-skinned faces than darker-skinned faces when it came to guessing the gender of a face. If we were to stop at this level of single-axis analysis—that is, looking at gender in isolation and skin type in isolation—the assumption would be made that when it came to gender, regardless of skin type, systems performed better on men than women. Conversely, without digging deeper we might also assume that when it came to skin type, performance would be better on lighter skin than darker skin regardless of gender. These assumptions, which you might attempt to justify with the single-axis analysis, would, however, miss a more complex story. We needed to go another step further in the analysis, because we cannot assume, if the data has not been collected and tested, that the system performs the same across all groups. Furthermore, the burden of proof of performance needs to be placed on the people developing systems, not those who are impacted by their use.

What accuracy doesn’t reveal are questions around failure. When a system fails, how are the errors distributed? We should not assume equal distribution of errors. Beyond just looking at the accuracies of a system, we can also learn more about performance by looking at the kinds of errors that are made. For error analysis, I took it one step further and looked at the results by each skin type, revealing even worse performance. When I looked at women with the darkest skin on the Fitzpatrick skin type scale, I found that for female-labeled faces of type VI, error rates were up to 46.8 percent. To my knowledge at the time these were some of the biggest accuracy disparities recorded regarding commercially sold AI products. My test had exposed significant cracks in the system not just from small research labs but from some of the oldest and most respected tech giants. If the tech giants had bias in gender classification, what other biases could be lurking in their AI products?

I paid a visit to Hal Abelson. “Your calculations do show differences in accuracy, but what’s the harm? Why should people care?” Hal’s questions lingered with me. “Is being misclassified harmful?” “Does it matter if a parliament member is misgendered?” Maybe the example I had chosen was not compelling enough, or maybe having a committee of all men made them less sensitive to the implications of misgendering women? To me this was akin to announcing, “She was hit by a brick; look at the evidence” and then being asked, “How do we know being hit by a brick is harmful?” I could not assume everyone would see that the issue with gender classification was problematic in and of itself and also indicative that there could be similar issues in other areas of facial recognition technologies and AI. Being misclassified as a criminal suspect and being wrongfully arrested certainly seemed high stakes to me.

I was frustrated because results that I thought were substantial now felt diminished. Maybe I was making too big a deal of what I had uncovered. Or maybe this was part of the academic rite of passage that required you to defend the importance of your work to a skeptical audience of senior academics. I lobbed back questions of my own to Hal and the rest of the committee. “Should I show the faces of those who are mislabeled?” None of the committee saw an issue, but showing the mislabeled faces directly didn’t sit well with me. It felt like I would be framing an insult that had been applied to someone. I decided to show average faces, composite images that blended multiple people together so no specific identity could be seen when I presented specific numbers on errors.

Over the summer, Hal was teaching me that I could not assume everybody would interpret gender classification or misclassification as being harmful. He was pushing me to provide more context and articulate why the work mattered. He was also preparing me for the skepticism that others in the computer science community would have. At the same time, I was not doing the work for the computer science community alone, but for those who could find themselves on the wrong side of a label. Hal guided me to emphasize that the techniques used for gender classification were being used in other areas of computer vision and machine learning, and therefore these findings were a warning sign. “This might seem obvious to you, but you are going to need to draw the connections so people understand what you are talking about.”

Now with an accepted paper, I began sharing the results with all the companies. I emailed IBM’s Ruchir as well as Microsoft and Face++ representatives, since their companies were all implicated by the paper. I heard back from Ruchir almost immediately. He wanted to know who the competitors were. I had sent the paper results with all the findings, but instead of including the company names I used companies A, B, and C. They would have to wait until the conference to know for sure who their competitors were.

I presented the “Gender Shades” work to Ruchir Puri, IBM chief scientist; Francesca Rossi, head of AI and Ethics; Anna Sekaran, head of communications; and members of the computer vision team, who sat nervously as I advanced through the slides showing performance results. “Who are the other companies?” they queried. “You are going to have to wait and see…” After the formal presentation, I enjoyed a lunch with the team. Again I was asked, “Who are the other companies?” The meal was good, but it wasn’t that good. “You will have to wait and see.” Later that evening I went out for a truly delicious Italian meal with a few IBMers who had been part of the drill team examining my research. For a third time, I was asked, “Who are the other companies?” As I took another spoonful of pasta, I nearly slipped—but then I caught myself. “You are going to have to wait and see…”

After the NYC meeting, I had a few meetings in the IBM Boston office. The team let me know that they had developed a new model and wanted me to share the results at the conference. I insisted that I would have to test their new model at my office, so they took the trip to the Media Lab and met me right around the corner from the Foodcam. Their dress clothes gave corporate vibes as they walked past the ping-pong table in the atrium. Sitting next to my LEGOs, surrounded by grad school artifacts and the white mask, the IBM team members shared their model with me. The numbers did in fact look significantly better than the older model. One of the computer vision team members put his hands together in a prayer pose and looked me dead in the eyes. “Please share the updated results.” I understood the look in his eyes. His reputation was on the line. Just as there are people whose lives are impacted by the labels and classifications that machine learning systems put out, there are people on the other side of the equation whose careers and livelihoods are linked to the products I was scrutinizing. Before my visits to the IBM offices, the developers of these commercial systems were as faceless to me as the users who are impacted by these systems might have been to them. They were in some ways “just” doing their job in the ways they had been trained to do. But when just doing your job can mean someone else’s life chances are at stake, there has to be a higher standard. Doing your job ...

the assumptions I’d made in Cordova did not quite hold up in Kombolcha. At the time, the Google Android tablets I was programming did not have Amharic keyboards, yet most of the people who were being surveyed and the health workers needed to use Amharic. Our project team developed a custom Amharic keyboard for the Android tablets that had to be loaded onto every new device. Defaults are not neutral. Years later I met some Android team members and asked why keyboards were not available in Amharic then. The answer was business economics. To Google, Ethiopia was not a priority market. In the age of AI, who will decide what our priorities are? What assumptions about using AI for good are being made by people far removed from the day-to-day realities of those they aim to help? My desire to show compassion through computation had to be measured by my ignorance of what would truly be of help. I had taken a parachute approach, jumping into a location I knew little about. Though I like to think that I made a meaningful difference by improving the Carter Center’s data collection efforts, my experience in Ethiopia revealed the limits of good intentions and the need for local context.

My problem with effective altruism is that the approach entrenches the status quo. Supporting exclusive charities sidesteps addressing the issues that led to the rise of charities in the first place and does not require changing existing power relations or company practices.

Longtermists follow a tradition of showing concern for future descendants. Many ancient cultures have emphasized the importance of taking care of planet Earth in order that future generations can breathe clean air and drink from the bounty of nature. However, safeguarding future generations means taking care of present and addressable dangers.

The term “x-risk” is used as a shorthand for the hypothetical existential risk posed by AI. While my research supports why AI systems should not be integrated into weapons systems because of the lethal dangers, this isn’t because I believe AI systems by themselves pose an existential risk as superintelligent agents. AI systems falsely classifying individuals as criminal suspects, robots being used for policing, and self-driving cars with faulty pedestrian tracking systems can already put your life in danger. Sadly, we do not need AI systems to have superintelligence for them to have fatal outcomes on individual lives. Existing AI systems with demonstrated harms are more dangerous than hypothetical “sentient” AI systems because they are real. One problem with minimizing existing AI harms by saying hypothetical existential harms are more important is that it shifts the flow of valuable resources and legislative attention. Companies that claim to fear existential risk from AI could show a genuine commitment to safeguarding humanity by not releasing the AI tools they claim could end humanity. I am not opposed to preventing the creation of fatal AI systems. Governments concerned with lethal use of AI systems can adopt the protections long championed by the Campaign to Stop Killer Robots to ban lethal autonomous systems and digital dehumanization.[*3]

Though it is tempting to view physical violence as the ultimate harm, doing so makes it easy to forget pernicious ways our societies perpetuate structural violence. Johan Galtung coined this term to describe how institutions and social structures prevent people from meeting their fundamental needs and thus cause harm. Denial of access to healthcare, housing, and employment through the use of AI perpetuates individual harms and generational scars. AI systems can kill us slowly.

people who would never be invited to a dinner like this, I doubted I would be receiving any future private dinner invitations. I thought about the excoded—people being harmed now and those who are at risk of harm by AI systems. When I think of x-risk, I also think of the risk and reality of being excoded. You can be excoded when a hospital uses AI for triage and leaves you without care, or uses a clinical algorithm that precludes you from receiving a life-saving organ transplant.[6] You can be excoded when you are denied a loan based on algorithmic decision-making.[7] You can be excoded when your résumé is automatically screened out and you are denied the opportunity to compete for the remaining jobs that are not replaced by AI systems.[8] You can be excoded when a tenant screening algorithm denies you access to housing.[9] All of these examples are real. No one is immune from being excoded, and those already marginalized are at greater risk.