Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor

by Virginia Eubanks

in the absence of sufficient public investment in building or repurposing housing, coordinated entry is a system for managing homelessness, not solving it.

“Welfare records are available to law enforcement officers simply upon request—without probable cause, suspicion, or judicial process of any kind.”

Police presence at a social service organization is sufficient to turn away the most vulnerable unhoused, who might have outstanding warrants for status crime tickets associated with being homeless.

said Blasi. “But homelessness is not a systems engineering problem. It’s a carpentry problem.”

Three-quarters of child welfare investigations involve neglect rather than physical, sexual, or emotional abuse.

The pairing of the human discretion of intake screeners like Pat Gordon with the ability to dive deep into historical data provided by the predictive risk model is the most important fail-safe of the system. “This is the place where we have the least information,” said Erin Dalton. “The callers don’t know that much. We know a lot about these families. There’s so much history [in the data]. We can make a more informed recommendation.” Pat walks me through Krzysztof’s case.

Data scientist Cathy O’Neil has written that “models are opinions embedded in mathematics.”8 Models are useful because they let us strip out extraneous information and focus only on what is most critical to the outcomes we are trying to predict. But they are also abstractions. Choices about what goes into them reflect the priorities and preoccupations of their creators. Human decision-making is reflected in three key components of the AFST: outcome variables, predictive variables, and validation data.

This is essential for us to remember and consider whenever we build models.

When Janine says that “everybody is a risk,” she doesn’t mean that anyone might hit their child. She means that every parent in her community could be profiled by the AFST, simply for being poor and Black. According to statistics gathered by the National Council of Juvenile and Family Court Judges, in 37 states, the Dominican Republic, and Puerto Rico, African American and Native American children are removed from their homes at rates that significantly exceed their representation in the general population. For example, in 2011, 51 percent of children in foster care in Alaska were Native American, though Native Americans make up only 17 percent of the youth population. In Illinois, 53 percent of the children in foster care were African American, though African Americans make up only 16 percent of the youth population. In 2016, 48 percent of children in foster care in Allegheny County were African American, though they made up only 18 percent of the county’s children and youth.

I wish we could get data to demonstrate that physical and sexual child abuse is equally common across race & economic status - whereas "neglect" might be more common among the poor because it is due to poverty (i.e. how can you feed your kid if you don't have food). People are tagged by the models because of how poverty has impacted humans decisions to report cases - but poverty is not really predictive of abuse, its just predictive of someone catching the abuser.

A quarter of the predictive variables in the AFST are direct measures of poverty: they track use of means-tested programs such as TANF, Supplemental Security Income, SNAP, and county medical assistance.

Another quarter measure interaction with juvenile probation and CYF itself, systems that are disproportionately focused on poor and working-class communities, especially communities of color.

According to the U.S. Department of Health and Human Services Administration for Children and Families, of the 3.4 million children involved in child welfare investigations in 2015, 75 percent were investigated for neglect, while only a quarter were investigated for physical, emotional, or sexual abuse.

Nearly all of the indicators of child neglect are also indicators of poverty: lack of food, inadequate housing, unlicensed childcare, unreliable transportation, utility shutoffs, homelessness, lack of health care.

CYF can offer parents a multitude of useful resources: respite care for a new mom who needs an hour off to do some laundry, early childhood education and development programs, even a visiting home aid to help with household chores. But professional middle-class families rely instead on private sources for family support, so their interactions with helping professionals are not tracked or represented in the data warehouse.

Like racial profiling, poverty profiling targets individuals for extra scrutiny based not on their behavior but rather on a personal characteristic: living in poverty. Because the model confuses parenting while poor with poor parenting, the AFST views parents who reach out to public programs as risks to their children.

Parenting while poor means parenting in public.

When we passed the anguished man near the Los Angeles Public Library and did not ask him if he needed help, it was because we have collectively convinced ourselves that there is nothing we can do for him. When we failed to meet each others’ eyes as we passed, we signaled that, deep down, we know better.

beautifully convicting

Rational discrimination does not require class or racial hatred, or even unconscious bias, to operate. It only requires ignoring bias that already exists. When automated decision-making tools are not built to explicitly dismantle structural inequities, their speed and scale intensify them.

historically, we have only made headway against persistent poverty when mass protest compelled substantial federal investment.

“Through our scientific and technological genius, we have made of this world a neighborhood and yet we have not had the ethical commitment to make of it a brotherhood,” he said. “But somehow, and in some way, we have got to do this.… We are tied together in the single garment of destiny, caught in an inescapable network of mutuality.”

Journalist Monica Potts suggests that we can only tolerate illustrations of suffering, litanies of misery, or morality plays of bad choices and their consequences. It is as if telling stories of economic hardship allows only two lessons, she writes: “‘You should feel sorry for the poor’ or ‘You shouldn’t.’”

The insistence that there is a “culture of poverty” takes on the character of a bizarre and delusional mantra when we understand that poverty is a majority experience in the United States.

Does the tool increase the self-determination and agency of the poor? Would the tool be tolerated if it was targeted at non-poor people?

imaginations and asking entirely different kinds of questions: How would a data-based system work if it was meant to encourage poor and working-class people to use resources to meet their needs in their own ways? What would decision-making systems that see poor people, families, and neighborhoods as infinitely valuable and innovative look like? It will also require sharpening our skills: high-tech tools that protect human rights and strengthen human capacity are more difficult to build than those that do not.

Oath of Non-Harm for an Age of Big Data I swear to fulfill, to the best of my ability, the following covenant: I will respect all people for their integrity and wisdom, understanding that they are experts in their own lives, and will gladly share with them all the benefits of my knowledge. I will use my skills and resources to create bridges for human potential, not barriers. I will create tools that remove obstacles between resources and the people who need them. I will not use my technical knowledge to compound the disadvantage created by historic patterns of racism, classism, able-ism, sexism, homophobia, xenophobia, transphobia, religious intolerance, and other forms of oppression. I will design with history in mind. To ignore a four-century-long pattern of punishing the poor is to be complicit in the “unintended” but terribly predictable consequences that arise when equity and good intentions are assumed as initial conditions. I will integrate systems for the needs of people, not data. I will choose system integration as a mechanism to attain human needs, not to facilitate ubiquitous surveillance. I will not collect data for data’s sake, nor keep it just because I can. When informed consent and design convenience come into conflict, informed consent will always prevail. I will design no data-based system that overturns an established legal right of the poor. I will remember that the technologies I design are not aimed at data points, probabilities, or patterns, but at...