Jeremy Howard's Blog, page 2

AI Harms are Societal, Not Just Individual

Not just Individual, but Societal Harms Case Study: Privacy and surveillance Case Study: Disinformation and erosion of trust Individual Harms, Individual Solutions Parallels with Environmental Harms Directions ForwardNot just Individual, but Societal Harms

When the USA government switched to facial identification service ID.me for unemployment benefits, the software failed to recognize Bill Baine��s face. While the app said that he could have a virtual appointment to be verified instead, he was unable to get through. The screen had a wait time of 2 hours and 47 minutes that never updated, even over the course of weeks. He tried calling various offices, his daughter drove in from out of town to spend a day helping him, and yet he was never able to get a useful human answer on what he was supposed to do, as he went for months without unemployment benefits. In Baine��s case, it was eventually resolved when a journalist hypothesized that the issue was a spotty internet connection, and that Baine would be better off traveling to another town to use a public library computer and internet. Even then, it still took hours for Baine to get his approval.

Journalist Andrew Kenney of Colorado Public Radio has covered the issues with ID.me

Baine was not alone. The number of people receiving unemployment benefits plummeted by 40% in the 3 weeks after ID.me was introduced. Some of these were presumed to be fraudsters, but it is unclear how many genuine people in need of benefits were wrongly harmed by this. These are individual harms, but there are broader, societal harms as well: the cumulative costs of the public having to spend ever more time on hold, trying to navigate user-hostile automated bureaucracies where they can��t get the answers they need. There is the societal cost of greater inequality and greater desperation, as more people are plunged into poverty through erroneous denial of benefits. And there is the undermining of trust in public services, which can be difficult to restore.

Potential for algorithmic harm takes many forms: loss of opportunity (employment or housing discrimination), economic cost (credit discrimination, narrowed choices), social detriment (stereotype confirmation, dignitary harms), and loss of liberty (increased surveillance, disproportionate incarceration). And each of these four categories manifests in both individual and societal harms.

It should come as no surprise that algorithmic systems can give rise to societal harm. These systems are sociotechnical: they are designed by humans and teams that bring their values to the design process, and algorithmic systems continually draw information from, and inevitably bear the marks of, fundamentally unequal, unjust societies. In the context of COVID-19, for example, policy experts warned that historical healthcare inequities risked making their way into the datasets and models being used to predict and respond to the pandemic. And while it��s intuitively appealing to think of large-scale systems as creating the greatest risk of societal harm, algorithmic systems can create societal harm because of the dynamics set off by their interconnection with other systems/ players, like advertisers, or commercially-driven media, and the ways in which they touch on sectors or spaces of public importance.

Still, in the west, our ideas of harm are often anchored to an individual being harmed by a particular action at a discrete moment in time. As law scholar Natalie Smuha has powerfully argued, legislation (both proposed and passed) in Western countries to address algorithmic risks and harms often focuses on individual rights: regarding how an individual��s data is collected or stored, to not be discriminated against, or to know when AI is being used. Even metrics used to evaluate the fairness of algorithms are often aggregating across individual impacts, but unable to capture longer-term, more complex, or second- and third-order societal impacts.

Case Study: Privacy and surveillance

Consider the over-reliance on individual harms in discussing privacy: so often focused on whether individual users have the ability to opt in or out of sharing their data, notions of individual consent, or proposals that individuals be paid for their personal data. Yet widespread surveillance fundamentally changes society: people may begin to self-censor and to be less willing (or able) to advocate for justice or social change. Professor Alvaro Bedoya, director of the Center on Privacy and Technology at the Georgetown University Law Center, traces a history of how surveillance has been used by the state to try to shut down movements for progress�� targeting religious minorities, poor people, people of color, immigrants, sex workers and those considered ��other��. As Maciej Ceglowski writes, ��Ambient privacy is not a property of people, or of their data, but of the world around us�� Because our laws frame privacy as an individual right, we don��t have a mechanism for deciding whether we want to live in a surveillance society.��

Drawing on interviews with African data experts, Birhane et al write that even when data is anonymized and aggregated, it ��can reveal information on the community as a whole. While notions of privacy often focus on the individual, there is growing awareness that collective identity is also important within many African communities, and that sharing aggregate information about communities can also be regarded as a privacy violation.�� Recent US-based scholarship has also highlighted the importance of thinking about group level privacy (whether that group is made up of individuals who identify as members of that group, or whether it��s a ��group�� that is algorithmically determined - like individuals with similar shopping habits on Amazon). Because even aggregated anonymised data can reveal important group-level information (e.g., the location of military personnel training via exercise tracking apps) ��managing privacy��, these authors argue ��is often not intrapersonal but interpersonal.�� And yet legal and tech design privacy solutions are often better geared towards assuring individual-level privacy than negotiating group privacy.

Case Study: Disinformation and erosion of trust

Another example of a collective societal harm comes from how technology platforms such as Facebook have played a significant role in elections ranging from the Philippines to Brazil, yet it can be difficult (and not necessarily possible or useful) to quantify exactly how much: something as complex as a country��s political system and participation involves many interlinking factors. But when ��deep fakes�� make it ��possible to create audio and video of real people saying and doing things they never said or did�� or when motivated actors successfully game search engines to amplify disinformation, the (potential) harm that is generated is societal, not just individual. Disinformation and the undermining of trust in institutions and fellow citizens have broad impacts, including on individuals who never use social media.

Reports and Events on Regulatory Approaches to Disinformation

Efforts by national governments to deal with the problem through regulation have not gone down well with everyone. ��Disinformation�� has repeatedly been highlighted as one of the tech-enabled ��societal harms�� that the UK��s Online Safety Bill or the EU��s Digital Services Act should address, and a range of governments are taking aim at the problem by proposing or passing a slew of (in certain cases ill-advised) ��anti-misinformation�� laws. But there��s widespread unease around handing power to governments to set standards for what counts as ��disinformation��. Does reifying ��disinformation�� as a societal harm become a legitimizing tool for governments looking to silence political dissent or undermine their weaker opponents? It��s a fair and important concern - and yet simply leaving that power in the hands of mostly US-based, unaccountable tech companies is hardly a solution. What are the legitimacy implications if a US company like Twitter were to ban democratically elected Brazilian President Jair Bolsonaro for spreading disinformation, for example? How do we ensure that tech companies are investing sufficiently in governance efforts across the globe, rather than responding in an ad hoc manner to proximal (i.e. mostly US-based) concerns about disinformation? Taking a hands off approach to platform regulation doesn��t make platforms�� efforts to deal with disinformation any less politically fraught.

Individual Harms, Individual Solutions

If we consider individual solutions our only option (in terms of policy, law, or behavior), we often limit the scope of the harms we can recognize or the nature of the problems we face. To take an example not related to AI: Oxford professor Trish Greenhalgh et al analyzed the slow reluctance of leaders in the West to accept that covid is airborne (e.g. it can linger and float in the air, similar to cigarette smoke, requiring masks and ventilation to address), rather than droplet dogma (e.g. washing your hands is a key precaution). One reason they highlight is the Western framing of individual responsibility as the solution to most problems. Hand-washing is a solution that fits the idea of individual responsibility, whereas collective responsibility for the quality of shared indoor air does not. The allowable set of solutions helps shape what we identify as a problem. Additionally, the fact that recent research suggests that ��the level of interpersonal trust in a society�� was a strong predictor of which countries managed COVID-19 most successfully should give us pause. Individualistic framings can limit our imagination about the problems we face and which solutions are likely to be most impactful.

Parallels with Environmental Harms

Before the passage of environmental laws, many existing legal frameworks were not well-suited to address environmental harms. Perhaps a chemical plant releases waste emissions into the air once per week. Many people in surrounding areas may not be aware that they are breathing polluted air, or may not be able to directly link air pollution to a new medical condition, such as asthma, (which could be related to a variety of environmental and genetic factors).

There are parallels between air polllution and algorithmic harms

There are many parallels between environmental issues and AI ethics. Environmental harms include individual harms for people who develop discrete health issues from drinking contaminated water or breathing polluted air. Yet, environmental harms are also societal: the societal costs of contaminated water and polluted air can reverberate in subtle, surprising, and far-reaching ways. As law professor Nathalie Smuha writes, environmental harms are often accumulative and build over time. Perhaps each individual release of waste chemicals from a refinery has little impact on its own, but adds up to be significant. In the EU, environmental law allows for mechanisms to show societal harm, as it would be difficult to challenge many environmental harms on the basis of individual rights. Smuha argues that there are many similarities with AI ethics: for opaque AI systems, spanning over time, it can be difficult to prove a direct causal relationship to societal harm.

Directions Forward

To a large extent our message is to tech companies and policymakers. It��s not enough to focus on the potential individual harms generated by tech and AI: the broader societal costs of tech and AI matter.

But those of us outside tech policy circles have a crucial role to play. One way in which we can guard against the risks of the ��societal harm�� discourse being co-opted by those with political power to legitimise undue interference and further entrench their power is by claiming the language of ��societal harm�� as the democratic and democratising tool it can be. We all lose when we pretend societal harms don��t exist, or when we acknowledge they exist but throw our hands up. And those with the least power, like Bill Baine, are likely to suffer a disproportionate loss.

In his newsletter on Tech and Society, L.M. Sacasas encourages people to ask themselves 41 questions before using a particular technology. They��re all worth reading and thinking about - but we��re listing a few especially relevant ones to get you started. Next time you sit down to log onto social media, order food online, swipe right on a dating app or consider buying a VR headset, ask yourself:

How does this technology empower me? At whose expense? (Q16) What feelings does the use of this technology generate in me toward others? (Q17) What limits does my use of this technology impose upon others? (Q28) What would the world be like if everyone used this technology exactly as I use it? (Q37) Does my use of this technology make it easier to live as if I had no responsibilities toward my neighbor? (Q40) Can I be held responsible for the actions which this technology empowers? Would I feel better if I couldn��t? (Q41)

It��s on all of us to sensitise ourselves to the societal implications of the tech we use.

View more on Jeremy Howard's website »

Like • 0 comments • flag

Published on May 16, 2022 07:00

There's no such thing as not a math person

On the surface, I may seem into math: I have a math PhD, taught a graduate computational linear algebra course, co-founded AI research lab fast.ai, and even go by the twitter handle @math_rachel.

Yet many of my experiences of academic math culture have been toxic, sexist, and deeply alienating. At my lowest points, I felt like there was no place for me in math academia or math-heavy tech culture.

It is not just mathematicians or math majors who are impacted by this: Western culture is awash in negative feelings and experiences regarding math, which permate from many sources and impact students of all ages. In this post, I will explore the cultural factors, misconceptions, stereotypes, and relevant studies on obstacles that turn people off to math. If you (or your child) doesn��t like math or feels anxious about your own capabilities, you��re not alone, and this isn��t just a personal challenge. The below essay is based on part of a talk I recently gave.

me, teaching sorting algorithms, at an all-women coding academy in 2015 Myth of Innate Ability, Myth of the Lone Genius

One common myth is the idea that certain people��s brains aren��t ��wired�� the right way to do math, tech, or AI, that your brain either ��works that way�� or not. None of the evidence supports this viewpoint, yet when people believe this, it can become a self-fulfilling prophecy. Dr. Omoju Miller, who earned her PhD at UC Berkeley and was a senior machine learning engineer and technical advisor to the CEO at Github, shares some of the research debunking the myth of innate ability in this essay and in her TEDx talk. In reality, there is no such thing as ��not a math person.��

Dr. Cathy O��Neil, a Harvard Math PhD and author of Weapons of Math Destruction, wrote about the myth of the lone genius mathematician, ��You don��t have to be a genius to become a mathematician. If you find this statement at all surprising, you��re an example of what��s wrong with the way our society identifies, encourages and rewards talent�� For each certified genius, there are at least a hundred great people who helped achieve such outstanding results.��

Dr. Miller debunking the myth of innate ability, and Dr. O'Neil debunking the myth of the lone genius mathematician Music without singing or instruments

Imagine a world where children are not allowed to sing songs or play instruments until they reach adulthood, after spending a decade or two transcribing sheet music by hand. This scenario is absurd and nightmarish, yet it is analogous to how math is often taught, with the most creative and interesting parts saved until almost everyone has dropped out. Dr. Paul Lockhart eloquently describes this metaphor in his essay,A Mathematician��s Lament, on ��how school cheats us out of our most fascinating and imaginative art form.�� Dr. Lockhart left his role as a university math professor to teach K-12 math, as he felt that so much reform was needed in how math is taught.

Dr. David Perkins uses the analogy of how children can play baseball wthout knowing all the technical details, without having a full team or playing a full 9 innings, yet still gain a sense of the ��whole game.�� Math is usually taught with an overemphasis on dry, technical details, without giving students a concept of the ��whole game.�� It can take years and years before enough technical details are accumulated to build something interesting. There is an overemphasis on techniques rather than meaning.

What if math was taught more like how music or sports are taught?

Math curriculums are usually arranged in a vertical manner, with each year building tightly on the previous, such that one bad year can ruin everything that comes after. Many people I talk to can pinpoint the year that math went bad for them: ��I used to like math until 6th grade, when I had a bad teacher/was dealing with peer pressure/my undiagnosed ADHD was out of control. After that, I was never able to succeed in future years.�� This is less true in other subjects, where one bad history teacher/one bad year doesn��t mean that you can��t succeed at history the following year.

Gender, race, and stereotypes

Female teachers�� math anxiety affects girls�� math achievement: In the USA, over 90% of primary school teachers are female, and research has found ��the more anxious teachers were about math, the more likely girls (but not boys) were to endorse the commonly held stereotype that ��boys are good at math, and girls are good at reading�� and the lower these girls�� math achievement�� People��s fear and anxiety about doing math��over and above actual math ability��can be an impediment to their math achievement.��

Research across a number of universities has found that more women go into engineering when courses focus on problems with positive social impact.

Structural racism also impacts what messages teachers impart to students. An Atlantic article How Does Race Affect a Student��s Math Education? covered the research paper A Framework for Understanding Whiteness in Mathematics Education, noting that ��Constantly reading and hearing about underperforming Black, Latino, and Indigenous students begins to embed itself into how math teachers view these students, attributing achievement differences to their innate ability to succeed in math�� teachers start to expect worse performance from certain students, start to teach lower content, and start to use lower-level math instructional practices. By contrast, white and Asian students are given the benefit of the doubt and automatically afforded the opportunity to do more sophisticated and substantive mathematics.��

The mathematics community is ��an absolute mess which actively pushes out the sort of people who might make it better��

Dr. Harron's website, and some of the coverage of her number theory thesis, including on the Scientific American blog

Dr. Piper Harron made waves with her Princeton PhD thesis, utilizing humor, analogies, sarcasm, and genuine efforts to be accessible as she described advanced concepts in a ground-breaking way, very atypical for a mathematics PhD thesis. Dr. Harron wrote openly in the prologue of her thesis on how alienating the culture of mathematics is, ��As any good grad student would do, I tried to fit in, mathematically. I absorbed the atmosphere and took attitudes to heart. I was miserable, and on the verge of failure. The problem was not individuals, but a system of self-preservation that, from the outside, feels like a long string of betrayals, some big, some small, perpetrated by your only support system.�� At her blog, the Liberated Mathematician, she writes, ��My view of mathematics is that it is an absolute mess which actively pushes out the sort of people who might make it better.��

These descriptions resonate with my own experiences obtaining a math PhD (as well as the experiences of many friends, at a variety of universities). The toxicity of academic math departments is self-perpetuating, pushing out the people who could make them better.

The full talk

This post is based on the first part of the talk I gave in the below video, which includes more detail and a Q&A. The talk also includes recommendations about math apps and resources, as well as a framework for how to consider screentime. Stay tuned for a future fast.ai blog post covering math apps and screentime.

View more on Jeremy Howard's website »

Like • 0 comments • flag

Published on March 14, 2022 17:00

There's no such thing as not a math person

On the surface, I may seem into math: I have a math PhD, taught a graduate computational linear algebra course, co-founded AI research lab fast.ai, and even go by the twitter handle @math_rachel.

Yet many of my experiences of academic math culture have been toxic, sexist, and deeply alienating. At my lowest points, I felt like there was no place for me in math academia or math-heavy tech culture.

It is not just mathematicians or math majors who are impacted by this: Western culture is awash in negative feelings and experiences regarding math, which permate from many sources and impact students of all ages. In this post, I will explore the cultural factors, misconceptions, stereotypes, and relevant studies on obstacles that turn people off to math. If you (or your child) doesn��t like math or feels anxious about your own capabilities, you��re not alone, and this isn��t just a personal challenge. The below essay is based on part of a talk I recently gave.

me, teaching sorting algorithms, at an all-women coding academy in 2015 Myth of Innate Ability, Myth of the Lone Genius

One common myth is the idea that certain people��s brains aren��t ��wired�� the right way to do math, tech, or AI, that your brain either ��works that way�� or not. None of the evidence supports this viewpoint, yet when people believe this, it can become a self-fulfilling prophecy. Dr. Omoju Miller, who earned her PhD at UC Berkeley and was a senior machine learning engineer and technical advisor to the CEO at Github, shares some of the research debunking the myth of innate ability in this essay and in her TEDx talk. In reality, there is no such thing as ��not a math person.��

Dr. Cathy O��Neil, a Harvard Math PhD and author of Weapons of Math Destruction, wrote about the myth of the lone genius mathematician, ��You don��t have to be a genius to become a mathematician. If you find this statement at all surprising, you��re an example of what��s wrong with the way our society identifies, encourages and rewards talent�� For each certified genius, there are at least a hundred great people who helped achieve such outstanding results.��

Dr. Miller debunking the myth of innate ability, and Dr. O'Neil debunking the myth of the lone genius mathematician Music without singing or instruments

Imagine a world where children are not allowed to sing songs or play instruments until they reach adulthood, after spending a decade or two transcribing sheet music by hand. This scenario is absurd and nightmarish, yet it is analogous to how math is often taught, with the most creative and interesting parts saved until almost everyone has dropped out. Dr. Paul Lockhart eloquently describes this metaphor in his essay,A Mathematician��s Lament, on ��how school cheats us out of our most fascinating and imaginative art form.�� Dr. Lockhart left his role as a university math professor to teach K-12 math, as he felt that so much reform was needed in how math is taught.

Dr. David Perkins uses the analogy of how children can play baseball wthout knowing all the technical details, without having a full team or playing a full 9 innings, yet still gain a sense of the ��whole game.�� Math is usually taught with an overemphasis on dry, technical details, without giving students a concept of the ��whole game.�� It can take years and years before enough technical details are accumulated to build something interesting. There is an overemphasis on techniques rather than meaning.

What if math was taught more like how music or sports are taught?

Math curriculums are usually arranged in a vertical manner, with each year building tightly on the previous, such that one bad year can ruin everything that comes after. Many people I talk to can pinpoint the year that math went bad for them: ��I used to like math until 6th grade, when I had a bad teacher/was dealing with peer pressure/my undiagnosed ADHD was out of control. After that, I was never able to succeed in future years.�� This is less true in other subjects, where one bad history teacher/one bad year doesn��t mean that you can��t succeed at history the following year.

Gender, race, and stereotypes

Female teachers�� math anxiety affects girls�� math achievement: In the USA, over 90% of primary school teachers are female, and research has found ��the more anxious teachers were about math, the more likely girls (but not boys) were to endorse the commonly held stereotype that ��boys are good at math, and girls are good at reading�� and the lower these girls�� math achievement�� People��s fear and anxiety about doing math��over and above actual math ability��can be an impediment to their math achievement.��

Research across a number of universities has found that more women go into engineering when courses focus on problems with positive social impact.

Structural racism also impacts what messages teachers impart to students. An Atlantic article How Does Race Affect a Student��s Math Education? covered the research paper A Framework for Understanding Whiteness in Mathematics Education, noting that ��Constantly reading and hearing about underperforming Black, Latino, and Indigenous students begins to embed itself into how math teachers view these students, attributing achievement differences to their innate ability to succeed in math�� teachers start to expect worse performance from certain students, start to teach lower content, and start to use lower-level math instructional practices. By contrast, white and Asian students are given the benefit of the doubt and automatically afforded the opportunity to do more sophisticated and substantive mathematics.��

The mathematics community is ��an absolute mess which actively pushes out the sort of people who might make it better��

Dr. Harron's website, and some of the coverage of her number theory thesis, including on the Scientific American blog

Dr. Piper Harron made waves with her Princeton PhD thesis, utilizing humor, analogies, sarcasm, and genuine efforts to be accessible as she described advanced concepts in a ground-breaking way, very atypical for a mathematics PhD thesis. Dr. Harron wrote openly in the prologue of her thesis on how alienating the culture of mathematics is, ��As any good grad student would do, I tried to fit in, mathematically. I absorbed the atmosphere and took attitudes to heart. I was miserable, and on the verge of failure. The problem was not individuals, but a system of self-preservation that, from the outside, feels like a long string of betrayals, some big, some small, perpetrated by your only support system.�� At her blog, the Liberated Mathematician, she writes, ��My view of mathematics is that it is an absolute mess which actively pushes out the sort of people who might make it better.��

These descriptions resonate with my own experiences obtaining a math PhD (as well as the experiences of many friends, at a variety of universities). The toxicity of academic math departments is self-perpetuating, pushing out the people who could make them better.

The full talk

This post is based on the first part of the talk I gave in the below video, which includes more detail and a Q&A. The talk also includes recommendations about math apps and resources, as well as a framework for how to consider screentime. Stay tuned for a future fast.ai blog post covering math apps and screentime.

View more on Jeremy Howard's website »

Like • 0 comments • flag

Published on March 14, 2022 07:00

7 Great Lightning Talks Related to Data Science Ethics

I have been organizing and facilitating a series of Ethics Workshops for the Australian Data Science Network, featuring lightning talks by Australian experts on a range of topics related to data science ethics, including machine learning in medicine, explainability, Indigenous-led AI, and the role of policy. Check out the videos from these thought-provoking lightning talks (with longer discussions at the end):

The False Hope of Explainability in Medicine

Differences between understandings of explainability.

Lauren Oakden-Rayner, the Director of Research for Medical Imaging at Royal Adelaide Hospital, is both a radiologist and a machine learning expert. She spoke about mismatched expectations between technical and non-technical communities on what questions explainability answers, based on her paper ��The false hope of current approaches to explainable artificial intelligence in health care��. Lauren��s talk is at the start of Video #1.

Critical Gaps in ML Evaluation Practice

Often unspoken assumptions underlying machine learning evaluation practices, and the gaps left by each

Ben Hutchinson is a senior engineer in Google Research based in Sydney. Practices for evaluating machine learning models are largely developed within academic research and rest on a number of assumptions that lead to concerning gaps when applied to real-world applications. Ben��s talk starts at 12 min mark of Video #1.

Indigenous-Led AI

On empowering, enabling, and informing Indigenous knowledge throughout the model development process.

Cathy Robinson is a principal research scientist at CSIRO, working on a project to center Indigenous data soveriegnty and Indigenous co-design in addressing complex ecological and conservation issues. Read more about CSIRO��s Healthy Country AI project or about CARE Indigenous Data Principles. Watch Cathy��s talk starting at 23 min mark of Video #1.

Near-Termism and AI Value Alignment

The differences between definitive and normative understandings of explainability.

Aaron Snoswell is a postdoctoral research fellow at QUT, with over a decade��s experience in software development, industry research, and robotics. He spoke about the issues with focusing primarily on long-termism in AI value alignment and the need to consider short-term issues. Starts at 36 min mark Video #1.

Narrow vs Broad Understandings of Algorithmic Bias among Stakeholders in Healthcare AI

The differences between narrow vs broad undersatndings of algorithmic bias.

Yves Saint James Aquino is a philosopher and physician, currently working on the project ��The algorithm will see you now: ethical, legal and social implications of adopting machine learning systems for diagnosis and screening�� as a postdoctoral research fellow at the University in Wollongong. For his talk, he drew on interviews with 70 different stakeholders in healthcare AI, including software developers, medical doctors, and startup founders, to explore different conceptions of how algorithmic bias is understood. Watch the first talk in Video #2.

Towards Human-Centric XAI using Eye Tracking in Chest Xrays

Using a multi-modal approach for machine learning on chest x-rays

Catarina Pinto Moreira is a Lecturer in Information Systems at Queensland University of Technology and a pioneer in non-classical probabilistic graphical models for decision making to empower human decision-making. Interviews with radiologists are crucial to her work; for example, interviews revealed that clinical notes are important for radiologists to use in diagnosis, even though this is not often mentioned in the literature. Her talk begins at 10 min mark of Video #2.

The Role of Policy in Data Ethics

AI policy should span the entire AI life cycle; focus on applications rather than underlying tech; and move beyond abstract principles.

Michael Evans crafted Australia��s National Artificial Intelligence Roadmap, contributed to the development of Australia��s national approach to governing autonomous vehicles, and represented Australia at the World Bank/IMF Annual Meetings. He gave an overview of the AI policy landscape, including policy tools, the disconnect between principles and application, and recommended ways forward. Watch Michael��s talk beginning at 20 min mark of Video #2.

Each talk is around 5 minutes long. Feel free to fast forward to those of particular interest, or watch them all!

The End The False Hope of Explainability in Medicine (Lauren Oakden-Rayner, Australian Institute for Machine Learning) Critical Gaps in ML Evaluation Practice (Ben Hutchinson, Google Sydney) Indigenous-Led AI (Cathy Robinson, CSIRO) Near-Termism and AI Value Alignment (Aaron Snoswell, Queensland Univ of Technology) Narrow vs Broad Understandings of Algorithmic Bias among Stakeholders in Healthcare AI (Yves Saint James Aquino, Univ of Wollongong) Towards Human-Centric XAI using Eye Tracking in Chest Xrays (Catarina Pinto Moreira, Queensland Univ of Technology) The Role of Policy in Data Ethics (Michael Evans, Evans AI)

View more on Jeremy Howard's website »

Like • 0 comments • flag

Published on March 13, 2022 17:00

7 Great Lightning Talks Related to Data Science Ethics

I have been organizing and facilitating a series of Ethics Workshops for the Australian Data Science Network, featuring lightning talks by Australian experts on a range of topics related to data science ethics, including machine learning in medicine, explainability, Indigenous-led AI, and the role of policy. Check out the videos from these thought-provoking lightning talks (with longer discussions at the end):

The False Hope of Explainability in Medicine

Differences between understandings of explainability.

Lauren Oakden-Rayner, the Director of Research for Medical Imaging at Royal Adelaide Hospital, is both a radiologist and a machine learning expert. She spoke about mismatched expectations between technical and non-technical communities on what questions explainability answers, based on her paper ��The false hope of current approaches to explainable artificial intelligence in health care��. Lauren��s talk is at the start of Video #1.

Critical Gaps in ML Evaluation Practice

Often unspoken assumptions underlying machine learning evaluation practices, and the gaps left by each

Ben Hutchinson is a senior engineer in Google Research based in Sydney. Practices for evaluating machine learning models are largely developed within academic research and rest on a number of assumptions that lead to concerning gaps when applied to real-world applications. Ben��s talk starts at 12 min mark of Video #1.

Indigenous-Led AI

On empowering, enabling, and informing Indigenous knowledge throughout the model development process.

Cathy Robinson is a principal research scientist at CSIRO, working on a project to center Indigenous data soveriegnty and Indigenous co-design in addressing complex ecological and conservation issues. Read more about CSIRO��s Healthy Country AI project or about CARE Indigenous Data Principles. Watch Cathy��s talk starting at 23 min mark of Video #1.

Near-Termism and AI Value Alignment

The differences between definitive and normative understandings of explainability.

Aaron Snoswell is a postdoctoral research fellow at QUT, with over a decade��s experience in software development, industry research, and robotics. He spoke about the issues with focusing primarily on long-termism in AI value alignment and the need to consider short-term issues. Starts at 36 min mark Video #1.

Narrow vs Broad Understandings of Algorithmic Bias among Stakeholders in Healthcare AI

The differences between narrow vs broad undersatndings of algorithmic bias.

Yves Saint James Aquino is a philosopher and physician, currently working on the project ��The algorithm will see you now: ethical, legal and social implications of adopting machine learning systems for diagnosis and screening�� as a postdoctoral research fellow at the University in Wollongong. For his talk, he drew on interviews with 70 different stakeholders in healthcare AI, including software developers, medical doctors, and startup founders, to explore different conceptions of how algorithmic bias is understood. Watch the first talk in Video #2.

Towards Human-Centric XAI using Eye Tracking in Chest Xrays

Using a multi-modal approach for machine learning on chest x-rays

Catarina Pinto Moreira is a Lecturer in Information Systems at Queensland University of Technology and a pioneer in non-classical probabilistic graphical models for decision making to empower human decision-making. Interviews with radiologists are crucial to her work; for example, interviews revealed that clinical notes are important for radiologists to use in diagnosis, even though this is not often mentioned in the literature. Her talk begins at 10 min mark of Video #2.

The Role of Policy in Data Ethics

AI policy should span the entire AI life cycle; focus on applications rather than underlying tech; and move beyond abstract principles.

Michael Evans crafted Australia��s National Artificial Intelligence Roadmap, contributed to the development of Australia��s national approach to governing autonomous vehicles, and represented Australia at the World Bank/IMF Annual Meetings. He gave an overview of the AI policy landscape, including policy tools, the disconnect between principles and application, and recommended ways forward. Watch Michael��s talk beginning at 20 min mark of Video #2.

Each talk is around 5 minutes long. Feel free to fast forward to those of particular interest, or watch them all!

The End The False Hope of Explainability in Medicine (Lauren Oakden-Rayner, Australian Institute for Machine Learning) Critical Gaps in ML Evaluation Practice (Ben Hutchinson, Google Sydney) Indigenous-Led AI (Cathy Robinson, CSIRO) Near-Termism and AI Value Alignment (Aaron Snoswell, Queensland Univ of Technology) Narrow vs Broad Understandings of Algorithmic Bias among Stakeholders in Healthcare AI (Yves Saint James Aquino, Univ of Wollongong) Towards Human-Centric XAI using Eye Tracking in Chest Xrays (Catarina Pinto Moreira, Queensland Univ of Technology) The Role of Policy in Data Ethics (Michael Evans, Evans AI)

View more on Jeremy Howard's website »

Like • 0 comments • flag

Published on March 13, 2022 07:00

Doing Data Science for Social Good, Responsibly

The phrase ��data science for social good�� is a broad umbrella, ambiguously defined. As many others have pointed out, the term often fails to specify good for whom. Data science for social good can be used to refer to: nonprofits increasing their impact through more effective data use, hollow corporate PR efforts from big tech, well-intentioned projects that inadvertently result in surveillance and privacy invasion of marginalized groups, efforts seeped in colonialism, or many other types of projects. Note that none of the categories in the previous list are mutually exclusive, and one project may fit several of these descriptors.

"Data for good" is an imprecise term that says little about who we serve, the tools used, or the goals. Being more precise can help us be more accountable & have greater positive impact. @sarahookr presents at @DataInstituteSF lunch seminar pic.twitter.com/efAMJxdQB8
— Rachel Thomas (@math_rachel) August 24, 2018

Picture from a presentation given in 2018 by Sara Hooker, founder of non-profit Delta Analytics and an AI researcher at Google, on Why ��data for good�� lacks precision.

I have been involved with data science for social good efforts for several years: chairing the Data for Good track at the USF Data Institute Conference in 2017; coordinating and mentoring graduate students in internships with nonprofits Human Rights Data Analysis Group (for a project on entity resolution to obtain more accurate casualty conflicts in Syria and Sri Lanka) and the American Civil Liberties Union (one student analyzed covid vaccine equity in California and another analyzed disparities in school disciplinary action against Black and disabled students) during my time as director of the Center for Applied Data Ethics at USF; and now as a co-lead of the Data Science for Social Good program at Queensland University of Technology (QUT). At QUT, grad students and recent graduates partnered with non-profits Cancer Council Queensland (well known for their Australian Cancer Atlas) and FareShare food rescue organisation, which operates Australia��s largest charity kitchens. While data for good projects can be incredibly useful, there are also pitfalls to be mindful of when approaching data for social good.

Some Questions & Answers

I recently spoke on a panel at the QUT Data Science for Social Good showcase event. I appreciated the thoughtful, nuanced questions from the moderators, Dr. Timothy Graham and Dr. Char-lee Moyle, who brought up some of the potential risks. I want to share their questions below, along with an expanded version of my answers.

What ethical and governance considerations do you think not-for-profits should consider when starting to adopt data science? Be specific about the goals of the project and how different stakeholders will be impacted:A series of interviews with African data experts revealed that power imbalances, failure to acknowledge extractive practices, failure to build trust, and Western-centric policies were all prevalent. Even in ��data for good�� projects, the people whose data is accessed and shared may not reap the benefits that those who control the project do. Stakeholders such as government bodies and non-profits have significantly more power and leverage compared to data subjects. There are issues where data gathered for one goal ends up being repurposed or shared for other uses. While Western ��notions of privacy often focus on the individual, there is growing awareness that collective identity is also important within many African communities, and that sharing aggregate information about communities can also be regarded as a privacy violation.�� Center the problem to be solved, not a flashy solution. Sometimes machine learning practitioners have a solution searching for a problem. It is important to stay focused on the root problem and be open to ��mundane�� or even non-technical solutions. One data for good project used the records of 15 million mobile phone owners in Kenya to quantify the movements of workers who migrate for seasonal work to an area with malaria, and made recommendations to increase malaria surveillance in their hometowns when they return. As a journalist for Nature reported, ��But it��s unclear whether the results were needed, or useful. Malaria-control officers haven��t incorporated the analyses into their efforts.�� The excitement around ��flashy�� big data approaches contrasts with the lack of funding for proven measures like bed nets, insecticides, treatment drugs, and health workers. Take data privacy seriously. Be clear about how the data will be stored, who has access to it, and what will happen to it later. Ask what data is truly needed, and if there are less invasive ways to get this information. Note that the above example tracking Kenyan mobile phone owners raises risks around lack of consent, invasion of privacy, and risk of de-anonymization. Include the people most impacted, and recognize that their values may be different from those of both non-profits or academic stakeholders involved. A recent article from AI Now Institute recommended that ��social good projects should be developed at a small scale for local contexts �� they should be designed in consultation with the community or social environment impacted by the systems in order to identify core values and needs.�� One example of differing values: Indigenous scholars highlighted that a set of open data principles developed primarily by Western scholars to improve data discovery and reuse created tension with Indigenous values. The FAIR principles, first developed at a workshop in the Netherlands in 2014 and elaborated on in this paper published in Nature, call for data to be findable, accessible, interoperable, and reusable. In response, Indigenous scholars convened to develop the CARE principles for Indigenous data governance, calling for collective benefit, authority to control, responsibility, and ethics, intended as a complement to the FAIR principles. Avoid answering the ��wrong problem.�� For instance, many European governments are currently using algorithmic approaches to justify austerity cuts. Arguments about reducing fraud often accompany these, even when fraud is minimal. Due to a biometric identity system in India, many poor and elderly people are no longer able to access their food benefits due to faded fingerprints, not being able to travel to scanners, or intermittent internet connections.

My impression is that some folks use machine learning to try to "solve" problems of artificial scarcity. Eg: we won't give everyone the healthcare they need, so let's use ML to decide who to deny.

Question: What have you read about this? What examples have you seen?
— Rachel Thomas (@math_rachel) November 18, 2020

Do you think that data science for social good can increase the surveillance and control of disadvantaged groups or certain segments of society?

Many well-meaning projects inadvertently lead to increased surveillance, despite good intentions. Cell-phone data from millions of phone owners in over two dozen low- and middle- income countries has been anonymized and analyzed in the wake of humanitarian disasters. This data raises concern of the lack of consent of the phone users and risks of de-anonymization. Furthermore, it is often questionable whether the results are truly useful, as well as if they could have been discovered through other, less invasive approaches. One such project analyzed the cell phone data of people in Sierra Leone during an Ebola outbreak. However, this approach didn��t address how Ebola spreads (only through direct contact with body fluids) or help with the most urgent issue (which was convincing symptomatic people to come to clinics to isolate).

What do you think is the role of government and universities in supporting and incentivising the not-for-profit sector in adopting data science?

Academia and government have a big role to play. Often non-profits lack the in-house data science skill to take advantage of their data, and many data scientists who are searching for meaningful and impactful real-world problems to work on. We will also need the government to regulate topics such as data privacy to help protect those who may be impacted. It is important to recognize that privacy should not just be considered an individual right, but also a public good.

What are your thoughts around the development of ethical frameworks to guide data science �� are they more than marketing tactics to increase trustworthiness and reputation of data science?

We need ethical frameworks AND regulation. Both are crucially important. Many people want to do the right thing, and having standardized processes to guide them can help. I recommend the Markkula Center Tech Ethics Toolkit, which includes practical processes you can implement in your organization to try to identify ethical risks BEFORE they cause harm. At the same time, we need legal protections anywhere that data science impacts human rights and civil rights. Meaningful consequences are needed for those who cause harm to others. Also, policy is the appropriate tool to address negative externalities, such as when corporations offset their costs and harms to society while reaping the profits for themselves. Otherwise, there will always be a race to the bottom.

What skills and training do you think the not-for-profits sector needs to embrace data science and what��s the best strategies for upskilling?

The people who are already working for an organization are best positioned to understand that organization��s problems and challenges, and where data science can help. Upskilling in-house talent is underutilized. Don��t feel that you need to hire someone new with a fancy pedigree, if there are people at your organization who are interested and eager to learn. I would start by learning to code in Python. Have a project from your not-for-profit that you are working on as you go, and let that project motivate you to learn what you need as you need to (rather than feeling like you need to spend years studying before you can tackle the problems you care about). One of our core missions with fast.ai is to train people in different domains to use machine learning for themselves, as they best understand the problems in their domain and what is needed. There are many myths that you need a super-elite background to use techniques like deep learning, but it��s not magic. Anyone with a year of coding experience can learn to use state-of-the-art deep learning.

Avoiding Data Disasters

Things can go disastrously wrong in data science and machine learning projects when we undervalue data work, use data in contexts that it wasn��t gathered for, or ignore the crucial role that humans play in the data science pipeline. A new multi-university centre focused on Information Resilience, funded by the Australian government��s top scientific funding body (ARC), has recently launched. Information Resilience is the capacity to detect and respond to failures and risks across the information chain in which data is sourced, shared, transformed, analysed, and consumed. I��m honored to be a member of the strategy board, and I have been thinking about what information resilience means with respect to data practices. Through a series of case studies and relevant research papers, I will highlight these risks and point towards more resilient practices.

Case study: UK covid tracking app

Data from a covid-symptom tracking app was used in a research paper to draw wildly inaccurate conclusions about the prevalence of Long Covid, the often debilitating neurological, vascular, and immune disease that can last for months or longer (some patients have been sick for 20 months and counting). The app suggested that only 1.5% of patients still experience symptoms after 3 months, an order of magnitude smaller than estimates of 10-35% being found by other studies.

How could this research project have gone so wrong? Well, the app had been designed for a completely different purpose (tracking 1-2 week long respiratory infections), didn��t include the most common Long Covid symptoms (such as neurological dysfunction), had a frustrating user-interface that led many patients to quit using it, and made the erroneous assumption that those who stopped logging must be fully recovered. The results from this faulty research paper were widely shared, including in a BBC article, offering false reassurance than Long Covid prevalence is much rarer than it is. Patients had been voicing their frustrations with the app all along, and if researchers had listened sooner, they could have collected a much higher quality and more accurate data set.

This research failure illustrates a few common issues in data projects:

The context of the data was not taken into account. The user-interface, the categories listed, the included features�� these were all designed to record data about a short-term mild respiratory infection. However, when it was used for a different purpose (long covid patients suffering for months with vascular and neurological symptoms), it did a poor job, and led to missing and incomplete data. This happens all too often, in which data gathered for one context is used for another The people most impacted (long covid patients) were ignored. They had the most accurate expertise on what long covid actually entailed, yet were not listened to. Ignoring this expertise led to lower quality data and erroneous research conclusions. Patients have crucial domain expertise, which is distinct from that of doctors, and must be included in medical data science projects. From the start of the pandemic, patients who had suffered from other debilitating post-viral illnesses warned that we should be on the lookout for long-term illness, even in initially ��mild�� cases.Data is Crucial

Collecting data about covid and its long-term effects directly from patients was a good idea, but poorly executed in this case. Due to privacy and surveillance risks, I frequently remind people not to record data that they don��t need. However, the pandemic has been a good reminder of how much data we really do need, and how tough it is when it��s missing.

At the start of the pandemic in the United States, we had very little data about what was happening�� the government was not tabulating information on cases, testing, or hospitalization. How could we know how to react when we didn��t understand how many cases there were, what death rates were, how transmissible the disease was, and other crucial information? How could we make policy decisions in the absence of a basic understanding of the facts.

In early March 2020, two journalists and a data scientist from a medication-discovery platform began pulling covid data together into a spreadsheet to understand the situation in the USA. This launched into a 15-month long project in which 500 volunteers compiled and published data on COVID-19 testing, cases, hospitalizations, and deaths in the USA. During those 15 months, the Covid Tracking Project was the most comprehensive source of covid data in the USA, even more comprehensive than what the CDC had, and it was used by the CDC, numerous government agencies, and both the Trump and Biden Administrations. It was cited in academic studies and in thousands of news articles.

A data infrastructure engineer and contributor for the project later recounted, ��It quickly became apparent that daily, close contact with the data was necessary to understand what states were reporting. States frequently changed how, what, and where they reported data. Had we set up a fully automated data capture system in March 2020, it would have failed within days.�� The project used automation as a way to support and supplement manual work, not to replace it. At numerous points, errors in state reporting mechanisms were caught by eagle-eyed data scientists notifying discrepancies.

This vision of using automation to support human work resonates with our interest at fast.ai in ��augmentedML��, not ��autoML��. I have written previously and gave an AutoML workshop keynote on how too often automation ignores the important role of human input. Rather than try to automate everything (which often fails), we should focus on how humans and machines can best work together to take advantage of their different strengths.

Speaking about AugmentedML vs. AutoML at ICML 2019 Data Work is Undervalued

Interviews of 53 AI practitioners across 6 countries on 3 continents found a pattern that is very familiar to many of us (including me) who work in machine learning: ��Everyone wants to do the model work, not the data work.�� Missing meta-data leads to faulty assumptions. Data collection practices often conflict with the workflows of on-the-ground partners, such as nurses or farmers, who are usually not compensated for this extraneous effort. Too often data work is arduous, invisible, and taken for granted. Undervaluing of data work leads to poor practices and often results in negative, downstream events, including dangerously inaccurate models and months of lost work.

Throughout the pandemic, data about covid (both initial cases and long covid) has often been lacking. Many countries have experienced testing shortages, leading to undercounts of how many people have covid. The CDC decision not to track breakthrough cases unless they resulted in hospitalization made it harder to understand prevalence of break-throughs (a particularly concerning decision since break-throughs can still lead to long covid). In September, it was revealed that British Columbia, Canada was not including covid patients in their ICU counts once the patients were no longer infectious, a secretive decision that obscured how full ICUs were. Some studies of Long Covid have failed to include common symptoms, such as neurological ones, making it harder to understand the prevalence or nature.

Data has Context

Covid is giving us a first-hand view of how data, which we may sometimes want to think of as ��objective��, are shaped by countless human decisions and factors. In the example of the symptom tracking app, decisions about which symptoms were included had a significant impact on the prevalence rate calculated. Design decisions that influenced the ease of use impacted how much data was gathered. Lack of understanding of how the app was being used (and why people quit using it) led to erroneous decisions about which cases should be considered ��recovered��. These are all examples of the context for data. Here, the data gathered was reasonably appropriate for understanding initial covid infections (a week or two of respiratory symptoms), but not for patients experiencing months of neurological and vascular symptoms. Numbers can not stand alone, we need to understand how they were measured, who was included and excluded, relevant design decisions, under what situations a dataset is appropriate to use vs. not.

As another example, consider covid testing counts: Who has access to testing (this involves health inequities, due to race or urban vs. rural), who is encouraged to get tested (at various times, people without symptoms, children, or other groups have been discouraged from doing so), varying accuracies (e.g. PCR tests are less accurate on children, missing almost half of cases that later go on to seroconvert), and making decisions about what counts as a ��case�� (I know multiple people who had alternating test results: positive, negative, positive, or the reverse�� what counts as a positive case?)

Datasheet for an electrical component. Image from 'Datasheets for Datasets'

One proposal for capturing this context is Datasheets for Datasets. Prior to doing her PhD at Stanford in computer vision and then co-leading Google��s AI ethics team, Dr. Timnit Gebru worked at Apple in circuit design and electrical engineering. In electronics, each component (such as a circuit or transistor) comes with a datasheet that lists when and where it was manufactured, under what conditions it is safe to use, and other specifications. Dr. Gebru drew on this background to propose a similar idea for datasets: listing the context of when and how it was created, what data was included/excluded, recommended uses, potential biases and ethical risks, work needed to maintain it, and so on. This is a valuable proposal towards making the context of data more explicit.

The People Most Impacted

The inaccurate research and incomplete data from the covid tracking app could have been avoided by drawing on the expertise of patients. Higher quality data could have been collected sooner and more thoroughly, if patients were consulted in the app design and in the related research studies. Participatory approaches to machine learning is an exciting and growing area of research. In any domain, the people who would be most impacted by errors or mistakes need to be included as partners in the design of the project.

The Diverse Voices project from University of Washington Tech Policy Lab involves academic papers and practical how-to guides.

Often, our approaches to addressing fairness or other ethics issues, further centralize the power of system designers and operators. The organizers of an ICML workshop on the topic called for more cooperative, democratic, and participatory approaches instead. We need to think not just about explainability, but about giving people actionable recourse. As Professor Berk Ustun highlights, when someone asks why their loan was denied, usually what they want is not just an explanation but to know what they could change in order to get a loan. We need to design systems with contestability in mind, to include from the start the idea that people should be able to challenge system outputs. We need to include expert panels of perspectives that are often overlooked, depending on the application, this could mean formerly or currently incarcerated people, people who don��t drive, people with very low incomes, disabled people, and many others. The Diverse Voices project from University of Washington Tech Lab provides guidance on how to do this. And it is crucial that this not just be tokenistic participation-washing, but a meaningful, appropriately compensated, and ongoing role in their design and operation.

Towards Greater Data Resilience

I hope that we can improve data resilience through:

Valuing data work Documenting context of data Close contact with the data Meaningful, ongoing, and compensated involvement of the people impacted

And I hope that when our data represents people we can remember the human side. As AI researcher Inioluwa Deborah Raji wrote, ��Data are not bricks to be stacked, oil to be drilled, gold to be mined, opportunities to be harvested. Data are humans to be seen, maybe loved, hopefully taken care of.��

Quote from AI researcher Inioluwa Deborah Raji

View more on Jeremy Howard's website »

Like • 0 comments • flag

Published on November 03, 2021 17:00