Gender Shades

Gender Shades
Title	Gender Shades
Authors	Joy Buolamwini; Timnit Gebru
Year	2018
Venue	Conference on Fairness, Accountability, and Transparency

Gender Shades

Gender Shades is a 2018 empirical study by Joy Buolamwini and Timnit Gebru that evaluated commercial facial analysis systems for demographic classification. The study compared automated gender classification tools from companies using a novel dataset and demonstrated marked disparities in performance across different demographic groups. It catalyzed debates among researchers, activists, policymakers, and technology companies about bias in automated decision-making systems.

Background and Motivation

Buolamwini drew inspiration from prior work on algorithmic bias exemplified by research from Latanya Sweeney, Frank Pasquale, Cathy O'Neil, and scholarship emerging from Fairness, Accountability and Transparency (FAccT), while Gebru's background included studies at Microsoft Research and Stanford University. The project was motivated by controversies involving facial recognition deployments in contexts connected to New York Police Department, Chicago Police Department, and public discussions triggered by reporting from outlets like The New York Times, Wired, and The Guardian. The authors situated their inquiry within literatures from MIT Media Lab, Harvard Kennedy School, and advocacy by organizations such as ACLU, Electronic Frontier Foundation, and Algorithmic Justice League.

Methodology and Dataset

The study assembled a dataset drawn from images associated with public figures and ethnographic benchmarks used in facial analysis research, referencing sources linked to work at IARPA, National Institute of Standards and Technology, and datasets employed by researchers at University of Oxford and Massachusetts Institute of Technology. Buolamwini and Gebru sampled images across phenotypic variables and annotated attributes following protocols related to projects at ImageNet contributors and teams at Google Research. They evaluated three commercial APIs provided by companies with ties to Microsoft, IBM, and Amazon Web Services by running standardized experiments similar to procedures used at Stanford Vision Lab and Carnegie Mellon University.

Key Findings and Results

The results revealed substantial disparities: error rates for darker-skinned females were significantly higher than for lighter-skinned males, echoing concerns raised by researchers at University of Toronto, University College London, and Princeton University. The paper quantified errors using performance metrics familiar to groups at OpenAI, DeepMind, and Facebook AI Research, showing near-perfect classification for some subgroups and much poorer performance for others. The study also identified dataset composition and training-set imbalance as root causes, a phenomenon discussed in literature from Cornell University, ETH Zurich, and University of Cambridge.

Impact and Reception

Following publication at FAccT Conference, the study received attention from academic venues such as NeurIPS, ICML, and AAAI, and coverage in mainstream media including BBC, NPR, and Reuters. It influenced conversations among regulators at bodies like the European Commission, committees within United States Congress, and advisory groups associated with Organisation for Economic Co-operation and Development. Advocacy organizations including Color Of Change and Access Now cited the work in campaigns, while it was discussed in panels at SXSW and workshops hosted by Berkman Klein Center.

Critiques and Limitations

Scholars at institutions such as University of Pennsylvania, Yale University, and University of California, Berkeley raised methodological queries about dataset representativeness, annotation protocols, and the generalizability of benchmarks to real-world deployments. Critics referenced debates exemplified by scholars from Princeton University, Columbia University, and Duke University on evaluation metrics, intersectionality, and the ethics of demographic labeling. The authors acknowledged constraints tied to sample selection and API versions, similar to concerns voiced in follow-up analyses from labs at University of Washington and University of Michigan.

Influence on Policy and Industry Practices

The study spurred policy responses including moratoria and procurement reviews by municipal governments such as San Francisco, legislative initiatives in California State Legislature, and deliberations at European Parliament committees. In industry, technology providers including teams at Microsoft Research, IBM Research, and Amazon Web Services announced audits, model cards initiatives promoted by Google, and new documentation practices informed by standards underway at IEEE and NIST. The report also contributed to curricular changes in programs at Massachusetts Institute of Technology, Stanford University, and University of Oxford emphasizing ethics in machine learning.

Category:Computer vision