Multigroup Fairness and the Validity of Statistical Judgment

Co-Organizers: Alexandra Chouldechova (CMU), Nika Haghtalab (UC Berkeley), Moritz Hardt (Max Planck Institute for Intelligent Systems), Michael Kim (UC Berkeley), Omer Reingold (Stanford)

In recent years, research on algorithmic fairness in machine learning and statistical analysis has exploded, gaining attention in multiple communities. The purpose of this workshop is to bring together experts in several areas to share and synthesize different perspectives on fairness and the validity of predictive tools.  Bringing together experts from machine learning, theoretical computer science, statistics, and relevant application areas, the workshop will begin with an overview of recent notions of multigroup fairness.  This overview will highlight the relationship between novel notions of fairness and traditional techniques in machine learning and statistics, emphasizing known applications in learning and statistical decision making.  Throughout, the workshop will also aim to identify challenges arising in application areas that can be addressed (or perhaps, are not well-addressed) by existing tools developed in the context of algorithmic fairness.

Multigroup fairness is a class of definitions of fairness proposed in the last five years. Instead of providing aggregate statistical guarantees for a few protected categories (defined by race, ethnicity, gender, age, and so on), these definitions suggest giving such guarantees to a large (often exponential) number of sets. The intuition is that providing meaningful fairness guarantees to a group requires extending these guarantees to subgroups that are relevant in the setting we consider. The identity of these groups may be hard to specify, a priori. Instead, these definitions apply to every large group that can be identified with a specific set of computational resources and information.  In some technical sense, such a guarantee is the best possible:  sets that cannot be identified, cannot be protected.

Multigroup fairness also relates to questions about the validity and justifiability of statistical decision making. While statistics can only give aggregate guarantees about a reference population, statistical decision making applies to the individual. On the basis of what guarantee should we trust the output of a statistical tool on an individual instance? Is the output of a probabilistic genotyping software valid in the defendant's case? What is the risk that this patient will go on to suffer cardiac arrest? What is the meaning of a 70% chance of repayment computed by statistical risk score on an individual loan applicant? Multigroup fairness suggests a process of particularization of statistical judgment that negotiates a middle ground between a population and the individual.

Since the introduction of multigroup fairness, a large body of research has extended initial results to online learning, unsupervised learning, affirmative action, and many other directions. The results have been shown to be applicable in the wild. Some of these connections are:

1. Practical methods for learning in a heterogeneous population, employed in the field to predict COVID-19 complications at an early stage of the pandemic.
2. A computational perspective on the meaning of individual probabilities.
3. A new paradigm for loss minimization in machine learning, through the notion of omnipredictors, that simultaneously applies to a wide class of loss-functions, allowing the specific loss function to be ignored at the time of learning. Some of this research suggests alternatives to traditional loss minimization in machine learning and to propensity scoring in statistics.
4. A method for adapting a statistical study on one probability distribution to another, which is blind to the target distribution at the time of inference and is competitive with wide-spread methods based on propensity scoring.

We would like to share these and other results with each other, to identify connections between research in the different fields, and to examine the applicability of techniques from one area to the challenges of another. This workshop is currently invite only with a limited number of spots open for public registration.