In our previous work titled “How to be fair? A study of label and selection bias,” we discussed how the interaction between bias and fairness can yield fruitful insights for fairness research. We considered the scenario where an initially unobservable fair distribution becomes corrupted by bias, leading to an observable unfair distribution. By employing various fairness definitions and types of bias, we derived valuable mathematical conditions and properties that the observable distribution must adhere to. This would allow practitioners to better understand bias and mitigate its efect in their models. In this paper, we delve into the significance of these findings, address their limitations, and explore potential future research directions.
The fairness-accuracy trade-of [
Furthermore, the ethical interpretation of the fairness-accuracy trade-of is inherently flawed,
since it is self-contradictory in nature. If we acknowledge the necessity of implementing fairness
interventions on our data, it implies that due to bias we have reason to believe that the data
may contain artifacts that could hurt the performance of a classifier trained on them. We’ve
already deemed our data flawed. It is then naive to expect that the accuracy of the classifier will
not decrease, since the efort to make the data suitable for classification will necessarily shift
the data distribution to a fairer one. If we pursue fairness, it is because we believe our data to
be biased, and if we believe our data to be biased, we should also believe they cannot provide a
good measure of the quality of the classifier. In reality, the trade-of should be considered as an
empty signifier: no meaning is conveyed by the trade-of, which should only be regarded as a
numerical fact. In recent years, an increasing number of fairness researchers have endeavored to
critically address the misconception surrounding the fairness-accuracy trade-of by highlighting
that when bias is introduced into the data, the accuracy of a fair classifier on the unbiased data
outperforms the accuracy of a fairness-agnostic classifier [
Despite this, the literature still lacks a clear understanding of the relationship between bias and fairness. Current fairness measures only acknowledge the presence of bias, but they are unable to comprehend its nature or provide a clear path to remove it. This could pose potential challenges when attempting to enforce fairness in real-world applications.
For instance, when considering the fairness measure “Demographic Parity”, it’s clear that the objective of the metric is to ensure that the probability of being classified as positive is the same for all sensitive groups.
However, it remains unclear whether the distribution of positive labels achieved through a fairness intervention enforcing demographic parity is indicative of the successful elimination of societal bias or merely an instance of afirmative action. In other words, it is unclear if the new distribution is unbiased or if it is just a diferent kind of bias. We believe that studying the interaction between bias and fairness should be a priority for the fairness community, and our previous work is a step in this direction.
In our previous work “How to be fair? A study of label and selection bias” [
Under these assumptions, asking whether fairness interventions are able to remove bias can be rephrased as whether fairness interventions manage to correct a classifier and align its score with the fair unobservable conditional probability. To mathematically formalize this, we consider a binary random variable representing bias, independent of the non-sensitive features , given the binary label and the sensitive attribute . In layman’s terms, for each data point in our initially fair dataset, we flip a weighted coin to decide whether to introduce bias, where the weight of the coin, i.e. the probability of bias, depends only on the sensitive attribute and the original label.
In this paper we analyze two types of biases: label bias and selection bias. Label bias arises when the assigned label for an individual does not accurately represent the label they should Selection Bias
Demographic Parity Necessary conditions on ( |, )
( |, ) satisfies fairness
Not detectable from ( |, ) ( |, ) doesn’t satisfy fairness
We’re all equal Necessary conditions on ( |, )
( |, ) satisfies fairness Necessary conditions on ( |, ) ( |, ) satisfies fairness have been assigned, whereas selection bias occurs when certain individuals are not represented in the dataset, resulting in a non-representative sample of the population.
To study these biases, a mathematical model is needed. We adopted the following: for label bias, when a data point needs to be biased, i.e. when it loses the coin flip, we change the assigned binary label; for selection bias, we instead remove the data point entirely.
We then examined the impact of these biases on two specific fairness definitions: “Demographic Parity" and “We’re all equal". Demographic parity, as previously mentioned, means that the distribution of the initial label is the same for both sensitive groups, formally =|. “We’re all equal”, on the other hand, requires that the distribution of the initial label depends solely on the non-sensitive features, formally =|| . Based on these notions, our research question now is:
Consider a distribution that satisfies either demographic parity or “we’re all equal,” which is transformed by either label or selection bias. Is it possible to detect or discern the bias based on a sample of the altered distribution? And if so, is there a fairness intervention that can counteract it? Everything combined, it means there exist four possible combinations of bias and fairness we can study, as shown in Table 1.
Having a clear mathematical framework proved to be a fruitful choice for our research, as we were able to prove multiple statements about each of the four combinations. In particular, what became evident from our research is that in 3 out of 4 combinations, when bias has influenced the distribution, it is possible to confirm it purely by looking at the conditional distribution of the label, since there are strict conditions that the observable distribution must satisfy. Moreover, since diferent conditions are satisfied for diferent combinations, it is possible to discern the nature of the bias.
This result has positive implications for the field of fairness, as it suggests that, at least theoretically, the data itself could assist in determining the best course of action. This is because certain choices might already be excluded when they do not align with the data distribution, alleviating practitioners from the burden of blindly applying fairness techniques and helping them to take a more informed decision.
On the other hand, the remaining combination, selection bias combined with demographic parity, yields an even more interesting and somewhat counter-intuitive result. In this case, we demonstrated that the original fair distribution never satisfies demographic parity when computed on the available biased data. In other words, the correct fair distribution does not appear completely fair according to the used measure. This simple condition has a significant impact on the application of many classifiers in terms of fairness. Indeed, many fairness techniques are designed to enforce demographic parity on the available data, which automatically fails to properly remove selection bias from the data as they cannot retrieve the original fair distribution. When bias influences the data, even the fairness measures themselves can become biased.
This also formally supports one of the most common criticisms of fairness, namely that we need a better understanding of the efect of fairness techniques on data.
Despite the merits of our work, we acknowledge that it also has its limitations. Firstly, even
though we utilized our theoretical framework to explain certain experimental results from the
literature [
In reality, it is often the case that bias may be influenced by both the features and the sensitive attributes. For instance, non-sensitive features that correlate with being part of a minority might inadvertently trigger confirmation bias in individuals with prejudices. This could subsequently exacerbate discrimination against certain groups of people, thereby perpetuating bias. A person with a foreign-sounding name might experience more rejections than a compatriot with a more common name when seeking employment. Police over-patrolling black neighborhoods gathers more data on residents of those neighborhoods than on black individuals residing in predominantly-white areas. In both cases, the bias is influenced by the features; label bias for the first example and selection bias for the second.
We believe this to be the most important point of improvement in our work. To overcome this limitation, in our future work we will explore what happens when the features are allowed to influence bias in our definitions. However, we also need to limit the extent to which the features influence the bias. This enables us to relax the independence assumption without altering the definitions of bias established thus far. We believe this to be a reasonable approach, as it allows us to still apply mathematical rigorousness when defining diferent kinds of biases.
Here we show three possible approaches: one based on diferential privacy, the second on the Hirschfeld–Gebelein–Rényi coeficient, and the third is a direct chance-constraint on the probability distribution.
1. (|, , ) ≤ (|, ) 2. HGR(, |, ) := sup corr( (), ()|, ) < for functions and . 3. (| ( = 1|, , ) − ( = 1|, )| ≥ ) ≤ All these definitions are suitable, albeit with diferent levels of generality and ease of implementation. They can further improve our understanding of the relationship between bias and fairness and help us develop more efective fairness interventions. We believe our work is a step in the right direction: a new research line to connect bias types and data assumptions to measurable properties in observed data. The implications are both theoretical and practical, leading to a better understanding of fairness measures and better tools to guarantee fair AI.