Motivated by the recital (67) of the current corrigendum of the AI Act in the European Union, we propose and present measures and mitigation strategies for discrimination in tabular datasets. We specifically focus on datasets that contain multiple protected attributes, such as nationality, age, and sex. This makes measuring and mitigating bias more challenging, as many existing methods are designed for a single protected attribute. This paper comes with a twofold contribution: Firstly, new discrimination measures are introduced. These measures are categorized in our framework along with existing ones, guiding researchers and practitioners in choosing the right measure to assess the fairness of the underlying dataset. Secondly, a novel application of an existing bias mitigation method, FairDo, is presented. We show that this strategy can mitigate any type of discrimination, including intersectional discrimination, by transforming the dataset. By conducting experiments on real-world datasets (Adult, Bank, COMPAS), we demonstrate that de-biasing datasets with multiple protected attributes is possible. All transformed datasets show a reduction in discrimination, on average by 28%. Further, these datasets do not compromise any of the tested machine learning models' performances significantly compared to the original datasets. Conclusively, this study demonstrates the efectiveness of the mitigation strategy used and contributes to the ongoing discussion on the implementation of the European Union's AI Act.
Discrimination in artificial intelligence (AI) applications is a growing concern since the adoption of
the AI Act by the European Parliament on March 13, 2024 [
Our paper mainly focuses on how to measure and mitigate discrimination in datasets where multiple
protected attributes are present. In our first contribution, we provide a comprehensive categorization
of discrimination measuring methods. Besides introducing new measures for some of these cases, we
also categorize existing measures from the literature. Some of the listed measures specifically address
intersectional discrimination and non-binary groups. The second contribution deals with bias mitigation.
For this, we use our published pre-processing framework, FairDo [
Color e p a h S r o l o C e p a h S
Non-intersectional
minimized. By implementing the introduced measures, we can therefore mitigate biases for multiple
protected attributes. Another advantage of FairDo is that it preserves data integrity and does not
modify the features of individuals during the optimization process, unlike other methods [
We evaluated our methodology on popular tabular datasets with fairness concerns, such as Adult [
The results of the bias mitigation process as well as the performance of the machine learning models
are promising. They indicate that achieving fairness in datasets with multiple protected attributes
is possible, and FairDo is a proper framework for this task. Overall, our work contributes technical
solutions for stakeholders to enhance the fairness of datasets and machine learning models, aiming for
compliance with the AI Act [
To handle multiple protected attributes, we define = {1, . . . , } as a set of protected attributes. It
can represent the set of sociodemographic features such as age, gender, and ethnicity. These factors
may make individuals vulnerable to discrimination. Each protected attribute ∈ is formally a
discrete random variable that can take on values from the sample space . In this context, we refer
to groups that describe distinct social categories of a protected attribute. For example, let represent
gender; then is a set containing the genders male, female, and non-binary. To avoid limitations to a
particular group fairness notion, we introduce a generalized notation based on the works of Žliobaitė
[
Definition 2.1 (Treatment). Let 1, 2 be events and be a random variable that can take on values from , then we call the conditional probability
(1 | 2, = ) treatment, where ∈ . 1 describes some favorable outcome, such as getting accepted for a job, while 2 often represents some additional information about the individual, such as their qualifications. Definition 2.2 (Fairness Criteria). With the definition of treatment, we can define fairness criteria that demand equal treatment for diferent groups. Let (1 | 2, = ) and (1 | 2, = ) be treatments, then we call the following equation:
(1 | 2, = ) = (1 | 2, = ) a fairness criterion, for all , ∈ .
Definition 2.2 allows us to define various group fairness criteria, including statistical parity [
In real-world applications, achieving equal probabilities for certain outcomes is not always possible.
Due to variations in sample sizes in the groups, it is common to yield unequal treatments, even when
they are similar. Thus, existing literature [
Definition 2.3 (Disparity). Let (1 | 2, = ) and (1 | 2, = ) be two treatments, then we refer to
(, , 1, 2) = | (1 | 2, = ) − (1 | 2, = )| as the disparity, for all , ∈ . Trivially, is commutative regarding , . In practice, it prevents reverse discrimination due to the absolute value.
Definition 2.4 (Discrimination). We use : D → R to denote some discrimination measure that quantifies the discrimination inherent in any dataset ∈ D. A dataset consists of features, protected attributes, and labels for each individual. The explicit form of depends on the cases introduced in Section 3.
We found that numerous scenarios arise when dealing with multiple protected attributes. We categorize
these scenarios based on the number of groups, denoted as ||, and the number of protected attributes,
denoted as ||. By going through all cases, we present possible approaches from the literature as well
as our own suggestions to measure discrimination.
3.1. Single Protected Attribute (|| = 1)
In the case of having only one protected attribute, i.e., || = |{1}| = 1, we distinguish between
cases by the number of available groups || in the dataset. We categorize the cases by || = 0, 1, 2, and
|| > 2.
3.1.1. No Groups (|| = 0)
When there are no groups, the measurement of discrimination is impossible if no assumptions are being
made. Discrimination can be assessed through proxy variables [
() = 0. 2. Diference to optimal treatment : Another way is to return the absolute diference of the group’s outcome to the optimal treatment. For example, group has an 80% chance of receiving the favorable treatment. Ideally, having a 100% chance would represent the optimal scenario. Therefore, the discrimination score is 20% in this case. It is given by:
() = | (1 | 2, 1 = ) − 1|. 3. Diference to expected treatment : We can use the expected treatment as a reference point. For example, we know that a company has a 50% acceptance rate for job applications. Now a machine learning classifier is trained to predict whether an applicant will be accepted and the model’s predictions result in a 60% acceptance rate for group . Hence, the model is positively biased towards group by 10%. This can be formulated as:
() = | (1 | 2, 1 = ) − expect.|, where expect. is the expected treatment. It can describe the average treatment across all groups [21] or some other prior information that is not included in the dataset. (1) (2) (3) (4) (5) (6) () =
max (, , 1, 2) ,∈,< = max (1 | 2, = ) − ∈
min (1 | 2, = ).
∈
Equation (5) describes the maximum discrimination obtainable between two groups. An alternative and
equivalent formulation is given in Equation (6) [
agg(1) (, , 1, 2),
,∈,<
where is the disparity as defined in Definition 2.3 and < ensures that each pair is considered
only once (assuming label-encoded groups). According to Žliobaitė [
A more general approach to measuring discrimination is to calculate some form of correlation
coeficient between the protected attribute and the outcome. The correlation coeficient can be calculated
using Pearson’s correlation [23], Spearman or Kendall’s rank correlation [24, 25]. The discrimination
measure can then be defined as the absolute value of the correlation coeficient:
() = |Corr(1, )|.
(7)
This approach can be applied to any number of groups. Fairlearn provides a pre-processing method
that removes the correlation between the protected attribute and the outcome by transforming the
data [
The central idea of intersectionality is that individuals experience overlapping forms of oppression or privilege based on the combination of multiple social categories they belong to. In the following, we will introduce definitions to formulate intersectional discrimination, which is based on the work of Kearns et al. [21].
Definition 3.1 (Subgroup [21]). Let = {1, . . . , } be a set of discrete random variables representing protected attributes that can take on values from corresponding groups 1, . . . , . A subgroup is defined as = (1, . . . , ) ∈ 1 × . . . × . In other words, a subgroup encompasses multiple groups from diferent protected attributes.
Definition 3.2 (Subgroup Treatment). Let be a subgroup as defined in Definition 3.1 and let {1, . . . , } be a set of discrete random variables. Subgroup treatment is then defined as: = (1 | 2, 1 = 1, . . . , = ).
Definition 3.3 (Subgroup Disparity). Let = {1, . . . , } be a set of discrete random variables. Let , ∈ 1 × . . . × be two subgroups with = (1, . . . , ) and = (1, . . . , ). The disparity between two subgroups is denoted as ^ and is given by:
^ (, , 1, 2) = | (1 | 2, 1 = 1, . . . , = ) − (1 | 2, 1 = 1, . . . , = )|. Similarly to Equation (4), we can calculate the discrimination score for multiple protected attributes by aggregating disparities across all subgroups. A subgroup can be treated like a normal group. According to Definition 3.1, there are theoretically at least 2 subgroups, where is the number of protected attributes. However, not all subgroups may be available in the dataset. For unavailable subgroups, the disparity cannot be calculated as the corresponding treatment is undefined.
Let us denote the set of available subgroups as avail ⊆ 1 × . . . × . To finally capture the discrepancies across all available subgroup pairs, an aggregation function agg(1) is applied to the subgroup disparities ^ : intersect() = agg(1) ^ (, , 1, 2).
,∈avail (8) Equation (8) represents the aggregated discrimination between all available subgroups in the dataset. When using the maximum function as the aggregator, the calculations are equivalent to Equation (5) and Equation (6). The only diference is that the conditionals are now subgroups instead of groups: = max (1 | 2, 1 = 1, . . . , = ) − ∈avail
min (1 | 2, 1 = 1, . . . , = ).
∈avail
Kang et al. [
The problem with measuring discrimination for intersectional groups is that it has an upward bias when using meta-metrics [22]. This is because the number of subgroups grows exponentially with the number of protected attributes. This leads to many subgroups where the number of samples in each subgroup is possibly small, resulting in larger noise in the treatment estimates [22].
Besides intersectional groups, Yang et al. [
}︃ agg(1) (, , 1, 2) .
,∈,< The first-level aggregator agg(1) aggregates disparities within a protected attribute, considering unique pairs of groups and . The second-level aggregator agg(2) then combines the results across all protected attributes. By applying both operators, we obtain a discrimination measure that captures disparities between groups across multiple attributes.
(9) (10) (11) 3.2.3. Example Let us consider a dataset with two protected attributes, age and sex (see Table 1). The set of protected attributes is = {1, 2} = {Age, Sex} and the set of available subgroups in the dataset is avail = {Old, Young} × {
Male, Female}. We measure discrimination using statistical disparity. For simplicity, all aggregation functions are set to the maximum function. The intersectional approach yields the following discrimination score: max ( = 1 | 1 = 1, 2 = 2) − ∈avail
min ( = 1 | 1 = 1, 2 = 2) = | ( = 1 | Age = Old, Sex = Male) − ( = 1 | Age = Young, Sex = Male)| = 1, (12) (13) while the discrimination score for the non-intersectional approach is given by:
︂{ indep() = max ∈
max ,∈,< (, , ( = 1), Ω)
︂} = max {︀ Age(Old, Young, ( = 1), Ω), Sex(Male, Female, ( = 1), Ω)}︀ = max{|0.5 − 0.5|, |0.5 − 0.5|} = max{0, 0} = 0.
The non-intersectional approach yields a discrimination score of 0 because the disparities for both protected attributes are 0. This is quite diferent from the intersectional approach, which reports a discrimination score of 1. As seen, the results can difer depending on the approach.
Our experimentation follows a pipeline consisting of data pre-processing, bias mitigation, model training,
and evaluation. To mitigate bias in tabular datasets with multiple protected attributes, we used the
sampling method, FairDo [
The tabular datasets employed in our experiments include the
Adult [
Each dataset was divided into training and testing sets using an 80/20 split, respectively. We ensured
that the split was stratified (if possible) based on protected attributes to maintain representativeness
across diferent groups in both sets.
Bias Mitigation Applying the bias mitigation method FairDo [
min (fair), (14) fair⊆ where is the training set of Adult, Bank, or COMPAS, and is the fairness objective function. We experimented with both intersect and indep as objectives functions. Bias mitigation is only applied to the training set and the testing set remains unchanged. FairDo internally uses genetic algorithms to select a subset of the training set that minimizes the objective function. We used the same settings and operators as provided in the package and only adjusted the population size (200) and the number of generations (400).
Model Training We utilized the scikit-learn library [27] to train various machine learning classifiers, namely Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), and Artificial Neural Network (ANN). These classifiers were trained on both the original and fair datasets. Classifiers trained on the original datasets serve as a baseline for comparison. We used the default hyperparameters given by scikit-learn package for each classifier.
Evaluation Metrics We evaluated the models’ predictions on fairness and performance using the test set. For fairness, we assessed intersect and indep. For the classifiers’ performances, we report the area under the receiver operating characteristic curve (AUROC) [28], where higher values indicate better performances. Because removing data points can compromise the overall quality of the data, we also report the number of subgroups before and after bias mitigation to check for representativeness. Trials For each dataset and discrimination measure combination, the bias mitigation process was repeated 10 times. The results were averaged over the trials to obtain a more robust evaluation.
Fair Dataset Generation Table 3 shows the average discrimination before and after mitigating bias in the training sets. On all datasets, discrimination was reduced after applying FairDo. Without considering group intersections, discrimination was reduced by 7%, 19%, and 25% for Adult, Bank, and COMPAS, respectively. When considering intersectionality, the discrimination was reduced by 15%, 18%, and 83%. Hence, discrimination was reduced by 28% on average across all datasets, thus answering RQ1 positively. When comparing the discrimination scores, it can be observed that the intersectional discrimination scores are generally higher. This is because in the intersectional setting, more subgroups are considered, which potentially leads to larger diferences between them [21].
We also report the number of subgroups before and after bias mitigation to assess the impact of the undersampling method on the dataset. The removal of subgroups can only be observed in the intersectional setting. In the COMPAS dataset 5.2 out of 34 subgroups were removed on average, indicating the largest amount of subgroups removed across all datasets. While the Bank dataset consists of 48 subgroups, only 1.8 subgroups were removed on average. Because the COMPAS dataset’s initial intersectional discrimination score is 100%, removing more subgroups seems inevitable to reduce bias. Model Performance and Fairness Figure 2 shows the results of the classifiers’ performances on the test set. The classifiers’ performances are displayed on the y-axis, while the discrimination values are shown on the x-axis. We note that the axes do not share the same scale across the subfigures for analytical purposes.
Classifiers trained on fair datasets did not sufer a significant decline in performance compared to those trained on original datasets. In all cases, only a slight decrease of 1%-3% in performance can be noted. This indicates that the bias mitigation process does not compromise the dataset’s fidelity and, therefore, the classifiers’ performances. Regarding discrimination, a significant reduction is evident. The x-axis scales are much larger than the y-axis scales, suggesting that changes in discrimination are larger than changes in performance. For example, the RF classifier trained on the Bank dataset (Figure 2g) shows a decrease in intersectional discrimination from 38% to 15%, while the performance only decreases by 2%. Similar results can be observed for the other classifiers and datasets as well, successfully addressing RQ2. The results suggest that FairDo can be reliably used to mitigate bias in tabular datasets for various measures that consider multiple protected attributes. Still, we advise users to carefully perform similar analyses when applying the method to their datasets.
The results of our experiments show that the presented measures detect discrimination in datasets with multiple protected attributes diferently. When using the intersectional discrimination measure, more groups are identified and compared to each other. While subgroups are not ignored by this measure, measuring higher discrimination scores by random chance becomes more likely [21, 22]. In contrast, treating each protected attribute separately prevents this issue but may lead to overlooking discrimination. The choice of measure is up to the stakeholders and depends on the context of the dataset and the regulations that apply to the AI system. We generally recommend using the intersectional discrimination measure if the number of individuals in each subgroup is large enough to draw statistically significant conclusions. Otherwise, treating each protected attribute separately is more suitable.
By using the mitigation strategy FairDo [
Discrimination
(a) Adult (LR)
statistical properties regarding fairness. Whether intersectionality was considered or not, reducing
discrimination in datasets was possible. At the current state, the AI Act [
Datasets often come with multiple protected attributes, which makes measuring and mitigating discrimination more challenging. Most existing studies only deal with a single protected attribute, and works that consider multiple protected attributes often focus on intersectionality. In opposition to this, we proposed a new non-intersectional measure that treats each protected attribute separately. This is more suitable when the number of subgroups is too large or the number of individuals in each subgroup is small. We used both intersectional and non-intersectional measures as objectives and applied the FairDo framework to mitigate discrimination in multiple datasets. The experiments show that discrimination was reduced in all datasets and on average by 28%. Machine learning models trained on the bias-mitigated datasets also improved their fairness while maintaining performance compared to models trained on the original datasets.