With the increase use of artificial intelligence systems and the associated concerns regarding automated discrimination, research in the field of fairness has increased in the past years. To evaluate their work in fair machine learning, researchers have often been using the same three datasets (Adult, COMPAS, and German credit) as benchmarks. However, those datasets each present serious limitations. In this work, we first explore what other datasets could potentially be used as replacement, specifically in a European context. We then use an experimental approach to compare Adult and COMPAS with a new candidate, Student Performance (a.k.a Student Alcohol Consumption). Our early results highlight the scarcity of easily accessible European datasets suitable as benchmarks for fairness evaluation of problems with positive or negative outcome, as well as the high influence dataset selection can have on experimental results.
With the increase use of artificial intelligence systems, legitimate ethical and legal concerns have been growing, including the risk that some people may be treated more negatively than others, thus resulting in discrimination [18,21,22]. This problem is prevalent in machine learning, where the training data is of paramount importance, while usually retaining historical and social biases that are then learned by the prediction models [21].
Research in the field of fairness has been constantly growing in the past few years, with the problem of fair classification receiving the most attention [21]. Many fairness metrics and bias mitigation methods have been developed in that sub-field, but less attention has been given towards quality benchmark datasets, specially regarding European data. A subset of only three datasets has been surpassing all others in term of popularity [12,17], namely Adult [3], COMPAS [10], and German credit [16]. Those popular datasets have nevertheless been shown to present serious limitations [12], such as, but not limited to, old age (Adult and German credit), noisy data [2,9,14], label bias (COMPAS [1]), or even coding mistakes preventing retrieval of the sensitive attribute (German credit [14]).
Given these observations, the emerging consensus is that their use as benchmarks should be avoided, lest duly justified. Efforts have been made to provide alternatives (for instance a dataset suite to replace Adult [9]), to disseminate good practices (like in [1]), or to facilitate access to other datasets (such as a large collection of fairness datasets [12] along with its search engine [13]). Building on these works, we explore potential alternatives to the three popular yet flawed historical benchmarks, while focusing on the European context. We also compare, with an experimental approach, results stemming from the use of Adult and COMPAS as benchmarks with a potential replacement, Student Performance [7]. More specifically, we compare the results of a bias mitigation method and several popular fairness metrics for models trained with each of those three datasets.
Our results highlight the scarcity of open access and easily accessible datasets for fairness evaluation in a European context. Further, the analysis of those three examples confirms that different datasets may lead to different results when evaluating bias mitigation methods and fairness metrics. This stresses the importance of dataset selection in experiments. We also aim to expand this work with a more thorough search for datasets and further experiments encompassing more bias mitigation methods, fairness metrics and sensitive attributes.
As mentioned in the introduction, the datasets Adult, COMPAS and German credit have been widely used as benchmarks in fair classification studies, even though they are not as suitable for fair machine learning research as previously thought. All three of them contain real life tabular data about individuals with one or more attribute(s) recognized as sensitive/protected and a target label that is deemed positive or negative for the individual.
Those key characteristics for fair classification should be shared by potential replacement datasets. Being open access also increased their appeal. We thus aim to find a potential new benchmark that is also easily accessible, specially considering the fast past environment of computer science conferences where there is often little time dedicated to the selection of datasets. Lastly, we focus on the European context. Indeed, the worldwide influence of the United-States and difference in data privacy culture and legislation across countries have made datasets from the USA dominant in the field, while data from other areas, including Europe, remains less accessible. This can cause further bias given that models and results from a certain place aren't necessary applicable to other geographical locations [9].
Merging the positive characteristics of the popular datasets with our context of interest, we formulate our dataset selection criteria as follows: An open access dataset with tabular data about European subjects that is no more than 25 years old and is adapted to the problem of fair classification leading either to a positive or negative outcome for the subject.
To find such dataset, we used the search engine for fairness datasets1 presented in [13] as it is the most complete collection of datasets for fair machine learning to the best of our knowledge. Its database comprises over 200 datasets for diverse domains and fairness tasks.
Filtering on tabular data and fair classification with a positive or negative outcome, we are left with 22 datasets (as of January 29 and February 2, 2024). Out of those, fourteen datasets contain data from the United-States, two from elsewhere in America, three have no mention of localization in their description, three are European, and only one is from Asia. There are thus no dataset referenced for Africa nor Oceania. Out of the three European datasets, only two contain data collected in this century, Dutch Census [6] and Student Performance [7] (a.k.a "Student" and "Student Alcohol Consumption"). Since Dutch Census is part of the IPUMS International collection 2 and requires approval to be accessed, only Student Performance fully fits our criteria. Presented in [8], this dataset contains social, gender and study information about students in two Portuguese schools for the core classes of Mathematics and Portuguese in secondary education (high school). It has been used in a few fairness studies and is referenced in the dataset survey [19].
With this experiment 3 , we compare different models trained with bias mitigation on Adult, COMPAS, and Student Performance. For Student Performance, we use data related to the Portuguese subject version of the dataset since it has the most instances (649). We consider the sensitive attributes sex and age, with students who are 18 or older as the protected group, as in [19]. We take the usual sensitive attributes sex and race for Adult and COMPAS. We did not include German credit, as it is impossible to retrieve its protected attribute (sex), making interpretation of results misleading.
For each of the considered datasets, we compute several classifiers using the training and in-processing bias mitigation meta-algorithm presented in [5]. This algorithm considers a constrained optimization problem that is approximately solved with provable guarantees. The constraint enforces a minimal value, the fairness penalty parameter 𝜏 , for a chosen fairness metric. We use the AIF360 [4] implementation with Statistical Parity ratio as the fairness constraint and gradient descent. We train different models with 𝜏 value ranging from 0 (no bias mitigation) to 1 (constraint of perfect statistical parity). We then evaluate the performance and fairness of the resulting models using different metrics to assess their evolution with different constraint levels. The fairness metrics we use are Statistical Rate difference (SR) [11], the most used metric and based on prediction only, Equality of Opportunity (Eq. Opp) and Equalized Odds (Eq. Odds) [15], the most used metrics based both on predictions and ground truth, and Consistency [24], the most used metric based on similarity [23,17]. We evaluate each metric for each classifier, then report the average over 10 folds and the corresponding confidence interval.
You can see in Figure 1 the results for the different models, each represented by the fairness penalty parameter 𝜏 it was trained with. Results for Adult and COMPAS with sensitive attribute race are close to those with sex and are not presented here due to space restriction.
Let us first note that Consistency, an individual fairness metric, is not impacted by the Statistical Parity mitigation, which is based on group fairness. We thus focus only on the three other metrics, related to group fairness, for the remainder of this section.
In Figure 1a, we see that the model trained on Adult without bias mitigation (𝜏 = 0) is already very close to fairness according to all metrics. Bias mitigation still induces a general improvement for all group fairness metrics. Eq. Opp indicates that the protected group (Women) becomes the one with a slightly higher true positive rate when the fairness constraint is greater than or equal to 0.7 . We also note a significant drop in accuracy and increase of the F1-score as soon as bias mitigation is introduced, but only marginal changes after that.
In Figure 1b, for models trained on COMPAS, there is first an overall decrease in fairness for all group metrics from 𝜏 = 0.1 to 𝜏 = 0.7. Fairness then increases again for higher values of 𝜏 , which is coincident with a significant drop in accuracy, while the F1-score remains steady.
In Figure 1c, the unconstrained model trained on Student Performance with sex as protected attribute is near perfect fairness according to SR and Eq. Opp. The constrained models never surpass these values. Eq. Odds shows more bias than both SR and Eq. Opp for all models but one. This metric is less mitigated than for Adult, even though the original value was higher to start with.
In Figure 1d, the model trained on Student Performance with age as sensitive attribute and no bias mitigation shows significant bias according to SR. This bias is efficiently mitigated by the model, even with the lowest level of fairness constraint. Bias reported by SR and Eq. Opp is very low even before bias mitigation. Except for a brief increase when 𝜏 is 0.1, their bias level is reduced, with Eq. Opp extremely close to equality.
Overall, even though the same algorithm as been used for the training and bias mitigation of all of these models, the results vary significantly for each of the different datasets studied. Additionally, for Adult and COMPAS the overall tendencies are similar for the two sensitive attributes considered (sex and race), but we see a very significant difference when considering sex or age for Student Performance.
On the one hand, the search for new potential benchmarks highlights the scarcity of European datasets for use in fair classification with a positive or negative outcome. Indeed, out of the over 200 datasets referenced in the search engine used [13], only Student Performance [7] fits our selection criteria. This dataset shows very little to no bias with regard to the sensitive attribute sex, at least regarding the most common fairness definitions, strongly reducing its interest when studying fairness related problems. The other most common sensitive attribute considered for this dataset is age. However, being an older student is most often a direct result of past failures, which is itself usually deemed an appropriate criterion to predict future exam results. It is thus questionable whether this attribute should be considered protected or not. Other attributes we have not explored here could also be of interest. For example, attributes related to alcohol consumption are studied in [20], which also extends the discussion to label bias.
So, despite the existing efforts to mitigate the collective data documentation debt and offer new alternatives to Adult, COMPAS and German credit, there is still a need to bring forward new European datasets, as well as data from other underrepresented continents. This may include improvement of the visibility and centralization of existing datasets or collection of new data adapted to fairness related questions.
On the other hand, our study illustrates that the same procedure applied to different datasets may lead to significantly different results, which is congruent with the results in [19]. This showcases the importance the choice of dataset can have on fairness evaluation and when presenting results. We thus recommend to use several different datasets since they may lead to varying results, to look beyond open access data if needed, and to avoid making broad claims based on few examples, which echoes some of the recommendations in [1] and [19]. We also encourage researchers to consider the selection of data not as a minor step, but as a meaningful part of the research, and to provide justification on the choices made in that regard.
Beyond this discussion, we aim to expand this work to study more datasets, include non-open access data, as well as more bias mitigation methods, fairness metrics and sensitive attributes.