We present novel experiments shedding light on a potential limitation of common fairness metrics for assessing the undesirable biases made by machine learning algorithms. Our experiments are based on the Bios dataset, for which the learning task consists in predicting the occupation of individuals based on their LinkedIn biography. This dataset is then particularly suited to reproduce Natural Language Processing (NLP) solutions dedicated to automatic job recommendation, which was identified as a High-Risk application of A.I. in the A.I. act. We specifically address an important limitation of theoretical discussions dealing with group-wise fairness metrics in the machine learning literature: they focus on large datasets, although the norm in many commercial A.I. applications is to use reasonably small training and test sets. Data annotation, which is mandatory in supervised learning, is indeed a time consuming and costly task. It is therefore common practice to stop annotating the training data when they reach a suficiently large size to get a desired level of prediction accuracy. This is typically the case when using active learning procedures, which have become very popular recently. We then question how reliable are diferent measures of bias when the size of the training and the test set is simply suficient to learn reasonably accurate predictions. We conclude our study by emphasizing the crucial need to take into account the stability of the bias metrics for small variations of the test set when auditing high-risk A.I. systems.
Potential biases introduced by Artificial Intelligence (AI) systems are now both an academic
concern and a critical issue for industry, as the European Commission plans to regulate AI
systems that could adversely afect individual users. The AI act1 will indeed require AI systems
sold in the European Union to have proper statistical properties with regard to any potential
discrimination they could engender. In particular, AI systems that exploit linguistic data for
automatic job recommendation fall into the category of high risks systems in the AI act and will
be tightly regulated. In this context, it will be necessary to define fairness metrics to quantify
the level of fairness of prediction models. We see two main problems with this: (1) Each fairness
metric measures the bias in a certain way and not all metrics are compatible with each other,
which was already discussed in [
To explore the second question of introduction, we used a new experimental protocol developed
hereafter: We fine-tuned the DistilBERT [
Let ˆ and be the predicted and the true target occupations, respectively. Let be a random variable representing the binary gender of the biography’s subject. For each model, we quantified the gender bias by using Group Parity , = (ˆ = | = ), True Positive Rate , = (ˆ = | = , = ) and Predictive parity , = ( = |ˆ = , = ). To measure the gender gap with these metrics, we computed the diference between binary genders and ˜ — for each occupation : _, = , − ˜,, where is , or .
All the models we trained reached a prediction accuracy ranging from 0.72 to 0.86, as in [
Although the model is trained to predict the occupations of bio authors from the 28 possible choices, we focus, in our study, on the analysis of the biases on two specific occupations: Surgeon versus the 27 remaining occupations, and Physician versus the other occupations. We chose these professions so that we could compare an occupation with an imbalanced gender distribution and one with balanced a gender distribution.
Our experiments clearly show that the lower the amount of observations in the training set, the more the fairness metrics vary on the test set. The samples with 10K and 20K observations present particularly unstable biases. For example, most TPR (resp. GP) Gender Gaps are negative (resp. positive) for surgeon (resp. physician) but some samples yield positive TPR (resp. negative GP) Gender Gaps. This is problematic since we cannot not deduce a priori that a particular sample should produce a discrimination one way or the other.
In addition, the average biases also depend on the sample size. Again, we obtained unstable average biases for small samples (10K, 20K). The bias indicators are estimated on the minority class: an amount of 41, 115, 334 and 903 predicted surgeons were obtained in the test set for the 10K, 20K, 50K and 120K sampling sizes. Hence, their estimation is unstable for small samples.
However, GP appears as more stable than the other metrics in our experiments, in particular when there was little observations. Its variance was indeed close to 0.01, which is much lower than the variances of 0.1 and 0.2 for GP and PP, respectively. We explain this because on our dataset, for TPR and PP, they do not use all predicted surgeons (unlike GP), but only the predicted surgeons who are also real surgeons (in 10k sampling, there are 41 predicted surgeons vs. 30 real surgeons and predicted surgeons, which is an information loss of 26,8%).
Even for large samples with 120K observations, biases sometimes difered from what we expected.
For the occupation surgeon (15% of females) the gender gap was negative for all metrics, which
was expected. For physician (49,5% of females), we also expected to have negative or zero gender
gap (see [
In this section, we confirm our analysis on the specific occupations of from a global point of view on all the classes of the model.
Surgeon and Physician 1. In the Figure 2, we have more and more important deviations on the variance of the metrics as the size of the data set decreases. And that on most trades. As explained before, the GP gender gap is more stable, because it has more data. 2. In the first table of Figure 3, the metrics give inconsistent results for several occupations: depending on the metric bias in favor of men or women for the same profession and the same model. This is particularly visible for the occupations: software engineer, poet, architect, attorney, and nurse.
These results give us guarantees on the generalization of our analysis carried out on the two classes previously. We find the same problems on the metrics and the size of the sample, regardless of the occupation being looked at.
Our paper used the Bios dataset to study the influence of the training set size on discriminatory biases. Our results shed light on new phenomena: (1) fairness metrics did not converge to stable results for small sample sizes, which precluded any conclusions about the nature of the biases; (2) even on large training samples, the biases discovered were not always those expected and varied according to the metrics for several occupations. These results give two clear messages to data scientists who must design NLP applications with a potential social impact. They should ifrst be particularly careful, as the decision rules they train may have unexpected discriminatory biases. In addition, a bias metric not only returns a score but has a strong practical meaning and may be unreliable, in particular when working with small training sets. So multiple metrics should be considered. Also, statistical methods to obtain the variance of the observed metrics are necessary to guarantee the fairness of a model.
This research was funded by the AI (Artificial Intelligence) Interdisciplinary Institute ANITI (Artificial and Natural InTelligence Institute.), which is funded by the French ‘Investing for the Future– PIA3’ program under the Grant agreement ANR-19-PI3A-0004.