Machine Learning (ML) systems, whether predictive or generative, not only reproduce biases and stereotypes but, even more worryingly, amplify them. Strategies for bias detection and mitigation typically focus on either ex post or ex ante approaches, but are always limited to two steps analyses. In this paper, we introduce the notion of Bias Amplification Chain (BAC) as a series of steps in which bias may be amplified during the design, development and deployment phases of trained models. We provide an application to such notion in the credit scoring setting and a quantitative analysis through the BRIO tool.
In recent years, research in the field of artificial intelligence has seen its impact grow in the lives of many people. In particular, algorithmic fairness, and its negative counterpart, algorithmic bias, have become hot topics: many experts, as well as ordinary people, have realized that systems based on Machine Learning (ML), whether predictive or generative, not only reproduce biases and stereotypes but, even more worryingly, amplify and reinforce them. This phenomenon, known as bias amplification, urgently requires addressing to start relying on automatic systems.
In academic research, risks of bias have been outlined for a decade now and solutions are being
sought, following two main approaches to algorithmic fairness:
• an ex post approach, in which fairness metrics are defined to identify and mitigate the presence
of bias [
Both of these approaches, despite their great diferences, are characterized by identifying an ideal,
abstract, probabilistic model to which an empirical, non-deterministic result must come as close as
possible. In the former approach, the evaluation of distance between the two is performed without
assuming any knowledge of the underlying model. In the latter approach, the analysis seeks to identify
a transparent underlying structure which minimizes distance from a given justification. Standard
theoretical computer science and philosophy of computing terminology [
However, when dealing with complex AI systems, and especially in the case of generative AI, there
are several levels to consider, and at each of these levels bias amplification phenomena can occur.
We will refer to the bias amplification phenomenon in a system with more than one level as Bias
Amplification Chains (BAC). Consider the case of text-to-image generation: one should first take into
account that “[f]airness [...] is not a purely technical construct, having social, political, philosophical
and legal facets" [7, p.1], and therefore consider how bias is reproduced and pre-amplified from society
to training datasets, via image captioning [
A perspective similar to ours on the way bias propagates and amplifies throughout the life cycle of
machine learning was given by [
In the present work we describe and exemplify the case of bias amplification in the sense of [
To show our methodology, we examine a simple amplification chain consisting of just three links. The first link identifies the amplification of bias occurring in sampling the training dataset from the real population, thus identifying the input bias of our ML pipeline. The second link corresponds to the divergence between the distribution on which the model is trained and the one to which it is applied (test set). Lastly, the third link identifies the amplification of bias outputted by our model, that is, how the bias is amplified from the dataset used to perform the analysis to the predictions produced by the model. Our goal is achieved through the following functionalities: • FreqVsRef : a functionality of BRIO which considers the first and the second link of the chain, by investigating how the dataset population distribution (Freq) amplifies bias on a property of interest as observed in the real population (Ref); • FreqVsFreq: a functionality of BRIO which considers the second and third links of the chain, by investigating how the distribution divergence between the sensitive groups increases or decreases in the predicted outcome, with respect to the true distribution; • an in-depth final analysis to inspect the entire chain, quantifying the contribution of each link to amplification, and identifying which of them most urgently requires mitigation interventions.
In practice, we conduct our analysis in the paradigmatic context of credit scoring. We use the UCI
German Credit Dataset [
The paper is structured as follows. First, we introduce BRIO’s main functionalities in Section 2. These will be used in subsequent Sections 3, 4, and 5 – each of which focuses on a specific link of the BAC – to compute the corresponding bias amplification. Finally, some final insights on the possible refinements of the notion of BAC will be given in Section 6.
• the predictions of a ML model encoded as a set of datapoints with relative features, • a set of parameters, including the designation of one or more sensitive features, also called protected attributes, • and a distribution of reference, which might be automatically computed on the input dataset of the model, or externally provided,
On this input, it returns an evaluation of the possibility that the model under consideration is unfair with respect to the designated features when comparing the predictions and the reference distribution. To do so, the system allows conducting two kinds of analyses, consisting in: 1. FreqVsRef : comparing the behaviour of the AI system against a desirable one; 2. FreqVsFreq: comparing the behaviour of the AI system with respect to a sensitive group 1 against another sensitive group 2 related to the same feature , with = 1, 2.
If the second analysis alerts of a possibly biased behaviour, one can conduct a subsequent check on some (or all the) subclasses of the considered sensitive classes. This second check is meant to verify whether the bias encountered at the level of the classes can be explained away by non-sensitive features of the individuals that it is morally acceptable to use for taking decisions.
One can use the result of analyses 1 and 2 and, keeping track of the (sub)classes where they fail, compute: 3. Hazard: an aggregate measure of the number of datapoints where a test of type 1 or 2 fails, of how far it is from not failing on some datapoints (when so), and of how dificult was it for it to fail.
Age 0-24 years old 25-39 y.o. 40-59 y.o.
Over 60 y.o.
Of this latter measure, fully detailed in [2, S5], two options are provided, one focusing on group fairness and one on individual fairness. In the present work, we only consider the one involving group fairness2.
Finally, it should be noted that, since the tool is model-agnostic, the same analyses 1, 2, and 3 described above can be applied to study the distribution of the ground truth (as it was the prediction of a perfect classifier). This feature will be used in Section 5.
In this section, we apply the FreqVsRef bias detection module of the BRIO tool in order to evaluate whether the process of labelling and sampling from the general population3 to produce the training set is biased in the sense that it inaccurately represents the real distribution Bias is in fact mathematically defined as a systematic deviation from the true estimation. Although in the ML literature this term is typically used in relation to the decisions of a model, when it comes to the training set, we are interested in checking for bias contained in the features, rather than in the labels (or decisions). In fact, as Figure 1 illustrates, the reference distribution against which we want to compare the training set contains no labels.
For our experiments, we use the UCI German Credit Dataset [
The first step of the analysis consists in running the FreqVsRef bias detection module of the BRIO tool in order to evaluate whether the distribution of individuals in the diferent classes of the training set diverges too much from the distribution indicated by the statistics about German society. The method returns the respective Kullback-Leibler divergences5 [20] with Laplacian smoothing, and compares it with the BRIO automated threshold set to high sensitivity, see Table 2.
The results show that the distributions of both gender and age in the training set do not match the corresponding distribution in real population: when further looking at the training set, in fact, one ifnds that males are extremely over-represented, and that people in age groups 40-59 and over 60 are greatly under-represented. The respective distributions of marital statuses are almost swapped.
The sampling to produce a test set for the prediction task is the next link of our BAC. By test set we mean the set of unseen data used to evaluate how the final model performs on unseen data, as opposed to the validation set which can be used to fine-tune the model itself. A major assumption of the ML approach is that the training and the test data share the same statistical distribution, being Independent and Identically Distributed (i.i.d.). However, in many practical applications, the i.i.d. assumption fails due to unforeseen distributional changes occurring once the ML system is deployed and used on new inputs (test set). This becomes relevant for our discussion because, in addition to a possible degradation of performance, the failure of i.i.d. might also amplify bias. We here distinguish two diferent perspectives from which the bias amplification occurring at the test set level can be thought of.
In a first sense, sampling unseen inputs for the test set may be amplifying bias in the properties of interest as they occur in the general population. In this case, an evaluation of this bias can be computed in the same vein as done for bias in the sampling of the training set.
In a second sense, the test set may amplify or compensate bias with respect to the training set. This is evidence of the fact that the sampling process of the test set was diferent from that producing the training set. Put diferently, a model trained on a given distribution is then used to predict on a dataset which sensibly difers for some property of interest. This relates to the Out-of-Distribution problem, which occurs when the ML system is used on a distribution that significantly diverges from the one learnt during the training phase [21]. Evaluating such bias diference can be done through the BRIO tool in the same vein as done in the next phase of our analysis (Section 5).
Note that: a. assuming a neutral bias amplification in sampling the training set, the bias input of the chain at the next stage will be ascribed entirely to the first sense of bias in the test set; b. assuming a neutral bias diference in the second sense for the test set, the bias input of the chain at the next stage will be ascribed entirely to the second sense of bias in the test set. 5The Kullback–Leibler divergence KL is mathematically expressed by the following equation:
KL( ‖ ) = ∑∈︁ () · log( (()) ) and quantifies the discrepancy of a considered probability distribution with respect to a reference probability distribution .
To determine the extent to which the model’s predictions amplify bias relative to the actual distribution of the sensitive groups of interest, we run the bias detection module of the BRIO tool on both the model’s predictions and the ground truth and compare the former result against the latter. For the present example, we run experiments for two diferent sensitive features: age and gender. Our predictive model is obtained through a standard credit scorecard modelling process based on the methodology of OptBinning6.
The bias detection module returns – along with a list of fairness violations alerts – a hazard value for each dataset and each sensitive feature that indicates how dangerous the violation is. This is the value that we will employ to compute the bias amplification between the ground truth and the result of passing it through the model.
The bias detection tests have been conducted with the following arguments.
• Target feature: ground truth labels (Table 3), and model predictions (Table 4). • Sensitive features: – gender (male and female); – age (four age groups: 0-24, 25-39, 40-59, over 60); – marital status (single and married/separated/divorced/widowed). • Divergence used to measure the discrepancy between frequencies: Jensen–Shannon [22]. • Threshold: automatically computed with sensitivity set to high. • Function to aggregate divergences between pairs of sensitive classes (only for the test on age): arithmetical mean. • Conditioning variables for double-checks on subclasses: – Attribute 3, “credit history” (no credits taken / all credits paid back duly, all credits at this bank paid back duly, existing credits paid back duly till now, delay in paying of in the past, critical account / other credits existing but not at this bank); – Attribute 10, “other debtors / guarantors” (none, co-applicant, guarantor).
Our results are collected in Table 3 and in Table 4. We display hazard values, in particular cumulative hazard is the sum of all estimated hazard values including without conditioning. The aim is to assess whether looking at subclasses increases the diference in behaviour or not. Recall that a full list of violations with their respective estimated hazards is available on Git7.
First, it should be noted that the hazard measure for gender and marital status is quite similar – it is not actually identical, see the full report in the Git repo, since it difers from the 15ℎ decimal point on. This is partly explained by the fact that all women in our dataset are married.8
Now we move on to the significance of our analysis. The most notable diference in comparing our analysis on the ground truth with that on the model predictions occurs with gender: while the ground truth appears to be non-biased with respect to gender,9 this is not the case for the model which
7https://github.com/DLBD-Department/BRIO_x_Alkemy/tree/main/BEWARE_2024
8This is perhaps another argument in favour of the thesis of [
ground_truth_hazard whose results are collected in Table 5.
Of course, as for the hazard values related to gender and marital status, we would need to divide 0.1673 by 0, so we cannot compute an actual percentage. When this is the case, in general, we have a situation in which the model introduces a bias which is not at all present in the ground truth. While we cannot compute a percentage value, we can still evaluate the seriousness of the situation by considering the hazard values computed by the BRIO tool relatively to the predictions of the model. For instance, as far as gender and marital status are concerned, the bias introduced is comparable: the tool yielded for both sensitive features an hazard value (without conditioning) of 0.0695 and a cumulative hazard value of 0.1673. As already mentioned, this is probably due to the extreme correlation between the gender value associated to women and the marital status value associated to married people. We leave as future work the task of defining a mathematical measure of bias amplification that enables us to homogeneously treat all these cases.
Bias amplification describes the process by which bias in the data propagates throughout the ML pipeline, ultimately leading to unfair outcomes. By building on this notion, in this paper, we introduced the concept of the Bias Amplification Chain (BAC), illustrated three links thereof, and used the BRIO framework to quantify each of these.
An interesting avenue for future research involves exploring more complex configurations of the Bias Amplification Chain itself. In particular, refining our BAC with additional links to the chain could provide a more fine-grained and realistic analysis of bias amplification. For instance, we want to investigate how bias is amplified during the phase of data curation, when in order to optimize the training certain variables are discarded (feature selection), and synthetic features are automatically crafted by the system from the initial predictors (feature engineering).
More generally, the notion of BAC provides a model to quantitatively reason about where and when unfairness is produced along the ML life cycle. Breaking down the phenomenon of ML unfairness in diferent links of a chain representing the ML pipeline can potentially yield valuable insights on the efectiveness of certain mitigation methods (e.g., increasing the data quality, increasing the sample size, learning fair representations, etc.). Finally, a major open question remains how to meaningfully aggregate the preliminary results here obtained in order to gain a holistic insight on the phenomenon of ML unfairness. 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’24, Association for Computing Machinery, New York, NY, USA, 2024, p. 642–659. URL: https://doi.org/10.1145/ 3630106.3658931. doi:10.1145/3630106.3658931. [20] S. Kullback, R. A. Leibler, On Information and Suficiency, The Annals of Mathematical Statistics 22 (1951) 79 – 86. URL: https://doi.org/10.1214/aoms/1177729694. doi:10.1214/aoms/1177729694. [21] J. Liu, Z. Shen, Y. He, X. Zhang, R. Xu, H. Yu, P. Cui, Towards out-of-distribution generalization: A survey, 2023. URL: https://arxiv.org/abs/2108.13624. arXiv:2108.13624. [22] J. Lin, Divergence measures based on the shannon entropy, IEEE Transactions on Information Theory 37 (1991) 145–151. doi:10.1109/18.61115.