Machine learning applications are becoming increasingly pervasive in our society. Since these decisionmaking systems rely on data-driven learning, risk is that they will systematically spread the bias embedded in data. In this paper, we propose to analyze biases by introducing a framework for generating synthetic data with specific types of bias and their combinations. We delve into the nature of these biases discussing their relationship to moral and justice frameworks. Finally, we exploit our proposed synthetic data generator to perform experiments on diferent scenarios, with various bias combinations. We thus analyze the impact of biases on performance and fairness metrics both in non-mitigated and mitigated machine learning models.
As society grows more digital, a greater amount of data becomes accessible for decision-making.
In this context, machine learning techniques are increasingly being adopted by businesses,
governments, and organizations in many important domains that afect people’s lives every
day. However, algorithms, like humans, are susceptible to biases that might lead to unfair
outcomes [
Bias is not a recent problem; rather, it is ingrained in human society and, as a result, it is
reflected in data [
Several projects and initiatives have been launched in recent years aimed at bias mitigation
and the development of fairness-aware machine learning models. Following to [
One common approach to investigate the nature of bias is to conduct experiments on ad hoc
scenarios through the generation of synthetic data [
With this work, we aim to contribute to the literature on fairness in machine learning in each of the three areas discussed above through the use of synthetic data.
In particular, we contribute to understanding bias by introducing a model framework for generating synthetic data with specific types of bias. Our formalisation of these various types of bias is based on the theoretical classifications present in the relevant literature, such as the surveys on bias in machine learning by Mehrabi et al. and Ntoutsi et al..
Against the background of the stream of literature about the relation between moral
worldviews and biases, and in particular following [
Finally, about mitigating bias, we leverage our framework to generate twenty-five diferent
scenarios characterized by the presence of various bias combinations. In each setting, we
investigate the behavior and efects of traditional machine learning mitigation strategies [
An open source implementation of the proposed model framework is available at github.com/rcrupiISP/ISParity.
Synthetic data generation is a relevant practice for both businesses and the scientific community. As a result, the literature has given it a lot of attention.
Main directions behind the generation of synthetic data are: the emulation of certain key
information in real dataset while preserving privacy [
Is important to remark that synthetic data generation is not a possible technique for data
anonymisation [
As introduced in the previous section, we are interested in producing fully synthetic data
to replicate some common biases that could afect the data, and investigates fairness-related
issues that arise from the development of machine learning models on them. In this regard,
there are numerous works in the literature that generate synthetic datasets to simulate desired
scenarios and, from these, testing discrimination-aware methods [
We contribute in this field of literature by introducing a modeling framework to generate synthetic data presenting specific forms of bias. Our formalisation of these diferent kinds of bias builds upon theoretical classifications present in the relevant literature, such as the works of Mehrabi et al. and Ntoutsi et al.. We leverage our proposed method to actually generate several datasets, each characterized by a specific combination of biases, and perform experiments on them to examine the efects of such biases on state-of-the-art mitigation approaches.
There is little consensus in the literature regarding bias classification and taxonomy. Moreover,
the very notion of bias depends on profound ethical and philosophical considerations, which is
likely one of the very causes for the lack of consensus. Diferent understandings of bias and
fairness depend on the assumption of a belief system beforehand. Friedler et al. and Hertweck
et al. talk about worldviews. In particular, [
Starting from the definition of three diferent metric spaces, these two perspectives difer because of the way they consider the relations in between. The first space is named Construct Space (CS) and represents all the unobservable realized characteristics of an individual, such as intelligence, skills, determination or commitment. The second space is the Observable Space (OS) and contains all the measurable properties that aim to quantify the unobservable features, think e.g. of IQ or aptitude tests. The last space is the Decision Space (DS), representing the set of choices made by the algorithm on the basis of the measurements available in OS. Note that shades of ambiguity are already detectable at this level, because the mappings between spaces are susceptible of distortions. Moreover, CS is by definition unobservable, thus we can only make assumptions on it.
According to WYSIWYG, CS and OS are essentially equal and any distortion between the two
is altogether irrelevant for the fairness of the decision resulting in DS. Contrarily, WAE doesn’t
make assumptions about the similarity of OS and CS, and moreover assumes that we are all
equal in CS, i.e. that any diference between CS and OS is due to a biased observation process
that results in an unfair mapping between CS and OS. With the distance between worldviews in
mind, the notion of fairness inspired by [25] afirming that individuals that are close in CS shall
also be close in DS (commonly known as individual fairness) appears diversified and diferently
achievable. If WYSIWYG is chosen, non-discrimination is guaranteed as soon as the mapping
between OS and DS is fair, since CS ≈ OS. On the other hand, according to WAE the mapping
between CS and OS is distorted by some bias whenever a diference among individuals emerges
(this diference is named Measurement Bias in [
Building on [
These extreme worldviews amount on the one hand to accept the situation as it is observed (WYSIWIG), on the other to infer some form of unfairness whenever there is some observed disparity (WAE). To avoid such extreme scenarios, philosophical theories around Equality of Opportunity (EO) ofer some suggestions and interpretive tools for approaching biases in diferent situations [26, 27, 28]. In this sense, western political philosophy and algorithmic fairness literature encounter themselves in the formulation of fairness around the concept of equal opportunities for all members of society. Heidari et al. list three diferent EO conceptions, going from more permissive to more stringent: Libertarian EO, by which individuals are held accountable for any characterizing feature, sensitive one included; Formal EO, by which individuals are not held accountable for diferences in sensitive features only; Substantive EO, by which there is a set of individual characteristics which are due to circumstances and others that are a consequence of individual efort , and people should be held accountable only on the basis of the latter. The choice of which characteristics fall in the level of circumstances and which can be considered as individual efort is far from obvious. Depending on the EO framework that one is willing to embrace, observed disparities may be seen as “just” or “unjust” forms of bias.
In the following section, we shall describe the most common biases, explaining how they relate to these fundamental concepts.
Considering that the assumptions about the worldview and the EO framework afect the conception as well as the assessment of biases, in what follows we focus on what we consider the fundamental building blocks of most types of biases, namely: Historical bias, Measurement bias, Representation bias, Omitted variable bias.
Historical bias —sometimes referred to as social bias, life bias, or structural bias [
Measurement bias occurs when a proxy of some variable relevant to a specific goal or target
is employed, and that proxy happens to be dependent on some sensitive characteristics. For
instance, one may argue that IQ is not a “fair” approximation of actual “intelligence”, and it
might systematically favour/disfavour specific groups of individuals. Statistically speaking, this
type of bias is not very diferent from historical bias —since it results in employing a variable
correlated with sensitive attributes— but the underlying mechanism is nevertheless diferent,
and in this case the bias needs not to be present in the phenomenon itself, but rather it may
be a consequence of the choice of data to be employed. In other words, this is an example of
bias from data to algorithm in the taxonomy of [
Representation bias occurs when, for some reasons, data are not representative of the world population. One subgroup of individuals, e.g. identified by some sensitive characteristic such as ethnicity, age, etc. may be heavily underrepresented. This under-representation may occur in diferent ways. It may be at random, i.e. the subgroup is less numerous than it should be, but without any particular skewness in the other characteristics: in this case this single mechanism is not suficient to create disparities, but it may exacerbate existing ones. Alternatively, the under-represented subgroup might contain individuals with disproportionate characteristics with respect to their corresponding world population, e.g. only low-income individuals, or only low-education individuals. In this last case, representation bias may be enough to create disparities in decision making processes based on those data. The mechanism underlying the representation disparities should be analyzed on the basis of the assumed worldview and/or chosen EO framework: e.g. if the data has an under-represented ethnic minority one should investigate why it is so. If the target population is itself diferent from world population (i.e it is not merely a poor data collection), then one should consider the reasons by which this ethnic minority is under-represented in the target population, e.g. in the Substantive EO framework one should understand if these reasons have to be regarded as circumstance or as consequence of individual efort . In the latter case, the representation disparities are not to be considered as “unfair” per se.
Representation bias is strictly connected to sampling bias, in that it embodies problems arising during data collection, e.g. by collecting disproportionately less observations from one subgroup, possibly skewed with respect to some characteristics. Like measurement bias, this is a form of bias going from data to algorithm.
Omitted variable bias may occur when the collected dataset omits a variable relevant to some specific goal or task. In this case, if the variables that are present in the dataset have some dependence on sensitive characteristics of individuals, then a machine learning model trained on such dataset will learn such dependencies, thus producing outcomes with spurious dependence on sensitive attributes. Assuming the Formal EO framework, sensitive features are omitted by default. While this may appear fairer because the decision is made solely on the basis of the relevant attributes, on the other hand it becomes arduous to mitigate on structural biases that afect achievements. Depending on worldview assumptions or on the chosen EO framework, the mechanism through which the residual variables happen to depend on sensitive individual characteristics should be as well analyzed to understand/decide whether this dependence is legitimate or if they are themselves a consequence of some bias at work.
The above list of biases should be seen as the set of the most important mechanisms through which “unfair” disparities happen to result in data-driven decision making. Notice, however, that in terms of consequences on the data, it may well be that diferent types of bias result in very similar efects. E.g. representation bias may create in the dataset spurious correlations among sensitive characteristics of individuals and other characteristics relevant to the problem at hand, a situation very similar to the correlations present as a consequence of historical bias. This reminds us that in reality we are not aware of the type of bias (or biases) afecting the data and that their interpretation depends on former assumptions.
We propose a simple modeling framework able to simulate the bias-generating mechanisms described in section 3.
The rationale behind the model is that of being at the same time suficiently flexible to accommodate all the main forms of bias generation while maintaining a structure as simple and intuitive as possible to facilitate human readability and ensure compactness avoiding unnecessary complexities that might hide the relevant patterns.
As noted in section 3, following [
A data-driven decision maker infers from a (training) set of samples {(̃︀, )}=1,..., , an estimate for that we label ^ , thus producing its best estimate for , namely = () + , ^ = ^ (̃︀ ). (1) (2) The use of ̃︀ rather than describes the fact that the set of variables employed to make inferences about a phenomenon may not coincide with the actual variables that play a role in that phenomenon. This is precisely what happens in some forms of biases such as measurement bias or omitted variable bias.
Notice that users to-data types of bias impact directly equation (1), while data-to-algorithm biases have a role at the level of equation (2).
It is possible to make a schematic representation of building blocks of biases as discussed in section 3 via Directed Acyclic Graphs (DAGs), representing causal impacts among variables (see e.g. [29, 30, 31]). In general, in order to provide an intuitive grasp on interesting mechanisms and patterns, we shall make reference to the following situation: - we label with variables representing resources of individuals —be them economic resources, or personal talents and skills— which are relevant for the problem, i.e. they directly impact the target ; - we label with variables indicating sensitive attributes, such as ethnicity, gender, etc.; - we label with proxy variables that we have access to instead of the original variable .
We may think of as soft skills, or talent, or intelligence, i.e. something that we cannot measure or sometimes even quantify directly; - we label with additional variables, that may or may not be relevant for the problem (i.e. impacting ) and that may or may not be impacted either by or , e.g. the neighborhood one lives in.
With these examples in mind, we can e.g. think of —i.e. the target of our decision making process— as the repayment of a debt or the work performance.
In particular, figure 1 shows 3 minimal graph representations of historical, omitted variable and measurement biases. Historical bias occurs when the relevant variable is somehow impacted by sensitive feature . Omitted variable bias occurs when, for some reasons, we omit the relevant variable from our dataset and we employ another variable which happens to be impacted by . Measurement bias occurs when the relevant variable is, in general, free of bias, but we cannot access it, thus we employ a proxy which is impacted by sensitive characteristic . The measurement bias could occur also on the target variable , when we can access only on a proxy of the phenomenon that we want to predict.
The following system of equations formalizes the relationships between variables that are used to simulate specific form of biases: =, ∼ ℬ (); = − ℎ + , ∼ ( , 2); =, | (, ) ∼ ℬ (, (, )),
︁( (, ) = sigmoid − ( − ℎ) ;
︁) = − − ℎ + , ∼ (0, 2); =1{>¯}. (3a) (3b) (3c) (3d) (3e) I) No historical bias
II) Historical bias combined
III) Historical bias on
IV) Historical bias on
V) Historical bias on and
When simulating measurement bias, either on resources or on target , we are going to use the following proxies as noisy (and biased) substitutes of the actual variables: = − + ; ∼ (0, 2 ); = − + ;
∼ (0, 2 ); = 1{>¯} (4a) (4b) (4c)
By varying the values of the parameters, we are able to generate diferent aspect of biases as follows: - ℎ determines the presence and amplitude of the historical bias on the variable ∈ {, , }; - , when the proxy is used instead of the original variable , governs the intensity of measurement bias on ∈ {, }; - , control the linear impact on ( and thus) of and , respectively; represents the intensity of the dependence of on .
Additionally, in order to account for representation bias, we undersample the = 1 group. The amount of undersampling is governed by the parameter defined as the proportion of the under-represented group = 1 with respect to the majority group = 0. We draw the undersampling conditioned on by selecting the = 1 individuals with lower . Finally, simulating omission bias is as simple as dropping the variable from the set of features the model uses to estimate .
We employ our modeling framework to generate a total of twenty-five diferent datasets by combining a set of five users-to-data biases with a set of four data-to-algorithm biases, plus a configuration without additional biases. The aim of the experiments is to investigate the efects of these bias combinations on machine learning algorithms.
As clarified in section 3, users-to-data biases impact the phenomenon to be studied and thus the dataset. The five scenarios that we have chosen to explore, represented in figure 2, are define as follows:
= (, ) + , , ⊥⊥ .
= (, ) + , = (), = (), ⊥⊥ .
= (, ) + , = (), ⊥⊥ .
= (, , ) + , , ⊥⊥ .
= (, , ) + , = (), ⊥⊥ .
We combine each scenario with the following data-to-algorithm biases, having an efect on the variables that we assume we can access in order to train the models: (5) (6) (7) (8) (9) - no additional bias; - measurement bias on ; - omission bias on ; - representation bias on = 1; - measurement bias on .
We thus generate twenty-five synthetic datasets, each of which characterized by a specific choice of the set of parameters discussed in section 4. Details on the values of the parameters and the complete python code to reproduce the experiments is publicly available at github.com/rcrupiISP/ISParity.
On each generated dataset, we train and test the following three algorithms: (i) a Random
Forest (RF) [32] that aims to maximize performance by utilizing all available variables; (ii) Blinded
Random Forest (BRF): a random forest that does not use the sensitive variable in order to
avoid creating decision-making patterns based on it; (iii) Equalized Random Forest (ERF): the
same model in (i) with a post-processing fairness mitigation block [
Fairness of outcomes is evaluated through Demographic Parity diference ( ΔDP) [33], i.e. the diference between the rate of ^ = 1 among the two sensitive groups = 0 and = 1, namely ΔDP = (^ = 1 | = 0) − (^ = 1 | = 1). ΔDP can also be computed for the “true” target variable , thus capturing the dependence of the target variable on the sensitive attribute . Performance is evaluated by overall Accuracy (Acc). As an additional metric both of fairness and of performance, we compute the diference in accuracy between = 0 and = 1 groups (ΔAcc).
Table 1 summarizes the results obtained by the twenty-five experiments on generated datasets, on the basis of which we draw the following non-exhaustive list of insights:
On the efect of equalization : as expected, the adoption of a model to enforce a group equalization of the outcomes results in lower performance in the majority of scenarios considered. This gap is proportional to the level of disparity present in the data. On the contrary, when there is representation bias, the efects on performance are less visible, since enforcing equalization impacts fewer instances. This suggests that observing significant reductions in ΔDP with negligible performance deterioration is not per se suficient to conclude that a mitigation strategy has no efects on performance —as expressed by numerous studies in the field of fairness— since the protected class happens often to be a minority. Observing the diference in accuracy (ΔAcc), on the other hand, is a much more robust indicator because it is unafected by the diferent numerosity of the protected subgroups.
On the efect of blindness : excluding sensitive attributes when training machine learning models —as suggested by the Formal EO framework— has a negative impact on performance only when is directly influenced by . When the factors that determine contain historical bias, this technique has no benefit in terms of demographic parity. It’s worth noting that in the presence of measurement bias in scenarios I, II, and III, the efects on ΔDP are quite worse than when is included in the model. This occurs because the machine learning model actually needs to remove the excess of dependence induced by the inclusion of the proxy.
On the performance-bias trade-of in moral frameworks : one crucial aspect to keep in mind when developing a decision system is that what is and what is not a bias depends on the moral framework and worldview chosen in the first place. In this regard, the employment of a modeling strategy over another (e.g. using BRF instead of ERF) is in response to the chosen worldview. Thus, given a specific moral view , there are circumstances where it is justifiable to prefer performance over group equality and vice versa. Notice that the choice of a framework needs often be complemented with additional (crucial) decisions, e.g. choosing Substantive EO per se is not enough to understand whether some phenomenon underlying data is or not a bias, since one needs to agree on which characteristics are to be considered as “individual efort” (as opposed to “circumstance”), and thus legitimately employed as rationale of decisions.
On the interpretation of results: in this work, we simulate the presence of one or more biases and observe the corresponding efects on data. In real-world scenarios, data are typically observed without knowing the exact bias-generation mechanism underlying them. It is the assumption on the data generation mechanism that shapes the interpretation of biases present in the context under consideration.
This concept becomes more evident when the target variable in the data serves as proxy for the true target. In our experiments with measurement bias on , we trained the models with the proxy and evaluated the results with the real variable. This approach, even if unlikely in real-world situations, demonstrates that reducing bias can also lead to improved performances. These set of experiments shows that under a specific worldview, mitigation strategies that result in lower performance on the target variable in the observed space, may lead to higher performance on the “true” target.
On the absence of bias: when operating in an ideal bias-free environment, i.e. when any possible relevant characteristic is independent of sensitive features, all machine learning strategies provide the same performance and fairness results. Furthermore, in this ideal situation, all worldviews would agree, even if they are mutually incompatible in the general case. Regardless of how trivial this conclusion may be, it is essential to mention it in order to promote any legal, social, or strategic initiative with the goal of eliminating society biases.
In this work we have investigated how diferent types of bias impact fairness and performance metrics of machine learning models by introducing of a unified framework for generating synthetic data. Regardless of whether the test scenarios used to draw conclusions are synthetic, we have discussed moral frameworks and worldviews that can be applied in real-world situations.
We release the presented model framework as open-source at github.com/rcrupiISP/ISParity in order to encourage further research into the challenge of bias in data. For example, one crucial aspect to be further analyzed is the sensitiveness of results with varying magnitude of parameters; another is to control the efect of random fluctuations by averaging results of many simulations for each choice of the parameters set. the nuances in the fairness metrics landscape, Scientific Reports 12 (2022) 1–21. [25] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, R. Zemel, Fairness through awareness, in: Proceedings of the 3rd innovations in theoretical computer science conference, 2012, pp. 214–226. [26] M. Fleurbaey, Fairness, responsibility, and welfare, OUP Oxford, 2008. [27] J. E. Roemer, A. Trannoy, Equality of opportunity, in: Handbook of income distribution, volume 2, Elsevier, 2015, pp. 217–300. [28] J. Rawls, A theory of justice, in: Ethics, Routledge, 2004, pp. 229–234. [29] J. Pearl, Causality, Cambridge university press, 2009. [30] J. Pearl, D. Mackenzie, The book of why: the new science of cause and efect, Basic Books, 2018. [31] J. Peters, J. M. Mooij, D. Janzing, B. Schölkopf, Causal discovery with continuous additive noise models, Journal of Machine Learning Research 15 (2014). [32] G. Biau, E. Scornet, A random forest guided tour, Test 25 (2016) 197–227. [33] S. Barocas, M. Hardt, A. Narayanan, Fairness and Machine Learning, fairmlbook.org, 2019. http://www.fairmlbook.org.