As the impact of Artificial Intelligence systems and applications on everyday life increases, algorithmic fairness undoubtedly constitutes one of the major problems in our modern society. In the current paper, we extend the work of Dimitrakakis et al. on Bayesian fairness [1] that incorporates models uncertainty to achieve fairness, proposing a practical algorithm with the aim to scale the framework for a broader range of applications. We begin by applying the bootstrap technique as a scalable alternative to approximate the posterior distribution of parameters of the fully Bayesian viewpoint. To make the Bayesian fairness framework applicable to more general data settings, we define an empirical formulation suitable for the continuous case. We experimentally demonstrate the potential of the framework from an extensive evaluation study on a real dataset and diferent decision settings.
(, ) = E [ ] − E [ ] =
E [ ] ( ) − E [ ] ( )
This idea was used in [1], which focused on fairness constraints related to balance [3], that states that the actions should be independent of the sensitive variable if conditioned to the true outcome i.e ⊥ | . The resulting deviation from fairness can be formalized as follows: ∫︁ Θ ∫︁ Θ
Since the DM does not know the true parameter * , they have a belief ∈ over a family of distribution = { | ∈ Θ} that it may contain the actual law, i.e. for some . The belief expresses the uncertainty of the decision maker about the word. In the Bayesian case, the belief is a posterior formed through his prior distribution ( ) and the available data.
The DM wishes to find a policy
maximizing expected utility [], where (, ) is a
utility function dependent on the DM’s action and the unknown outcome. At the same time,
though they must take into account fairness constraints . In particular, the optimal policy for
a given belief will maximize the following utility-fairness tradeof:
(
∑︁ (|)[ (, |) − (|) (|)] ‖ while on the other hand, replacing the unknown with the marginal model P = ∫︀ results in the Marginal Balance Rule, which in practice can be very unfair for high-probability Θ ( ), models. This approximates the expected fairness violation as:
EP [ ] = ∑︁ ‖
∑︁ (|)[P (|, ) − P (|)] P (|) ‖ the Marginal balanced rule on the COMPAS [8] dataset.
In the context of the Bayesian fairness framework, the decision maker has a belief that is expressed through a distribution = { |
∈ Θ} over diferent parameters. Instead of using the posterior distribution, as in Bayesian inference, one can use the sampling distribution of an estimator ^ obtained by sub-sampling data from the original dataset . The idea of the bootstrapping method is to get diferent datasets by sampling diferent datasets of size where = ^(), and our optimisation problem becomes || with replacement, called bootstrap datasets. We then obtain a set of samples 1, . . . , , B*ootstrap = arg max ∑︁ E [ − ].
=1
In addition to the bootstrapping method, we need an empirical estimate of the fairness
violation 2 when our observations are continuous variables. In particular, for infinite finite
, and the equation 2 becomes:
(
For the above equation we need to estimate (), (|), (|), (|, ). In the case where , is finite it’s easy to estimate
(), (|) with discrete models, whereas for optimal policy of 1 we can make use of a gradient descent algorithm. (|), (|, ) we can use continuous predictive models to estimate them. To find the
We empirically evaluate our approach by comparing the policies obtained from the diferent balance rules on the COMPAS dataset. We performed two diferent experiments to study the efects of our algorithmic solution using discrete and continuous versions of the dataset. More specifically, in the discrete case, we can compare the bootstrap and the marginal methods to the closed-form Bayesian approach by employing a fully discrete Bayesian network model as described in [1]. In the second experiment, we only compare the policies obtained from Bootstrap and the marginal rules using the empirical formulation 5. To evaluate our policies, we calculate the fairness and utility with respect to the empirical model from a hold-out dataset, performing the experiments 50 times, each time shufling our dataset and using 6000 ( 70%) data points for training and the rest 1214 (30%) for testing, reporting the average and the variance of the aforementioned metrics. Our code https://github.com/a-athanasopoulos/Bayesian-fairness reproduces all presented experiments.
Experimental Setup. To study how DM uncertainty afects fairness we consider the following experimental setting which is operating over multiple steps , each time increasing the amount of training data. Intuitively, in the early steps of the process, the DM is more uncertain about the model parameters as he has assessed fess data compared to the later steps, so we expect the Bayesian methods to perform better regarding fairness as we account for uncertainty. We use 12 steps each time considering = * 500 data points.
Models and Policies. The graphical model is fully connected, so the model uses the factorization (,,)=(|,)(|)() from which we can calculate all the relevant models to optimize our policy. To form the posterior distribution for the discrete case we use the Dirichlet-Multinomial model with a non-informative initial Dirichlet parameter of 1/2 from which we can sample diferent model parameters, while we can also calculate the marginal model in closed form. We use parameterized policies of the form (|) = . For the continuous case, we calculate the marginal models of (), (|) as in the discrete case, while for the models (|), (|, ) and the policy (|) that contains the continuous observations we use logistic regression models. The sampling distribution of the bootstrap method uses the marginal model using diferent bootstrap datasets for both continuous and discrete cases as described in section 2.
The Algorithm. We use gradient descent to optimize the policies by minimizing the negative total expected utility as in 1. In particular, for the Bayesian approaches we sample a model from the distribution and then take a step in the gradient direction in each iteration. More specifically for both bootstrapping and the fully Bayesian approach, we use 16 models to form the , obtained either by using a diferent bootstrap dataset or directly sampling from the true posterior accordingly. The marginal policies simply perform the steepest gradient descent for the marginal model. We make 1500 gradient decent iterations in each policy update, with a learning rate of 0.01.
Results. In Figure 1 we illustrate the diferent trade-ofs between utility and fairness for both discrete and continuous experiment versions in each step of the process using = 1. Both experiments indicate that the policies obtained from the Bayesian approaches outperform the marginal one in both fairness and total utility . In particular, for the continuous version, the efect is more evident in the early steps of the algorithm where we have a limited amount of data and thus greater model uncertainty. For the discrete data, we observe a constant advantage of the Bayesian approach over the marginal one. In addition, we can say that the bootstrap method is a good approximation of the Bayesian posterior. Finally, the policies obtained from the Bayesian approaches result in less variance in both continuous and discrete experiments.
In this work, we ofer an algorithm solution to scale the concept of Bayesian fairness [ 1]. We propose the use of bootstrapping as an alternative to the posterior distribution of the parameters, along with an empirical formulation of the balance fairness metric suitable for continuous data scenarios. The results of our empirical study emphasize the significant role of accounting for uncertainty in the context of algorithmic fairness. Interesting research directions include the consideration of alternative fairness metrics, the incorporation of Bayesian fairness in reinforcement learning, and the application of the framework in modern deep learning models to investigate the advantages of Bayesian fairness in huge and complex datasets with higher degrees of uncertainty. as a form of transparency: Measuring, communicating, and using uncertainty, in: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, Association for Computing Machinery, 2021, p. 401–413. doi:10.1145/3461702.3462571. [8] J. Angwin, J. Larson, S. Mattu, L. Kirchner, Machine bias, in: Ethics of data and analytics, Auerbach Publications, 2016, pp. 254–264.