Fairness in Multi-Agent Systems (MAS) has been extensively studied, particularly in reward distribution among agents in scenarios such as goods allocation, resource division, lotteries, and bargaining systems. Fairness in MAS depends on various factors, including the system's governing rules, the behaviour of the agents, and their characteristics. Yet, fairness in human society often involves evaluating disparities between disadvantaged and privileged groups, guided by principles of Equality, Diversity, and Inclusion (EDI). Taking inspiration from the work on algorithmic fairness, which addresses bias in machine learning-based decision-making, we define protected attributes for MAS as characteristics that should not disadvantage an agent in terms of its expected rewards. We adapt fairness metrics from the algorithmic fairness literature-namely, demographic parity, counterfactual fairness, and conditional statistical parity-to the multi-agent setting, where self-interested agents interact within an environment. These metrics allow us to evaluate the fairness of MAS, with the ultimate aim of designing MAS that do not disadvantage agents based on protected attributes.
Multi-Agent Systems (MAS) consist of agents interacting with each other and their surrounding
environment to achieve their individual or shared goals. The achievement of an agent’s goals may
depend on the actions it takes, the actions of other agents, the environment they are situated in, and
the rules that govern the MAS. Similarly, fairness in MAS depends on multiple factors. Fairness can be
influenced by agents’ decision-making processes, as evidenced by research in reinforcement learning
focused on developing fair and eficient policies [
In human societies, fairness is often defined in terms of characteristics that should not disadvantage
an individual or group, such as age, race, disability or gender. For example, in the UK Equality Act 20101
these are identified as protected characteristics, and UK law states that individuals cannot be discriminated
against on the basis of these. These protected characteristics typically define subgroups of the population
who have historically been disadvantaged in particular situations, such as age discrimination in the
workplace, unequal access to healthcare or barriers in education for people with disabilities and gender
disparities in political representation, among others. Driven by the bias that often exists in the training
data as a result of these systemic inequalities, machine learning approaches often produce biased results
(e.g., discrimination in credit market [
Taking inspiration from the UK Equality Act 2010, we define the concept of protected attributes within a multi-agent system, which are any attributes that have been deemed should not disadvantage an agent in terms of its performance within that system. For example, consider a multi-agent setting that includes both artificial agents in the form of autonomous vehicles and human agents who drive their own cars; we may want to ensure that the human agents are not disadvantaged in such a setting. We adapt the following fairness metrics from the algorithmic fairness literature to our multi-agent setting. • Demographic parity – Agents with and without protected attributes should obtain the same expected rewards. • Counterfactual fairness – In both a factual and a counterfactual scenario, where the only diference is whether the protected attributes hold for an agent, agents should obtain the same expected rewards. • Conditional statistical parity – Within a group of agents characterised by a legitimate factor influencing rewards, agents with and without protected attributes should obtain the same expected rewards.
We are able to evaluate diferent MAS according to these metrics, with the ultimate aim of designing
fairer MAS (for example, by configuring the environment in which agents operate to optimise for
fairness). Such an approach is inspired by other works outside MAS, such as designing accessible
buildings [
To summarise, the contributions of this paper are as follows. We introduce protected attributes to MAS – characteristics that should not impact an agent’s expected rewards, all other things being equal. We adapt the concepts of demographic parity, counterfactual fairness and conditional statistical parity from the algorithmic fairness literature to the MAS context. The future aim of this work is to use these metrics to evaluate and optimise MAS for fairness.
Motivating example. In future urban environments, we may see vehicles operated by humans and vehicles operated by AI undertaking journeys within the same road network. These human and AI agents navigate city streets to reach their destinations, with the rewards they receive dependent on things like time taken and cost of journey. AI-driven vehicles excel by analysing trafic data in real-time, optimising routes, and communicating with other AI vehicles, providing them with an advantage over the human agents in the system, who are generally less eficient at route optimisation and less well-equipped to coordinate with other road users. To mitigate this advantage of AI agents, we might consider altering the road infrastructure, for example, by providing dedicated lanes for human-controlled vehicles.
Fairness has attracted the attention of Game Theory and MAS researchers for decades alongside
psychologists and economists [17, 18, 19, 20]. Factors such as the rules that govern the system can
influence fairness in MAS. For instance, this can be seen in the Ultimatum Game, where fairness is
influenced by the dynamics between proposers and responders [ 21, 22, 23]. In goods allocation or
cake-cutting games, the rules depend on the type of good being allocated, for example, whether they
are divisible or indivisible, goods or chores [24, 25], and fairness depends on the distribution of goods
among the agents [
Agent behaviour can also influence fairness. Fair behaviours often balance the rewards collected by the community and individuals. For example, Zhang and Shah [28] propose a minimum reward for the worst-performing agent while improving the overall rewards of the whole community of agents. However, fairness and reward optimisation can be in tension, and compromises must be made regarding one of the two sides. Jiang and Lu [29] propose a two-step solution consisting of a single policy for each agent based on fair and optimal rewards, with a controller agent who decides which sub-policies to implement to maximise environmental rewards and fairness. Other works [30, 31, 32] implement fair optimisation policies within cooperative multi-agent systems, aiming to integrate individualistic and altruistic behaviours. Grupen et al. [33] introduce a new measure of team fairness, demonstrating how maximising team rewards in cooperative MAS can lead to unfair outcomes for individual agents.
In contrast to these works, which do not distinguish agents that may be particularly disadvantaged within a system, we consider fairness across agents who do or do not possess protected attributes. We adapt demographic parity [34, 35], counterfactual fairness [35] and conditional statistical parity [36] fairness metrics from the algorithmic fairness literature to the MAS setting.
A multi-agent system consists of multiple decision-making agents who act and interact in an
environment to achieve their goals. A multi-agent system S = (, , , , , , ) is
characterised by: the set of possible environment states ; the starting state 0; the set of available actions
that may be performed by an agent in the environment (including a null action); a population
= {1, . . . , } of agents; the attributes = {1, . . . , } available to the agents in ; the
protected attributes ⊂ { 1, . . . , }; and the non-deterministic state transformer function
: × 1 × . . . × → × [
An agent within a multi-agent system (, , , , , , ) (where ∈ ) is defined as
a tuple (, , , ) where: the attribute evaluation function : → {0, 1} specifies which
attributes hold true for the agent; ⊆ are the actions available to the agent; the non-deterministic
policy : → × [
A possible run within a multi-agent system S = (, , , , , , ) (where consists of agents) is denoted = (0, (11, . . . , 1), 1, . . . , (1 , . . . , ), ) where: for each ∈ and for each such that 0 < ≤ , (, ) ∈ (− 1) and > 0; and for each such that 0 ≤ < , (+1, ) ∈ (, (1 , . . . , )) and > 0. The set of all possible runs within a multi-agent system S is denoted ℛS .
Let = (0, (11, . . . , 1), 1, . . . , (1 , . . . , ), ) ∈ ℛS where S = (, , , , , , ). We can determine the probability will occur, denoted ( | S ), as follows. ( | S ) = ︃( − 1(︃ ∏︁ ∏︁ where (+1, ) ∈ () =0 =1 )︃)︃ (︃ − 1 · ∏︁ where (+1, ) ∈ (, (+1, . . . , +1))
1 =0 )︃ For a run = (0, (11, . . . , 1), 1, . . . , (1 , . . . , ), ), the reward achieved by an agent is (, ) = ∑︀
=1 (− 1, ).
The expected reward of an agent within a system S , denoted (, S ), is thus (, S ) = ∑︁ (, ).( | S ).
∈ℛS Motivating example, continued. The city trafic consists of a population of cars, each capable of steering, accelerating or braking. Cars also possess attributes like speed or safety features. Cars are either driven by AI or humans, and we consider being driven by humans to be a protected attribute of cars. AI-driven cars can find optimal paths to reach their destination more eficiently than human-driven ones. If we consider agents reaching a hospital, we can foresee fairness problems as AI-driven cars would be advantaged. When the cars act with a specific probability, the environment changes state. Also, each car obtains a reward when reaching its destination. A car’s policy is a decision rule based on the state of the crossroads.
We define fairness by comparing, in diferent ways, the rewards gathered by individuals or groups of agents possessing and not possessing protected attributes. We adapt demographic parity [34, 35], counterfactual fairness [35] and conditional statistical parity [36] to MAS.
Demographic parity in MAS is achieved when the expected rewards of agents are not influenced by whether or not they possess protected attributes, all else being equal.
Definition 1 (Demographic Parity). Let S = (, , , , , , ) be a system and let ∈ be the protected attribute under consideration. Demographic parity is satisfied for in S if and only if: for all , ∈ , if () = 1, () = 0, and for all ′ ∈ ∖{}, (′) = (′), then (, S ) = (, S ).
Where demographic parity is not satisfied for a particular protected attribute, we can measure the extent to which this is the case, denoted (, S ), as follows. ,∈ such that ()=1, ()=0,
and for all ′∈∖{},(′)=(′) Note that if demographic parity holds for in S then (, S ) = 0.
Counterfactual fairness in MAS is achieved when the expected rewards of agents remain the same in both a factual and a counterfactual world, where in the latter, we change the protected attribute of the agents while keeping all other elements the same.
Definition 2 (Counterfactual Fairness). Let S = (, , , , , , ) be a system where = {(1, 1, 1, 1), . . . , (, , , )}, and let ∈ be the protected attribute under consideration. Let S ′ = (, , , ′, , , ) be the counterfactual of S such that ′ = {(′1, 1, 1, 1), . . . , (′, , , )} where for all such that 1 ≤ ≤ : if () = 0, then ′() = 1; if () = 1, then ′() = 0; and for all ∈ ∖ {}, () = (). Counterfactual fairness is satisfied for in S if and only if: for all = (, , , ) ∈ , for all ′ = (′, , , ) ∈ ′, (, S ) = (′, S ′). Where counterfactual fairness is not satisfied, we can measure the extent to which this is the case, denoted (, S ), as follows.
(, S ) =
∑︁ ∈ such that ()=1 (, S ) − (′, S ′) (2) Note that if counterfactual fairness holds for in S then (, S ) = 0.
Conditional statistical parity in MAS is achieved when the expected rewards of agents are not influenced by whether or not they possess protected attributes when conditioning on a legitimate factor, assuming all other elements are the same. A legitimate factor is an attribute that has been identified as one that may legitimately afect an agent’s reward.
Definition 3 (Conditional Statistical Parity). Let S = (, , , , , , ) be a system, let ⊆ ( ∖ ) be the set of legitimate factors, and let ∈ be the protected attribute under consideration. Conditional statistical parity is satisfied for with in S if and only if: for all , ∈ , if () = 1, () = 0, ( ) = ( ) = 1 for all ∈ , and for all ′ ∈ ∖ {}, (′) = (′), then (, S ) = (, S ).
Where conditional statistical parity is not satisfied, we can measure the extent to which this is the case, denoted (, , S ), as follows. (, , S ) = ∑︁ ,∈ such that ()=1, ()=0,
( )=( )=1 for all ∈, and for all ′∈∖{}, (′)=(′) (3) Note that if conditional statistical parity holds for with in S then (, , S ) = 0.
Conditional statistical parity is demographic parity within subsets of the population characterised by legitimate factors. For example, in algorithmic fairness, such a metric is used to verify whether the probability of predicting re-ofence for male and female prisoners is the same for similar age groups, which is the legitimate factor [37].
Motivating example, continued. In the city trafic example, demographic parity would be achieved if the sum of the expected rewards obtained by AI-driven cars and human-driven cars were equal, all other things being equal. In other words, the protected attribute should not afect the expected rewards gathered by the human-driven cars compared to the AI-driven ones. Counterfactual fairness is achieved if the sum of the expected rewards of the cars remains the same in both a factual and a counterfactual world, where in the latter, agents possess the protected attribute (i.e., cars are driven by humans) while keeping all other factors constant. Conditional statistical parity is achieved if the sum of the cars’ expected rewards is not influenced by whether or not they possess protected attributes when conditioned on a legitimate factor, e.g., a certain range of speed capacity of the cars, assuming all other elements are the same.
We can use the metrics above to measure fairness of diferent systems. Our ultimate goal is to optimise systems for these diferent fairness measures, for example by adjusting the starting state of the environment, or the way the environment responds to the agents’ actions.
This paper is a first step towards ensuring that certain sub-groups of agents are not disadvantaged in multi-agent systems. We identify protected attributes, which are characteristics that should not disadvantage an agent in terms of its expected rewards. Inspired by algorithmic fairness, we adapt demographic parity, counterfactual fairness and conditional statistical parity to analyse fairness in MAS. Our metrics assess fairness from various perspectives in any multi-agent system where expected rewards are applicable. Additional metrics from the algorithmic fairness literature, such as equal opportunity, equalised odds [38], disparate impact [39], or other metrics based on causal reasoning [40, 41] could be adapted to this setting to capture other aspects of fairness. Our methodology applies to MAS, involving both human and AI agents, as motivated by our example. It could also be used to improve the fairness of human societies by modelling these as multi-agent systems and seeing how changes to the system afect the various fairness metrics defined here.
In future work, we plan to analyse these fairness metrics experimentally in diferent settings, both competitive and cooperative, to find system configurations that enhance fairness. We will use techniques such as Bayesian optimisation [42], evolutionary algorithms [43] and sparse sampling techniques [44] to try to identify system configurations that optimise for the diferent fairness metrics.
This work was supported by UK Research and Innovation [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (www.safeandtrustedai.org). multi-agent reinforcement learning, 2021. arXiv:2012.09421. [33] N. A. Grupen, B. Selman, D. D. Lee, Cooperative multi-agent fairness and equivariant policies, 2022. arXiv:2106.05727. [34] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, R. Zemel, Fairness through awareness, in: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, Association for Computing Machinery, New York, NY, USA, 2012, p. 214–226. URL: https://doi.org/10.1145/2090236. 2090255. doi:10.1145/2090236.2090255. [35] M. Kusner, J. Loftus, C. Russell, R. Silva, Counterfactual fairness, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Curran Associates Inc., Red Hook, NY, USA, 2017, p. 4069–4079. [36] S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, A. Huq, Algorithmic decision making and the cost of fairness, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 797–806. URL: https://doi.org/10.1145/3097983.3098095. doi:10.1145/3097983. 3098095. [37] R. Berk, H. Heidari, S. Jabbari, M. Kearns, A. Roth, Fairness in criminal justice risk assessments: The state of the art, Sociological Methods & Research 50 (2021) 3–44. doi:10.1177/ 0049124118782533. [38] M. Hardt, E. Price, N. Srebro, Equality of opportunity in supervised learning, 2016.
arXiv:1610.02413. [39] M. Feldman, S. Friedler, J. Moeller, C. Scheidegger, S. Venkatasubramanian, Certifying and removing disparate impact, 2015. arXiv:1412.3756. [40] N. Kilbertus, M. Rojas-Carulla, G. Parascandolo, M. Hardt, D. Janzing, B. Schölkopf, Avoiding discrimination through causal reasoning, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Curran Associates Inc., Red Hook, NY, USA, 2017, p. 656–666. [41] R. Nabi, I. Shpitser, Fair inference on outcomes, in: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18, AAAI Press, 2018. [42] J. Snoek, H. Larochelle, R. P. Adams, Practical bayesian optimization of machine learning algorithms, 2012. arXiv:1206.2944. [43] P. A. Vikhar, Evolutionary algorithms: A critical review and its future prospects, in: 2016 International Conference on Global Trends in Signal Processing, Information Computing and Communication (ICGTSPICC), 2016, pp. 261–265. doi:10.1109/ICGTSPICC.2016.7955308. [44] M. Kearns, Y. Mansour, A. Y. Ng, A sparse sampling algorithm for near-optimal planning in large markov decision processes, in: Proceedings of the 16th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’99, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1999, p. 1324–1331.