Avoiding bias and understanding the consequences of artificial intelligence used in decision making is of high importance to avoid mistreatment and unintended harm. This paper aims to present an impact focused approach to model the information flow of a socio-technical decision system for analysis of bias and fairness. The framework roots otherwise abstract technical accuracy and bias measures in stakeholder efects and forms a scafold around which further analysis of the socio-technical system and its components can be coordinated. Two example use-cases are presented and analysed.
Artificial Intelligence (AI) is rapidly becoming more capable and becoming more and more
integrated in everyday life and society. Algorithmic decision-making is now commonly used in
many contexts, from the personal sphere, to business, and public and private organisations[
Several recent and comprehensive reviews make clear that there is a strong connection
between large-scale societal change and development and adoption of AI [
The current prevailing directions with growing taxonomies of bias types and fairness
definitions or measures have met critique on several levels [
The aim and contribution of this paper is to link the technical notion of algorithmic fairness to the socio-technical sphere where the connection between technical accuracy and stakeholder impacts can be established. Focusing on stakeholder impacts, AI fairness is realigned to avoid disproportionate harm to real individuals or groups. Since potential impacts can be connected to measurable efects, informed actions can be focused on the areas and problems which are actually causing harm.
This paper will be laid out as follows. Section 1 explores the background on fairness and impact assessment. Section 2 presents the main theoretical framework. In Section 3, we demonstrate the system on two illustrative case studies. Section 4 concludes the paper with a discussion on future directions.
The main current in AI Fairness focuses on the one side on taxonomies of biases which can
occur in relation to data collection, datasets, algorithms, evaluations, and other components of
technical or social systems. These biases are generally expressed as deviations from truth or
leading to an incorrect outcome [
The fairness definitions generally do not directly correlate to the degree the various classified
biases are expressed but depend on a great variety of additional factors, such as ground truth
availability or explanatory variables [
Algorithmic Impact Assessments (AIA) are methods for ex ante estimation of the societal consequences of an AI system before its development and iteratively during its life-cycle [18]. There is no clear consensus on the exact way to perform AIA but impact assessments in general typically build on stakeholder identification, participation and risk assessment with accountability as a main result [18]. Algorithmic impact assessment is also at least partially overlapping with Data Protection Impact assessments (DIPA) which are involved in GDPR as well as in Fundamental Rights Impact Assessments (FRIA) [18].
Conducting AIA takes the first step to answering the ‘question zero’ of AI ethics, namely
if AI should be used for this case at all by exploring the potential foreseeable consequences
[19, 18]. A focus on social impacts to afected stakeholders has the potential to bridge the
abstract notions of algorithmic fairness with actual social efects [
Another hindrance to the application of high-level ethical guidelines to AI ethics is what is termed as the abstraction gap, i.e., the seeming disconnect between values and guidelines on an abstract level and the specific technical features of a particular AI project. This also applies to algorithmic fairness where all biases and cases of unfair treatment need to be considered for a general and undefined application, but only a small subset of them apply to any real application.
A way to bridge this gap is to capture the context in such a manner that it singles out the issues
which are relevant for this particular case. RAIN [
In order to approach algorithmic fairness in a similar manner it would be necessary to characterize the socio-technical domain and links to stakeholders in a way to connect them with the technical measures of fairness and bias. This is what is done in the current paper.
We propose the Impactframework to specify a structural scafold over the information flow of a Socio-Technical System (STS) into which a Technical System (TS), i.e., an automatic decision making or decision support system, is embedded. Into this structure, the outputs, technical and social factors are connected to stakeholder impacts. This structure provides a reference frame for connecting technical measurements to actual efects and to track the risk and efect of potential sources of bias.
This framework does not aim to replace the existing work on stakeholder elicitation, algorithmic impact assessment or accuracy measurements. Rather, it connects the outputs and insights given by these approaches and tools. The modular dependency on these other methods means the framework can be applied in varied use-cases where diferent approaches might be ideal.
Applying the Impactframework in practice goes through the following steps: 1. Model the information flow. 2. Identify afected stakeholders and impacts. 3. Analyze the structure to determine bias and measures. 4. Perform testing to verify how the system performs. 5. Take action to handle risks of negative impact.
In the following the focus will be on the first three steps, first describing the capture of the system structure and information flow followed by presenting connecting impacts and potential measures to this structure.
We will formally characterize a Socio Technical System (STS) using the elements of Information Channel Theory [20, 21] that defines an information network consisting of three types of objects sites, types, and channels. Channels connect sites with each other and sites are of a type [20, 21]. In this paper, types will be assumed to be unique to each site and we can thus combine the notion of type and site.
We will see each system, — STS or TS — as a series of interactions through which some information becomes a particular output decision. Using the Information Channel Theory elements, sites will represent information passing through the system while channels will represents sub-systems which alter the information, like by making decisions.
Minimally, a system on this conception which takes some information as input and have outputs can be written as
⟼ . The output needs to relate to the input such that all information relating to a particular case upon which the decisions taken by to give can be found in . We can say that ⊆
and this requirement impose a partial ordering over the sites. A system can consist of any number of sub-system components, each of which is modeled as input-output maps as described above. Together these nested and chained systems form a directed acyclic graph from the base information through social or technical sub-systems (channels) until finally reaching one or more output sites representing the decisions taken.
A socio-technical decision-making system (STS) can be seen as a system on the above conception, where at least some component is a Technical System (TS), e.g., an artificial intelligence component embedded into otherwise social components, i.e., people.
Social systems often have non-linear and dynamic interactions between their participants. They can consist of feedback loops and back and forth passing of information. In this framework we will not be modeling this interaction directly but rather follow the transfer of information on an abstract level such that the model contains but underdetermines the actual interactions. Each channel represents a change between one state of information or site to another state of information rather than individual interactions of participating agents and components.
The simplest possible STS can be illustrated by a single user interacting with an application illustrated in Figure 1a. Seen as an information flow, we see the initial information state and the channel 1 representing what the user will share with the application (which is constrained by both the user and the application). A channel is determined by the application, modeling what contribution or change the interaction produces. Finally we have the users channel 2 to the output site which describes how the information now with the user contributes to a decision. This simplest socio-technical system could be described in this way ⟼1 1 ⟼ 2 ⟼2 , where 1 and 2 both represent the same user, but in diferent roles with regards to the information flow — and represents the behaviour of the application. A schematic over this information flow is presented in Figure 1b. A real system can be much more complicated and contain both more elements and branches. Because sub-systems and systems are modeled in the same manner we can imagine any number of levels of nesting where systems are in turn described using sub systems. This nesting is very useful for transparency-modeling, but we will focus on single level descriptions in this paper.
Since we are mapping information flow, it is a requirement that ⊆ , i.e., that the output does not contain information which is not in the input. A case of particular interest for AI decision making systems is training data. Suppose that the application in the previous example employs an AI model with training set . We then write
1 ⟼ 1, ( 1 ∪ ) 2 ⟼ 2 ⟼ making it explicit that is part of the input to the system .
Informally, a useful rule of thumb when building graphs like these is to associate each system
that has a functional description with a by-phrase explaining how the output is reached from
the input. For example, “ gives 2 by comparing 1 with .”. This describes functionally,
and if such a description is easy to give, then the relevant information is accounted for,
otherwise something is missing. By making it clear in this manner what information a decision is
made upon it is also made clearer which biases and problems might apply to such a decision.
Transparency thus builds the foundation of responsible AI [
There are many types of stakeholders for most systems, including decision makers, data subjects, and bystanders. An important class of stakeholders have already been identified, namely the participants of the decision making process — but the focus here is to identify those impacted by the system. This can be done using any appropriate stakeholder elicitation method [22]. Details of which are out of scope for this paper and use case dependent.
Impacted stakeholders fall into two categories. Firstly those who are impacted but in a way largely independent of particular decisions taken by the system - we call these independent subjects and they include for instance privacy invasion of the subjects of training data, efects on the general public or bystanders for the mere existence or widespread use of the system. These types of impact take an important role in answering the so called Question Zero; ‘Should AI be applied here?’ [19]. Secondly there are the stakeholders whose impacts are directly dependent on the decisions taken by the system and we call them dependent subjects, . These are the ones which the following framework will foremost focus on and which constitute the recipients of the systems outputs directly or indirectly. It is important to note that the outputs here refer to the outputs of the socio-technical system, not merely embedded technical systems.
All impacts — dependent or independent — apply to a group of subjects, and both outputs and impacts are distributed over subjects as well as over subject attributes. If all subjects are indistinguishable from each other on some attribute then any impact will be homogeneously distributed on that attribute by the system. This holds just as well for social and technical components of the system.
If the subjects are dissimilar however, they could be treated diferently. We will say that
subjects are discriminable when distribute unequally over on some attribute. If so, then the
output will also distribute unequally over the same attribute of . We can see this as for every
sensitive attribute of , will be subdivided into subgroups and any corresponding impacts will
be similarly grouped. This goes in both ways so in order for to discriminate over some subject
sub-groups then must also be discriminable over the same subgroups. Discriminability can be
both positive and negative as there is a potential for biased impact in both unfair discrimination
and lack of discrimination where such is needed [
Sensitive attributes of might be explicit like a gender attribute or indirect in the form of proxy attributes from which sensitive attributes can be derived. It is because of proxy-attributes and unequal representation not always be immediately clear which subgroups exist within a site. It is therefore beneficial to have extra control data — attributes which are part of datasets or measurements but are not necessarily used for the decision making or training data. Such attributes can be used to explore the behaviour of a system and help detect proxy attributes. An example of this is that the presence of a gender attribute in a dataset, which is not used for training or input, can be used to assess the existence of proxy attributes for gender in the data which is used. If no such gender attribute exist associated with the dataset, this examination might be impossible.
Having made an initial identification of the dependent subjects it is possible to consider impacts. Impacts are any normatively valued efects of the system, such as benefits or harms and can not occur without connection to a subject. There is always someone harmed or benefited for the impact to exist. Diferent groups of subjects of a decision making system can be impacted in diferent ways. Identifying the impacts in a particular case can be done using Impact Assessments [23].
Like with subjects, dependent impacts are derived from the systems output together with
the Confusion Matrix (CM) [
There are several kinds of measurements which can be made on a system which will show how bias propagate and explain efects on impacts.
This paper will focus on the distribution of the input subjects over discriminatory attributes and the confusion matrix with respect to the end result or partial results. These measures are relatively easily obtainable and as objective as the control data they are compared to, but taken by themselves it is unclear what they mean, if anything. By combining these measurements with their valuation of the impact functions they become contextualized. Valued results can be traced backwards to their source biases. Testing and measurements, as well as elicitation of stakeholders and bias needs to be repeated throughout the system life-cycle.
In order to obtain the confusion matrix of the system output, test data ideally containing the output of every subsystem as well as comparison data representing the correct output for each input is necessary. In some cases it might be dificult to obtain a correct answer, then some other attributes might fill that role by proxy — if so, conclusions must then be modified accordingly.
Given test data the CM describing the distribution of outcomes can be calculated. In case there are several outputs and impacted groups, a confusion matrix for each case needs to be calculated separately. By applying the impact functions to the outputs we can obtain a measure of the impact of the system on the specified dependent subjects. It is not suficient to test just the technical component in isolation to understand the outputs from the full socio-technical system. Pre-selection of subjects before the technical system as well as how technical outputs are transmitted into final outputs might make a lot of diference to the impacts, where efects of e.g., social biases, trust in algorithmic decisions (low or high) and steps to confirm conclusions might play significant parts. For this reason, test data should ideally be of the outputs of each subsystem for every input. With such data available it is possible to see how the final CM and impacts develop and originate. Some steps can well be seen as partial outputs in their own right if they have independent impacts.
Impacts are typically not best seen as a single quantity but rather as distinct classes of impacts, some of which might be more severe than others and might be best seen as risks of harm. Rather than looking at all impacts together it might be more informative to understand particular classes of harmful impacts, how they are caused and possibly prevented.
Dependent subjects are often not a homogeneous group and impacts might be very unequally distributed. For this reason it is not fully suficient with test data which tracks correct or incorrect decisions but also how discriminable attributes propagate. With control data in addition of the information actually passing through the system it is possible to determine how impacts distribute, and also to what degree proxy attributes contribute to make the input and output discriminable. For example, with control data with gender information about subjects it is possible to determine if impacts are equally distributed over genders — and also where in the information flow such bias appear.
In efect, a separate CM should be calculated for every discriminable attribute as well as
intersections of attributes. These sub-group CMs with associated impacts will describe how
impacts distribute over the subjects. Aggregation bias i.e. incorrect generalisation on an
individual level from group data could be detected by an uneven sub-group distribution [
Further common biases can be detected by comparing the distributions over sensitive attributes of training sets and other reference inputs (recall that training sets are treated as sub-system inputs). If static or reference information is not representative for the distrubution in the information it is compared to the result might be biased or plain incorrect. Sampling bias, population bias and representation bias among other problems can be detected this way and connected to impacts. The connection to impacts help determine if a possible problem is actually a problem in the particular use case of the system.
It would be possible to take this analysis much deeper to track how subsets within the information at each stage contribute to respective decision, but this analysis is outside the scope of this paper. It is worth mentioning however that there are interaction of this type which can not be easily analysed if looking at onlye a single sub-system, such as a singular AI model, but rather become available for analysis when the interaction of multiple steps and sets of information is considered. An example of this is when information of diferent levels of generalisation are separated or combined to produce a joint output (which risks aggregation bias, overgeneralization and stereotyping). On a similar vein additional analysis when it comes to system transparency and accountability can also be performed given that the system structure is charted and connected to impacts.
Here two illustrative example cases are presented. The cases are fictional, and while inspired by real situations they are devised to highlight issues which demonstrate the use of the Impactframework. Were they real, the example case would fall under European AI Act’s High Risk classification 1. 1https://artificialintelligenceact.com/title-iii/chapter-1/article-6/
This example case concerns a decision support for assisting doctors in potential cases of child abuse or neglect and is of interest because it an example of a case with low signal ratio and a high negative impact for false positives. When children coming to emergency services with suspicious bruises or burns they might be flagged by emergency technicans (EMTs) and will also be given a doctors examination. Into this situation a proposed decision making support system is added. The system will be given image data of injuries as well as demographic data for risk estimation, under the assumption that abuse is more common in some demographics.
A schematic for this case is given in Figure 2 where a) presents the information flow and b) is the impact function with confusion matrix.
( 1 ∪ ℎ ( ∪ 1 ∪ ℎ ⟼ 1, ) ⟼1 (
2 ) ⟼
∪ ℎ ⟼ Impacts • 0 The injured individual. • 1 The injured individual with potential EMT flag
• TS recommendation. • Decision to classify as malice or accident.
Channels, i.e sub-systems can be described as: • Medical personnel making an initial labeling. • 1 Doctor making the decision on to use the support system and what information to enter. • Decisions support system doing image analysis and risk analysis.
• 2 The final decisions are made by people but will be influenced by the systems output.
A particularity of this case is that while false negatives i.e failure to detect and actual case of neglect or abuse is terrible, so is false positives which might cause wrongful accusations, problems and pain for afected families and children. Another is that most cases are not abuse.
Given this impact estimation it is essential in this case to keep the false positive rate low, while still detecting the true positives. Using this lens we can examine the use of deographic information in the training data . The rationale for including this sort of information at all would be because abuse and neglect might be more common in certain demographics.
By considering the impact function, we can also see that if even in the highest risk demographic the abuse cases are not the majority it means any direct application of a demographic derived factor will lead to higher false positives in the high risk group and higher false negatives in low risk groups. It thus seems, just from the impact distribution, that using demographic information in a mixed manner with the image analysis will always have detrimental impacts regardless of how well the system otherwise performs.
We can also observe from the structure in Figure 5.1 a) that it will be dificult for the doctor to understand when is based upon or ℎ if the graph represents the structure of the real system. This is a transparency problem and could be solved by separating the two modalities into two systems, allowing to make an informed choice between injury assessment and risk assessment. Beyond this structure analysis, real measurements can be made of at any sub-system. With such measurements the real ratios between outcomes can be established. Combined with the system structure it can be determined how real impacts are caused by the actual system and it’s parts.
It is also important to point out that addressing the impacts could also be approached by adding another social step after the outputs specifically to carefully handle the risk of false positives. If such eforts would reduce the negative impact of false-positives significantly, that would in turn potentially allow for more sensitivity in TS (more TP and FP).
In this example case a reinforcement learning (RL) filter is proposed to be added to a hiring system. Prior to this addition the hiring system is constructed as a filtering pipeline, narrowing down the list of applicants in several steps until the final candidates are presented to the employer and selected for interviews. The first steps are performed using rule based filters with the express aim to not treat protected groups or attributes diferently.
After passing through this filtering the potential employers make a shortlist from applications and finally call candidates for an interview.
The proposed RL system would use scores from the interview and shortlist steps to perform a better filtering on applicants before they are presented to the employers. Figure 3 gives an overview of the structure of both the original and the proposed system. The Original system uses a pre-filtering step, a manual shortlisting step and an interview. The Proposed system adds another filtering step based upon the shortlist and interview steps.
⟼ , ⟼
(
Sites from 0, 1, 2, 3, 4, and their corresponding information can be described by these elements: • •
• A training set based on interview scores.
Channels, i.e sub-systems can be described as: • Pre Pre-screen applications by comparison between • RL RL-screen applications by comparison between • Shortlist Employer makes shortlist for interview based on . • Interview Employer makes selection based on interview ( ). and . and the job posting.
The main impacted stakeholder groups in this system are the applicants and the employers. This case is a good example of where there is no simple true answer if a job applicant fits for a job or not. A proxy ‘truth’ which can be used however is if the candidate was hired.
Using this we can construct a confusion matrix with eight fields representing where in pipeline the candidate was rejected. This test data can be obtained by letting a number of random candidates pass through the whole system an noting if they would be hired or rejected. From the employers perspective, it is a bad outcome if a candidate who would have been hired is eliminated early and also a bad outcome if a candidate who would not have been hired is passed to interview or shortlist, because this increases the employers workload and potentially obscures other more suitable candidates. From the applicants perspective, getting eliminated early represents a loss of opportunities, contacts and feedback even if they were not hired. If there is no chance of being hired, getting eliminated early might save time for the applicant, but they are also shut out from direct interaction.
The applicants are not a homogeneous group however. We can assume there are favored and disfavored sub-groups. Such favored groups could disproportionally be chosen over the disfavored group at the interview step and/or the application step. There might also be groups calling for their own impact function — a severe risk which become apparent with a broader perspective is that some groups might be efectively shut out from entering the job market at all and a badly constructed hiring system might exacerbate that problem. This can be illustrated with high impact for early and systematic rejection, especially on the RL step.
We can see that if Pre makes a pre-sampling without considering protected attributes then the subjects after this step will be homogenized on non-protected attributes. If so, then the selection by the employer in the shortlist and interview will to a large degree be about the protected attributes (since these are now what mostly set the applicants apart). The RL feedback-loop then amplifies this section by efectively applying the same bias twice. Taken together this means there is high risk for very unequal treatment between groups and especially so because of the combination of the RL step with the prior Pre filtering step.
These steps combined mean that the reinforcement learning must operate on the subset of attributes which are not made similar by the initial selection but are reflecting some part of the employers judgement and can be found in the application information directly or via proxy.
This situation can be confirmed (or rejected) as influencing by using control data which allows tracking of protected attributes. It can then be determined if there is an unequal distribution at each step. Such data can also confirm biases which occur due to pre-selection e.g who need to use a system like this and who are allowed to use it?
While the above analysis focused on the RL part of the system, the so called fair filtering could also act as a gatekeeper if the criteria e.g particular degrees, are not equally obtainable. This situation can be analyzed with measures in a similar way.
In this paper, context has been described in terms of information flow and impacts. With a general way to describe context, it is made clear which specific features are problematic and what can be addressed to devise a solution. The distribution of impacts over the technical system could also be used to guide choices of the fairness definitions to apply if that approach is desired.
Distribution and aggregation problems on the level of datasets which are commonly discussed roots of algorithmic biases are also put into context by relating them to their role in the sociotechnical system and the flow of information.
The framework is to some sense primarily addressing the transparency of algorithmic fairness by contextualizing where biases occur and how they connect to impacts. Just like transparency is essential for well-motivated trust in algorithmic systems, it is also essential for understanding a systems role in contributing or alleviating social problems. It is not possible for the users or the afected subjects of an algorithmic decision to apply their own faculties of judgement unless the process behind the decision is suficiently transparent. One way of achieving such transparency is subdivision into meaningful and meaningfully connected parts [24, 25].
The connection to impacts also allow clearer connections from fairness to risk and safety. Taken in isolation biases and their distributions tend towards the balancing of unavoidable abstract quantities. Looking at impacts however opens other avenues. It might be possible to prevent some harmful impacts entirely just by adding or changing steps, such as extra validation of conclusions with the risk of harmful mistakes. While some issues of fairness really are about balancing resources, some can be meaningfully seen from a safety perspective where risks of mistreatment or wrongful conclusions are contained and removed as much as possible rather than evenly distributed.
The presented framework provides means for modeling the information flow of a socio-technical decision system such that technical measures and stakeholder impacts can be connected. This initial work however leaves many directions unexplored.
As mentioned above, deeper analysis of the connection between sub-sets within the information flow and datasets will further the detection of biases and their contribution to the impacts. Related to this is also impact driven proxy attribute detection which emphasize that discriminability of attributes can take on both positive and negative roles based upon the composition with other social and technical systems — a dimension which is dificult to capture without an overarching connecting structure.
On the socio-technical side a closer integration with existing initiatives in stakeholder elicitation and impact assessments would be valuable. It might be possible to derive input functions and information flow together and in tandem with procedures for impact assessment and other assessment and evaluation tools. Similarly valuable would be a closer association with frameworks for algorithmic accountability, contestability and risk-analysis. Further validation work is of critical importance.
The work has been supported by the AEQUITAS project funded by the European Union’s Horizon Europe Programme (Grant Agreement No. 101070363). [18] A. Calvi, D. Kotzinos, Enhancing ai fairness through impact assessment in the european union: a legal and computer science perspective, in: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 2023, pp. 1229–1245. [19] L. Munn, The uselessness of ai ethics, AI Ethics (2022). [20] J. Barwise, J. Seligman, et al., Information flow: the logic of distributed systems, Cambridge
University Press, 1997. [21] J. Barwise, D. Gabbay, C. Hartonas, On the logic of information flow, Logic Journal of
IGPL 3 (1995) 7–49. [22] G. I. Pacheco C., A systematic literature review of stakeholder identification methods in requirements elicitation, J. Syst. Softw. 85 (2012) 2171–2181. [23] H. D, Impact assessment methodologies for microfinance: theory, experience and better practice 28 (2000) 79–98. [24] Z. C. Lipton, The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery., Queue 16 (2018) 31–57. [25] A. Páez, The pragmatic turn in eXplainable Artificial Intelligence (XAI), Minds and Machines 29 (2019) 441–459.