Fairness and transparency in artificial intelligence (AI) continue to become more prevalent as topics for research, design and development. General principles and guidelines for designing ethical and responsible AI systems have been proposed, yet there is a lack of design methods for these kinds of systems. In this paper, we present CoFAIR, a novel method to design user interfaces for exploring fairness, consisting of series of co-design workshops, and wider evaluation. This method can be readily applied in practice by researchers, designers and developers to create responsible and ethical AI systems.
in practice, it is up to the designer or
developer to craft appropriate ways to implement
It has been realised that Artificial intelligence this guideline.
and machine learning pose unique design chal- Google’s Responsible AI practices [
Microsoft’s Guidelines for Human-AI In- present a rationale of why these aspects
reteractions [
In addition, there is a emerging body of re- is transparency design [5]. This work prosearch that aims to investigate what is most poses a stage-based process to first investiefective in terms of user interfaces that pro- gate mental models of experts and then users vide explanations. A lot of work has focused to establish a target mental model of what on what information should be available to needs to be explained, before iteratively prousers and how this information should be com- totyping the user interface to establish how municated via text, graphics or visualizations to communicate the explanations and then [18, 19, 20, 21, 14]. A recent efort to start de- evaluating it. To develop the mental modveloping design patterns [4], backed by cog- els of experts, interviews and workshops are nitive psychology, has suggested links (or pat- suggested, while to investigate users’ menterns) of how people should reason, how peo- tal models it is suggested to employ surveys, ple actually reason, and how to generate ex- interviews, task-based studies and drawing planations that support reasoning. tasks. For developing the target model, card sorting, interviews and focus groups were pro2.3. Design Methods posed. Designing and evaluating the user interfaces can involve focus groups, workshops, There is only scarce considerations of design and think-aloud studies. There are now sevmethods for telling designers and developers eral case studies that have used this process how to design ethical and responsible AI sys- to successfully implement explanations in AI interfaces [23, 24, 25].
Our work is concerned with investigating
3. The CoFAIR method
design methods for user interfaces that can 3.1. Co-design workshops
help with making the fairness of AI algorithms
transparent, and then help with mitigating To start, CoFAIR comprises a series of
workfairness issues by incorporating user feedback shops to work closely with a limited number
back into the algorithm. of participants to research the topic area, to
develop some designs, and then to test those
designs. To set up these workshops, a
number of considerations will need to be made:
We present here our method to Co-design Fair 3.1.1. Recruitment
AI InteRactions, CoFAIR (Fig. 1). This method
is based on a co-design process [26] which Participants in a workshop should be the
taraims to work closely with users to develop geted users of an AI system. The aim is to
solutions through a participatory design ap- closely involve these participants in
designproach. As other co-design approaches, It is ing a solution that is right for them, and to
characterised by very close involvement of align the design with their requirements. If
a small number of users in all stages of de- there are a number of diferent user groups
signing a solution, in which these users are that are distinct in their background, use cases
empowered to be on equal footing with re- or tasks, then separate workshops should be
searcher and designers. Co-design has been organised for them. The users will not need
successfully adopted to design human-centred to have a detailed technical understanding or
technology in other settings [
Our proposed method for responsible AI should be relatively representative of the user includes a series of co-design workshops with group in terms of background and demographparticipants/co-designers that focus on user ics. For each workshop, the number of particresearch, conceptual and detailed design, and ipants should be kept low, between 3-6 peoinitial testing, which is then broadened in a ple, so as to encourage interaction between ifnal user evaluation stage. participants. 3.1.2. Workshop Aims and Structure involve real or hypothetical scenarios and the user experiences around the topic. Activities We suggest that the workshops aim to cover would typically explore problematic aspects three main steps in user-centred design: user and the challenges that users face in carrying research, conceptual and detailed design, and out a user task. They would also probe for testing. User research in these workshops basic understandings and conceptualisations should investigate the users’ current concep- around the topic of investigation. These can tualisations and experience within the topic be (but don’t have to be) documented in perarea, pain points, and high-level needs and sonas, and recent work has shown how these wants. This user research can be formalised personas can be co-created with co-design parand communicated through co-created per- ticipants [27, 29]. sonas that reflect the target user group [27, Activities that aim to support design are 29] or could be more informal as simple lists also kept very concrete. Typically this would of requirements. Conceptual and detailed de- investigate a scenario of use, either real sign will involve the participants in surfacing or fictitious. As part of conceptual design, what information and interactions are needed participants would usually be invited to go to achieve their tasks while also clarifying through the scenario of use and indicate what how to present this in the user interface. This they would look for, what interactions they might be documented in storyboards, user would expect, and what information the sysjourneys, and sketches, or produce scenario- tem would need to communicate. It is somebased object-action analyses. Last, these de- times helpful to develop storyboards or user sign should be prototyped, either using low- journeys with co-design participants. Detailifdelity paper prototypes or more high-fidelity ed design can flesh out design options through clickable wireframes, and then tested with par- sketches, however, this needs to be carefully ticipants. supported and scafolded as participants are
Depending on the complexity of what is often too timid to sketch themselves. to be designed, these steps need to be spread In testing, a prototype, often created or reover a series of workshops. Most naturally, ifned by a designer/developer ofline, is exthese steps suggest three sequential work- posed to evaluation by co-design participants. shops, each with a distinct focus on user re- Again, a real or fictitious scenario is used to search, design, and testing. It might be possi- explore how the prototype might be used and ble to combine user research and design, and what improvements are necessary for a subthus reduce the number of workshops to two. sequent iteration.
However, more iterations might be needed to explore design options and iteration of prototypes, and thus more workshops might need 3.2. Broader Evaluation to be scheduled. Our method is flexible enough to accommodate this. 3.1.3. Workshop Activities To achieve the aims of the workshops, codesign usually proceeds with group-based, hands-on activities and discussion around these activities. For user research, these could A common criticism of co-design is the limited number of participants that are involved in developing a solution. This leads to the fear that while the solution is optimally adapted to the 3-6 participants in the workshops, it is unsuitable for the wider user population.
Our method suggests that co-design is always followed by broader evaluation of the designed system through evaluations with users.
This can take various forms, such as think- in Japan through social media and personal aloud user testing or large-scale crowd-sourc- contacts. All participants’ ethnicities broadly ed system use. reflected the population of the country, and most participants had been educated to a Bachelor degree level. We paid an incentives of £40, or equivalent in the local currency. 4.1.2. Workshop procedure For each country, we held 2 co-design workIn order to show how our method can be in- shops; these two workshops were 3 weeks stantiated in practice, we present a case study apart. Both workshops lasted 2 hours. in which we investigated how to develop user In workshop 1, we conducted user research interfaces that allow users to explore the fair- and conceptual design. For the user research ness of AI loan application decisions. Loan part, we investigated how participants definapplications decisions are increasingly being ed fairness, and then how they explored fairautomated or supported using AI models (typ- ness in loan decisions. For investigating how ically, employing logistic regression). This participants viewed AI fairness, we first got study targeted three diferent user groups in participants to tell us about their own experitwo iterations: non-expert members of the ences of fair or unfair decisions that afected public (iteration 1), loan oficers and data sci- them, especially if they encountered AI in that entists (iteration 2). Iteration 1 details how decision-making. We then also probed them we instantiated the method with non-expert to consider fairness of using AI systems in customers, while iteration 2 is concerned with hiring or making medical decisions and what the method used with loan oficers and data makes AI systems fair or unfair. scientists. We focus on the techniques em- To continue user research and start on conployed in our method; we will report on the ceptual design, we constructed an activity inifndings of these studies elsewhere. volving four fictitious loan application scenarios (Fig. 2). This allowed us to further in4.1. Iteration 1: Non-expert vestigate what attributes and information they Members of the Public were looking for to assess the fairness of the applications’ outcomes and potentially what We ran a series of co-design workshops with they would change to make the decisions fairer. a total of 12 participants in the USA, UK, and Each scenario was discussed in turn, whether Japan. Because of COVID-19 restrictions we it was fair, why (based on the information inhad to change our planned face-to-face work- cluded in the application or their experience shops to be conducted entirely online. of the decisions they had seen), and what information would have been useful for them 4.1.1. Co-Design participants to assess fairness better. We changed some of the application scenario details to localize We recruited 3 participants (2 women, 1 man, them to each country (e.g. names, currency, mean age 47.3) for the co-design workshops dates) but otherwise kept them the same. We held in the USA, 5 participants (3 women, 2 showed participants information that is usumen, mean age 34.2) in the UK, and 4 par- ally collected as part of a loan application proticipants (3 women, 1 man, mean age 33.75) cess, based on the application form of a wellknown international bank. Application 1 formation that they used for fairness assess(USA/UK: Mark Benson or Kazufumi Taka- ments and requests for further information hashi) was always approved, as it was a ’safe’ obtained in workshop 1 to interface design application, with a homeowner with a very elements, and we did not involve participants good credit score applying for a small loan to in detailed design activities. buy a used car. Application 2 (USA/UK: Sadia In workshop 2, we moved on to a testing Mohammed or Chihe Pak) was rejected, as it activity. We structured our discussion on the was a more ‘risky’ application with low in- clickable wireframes, and developed some scecome, part-time job and low credit score. We narios to explore fairness using the clickable also included her application to investigate prototype. Going through each screen’s funcany potential minority or age biases. Appli- tionality, we discussed what helped to undercation 3 (USA/UK: Jennifer Clary or Maika stand if the application decisions were fair, Suzuki) was also rejected but crucially her what additional information would they like details were very similar to Mark Benson. This to determine fairness, and what feedback they was to introduce an application that seemed, would like to give to mitigate fairness. without any further information, to be blatantly unfair. Finally, application 4 (USA/UK: 4.1.3. Broader evaluation Kwame Odejima or D u˜ng Nguyên,) was accepted although it seemed more ‘risky’. Following the co-design workshops, we im
After the workshop, two researchers re- plemented an improved interface. We then viewed the workshop recordings and anal- set up an online study to investigate how this ysed the participants’ definitions of AI fair- prototype is employed by end-users to assess ness and how they thought AI could be made the fairness of an AI system, and how sugfairer. For each scenario, we analysed what gested changes to the model afect fairness. criteria they used to assess fairness, how they We recruited 388 participants (129 female, were using information to explore fairness, 256 male, 2 Other and 1 preferred not to say) and what other information they wanted to through Prolific 2, an online research platform, be able to assess whether a loan application and paid them £3.50 for an expected 30-minute decision was fair, or potentially biased. Based session. About half of of our participants had on this analysis, we constructed clickable wire- some programming experience and familiarframes to instantiate their input in an inter- ity with AI, machine learning or statistics, and face. We did this by carefully mapping in- 2https://www.prolific.co/ 146 participants had at least a Bachelor de- [30]. gree. On study completion, we analysed the in
We asked participants to interact with the teractions with the prototype to evaluate interface to a assess the fairness of an AI sys- whether this prototype was efective in suptem. Instead of using an open-source dataset, porting users in exploring the fairness of an the AI system we developed was based on AI model. an anonymized loan decisions dataset we obtained from Intesa Sanpaolo. This dataset con- 4.2. Iteration 2: Loan Oficer and tains decisions made on 1000 loan applica- Data Scientists tions and has 35 attributes including the label of whether the loan application was accepted 4.2.1. Co-design Participants or rejected. These attributes include demographic information of the applicant (age, gen- This iteration was focused on exploring how der, nationality, etc), financial information to support loan oficers and data scientists to (household income, insurance, etc), loan in- explore the fairness of loan application deformation (amount of loan requested, purpose cisions. These two stakeholder groups are of loan, loan duration, monthly payments, etc), diferent: loan oficers typically act as interas well as some information of their financial mediaries between the bank and customers and banking history (years of service with and had practical experience of loan decision the bank, etc). There were also some attributes making, while data scientists have experience that related to internal bank procedures, such in modelling and supporting and/or investias a money laundering check and a credit score gating customer application decisions. For developed by the bank. We developed a lo- this study, we recruited six loan oficers (5 gistic regression model after removing sparse men, 1 woman, mean age 36.5) and six data values, or where multiple attributes had simi- scientists (3 men, 3 women, mean age 29.7) lar values; the accuracy of the resulting model through Intesa Sanpaolo. was 0.618. Note that the model was unfair with respect to the nationality attribute: ’for- 4.2.2. Workshop Procedure eign’ applicants tended to be rejected more Due to Covid-19 and logistical limitations, all frequently than citizens, using disparate im- interactions with the users were conducted pact as a fairness metric. online. We structured the activities into two
The evaluation consisted of a brief pre-ques- workshops, each lasting 2 hours. Both worktionnaire and tutorial, 20 minutes of free use shops were repeated for each separate stakeof the interface to assess fairness, and a post- holder group. questionnaire. To evaluate the use of this pro- As with the previous iteration, the aim of totype we captured participants’ ratings of workshop 1 was to conduct user research into the AI fairness and key interactions with the how fairness was perceived by these user user interface where logged. We also asked groups, and to carry out initial conceptual dethem to describe in their own words what sign. Workshop 1 started of by discussing strategies they used to assess the fairness of the aspects that make decisions in loan apthe system, any systematic fairness issues they plications fair or unfair to get an insight on had spotted, and their views on suggesting participants’ loan application experience and changes and addressing fairness. We then fin- unfair scenarios that they may have come up ished the study by asking them to rate their against. This was followed by how AI could task load using the NASA-TLX questionnaire impact loan application decision-making and loan amount and number of instalments in fairness. detail.
To further our user research and also un- The aim of workshop 2 was to informally derstand what key information is important test the clickable wireframe. This wireframe to use in conceptual design, we then intro- was screen shared and the researcher ’drove’ duced an activity to explore the anonymized the interactions with it and acted as an loan decisions dataset we obtained from In- extension on the participants’ behalf, clicktesa Sanpaolo. The dataset was sent ahead ing through it. The researcher then stepped of the workshop so that participants could through it with the respective user groups, have time to look at it and have it available on and probed whether they understood how it their computers during the session. The dis- worked, wheth-er the information was usecussion elicited information on participants’ ful for exploring fairness, or what could be process, information needs, and the function- improved. ality required to develop an interface. To help Analysis of the second workshop investiparticipants investigate the dataset, a data vi- gated changes that needed to be made to imsualisation tool was created which was used prove the clickable prototype for broader evalto present the dataset should participants re- uation. Based on this analysis the researchers quire it. It provided the ability to slice the fea- designed a prototype (Fig. 4). tures on the fly and present them using various chart types such as histograms, scatter 4.2.3. Broader Evaluation plots, bar graphs and a strip plot.
Next, we introduced an activity to reflect The evaluations were conducted as one-toon a causal graph, showing causal relation- one user tests, unlike the workshops in the ships between the dataset attributes. This previous phase. A total of 17 participants were causal graph was derived through automatic recruited through Intesa Sanpaolo: 8 loan ofdiscovery, showing how attribute values and ifcers (5 men, 3 women, mean age 38) and 9 the loan application decisions are related to data scientists (5 men, 4 women, mean age each other. Through this activity we aimed to 31.8). All participants held a master’s degree understand how these users might interpret or higher. the causal graph and how this might be em- We developed ten tasks for participants to ployed in exploring the dataset for fairness. go through the prototype, from setting up the
After the first workshops, a researcher anal- dataset to explore to investigating the dataset, ysed the audio recordings to derive findings using diferent components of the user interabout how these user groups judged whether face. The study concluded with a post-quesloan applications were fair, how these users tionnaire used to evaluate users’ experience. explored the dataset to determine fairness, and This questionnaire comprised ratings aimed how they interpreted the causal graph. Based at quantifying how efective the prototype was on this analysis the researcher developed a in supporting users in assessing fairness inclickable wireframe to be used in workshop cluding information, functionality and rea2 (Fig. 3). Again, we did not involve the users soning, free comments to express their feedin detailed design. Due to implementation back about the prorotype, and the NASA TLX constraints, we only made a selection of the questionnaire [30]. wireframe interactive, and focused on a sce- The broader evaluation was analysed as to nario in which to explore the relationships what worked well and what did not, in order between citizenship, gender, credit risk level, to develop functioning interfaces in future.
design activities and compressed them into two workshops of two hours each. Ideally We have gained some experience from apply- we would like to extend them to span three ing co-design in other application domains, workshops and for a longer duration. Secand through a case study where we imple- ond, facilitation of online discussions is very mented the CoFAIR method to develop inter- dificult, and ideally we would have brought faces for exploring fairness. This showed that users together to discuss this more freely facethis method can be successfully employed to to-face. Last, we would have liked to involve design interfaces for responsible AI systems. users much more in conceptual and detailed However, we encourage other researchers and design, for example, through sketching or papractitioners to adopt this method and gen- per prototyping but this is very dificult to do erate more data points to improve this ap- virtually. proach, and also to validate it. In addition, We can also note some general limitations CoFAIR was so far employed under COVID- of the CoFAIR method which should be con19 restrictions which meant that all workshop sidered before it is chosen as a design approach. activities and testing had to be conducted re- First, as with all co-design there is a danger motely online, which impacted what we were that interfaces are developed that only fit the able to do. If we had not been placed in this small number of people that were involved situation, we would have made diferent choic- as users in the workshops. This can be allevies as how to conduct the workshops. First, ated through conducting broader evaluations due to the online nature we shortened the co- that ensure that the designs are fit for purpose. Second, it is not a ’discount’ methodol- of working. How to successfully mitigate fairogy that is fast and easy to apply. Implement- ness issues, especially through a human-ining it requires several lengthy workshops with the-loop approach, is still an open research users to be organised, separated in time so question. that researchers and designers can analyse We believe that our method is another step and produce new materials in subsequent ac- to strengthen the design of responsible and tivities. This means that even relatively small ethical AI. A major advantage of CoFAIR is projects can spread over several months, from that it produces designs and interfaces that initial recruitment of users to a fully refined focus heavily on what specific target users and evaluated interface. Because we want to need and want. It thus produces ’shrink-wrapguard against ’overfitting’ designs to small ped’ interfaces that should be eminently suitnumbers of participants, it is not advisable to able to communicate with a specific user group. cut short this process and skip the broader Taken together, this method could be easily evaluation to save on time. Last, this method extended to investigate what and how to exfocuses very much on the mental model of plain machine learning systems, in order to users and does not account for the input of design more responsible and ethical AI sys’experts’ or consider how people should rea- tems. son. Hence, it is possible that we might build in possible biases that users have back into these interfaces, and only support current ways
Personas: Engaging and Empowering Users with Diverse Needs Within the Design Process, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, ACM, New York, NY, USA, 2019, pp. 650:1–650:12. URL: http://doi.acm.org/ 10.1145/3290605.3300880. doi:10.1145/ 3290605.3300880, event-place: Glasgow, Scotland Uk. [30] S. G. Hart, L. E. Staveland, Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, in: Peter A. Hancock and Najmedin Meshkati (Ed.), Advances in Psychology, volume Volume 52 of Human Mental Workload, NorthHolland, 1988, pp. 139–183. URL: http://www.sciencedirect.com/science/ article/pii/S0166411508623869.