Understanding artificial intelligence (AI) and machine learning (ML) approaches is becoming increasingly important for people with a wide range of professional backgrounds. However, it is unclear how ML concepts can be effectively explained as part of human-centred and multidisciplinary design processes. We provide a qualitative account of how AI researchers explained and non-experts perceived ML concepts as part of a co-design project that aimed to inform the design of ML applications for diabetes self-care. We identify benefits and challenges of explaining ML concepts with analogical narratives, information visualisations, and publicly available videos. Co-design participants reported not only gaining an improved understanding of ML concepts but also highlighted challenges of understanding ML explanations, including misalignments between scientific models and their lived self-care experiences and individual information needs. We frame our findings through the lens of Stars and Griesemer's concept of boundary objects to discuss how the presentation of user-centred ML explanations could strike a balance between being plastic and robust enough to support design objectives and people's individual information needs.
Understanding artificial intelligence (AI)
approaches is becoming increasingly important
for industry practitioners with a wide range of
professional backgrounds and academic
researchers working in interdisciplinary fields,
such as human-computer interaction (HCI).
While HCI and AI research have often been
characterised as having quite distinct views of
the relationship between humans and
technology [
Pointing out that the manual work and
human factors of ML research can be
overlooked, Gillies et al. [10] and Clarke et al.
[5] encourage researchers to draw on
humancentred approaches to investigate the situated
and collaborative facets of ML practices and the
design of usable ML support tools. Taking up
this call, Muller et al. [
As ML plays an increasingly important role
in the design of products, not only data
scientists but also designers engage with ML
[
Moving on from how data scientists and
designers work with AI concepts and tools,
prior work has drawn on participatory
approaches to investigate end-users’
perceptions and the ethical implications of AI
systems [
The objective of this study was to investigate how ML explanations were presented and perceived as part of a co-design project that aimed to co-design ML-based decision support concepts and co-create suitable machine learning approaches. The project involved HCI researchers, AI researchers, and industry practitioners, as well as fifteen participants with T1 diabetes. This paper focuses on one workshop that specifically mediated ML concepts to workshop participants. We did not aim to evaluate the effectiveness or efficiency of the ML explanations. Instead, we investigated the following research questions: • • •
How did AI researchers explain ML concepts to co-design workshop participants? How did co-design workshop participants perceive the presented ML explanations? What are the transferable implications for designing user-centred ML explanations? The first author conducted 18 interviews via phone and video conference systems. Interviews involved eight people with T1D who participated in the co-design project (referred to as P1, P2, etc.), three HCI researchers (e.g. HCI1), and three AI researchers (e.g. AI1). To support recollection before the interviews, a slide deck was shared with participants including ML explanations used throughout the workshop. Interview topics covered prior experiences with AI/ML and perceptions of ML explanations. Interview questions were adjusted for each group of interviewees and lasted approximately 30 minutes. The audio recordings were transcribed verbatim. This interview study received an ethical approval from the Faculty Ethics Committee.
Data collection and analysis was conducted in a staggered way according to project roles. A qualitative data analysis software was used by the first author to thematically code data [3]. As some participants were authors, each interviewee was sent the representative quotes for the codes and explicitly agreed to their use before group analysis was conducted. The data corpus was iteratively analysed in an inductive fashion drawing on open coding by all the authors [3].
We first report on how AI researchers explained ML concepts to participants as part of a co-design workshop using different types of explanations, including analogical narratives, data visualisations, and publicly available videos. We then describe how workshop participants, including HCI researchers and people with diabetes, perceived the presented ML explanations and what benefits and challenges they experienced. 3.1.
Since the objective of the co-design project involved the design of ML based applications for diabetes self-management, AI researchers used different methods to explain ML approaches to workshop participants, including data visualisations, analogies and videos of real-world AI applications.
The concept of anomaly detection was explained with the help of two line graphs (see Figure 1). The first line graph showed continuous blood glucose measures over time in milligrams per decilitre. Representing a binary machine interpretation, the second line graph highlighted four anomalies in the continuous blood glucose data of the first line graph. Participants reported being used to reflect on line graphs when using different health and wellbeing applications [14]. However, they wished to hear narratives that described the real-word context and experiences of the person who collected the data to be able to relate and make sense of the anomaly explanation. For example, P8 made it clear that it is important not only to understand the contributing factors of anomalies but also how anomalies could be managed: “What you’re not really seeing is why those anomalies are happening. […] if we’re talking about diabetes, I think the ‘why’ is just as important in order to understand how to tackle those anomalies.” (P8)
Moreover, participants highlighted that binary representations of anomalies (see Figure 1, second line graph) may be useful to explain the concept of anomaly detection, however, potentially not suitable to support sense-making and decision-making in everyday life. They felt more comfortable with data visualisations that supported their agency in identifying and dismissing anomalies based on their lived experience. For example, high blood glucose values in daily life were not necessarily anomalous if participants were able to make educated guesses about contributing contextual factors and manage these situations.
Another ML concept that was explained as part of the co-design workshops was reinforcement learning. AI researchers mediated the concept of reinforcement learning with the help of the analogy of training a dog. “At first, it was a bit like, ‘What!?’ and then, when it was explained, it was like, ‘Oh, yes, that makes sense,’” P5 remembered, indicating that understanding this analogy requires translating the act of training a dog to the act of training a software agent that aims to maximise reward in a given environment. Participants reused the analogy of training a dog in different contexts, such as P8 who wished to be able to use a semiautomated self-tracking approach [4] that empowers people to manually stop false machine interpretations: “So, you could use the dog example again, where it might be learning something which necessarily isn’t correct, if that makes sense, like it might find a pattern which you don’t want it to learn. So, I think... I don’t think it’s a question of like manually versus automatic. I think they need to work together in some shape or form. […] there needs to be some sort of manual input to tell the machine learning aspect, ‘Please don’t learn this.’” (P8)
Participants also perceived limitations of using the analogy of training a dog with cookies. For example, P3’s account refers to the challenges of transferring anticipated emotions, such as the desire to learn, to machines and the challenges of translating the analogy to the design space of digital applications: “If a machine has desire or it’s how you explain the one for a cookie. I think that’s the bit I find it hard to get my head round with a machine […] So, I don’t know how you reward an app like a machine” (P3).
Furthermore, P10 politely critiqued the use of the term ‘cookie’ in the context of diabetes management, considering that cookies can be associated with dietary challenges people with diabetes can experience: “I’ve got dogs and I give them treats, little dog treats. I think the use of the word cookie I found amusing shall we say. Because cookies are not a reward for us diabetics. In fact, that’s a challenge.” (P10)
In addition, researchers used a seminal video [8], that is widely cited in the machine learning community, to demonstrate how agents learn to play the game of hide-and-seek. The video showed how agents developed strategies and counterstrategies over time, such as jumping on cubes and moving cubes to block doors. All participants described the video as a wellproduced, powerful and memorable exemplar that mediated machine learning driven multiagent behaviour with advanced character design and an entertaining narrative: “The way the video showed how they sort of developed and how they learned was really clear, and the characters are quite cute, so I think it was quite funny as well, at the same time. Again, that was a great example to show how machine learning can work.” (P5).
However, similar to the analogy of training a dog, participants found it challenging to transfer the hide-and-seek game to their diabetes self-management practices, highlighting that machine learning explanations need not only be abstracted but also transferred to a personally meaningful and researchspecific context: “I’m not sure how to transfer that to a diabetic situation in a way, that particular format. I mean there must be one, I haven’t really thought that one through. […] what have you got to have? You’ve got to have something whereby you’re correlating eating or carb intake, exercise and taking insulin. So, those three factors, I think.” (P6)
HCI researchers and workshop participants reported gaining an improved understanding of the presented ML approaches. Participants explained that even though they might not fully understand the “inner workings” (P3) of ML approaches, it was important to gain some knowledge of ML concepts to develop trust in the design process and potential ML implementations, though some noted the importance of it being presented in understandable terms: “I don’t think you just blindly follow stuff, particularly when designs are being made in the background […] it’s better to put it into terms that we could understand, which is quite difficult when it can be so complex, but I do think it’s quite important to give us some understanding of how and what’s going on in the background.” (P5)
HCI researchers and participants reported that learning about ML approaches as part of the workshops changed their prior understanding of the benefits and limitations of ML based technologies. “Before, it was kind of like, you know, computers being able to think for themselves or like have a sentience,” P8 explained, exemplifying that some participants’ prior understanding of AI was based on science fiction narratives that typically portray AI technologies with potentially dangerous autonomous and emotional capacities. Reflecting on their co-design workshop experiences, participants demonstrated differing degrees of ML literacy in creative ways. For example, participants used existing digital consumer services as examples to explain ML functionality, such as recommendations: “I think the term ‘artificial intelligence’ is a bit more specific than that, I think. It’s more to do with machine learning, […] So it’s things like, you know, how Netflix decides what you watch, kind of thing, or how you choose your recommendation. I think it’s algorithms, really.” (P3)
Participants described AI research and AI concepts, such as ML, as data driven algorithms that are written by humans and run on computers. “AI is computers that learn, that once you set certain criteria up or whatever, they can gain knowledge themselves without being told to gain knowledge, yeah. I think that is the simplest form,” explained P10, referencing the learning capabilities of AI technologies. Participants with diabetes also reflected on potential limitations of ML approaches, including differences between manual and automatic data collection, roles of data quality and potential limitations of predictive functionalities: “If it’s showing information based on weeks and weeks of data-gathering and it’s basically giving you your average day, I mean, I suppose that could be useful. But then, if you suddenly change your physical activity, or you’re eating something at a time that you don’t usually eat something, then I guess that could disrupt it.” (P8).
Understanding AI approaches is becoming increasingly important for people with a wide range of professional backgrounds in industrial and academic settings. We have provided a qualitative account of how AI researchers explained ML concepts to HCI researchers and people with diabetes as part of a co-design project that aimed to inform the design of ML applications for diabetes self-care. Here we discuss our findings through the lens of Stars and Griesemer’s concept of boundary objects to outline how the presentation of user-centred ML explanations could strike a balance between being plastic and robust enough to support design objectives and people’s individual information needs as part of multidisciplinary projects.
Star and Griesemer’s [
In their study of how amateurs, professionals, and administrators collaborate in a museum setting, Star and Griesemer distinguish between four types of boundary objects: (1) repositories provide a central location where objects, such as samples, are systematically stored and are available for people to be used; (2) ideal type is an object, such as a diagram, that provides an abstracted representation that can be adapted by others; (3) coincident boundaries are objects, such as tailored maps: they are defined by common (geographical) boundaries but can have different contents, purposes, and styles; (4) standardised forms are boundary objects that are used as formal methods of communication across different actors. While these four types of boundary objects can be used in different ways and can have different meanings for different actors from different social worlds, they typically support communication and facilitate collaborations. Although boundary objects aim to resolve conflicts, they are not neutral. The creation of boundary objects requires carefully managing power relationships to avoid forced use of predefined representations that can cause systematic exclusion, discrimination, and injustice.
In our case, AI researchers used different types of ML explanations to support HCI researchers and people with diabetes in codesigning possible ML systems. To foster a shared understanding of ML concepts, they used analogical narratives to explain reinforcement learning, data visualisations to explain anomaly detection, and publicly available videos to explain multi-agent behaviour. These explanations can be characterised as ideal types, based on Star and Griesemer’s types of boundary objects. Framing these ML explanations as boundary objects poses the question what the theory of boundary objects and the key properties of boundary objects - robustness and plasticity imply for the design of ML explanations.
While the robustness of a ML explanation can be described with features, such as being algorithmically correct and transferable to different research settings, the plasticity of a ML explanation can be associated with features, such as being adaptable to people’s lived experiences, reflective capacities, and information needs. Design techniques, such as personalisation and customisation are particularly suitable to support people’s individual needs and experiences of agency, such as sense of identify and ownership [1].
A robust and plastic enough ML explanation support actors, such as a co-designer, product manager, and end-user, in making sense of and acting on a ML explanation.
In our study, we have observed that participants made sense of ML explanations based on their prior knowledge of AI narratives and technologies, reused ML explanations, such as the analogy of training a dog, as part of co-design activities, and co-created mockups that visualised possible ML-based functionalities, such as predicting blood glucose values.
An important contributing factor for adopting a ML explanation was familiarity: participants particularly valued the analogical narrative of training a dog, since it seemed to help bridge the unknown concept of reinforcement learning and the known practice of training a dog. Barriers to adopting and using a ML explanation seemed to be a lack of abstraction and associations with people’s lived self-care experiences.
The sociocultural underpinning of boundary objects suggests that co-designing a plastic and robust enough ML explanation involves not only representing a specific ML concept correctly and evaluating whether the ML explanation was correctly understood, but also gaining a holistic and non-judgemental understanding of how the ML explanation was appropriated and experienced within a certain context. For example, our qualitive inquiry has revealed the importance of tailoring general ML explanations to specific cases, such as selfmanaging diabetes, to avoid misalignments between people’s lived experience and scientific concepts of ML.
Conceptualising ML explanations as boundary objects means to acknowledge that abstraction and ambiguity can lead to divergent viewpoints, misinterpretations, and misunderstandings. Our findings suggest that gaining a good enough understanding of ML explanations can support participants in developing trust in design processes, data collection and analysis technologies, and overarching research objectives. However, what a good enough understanding is and whether a good enough understanding of ML explanations and functionalities is ethically responsible depends on contextual factors, such as the sensitivity of a research setting. While participants with diabetes sketched predictive functionalities during co-design activities, AI researchers highlighted fundamental differences between the desirability and feasibility of ML-driven systems considering fatal implications of false predictions and recommendations in the case of continuous blood glucose monitoring and management.
Developing a plastic and robust enough ML explanation can require an iterative and multidisciplinary design process with a detailed understanding of ML approaches, user groups, and the intended purpose of a ML explanation.
Considering that design methods and tools
to facilitate co-design are recognised
methodological contributions [2, 16], we
encourage researchers and practitioners to
explore the design space of “learner-centered”
[
Content could be presented in engaging
ways, as demonstrated by the creative
presentation of AI as a monster metaphor [7],
the use of tangible cards in the context of data
protection regulations [
We have provided a qualitative account of how AI researchers explained and non-experts perceived ML concepts as part of a co-design project that aimed to inform the design of ML applications for diabetes self-care.
We have identified benefits and challenges of explaining ML concepts with analogical narratives, information visualisations, and publicly available videos. Co-design participants reported not only gaining an improved understanding of ML concepts but also gaining trust in the co-design process of ML based technologies, data collection and analysis technologies, and overarching research objectives. However, co-design participants also highlighted challenges of understanding ML explanations, including misalignments between scientific models of ML and their lived self-care experiences and prior knowledge of AI and ML approaches.
Based on this understanding, we have framed our findings through the lens of Stars and Griesemer’s concept of boundary objects to discuss how the presentation of user-centred ML explanations could maintain a delicate balance between being plastic and robust enough to support design objectives and people’s individual information needs as part of multidisciplinary projects.
This project was funded by an Innovate UK Digital Catalyst Award - Digital Health. RSR is partially funded by the UKRI Turing AI Fellowship EP/V024817/1. Many thanks to all study participants and reviewers.
[1] Ayobi, A. et al. 2020. Trackly: A
Customisable and Pictorial Self-Tracking App to Support Agency in Multiple Sclerosis Self-Care. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20) (2020). [2] Bødker, S. et al. 2000. Co-operative Design—perspectives on 20 years with ‘the Scandinavian IT Design Model.’ proceedings of NordiCHI (2000), 22–24. [3] Braun, V. and Clarke, V. 2006. Using thematic analysis in psychology.
Qualitative Research in Psychology. 3, 2 (Jan. 2006), 77–101.
DOI:https://doi.org/10.1191/1478088706qp 063oa. [4] Choe, E.K. et al. 2017. Semi-Automated Tracking: A Balanced Approach for SelfMonitoring Applications. IEEE Pervasive Computing. 16, 1 (Jan. 2017), 74–84.
DOI:https://doi.org/10.1109/MPRV.2017.1 8. [5] Clarke, M.F. et al. 2019. Better Supporting Workers in ML Workplaces. Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (Austin TX USA, Nov. 2019), 443–448. [6] Dove, G. et al. 2017. UX Design
Innovation: Challenges for Working with Machine Learning as a Design Material. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver Colorado USA, May 2017), 278– 288. [7] Dove, G. and Fayard, A.-L. 2020.
Monsters, Metaphors, and Machine Learning. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu HI USA, Apr. 2020), 1–17. [8] Emergent Tool Use from Multi-Agent Interaction: 2019. https://openai.com/blog/emergent-tooluse/. Accessed: 2020-09-17. [9] Fu, Z. and Zhou, Y. 2020. Research on human–AI co-creation based on reflective design practice. CCF Transactions on Pervasive Computing and Interaction. 2, 1 (Mar. 2020), 33–41. [10] Gillies, M. et al. 2016. Human-Centred Machine Learning. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems CHI EA ’16 (San Jose, California, USA, 2016), 3558–3565. [11] Girardin, F. and Lathia, N. 2017. When User Experience Designers Partner with Data Scientists. AAAI Spring Symposia (2017). [12] Harper, R.H.R. 2019. The Role of HCI in the Age of AI. International Journal of Human–Computer Interaction. 35, 15 (Sep. 2019), 1331–1344.
DOI:https://doi.org/10.1080/10447318.201 9.1631527. [13] Katan, S. et al. 2015. Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15 (Seoul, Republic of Korea, 2015), 251–254. [14] Katz, D.S. et al. 2018. Data, Data Everywhere, and Still Too Hard to Link: Insights from User Interactions with Diabetes Apps. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2018), 503:1-503:12. [15] Kaur, H. et al. 2020. Interpreting
Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu HI USA, Apr. 2020), 1–14. [16] Kensing, F. and Blomberg, J. 1998.
Participatory design: Issues and concerns. Computer Supported Cooperative Work (CSCW). 7, 3–4 (1998), 167–185.