The adoption of AI approaches is increasing across domains, yet the study of explainable AI user interfaces for domain experts remains limited. One potential benefit provided by this type of user interface is to facilitate better data collection and improve in-domain model training. With the advancement of explainable AI (XAI) methods for end-users, domain experts can more easily collaborate with AI researchers and contribute to the process of building and deploying domain relevant AI models. We propose an XAI interface for domain experts in a manufacturing setting, that provides transparency into multi-modal AI systems and supports domain expert collaboration with AI researchers to fine-tune models through active feedback. In this paper, we report early findings of a user study with this XAI interface, with participants including both domain experts and AI researchers. These early findings hold promise for supporting improved system understanding by end users as well as cross-functional collaboration between domain experts and AI researchers.
The field of artificial intelligence (AI) has progressed remarkably in recent years. It enables new services, such as AI assistance systems which will be adopted not only by households, but also professionals. AI support can streamline operations, support human performance in working environments, and augment human capabilities across tasks.
However, AI assistance systems in these areas encounter very real problems due to a very limited amount of data for training, issues with the incorporation of domain knowledge, and difficulties in fine-tuning the models.
Having domain experts collaborate with AI researchers in development of the AI assistance system could potentially help alleviate the problems mentioned above, since expert feedback is a rich source of data incorporating domain knowledge. However, such collaboration requires the domain expert to also be familiar with the design and back-end knowledge of such AI assistance systems, which is almost impossible without an explainable AI approach.
In this paper, we report a user interface designed to support explainable AI (XAI) for a complex multi-modal AI assistance system with a goal of facilitating collaboration between domain experts in manufacturing (e.g., technicians on the factory floor) and developers working on multimodal ambient sensing systems for human performance and task support.
In the first few years of its history, the XAI field focused on developing algorithms to explain or
interpret the behavior of "black box" models [
Our contribution builds on this work to facilitate collaboration between two groups: AI researchers and one type of end-user, users who have expert-level knowledge of their domain of work, i.e., domain experts. Facilitating collaboration between these two groups can address key challenges of data availability, data quality, and the resource demands of model fine-tuning. We propose an XAI interface through which domain experts collaborate with AI researchers to give direct input into AI models. A key innovation is the step of actionability: we intend our UI to support end-user action and cross-functional collaboration, in the form of providing feedback on model performance provided by domain experts, to improve system performance.
We developed a POC (proof of concept) XAI user interface prototype for a multi-modal AI system providing performance support in a manufacturing scenario. As this prototype is a kind of perceptually guided task guidance system, we call it “Task Guidance System” (TGS). TGS, as shown in Fig. 1, is designed to facilitate collaboration between domain experts and AI researchers in finetuning the various components of the multi-modal AI system.
The TGS visualizes multiple models used by the multi-modal AI supporting system for Action
Recognition (MSTCN[
In addition, TGS allows domain experts with ground-truth knowledge to critique these predictions through its dialogue system and, thereby, lets AI researchers fine-tune the AI models. To support this collaboration, prompts encourage users to provide feedback at the models' request. These prompts include an uncertainty score for vision, an importance score for noting which words in an utterance were most indicative of a match, highlighted inconsistencies between different models, etc.
The contribution of this work is significant because the obstacles mentioned earlier may hold us back from building systems that can help humans achieve their full potential. The XAI UI we propose is intended to directly address these obstacles.
The XAI UI was evaluated remotely with two user studies, one with domain experts and one with AI researchers. Wizard-of-Oz presentations of the UI were combined with qualitative interview protocols for both studies.
The first study was conducted with two domain experts interacting with the AI assistance system including the XAI UI. Both experts had several months of experience working with the AI system (without the XAI UI). A semi-structured interview protocol was designed and carried out to elicit and capture data about the domain experts' cognitive model of system and TGS, whether the information displayed had any impact on their understanding of model predictions, or their willingness to provide feedback to AI researchers.
The second study looked at three AI researchers experienced in multi-modal AI assistance systems. A second semi-structured interview protocol was designed to elicit and capture perceptions of the UI, what domain knowledge might be important for their goals as AI researchers, and how they feel this knowledge can be adequately captured from domain experts through TGS. All interview results were analyzed collaboratively by two of the authors using thematic analysis.
All five interviews took place remotely, via a video conferencing platform. Most interviews took between 30 and 90 minutes, with one outlier Wizard/interview process taking roughly two hours.
AI researchers confirmed that they could derive value from capturing domain experts' knowledge. They expressed that this data could be particularly valuable for fine-tuning models in domains characterized by high degrees of change, or domains or applications that may be made more complex by the need to serve multiple users or multiple different types of users.
AI researchers report it is important to see the entire data flow from domain experts' raw performance data to the final output. For example, with a multi-modal AI system, one models' output may rely on another's output.
Domain experts demonstrated a willingness to help AI researchers using TGS, and identified novel challenges AI researchers might face in negotiating a space of multiple or changing approaches in methods or objects. All domain experts reported that comprehending the information in the UI was challenging. However, even then, domain expert interaction with TGS provided valuable feedback for AI researchers. For example, domain experts pointed out instances where the task model was too specific and pointed to variants on the task that should be supported by the AI assistance system. Without access to the nuanced domain knowledge held by domain experts, AI researchers could face risks when fine-tuning their models of incorrectly labeling actions or objects that they might not know are acceptable alternative methods for accomplishing a task. We found that when domain experts interpret data differently than AI researchers, XAI information can reveal to domain experts that their input is needed to rectify an inaccurate model prediction.
When domain experts saw incorrect model predictions, they verbally challenged the discrepancies between the prediction and their understanding of correctness of the process being analyzed: one domain expert asked, unprompted, “Why does [the UI] think I'm holding [item] when I'm clearly holding [a different item]?” when shown the TGS UI. We believe that this promises that further engagement will see an increased willingness to engage with and correct inaccurate predictions. This was confirmed by explicit inquiries about their willingness to provide feedback: both domain experts expressed that they would be both willing and enthusiastic to provide input back to AI researchers via the interface.
The paper presents our ongoing XAI dashboard and findings of an early user study. A larger-scale user study will be performed to more robustly evaluate the visual content and affordances of the UI and dashboard. We plan to design additional XAI visualizations across multiple form factors, including popular approaches like heatmaps. We also plan visualizations plotting prediction history to identify outlier predictions and generate user-facing alerts. Our plans also include providing more options to edit models directly through TGS, which we hope will make fine-tuning models faster and more flexible. We hope our application can support scaling systems more quickly and easily across domains.