Explanations are necessary for building users' understanding and trust in machine learning systems. However, users may abandon systems if these explanations demonstrate consistent errors and they cannot affect change in the systems' behavior in response. When user feedback is supported, then the utility of explanations is to not only promote understanding, but also enable users to help the machine learning system overcome errors. We suggest an experiment to examine how users react when a system makes explainable mistakes with varied support for user feedback.
While significant work explores this need for more
explainable machine learning – whether by making the
algorithms themselves more explainable [
In this position paper, we discuss prior work on explainable systems that do and do not take user feedback into account and outline a study design to examine how users react when a system makes explainable mistakes with varied support for user feedback.
MOTIVATION Imagine the following example (from a frequently-cited explainable machine learning paper [10]): an explainable system, specifically an image classification tool, makes an error, such as classifying an image of a husky incorrectly as a wolf. The user then requests an explanation of why the system produced this incorrect classification. Using attention,1 the system can explain its mistake by displaying to the user that the presence of snow in the image led to the wolf misclassification. At this point, the user now understands why the system made this initial mistake, but what happens when the system makes the same or similar mistake again? Here we consider two possible outcomes. One possibility is that the user will be frustrated or choose not to use the system if they know that the it errs on certain types of data or problems, but they cannot do anything about it. Alternatively, the system may be deceiving to users who believe that the it can learn from mistakes (as a human who admits to making a mistake is expected to), but in fact it will continue to make the same mistake. In fact, Ribeiro et al. [10] find that while 10 of 27 participants trust the model that misclassifies a husky for a wolf without any explanation, only three out of 27 participants trust the model when it explains the mistake. Thus, without a way to provide user feedback to improve 1In image classification, attention [8] can be used to determine the portion of an image that most affected the system’s classification, or the part of the image that the system “attended to” the most when making a classification. the system, explaining predictions is more likely to be utilized as a method for knowing when not to trust the system.
Alternatively, in our prior work [13], we developed a system for intelligence analysts that both provides evidence for its decisions and supports analyst feedback to improve the underlying model. This system automatically clusters entity mentions (people, places, and organizations) from large unstructured corpora to overarching entity clusters.2 For example, clustering entity mentions throughout a large news corpus, such as Mr. Obama, President Obama, and Barack, into one entity cluster, President Barack Obama. The system provides as evidence the entity mention in context as well as the other entity mentions in the cluster. While this evidence may help the analyst to understand why certain mentions were incorrectly placed in clusters or other mentions are missing from a cluster, simply understanding the system’s mistakes is not sufficient for supporting trust and utilization. To this end, the system supports interactive feedback mechanisms, such as accepting and rejecting mentions as well as merging clusters. While no formal user experiment has been performed with this system, we have received positive feedback from analysts regarding the interactive feedback mechanisms.
EXPERIMENT DESIGN We outline an experiment design to examine how users react to an explainable system with varied support for user feedback. Specifically, we suggest two possible study methods: the first, which aims to explore user frustration or confusion that occurs when an explainable system (that does not update with user feedback) continues to make the same or similar mistakes, and the second, which compares users’ reactions to versions of a system based on the amount of control given to the user.
Research Questions The goal of this proposed experiment is to answer the following research questions: Q1: Do users assume an explainable system learns from mistakes? We hope to better understand what users expect when utilizing an explainable system. Whether or not users expect the system to continue making the same or similar mistakes impacts how negatively the users will be affected when it does. Furthermore, we would like to understand whether users’ expectations change if we vary how explanations are attended to or the form of explanations. Shneiderman [11] and Lanier [6] argue that systems (intelligent agents, in particular) should not have humanlike characteristics as these lead users to believing that the system may act rationally or take some responsibility for its actions [3]. Therefore, we hypothesize that conversational 2This technique utilizes inter and intra-document entity coreference, meaning it clusters entity mentions within and across documents. (or apologetic) explanations may be more likely to lead users to thinking the system will learn from a mistake. Similarly, whether users expect a system to improve may vary based on their interactions with explanations, such as simply clicking ‘ok’ to dismiss explanations as opposed to interactions such as ‘accepting’ or ‘rejecting’ classifications. The latter may lead users to believe they are correcting the system.
Q2: How is trust of and frustration with an explainable
system affected by varied supports for user feedback?
Prior work implied, albeit without a formal experiment, that
users may trust systems less when they explain their
mistakes [10]. Similarly, Lim and Dey [7] find that users’
impressions of a system are negatively impacted when
systems are highly uncertain of their decisions (even when
they behave appropriately). While supporting user
feedback, particularly in cases of system error or high
uncertainty, could mitigate these issue, the level of control
given to the user may have varied effects on trust and
frustration. For example, in prior work we discuss whether
user feedback should be taken as a command or a
suggestion for different types of interactive systems [12].
Method
To support examination of the identified research questions,
we outline the following two-part study methodology.
The first part of the study will be performed as an interview
study following a think-aloud methodology followed by a
post-task survey. First, users will be shown an explainable
system. When the system errs, it will provide an
explanation. We will then ask users whether they believe
the system will make the same or similar mistakes followed
by measuring frustration and/or surprise when it does
continue to do so. Frustration will be measured on the
incident level and overall level following the methodology
described by Bessier et al. [
The second part of the study will be performed as a
crowdsourced survey. In this case, we will incorporate user
feedback into an explainable system. We will vary the
system only in how it incorporates user feedback,
representing the amount of control the user has over the
system. We propose three system variants: one that ignores
all user feedback, one that takes feedback into account as a
suggestion, and one that takes feedback into account as a
command. We will then measure how user trust, frustration,
and other user reactions differ between these variants.
Frustration will again be measured on the incident and
overall level [
CONCLUSION In this position paper, we argue that while explainable systems are important, incorporating user feedback into these systems is equally important for supporting trust and continued use. And this goes both ways – systems that support user feedback must also ensure users understand how they work, such that they can give appropriate feedback. A truthful explanation into the system’s black box improves users’ understanding, which better prepares them for providing feedback to improve the system. We propose an experiment to provide additional evidence for this argument.
Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2017.2731842
comprehensible, predictable and controllable user interfaces. In Proceedings of the 2nd international conference on Intelligent user interfaces - IUI ’97, 33–39. https://doi.org/10.1145/238218.238281
Systems. In Designing for Uncertainty Workshop at CHI 2017.
Conference on Next-Generation Analyst V.