This paper compares two main perspectives on explanations in intelligent systems: 1) A normative view, based on recent legislation and ethical considerations, which motivates detailed and comprehensive explanations of algorithms in intelligent systems. 2) A pragmatic view, motivated by benefits for usability and efficient use, achieved through better understanding of the system. We introduce and discuss design dimensions for explanations in intelligent systems and their desired realizations as motivated by these two perspectives. We conclude that while the normative view ensures a minimal standard as a “right to explanation”, the pragmatic view is likely the more challenging perspective and will benefit the most from knowledge and research in HCI to ensure a usable integration of explanations into intelligent systems and to work on best practices to do so.
INTRODUCTION
Explaining how a system works and thus making its
underlying reasoning transparent can contribute positively to user
satisfaction and perceived control [
ExSS ’18, March 11, Tokyo, Japan.
explainable [
We often trust human decision making without completely understanding the rationale behind it. Why do we not invest the same trust in AI calculations that consistently yield good results? In this position paper we analyze two arguments for transparency: a normative one emphasizing the right to receive explanations and a pragmatic one viewing transparency as a precondition for effective use. We illustrate how both perspectives differ and how they affect the design of explanations in intelligent systems.
THE NORMATIVE VIEW: A RIGHT TO EXPLANATION
“[Algorithmic] decisions that seriously affect
individuals’ capabilities must be constructed in ways that are
comprehensible as well as contestable. If that is not
possible, or, as long as this is not possible, such decisions
are unlawful [...]” [
THE PRAGMATIC VIEW: FOCUS ON USABILITY
“No, no! The adventures first, explanations take such a
dreadful time.” [
DESIGN DIMENSIONS
We describe several design dimensions to characterize
possible explanations which might arise from either one of the two
presented perspectives. Some of these dimensions, such as
Spatial Embedding or Temporal Embedding, have been
similarly presented in prior work, e.g., on system intelligibility [
In contrast, the pragmatic view employs explanations to achieve a (possibly limited, non-comprehensive) level of understanding that facilitates usability and effective use of the system (see Presentation). Thus, it is necessary that users encounter explanations at some point before or during their main tasks with the system (see Temporal Embedding). To ensure this, systems may want to integrate explanations more closely (see Spatial Embedding) to achieve what we might call “foreground trust”.
Foundation The Foundation informs the content of the explanation (i.e., what to explain?).
The normative view may take into account an expert’s mental model as a “gold standard” to cover all details of the underlying algorithm in a comprehensive, but still human-readable form.
In contrast, the pragmatic view also puts more emphasis on
considering the users’ mental models, for example to tailor
explanations to particularly assess and address incorrect or
incomplete aspects of these models [
Presentation The Presentation dimension covers how the explanation is presented to the user.
To achieve a comprehensive detailed understanding, the normative view could employ almost any format, including videos, plots, interactive exploration and dedicated contact/help options, possibly even a “hotline” service.
In contrast, the pragmatic view aims for a presentation that facilitates a balance between explanation and the actual main UI elements. This might be achieved, for example, with markers/icons, details-on-demand techniques, textual or pictorial annotations, or modifications of layout and UI elements. Level of Detail The desired Level of Detail of the explanation also varies between the two perspectives: The normative view favors a highly detailed explanation with the goal of comprehensive understanding of the intelligent system’s underlying algorithms.
In contrast, the pragmatic view may favor a less detailed overview to facilitate a basic understanding. To do so efficiently, this view may focus on certain aspects and neglect others deemed less important. This focus could be informed by a user-centred design process (see Foundation). Spatial Embedding The Spatial Embedding describes how the explanation is integrated into the system’s GUI overall.
underlying algorithms in general specific content, e.g., a specific recommendation understanding, background trust expert mental model videos, plots, interactive exploration, contact/help options
high, comprehensive
separate view, “help page”
accessed before/after main tasks The normative view motivates a detailed explanation which might thus not be embedded into the main GUI at all. Instead, systems could add a separate view, such as a “help page”. In contrast, the pragmatic view is motivated to embed explanations directly into the GUIs used for the main tasks of the system. This dimension is thus strongly linked to the presentation choices (see Presentation).
Temporal Embedding The Temporal Embedding describes how the explanation is integrated into the temporal workflow with the system. The normative view motivates a detailed explanation which might thus not be embedded into the main task workflow at all. Instead, the user might optionally access it before or after the main task (e.g., on a separate page, see Spatial Embedding). Hence, once accessed, the full explanation is revealed at once. In contrast, the pragmatic view is motivated to embed explanations directly into the workflow, for example using annotations or other details-on-demand within the main GUI views. This implies that the explanation is revealed gradually over the course of the user’s main tasks with the system.
Reference The Reference dimension describes to which elements the explanations relate to primarily.
The normative view aims to reveal the underlying algorithms, yet may not be interested in doing so for specific cases that users encounter during their individual workflow. In contrast, by integrating explanations more directly, the pragmatic view’s references for explanations are the specific cases encountered by the individual user during their interactions. CONCLUSION In this paper, we sketched out two perspectives on transparency in intelligent systems – a normative and a pragmatic view. The distinction between these two allows us to discuss different usability, effective use, foreground trust symbiosis of expert and user mental models markers, details-on-demand, UI elements and annotations overview, efficient directly integrated into UI interleaved with main tasks approaches to designing explanations. If one takes a normative standpoint, the mere option to receive explanations about an algorithm is critical and sufficient. Explanations need to be detailed enough to satisfy users’ needs for information. To avoid cluttering the interface, these detailed holistic explanations might be separated from the main interface, e.g., in a help function. If one takes a pragmatic standpoint, explanations detached from the interface and workflow are unlikely to be effective, as one can expect that very few users will make use of this option. The goal of the pragmatic approach is rather to integrate small bites of explanations into the interface to increase users’ understanding of the system slowly and effortlessly over time. It is the design of such well thought-through interface concepts that reveal the systems functioning during the interaction where HCI knowledge and research will be most needed and impactful.
That said, both perspectives are not to be regarded as mutually exclusive but can likely be combined appropriately. The normative perspective can then be regarded as “must have” and the right to receive explanations as a minimal standard, even if explanations are not integrated in a user-friendly fashion. Integrating explanations elegantly where they are interesting and useful for users will then be the challenge to work on and we invite HCI researchers to jointly work on this already now.