The ability to explain actions and decisions is often regarded as a basic ingredient of cognitive systems. But when researchers propose methods for making AI systems understandable, users are usually not involved or even mentioned. However, the purpose is to make people willing to accept the decision of a machine or to be better able to interact with it. Therefore, I argue that the evaluation of explanations must involve some form of user testing.
The idea to build comprehensible technical devices is much older than the
current trend in AI. Many usability guidelines from the elds of design and
humancomputer interaction include implicit or explicit explanation to users. Norman
[
Thus, any device should be designed in a way that users are aware of the system's capabilities and current state. This can be interpreted as a kind of explanation, one can also say that such a system does not need any additional explanation. For AI researchers this means that 1) comprehensible systems can often be built by adhering to well-known usability principles, 2) the design and usability of the overall system is often more important than AI features to result in an overall positive user experience. 3
Embodied and virtual agents di er from typical human-computer interaction in that they can perform more pronounced actions and often have additional sensing capabilities. What is an explanation in this context? Of course, such agents could generate explanations of their actions, possibly in natural language. But in many cases a constant stream of explanations may annoy users and its e ectiveness is questionable. When we look at human-human interaction, we hardly ever need to explain our actions to others. So why should a machine do so?
A more natural way of interaction could be to implicitly communicate goals
and the necessity of actions. Lichtenthaler and Kirsch [
Robot behavior is legible if: (Factor 1) a human observer or interactor is able to understand its intentions, and (Factor 2) the behavior met the expectations of the human observer or interactor.
This de nition centers on the human observer, implying that legibility can only be determined by experiments involving users. Such tests can again be inspired by standard usability testing, but the quality criteria and setup must often be adapted to the task and embodied situation of the agent. In addition, if physical robots are involved, there are additional questions of how to ensure the safety of participants.
As an example, consider the task of legible robot navigation [
Performing user tests is time-consuming. Therefore, it would be nice to have
a general set of measures that determine legibility, but that can be measured
without direct user involvement. An experiment of Papenmeier et al. [
Legibility as used above is a criterion for passive interaction: even though a person has no direct contact with the agent, the behavior should be legible. But it is also a prerequisite for direct interaction, for example a person trying to initiate an interaction would have to be able to predict the robot's movements. But direct interaction may require more than just a basic, implicit understanding of the other's intentions, it may at times require a more explicit explanation.
Decision methods can be designed in a way that resembles human decision
making. The Heuristic Problem Solver [
In addition, such an explicit decision-making paragidm enables an interactive process of humans and machines jointly determining a solution. It is practically impossible to model all the knowledge a person has with respect to a task. But if people can interact with the decision process of the machine, they can use this additional knowledge without having to formalize all of it. Thus, a person could propose or delete alternatives, or change the evaluations of alternatives.
Here again, the question is whether the explanation is adequate. Ideally, the machine should provide enough information for a person to be satis ed with or even be willing to take responsibility for the decision. Or in the case of interaction, a possible measurement is whether the joint decision-making is more e cient and e ective than a purely computational or purely human one.
The ability to explain one's decisions and actions is often unquestioningly required from cognitive systems. But the frequency, form and implementation of explanations depend on the speci c application and context. In many cases, standard feedback mechanisms known from human-computer interaction are all that is needed. Other applications may demand that users are able to identify the source of errors. Even others may need an explicit dialogue between human and machine that allows the machine to request help if necessary. Explanations can take di erent forms, from standard interaction methods of ashing lights, sounds and graphical displays, to language or the display of legible actions. In the last section I have mentioned systems that generate an explanation along with the solution. An alternative is the post-hoc generation of explanations for black box algorithms, which is currently popular with the rise of statistical machine learning.
In all respects, the users are the leveling rule. An explanation is not a mathematical construct, an explanation is good if people nd it helpful in the speci c context. Therefore, the AI community should expand its evaluation metrics beyond optimization criteria to user-centered measures and evaluation procedures.
.