- Lack of trust in autonomy is a recurrent issue that is becoming more and more acute as manpower reduction pressures increase. We address the socio-technical form of this trust problem through a novel decision explanation approach. Our approach employs a semantic representation to capture decision-relevant concepts as well as other mission-relevant knowledge along with a reasoning approach that allows users to pose queries and get system responses that expose decision rationale to users. This representation enables a natural, dialogbased approach to decision explanation. It is our hypothesis that the transparency achieved through this dialog process will increase user trust in autonomous decisions. We tested our hypothesis in an experimental scenario set in the maritime autonomy domain. Participant responses on psychometric trust constructs were found to be significantly higher in the experimental group for the majority of constructs, supporting our hypothesis. Our results suggest the efficacy of incorporating a decision explanation facility in systems for which a sociotechnical trust problem exists or might be expected to develop.
Keywords-Semantic modeling; Maritime Autonomy; Trust in Autonomy; Decision Explanation.
I.
INTRODUCTION Large organizations such as the Department of Defense rely heavily on automation as a means of ensuring high-quality product, as well as cost control through manpower reduction. However, lack of user trust has repeatedly stood in the way of widespread deployment. We have observed two fundamental forms of the problem: the technical and the socio-technical form. The technical form is characterized by user reservations regarding the ability of a system to perform its mission due to known or suspected technical defects. For example, an automated detection process might have a very high false positive rate, conditioning operators to simply ignore its output. Trust in such a situation can only be achieved by addressing the issue of excessive false detections, a technical problem suggesting a purely technical solution. As another example, consider a situation in which automation is introduced into a purely manual process characterized by decision making in high-pressure situations. In such a situation, operators might reject automation in favor of the This research was supported by Raytheon Corporate IR&D. trusted, manual process for purely non-technical reasons. In other words, in the absence of any specific evidence of limitations of the automation, the automation could nonetheless be rejected for reasons stemming from the social milieu in which the system operates. This is the sociotechnical form of the problem.
One might address the socio-technical problem through education: train the operators with sufficient knowledge of system specifications and design detail to erase doubts they may have regarding the automation. Such an approach is costly since every operator would have to be trained to a high degree. Operators would essentially have to be system specialists. Instead, we propose an approach intended for nonspecialist operators, stemming from the insight that the sociotechnical trust problem results from a lack of insight into system decision rationale. If an operator can be made to understand the why of system behavior, that operator can be expected to trust the system in the future to a greater degree, if the rationale given to the operator makes sense in the current mission context.
Explanation mechanisms in expert systems have focused on
the use of explicit representations of design logic and problem
solving strategies [
Type 1 explanations are concerned with explaining why certain decisions were or were not made during the execution (runtime) of the system. These explanations use information about the relationships that exist between pieces of data and the knowledge (sets of rules for example) available for making specific decisions or choices based on this data. For example, Rule X fired because Data Y was found to be true.
Type 2 explanations are concerned with explaining the knowledge base elements themselves. In order to do this, explanations of this type must look at knowledge about knowledge. For example, knowledge may exist about a rule that identifies this rule (this piece of knowledge) as being applicable ninety percent of the time. A type 2 explanation could use this information (this knowledge about knowledge) to justify the use of this rule. Other knowledge used in providing this type of explanation consists of knowledge that is used to develop the ES but which does not affect the operation of the system. This type of knowledge is referred to as deep knowledge.
runtime control strategy used to solve a particular problem. For example, explaining why one particular rule (or set of rules) was fired before some other rule is an explanation about the control strategy of the system. Explaining why a certain question (or type of question) was asked of the user in lieu of some other logical or related choice is another example. Therefore, type 3 explanations are concerned with explaining how and why the system uses its knowledge the way it does, a task that also requires the use of deep knowledge in many cases.
Design considerations for explanations with dialog are
discussed in a number of papers by Moore and colleagues ([
Weiner [
In Carenini and Moore [
In Gruber and Gautier [
Approaches to producing explanations in MDPs are
proposed in Elizalde et al. [
An explanation application for JAVA debugging is
presented in Ko and Myers [
Explanations in case-based reasoning systems are examined
as well. Sørmo, Cassens, and Aamodt [
Explanation of decisions made via decision trees is
considered in Langlotz, Shortliffe, and Fagan [
Lieberman and Kumar [
Bader et al. [
In Pu and Chen [
The important of the use of context in explaining the
recommendations of a recommendation system was
investigated in Baltrunas et al. [
Explanation capabilities have also been shown to aid in
increasing user satisfaction with and establishing trust in
complex systems [
Our approach to providing transparency, the Why Agent, is a decision explanation approach incorporating dialog between the user and the system. Rather than attempting to provide monolithic explanations to individual questions, our dialogbased approach allows the user to pose a series of questions, the responses to which may prompt additional questions. Imitative of natural discourse, our dialog approach allows a user to understand the behavior of the system by asking questions about its goals, actions or observables and receiving responses couched in similar terms. We implemented our approach and conducted an evaluation in a maritime autonomy scenario. The evaluation consisted of an experiment in which two versions of an interface were shown to participants who then answered questions related to trust. Results of the experiment show response scores statistically consistent with our expectations for the majority of psychometric constructs tested, supporting our overall hypothesis that transparency fosters trust. The rest of this paper is organized as follows. Section II describes the problem domain and the technical approach. Experiments and results are presented in Section III. In Section IV, we provide some concluding remarks and future research directions.
II.
Our approach to demonstrating the Why Agent functionality and evaluating its effectiveness consisted of a simulation-based environment centered on a maritime scenario defined in consultation with maritime autonomy SMEs. The notional autonomous system in our scenario was the X3 autonomous unmanned surface vehicle (AUSV) by Harbor Wing Technologies 1 . Raytheon presently has a business relationship with this vendor in which we provide ISR packages for their AUSVs.
The X3 was of necessity a notional AUSV for our demonstration because the actual prototype was not operational at the time of the Why Agent project. For this reason, a live, on-system demonstration was not considered. Instead, our demonstration environment was entirely simulation-based. An existing route planning engine developed under Raytheon research was modified to serve as the AUSV planner. Additional code was developed to support the simulation environment and Why Agent functionality, as described below.
Our software architecture consists of four components interacting in a service-oriented architecture, as shown in Figure 1.
The Planner component performed route planning functions based on a plan of intended movement. A plan of intended movement is input in the form of a series of waypoints. These waypoints, along with environmental factors, such as weather forecast data, are used in the planning algorithm to determine 1 http://www.harborwingtech.com an actual over-ocean route. The planner was a pre-existing component developed on R&D that the Why Agent leveraged for the demonstration. Modifications made to the planner to support the Why Agent project include changes to expose route change rationale to the controller and inform the controller of weather report information. available to the user in the context-sensitive menu for the
item and the associated why? option, a query is generated that contains IDs associated with the ConductPatrol node and the servesPurpose link. The linked node, in this case MissionExecution,is then returned to the user as the result of a query against the associated OWL model.
The Controller represents the embodiment of the majority of the simulated AUSV decision logic and simulation control logic. Because we did not employ an actual AUSV for the Why Agent project, much of the decision logic of an actual AUSV had to be simulated for our demonstration, logic implemented in the Controller. The input to the Controller consisted of a test control file that defined the event timeline for the simulation. In addition to orchestrating simulation events defined in the control file, the Controller mediated queries and responses between the user interface and the semantic service.
The graphical user interface was implemented as a web application. Two versions of the GUI were developed, one with and one without the Why Agent explanation facility. The Why Agent version is shown in Figure 2. It has four screen regions: a map, a status panel, a log data panel and an explanation panel. The map, implemented with Google Map technology, shows the current location and route of the AUSV. The status panel shows various AUSV status values, such as location, speed, current mode, etc. The log panel shows a time-stamped series of event descriptions. Various items in the log panel are user-selectable and have context-sensitive menus to support the user interface functionality of the Why Agent facility. When a user makes a selection, the response from the semantic service is shown in the bottom (explanation) panel. Additionally, responses in the explanation panel are also selectable for further queries. In this manner, the user can engage in a dialog with the system.
The semantic service contains the knowledgebase underlying the decision rationale exposed by the Why Agent. The knowledge consists of event and domain ontology models represented in web ontology language (OWL) format. The semantic service provides responses to queries from the controller through queries against its underlying models.
An example of a domain model is shown in Figure 3. Relationships in this figure encode potential queries linking concepts and events that can be displayed in the user interface. For example, the activity ConductPatrol relates to the function
relationship is statically associated with the query why? at the user level. Thus, the existence of this link connected with the node ConductPatrol implies a why? option being made
Our evaluation approach consisted of an experiment in which the Why Agent was the treatment. Two versions of a prototype operator interface were developed. One version incorporated the Why Agent functionality and the second did not. The two versions were otherwise identical. Screenshots of the two interface versions are presented in Figures 4 and 5.
The demonstration scenario consisted of autonomous fishing law enforcement in the Northwestern Hawaiian Islands Marine National Monument. The CONOP for this mission is as follows:
The AUSV operator selects waypoints corresponding to a patrol area.
The AUSV route planner finds a route through the waypoints and a patrol is conducted.
RADAR is used to detect potential illegal fishing vessels (targets) Targets are investigated visually after AUSV closes to an adequate proximity.
Automated analysis of the visual data is used to confirm the target is engaged in illegal fishing.
Targets engaged in illegal activity are visually identified for subsequent manned enforcement action.
Non-lethal self-defensive actions can be taken by the AUSV in the presence of hostile targets.
To support this demonstration, a software-based simulation environment was developed. The demonstration consisted of capturing video of user interactions with the baseline and Why Agent versions of the operator interface while a scripted series of events unfolded over a pre-determined timeline.
Our experiment consisted of a single-factor, randomized design. The factor is interface type and has two levels: baseline (control) and Why Agent (experimental). Thus, we have two treatment levels, corresponding to the two factor types. The experimental subjects were Raytheon employees, recruited across multiple Raytheon locations, during the project.
Our general hypothesis is that the Why Agent fosters a
more appropriate level of trust in users than the baseline
system. By utilizing the information provided by the Why
Agent, users will be more able to calibrate their trust [
Trust in a particular system, being an unobservable mental aspect of a user, necessitates the use of psychometric readings of constructs related to the overall concept of trust. Given the broad nature of this concept, multiple constructs should be defined. Using our domain insight and engineering judgment, we selected the following set of five psychometric constructs: 1. General Competence, 2) Self-Defense, 3) Navigation, 4) Environmental Conservation and 5) Mission. Each construct is intended to capture the users‘ belief regarding the system‘s ability to effectively perform in regard to that construct, i.e. the user‘s level of trust for that construct. For example, the construct Mission attempts to encompass user attitudes toward the ability of the system to successfully execute its mission. The Environmental Conservation construct was included as an example of a construct under which we would not expect to see a difference in psychometric responses.
For each construct, we have a set of possible trust levels and a set of psychometric participant response scores. Define these as follows (for this study, k=5):
Set of k constructs C = {cj : 1 ≤ j ≤ k} Set of trust levels L = {low, high}
Control: RC = {rjC : 1 ≤ j ≤ k }
Experimental: RE = {rjE : 1 ≤ j ≤ k }
Here, we take the simplest possible approach, a binary trust level set. We simply assume that the trust level for a particular construct should either be low or high, with nothing in between. Clearly, many other trust models are possible. To operationalize the notion of ―more appropriate level of trust‖, we need to define, for each construct, a ground truth assignment of trust level. Thus, we need to define the following mapping T:
o o
T(j) = low: People should not trust the system regarding construct j T(j) = high: People should trust the system regarding construct j.
Additionally, we need to map the elements of the trust set to psychometric scale values. In other words, we need to normalize the scale as follows:
S: S(low) = 1; S(high) = 5.
At this point, we can define the concept of ―appropriate level of trust‖ in terms of the psychometric scale through a composition of the above mappings S and T. In other words, for each construct, the appropriate level of trust is the psychometric value associated with the trust level assigned to that construct:
intent A = {aj : 1 ≤ j ≤ k }
For each construct cj, the appropriate level of trust aj for that construct is given by aj = S(T(j)), 1 ≤ j ≤ k (1)
A key aspect of the above definition is the qualifier with respect to design intent. We assume the system functions without defects. With respect to design intent simply means ―it should be trusted to accomplish X if it is designed to accomplish X.‖ We make this assumption for simplification purposes, fully acknowledging that no real system is defectfree. In the presence of defects, the notion of appropriate level of trust becomes more complex.
Having defined appropriate level of trust, we are finally in
a position to define the key concept, more appropriate level of
trust. The intuition underlying this notion is the observation
that if one‘s trust level is not appropriate to begin with, any
intervention that moves the trust level toward the appropriate
score by a greater amount than some other intervention can be
said to provide a ―more‖ appropriate level of trust. The Why
Agent specifically exposes information associated with the
purpose of AUSV actions. Such additional information serves
to build trust [
response scores rjC and rjE for construct j, the experimental response rjE reflects a more appropriate level of trust when the following holds rjE - rjC < 0 if aj = 1 rjE - rjC > 0 if aj = 5
We expect the Why Agent to affect observed trust levels only for those constructs for which relevant decision criteria are exposed during the scenario. In these cases, we expect Equations (2)-(3) to hold. In all other cases, we do not. For example, since the AUSV is not designed to protect marine life, we assert that the appropriate level of trust for the Environmental Conservation construct is ―low.‖ However, we do not expect to observe response levels consistent with Equations (2) – (3) unless dialog exposing decision rationale relevant to this concept is included in the scenario.
Based on this reasoning, we expect the effect of decision explanation to be one of pushing response scores up or down, toward the appropriate trust level but only in cases where explanation dialog related to the construct under test is exposed. In other cases, we expect no difference in the response scores, as indicated in Table 1. We note that the null hypotheses are derived as the complementary sets to the equations in Table 1. E.g., the ‗low, with relevant dialog‘ null hypothesis equation would be rjE – rjC ≥ 0.
A total of 44 control and 50 experimental subjects were recruited for the Why Agent study. The experiment was designed to be completed in one hour. Following a short orientation, a pre-study questionnaire was presented to the participants. The pre-study questionnaire contained questions regarding participant demographics and technology attitudes. The purpose of the pre-study questionnaire was to determine whether any significant differences existed between the experimental and control groups. Following the pre-study questionnaire, participants were given a short training regarding the autonomous system and their role in the study. Participants were asked to play the role of a Coast Guard commander considering use of the autonomous system for a drug smuggling interdiction mission. Following the training, participants were shown the scenario video which consisted of several minutes of user interaction with either the baseline or Why Agent interface. Following the video, participants completed the main study questionnaire. The system training was provided in a series of powerpoint slides. Screenshots taken from the study video were provided to the participants in hardcopy form, along with hardcopies of the training material. This was done to minimize any dependence on memory for participants when completing the study questionnaire.
To investigate whether significant differences exist between the control and experimental groups in terms of responses to the technology attitudes questions, ANOVA was performed. The results are shown in Table 2. Cronbach reliability coefficients, construct variances and mean total response scores are shown for the control and experimental groups in Tables 3 and 4.
To investigate whether significant differences exist between the control and experimental groups in terms of responses to the study questions, ANOVA was performed. For this study, we focused our analysis on individual constructs. Thus, we do not present any statistics on, for example, correlations among responses related to multiple constructs for either the control or experimental group. The results are shown in Table 6.
T-test results for each construct are shown in Table 5. Two p-values are shown for each construct; p1 represents the pvalue resulting from use of the pooled variance while p2 represents the p-value resulting from use of separate variances.
The ANOVA results shown in Table 2 indicate that the experimental and control groups did not significantly differ across any attribute in terms of their responses to the technology attitudes questions. In other words, we do not see any evidence of a technology attitude bias in the study participants. For constructs one and two, the experimental response was greater than the control response (p = 0.001 and 0.004, respectively), consistent with our expectations. For construct four, environmental conservation, we see no significant difference between the experimental and control responses (p = 0.16), which is also consistent with our expectations as this construct had no associated decision explanation content exposed to the experimental group. The experimental response for construct 3 was not significantly higher than the control response, which is inconsistent with our expectations, although the difference is only marginally outside the significance threshold (p = 0.059).
While the test results indicate moderate support for the efficacy of the Why Agent approach, they are decidedly mixed, so it is not possible to draw any definitive conclusions. As discussed below, we recognize that a number of significant limitations also hinder the application of our results. A pilot study would have helped to create a stronger experimental design and recruit a more representative sample population, but this was not possible due to budget and schedule constraints. Nevertheless, the study has provided initial evidence for how and to what extent the Why Agent approach might influence trust behavior in autonomous systems, and given impetus for continued investigations.
Construct Reliability: Referring to Table 4, we see that reliability coefficients for some constructs are not above the commonly-accepted value of 0.7. Had schedule permitted, a pilot study could have uncovered this issue, providing an opportunity to revise the questionnaire.
Experiment Limitations: Clearly a variety of limitations apply to our experiment. One is that participants did not interact directly with the system interface; instead entire groups of participants were shown a video of someone else interacting with the system. Also, the participants were not drawn from the population of interest. Consequently, our results may not apply to that target group. Additionally, subjects were asked to play a role with much less information than a real person in that role would have. Also, as noted by a reviewer, the experimental design does not allow us to determine whether decision correctness is related to trust when clearly it should be; an intervention that raises trust regardless of correctness is not desirable. Finally, execution of the experiment could have been improved. In particular, our maritime autonomy SME noted: The Mode should have reflected the simulation events; The LRAD light should have illuminated during the approach phase with an audio warning; The subjects should have been trained on the nonlethal defense functions.
Semantic Modeling: A potentially significant drawback to our approach is the manually-intensive nature of the semantic modeling effort needed to populate our knowledgebase. Identifying ways to automate this process is a key area of potential future work related to this effort.
CONCLUDING REMARKS
We draw the following specific conclusions based on the quantitative results reported above. First, the experimental and control groups do not significantly differ across any attribute in terms of their responses to the technology attitudes questions. The experimental and control groups do not significantly differ across any non-Group attribute in terms of their responses to the study questions with the exception of gender differences for construct. Construct reliability is low in some cases, indicating the need for a prior pilot study to tune the psychometric instrument. We accept the null hypothesis for construct 4 and reject it for constructs 1 and 2, as predicted under our assumptions. We cannot reject the hypothesis associated with construct 3, although this is a very marginal case. The results of construct 5 are contradictory to our expectations. Overall, we conclude that the Why Agent approach does increase user trust levels through decision transparency.