A lack of algorithmic transparency is a major barrier to the adoption of artificial intelligence technologies within contexts which require high risk and high consequence decision making. In this paper we present a framework for providing transparency of algorithmic processes. We include important considerations not identified in research to date for the high risk and high consequence context of defence intelligence analysis. To demonstrate the core concepts of our framework we explore an example application (a conversational agent for knowledge exploration) which demonstrates shared human-machine reasoning in a critical decision making scenario. We include new findings from interviews with a small number of analysts and recommendations for future research.
• Information systems → Information systems applications;
With advances in artificial intelligence (AI) technologies, reasoning like cognitive processes are no longer restricted to the human mind. In cases this has led to shared human-machine reasoning, where both parties are able to explore information by interpreting, inferring, and learning, before reaching a common understanding. One example of shared reasoning can be found in conversational agent applications which reason from semantic knowledge graphs. These applications are the focus for this paper, however, the principles identified within the framework presented also apply to other types of application which exhibit shared human-machine reasoning for critical decision making. 1.1
Conversational agents, namely applications which allow users to communicate with machines through natural language, are becoming commonplace in many business and home environments. Technologies such as Google Home, Siri and Amazon Alexa present us with an easy way to access music, films, or plan our day. Many services, for example banking, have incorporated chatbots into existing processes to manage interactions with customers, including to direct them to the right information or department. This saves companies money and can save customers time waiting in a queue.
Typical applications for conversational agents tackle concise user tasks for mundane processes which can be translated to a finite set of user intentions. Here the risks of an incorrect or misleading response are low and the resulting consequences limited, particularly given the ease with which a user can validate results against an expected and desired conclusion to their interaction. Take the example of a user wishing to listen to a playlist of a specific music genre. They can task the conversational agent with finding and playing such a playlist. It does not necessarily matter to the user exactly how the agent has reasoned what music should be played in the playlist, what the track order should be, or many other aspects. The users intention is straightforward and the consequences of an unwanted track are limited i.e. the user will make an assessment of the track as soon as they hear it, at which point they may decide to skip the track, or ask for a diferent playlist. The result of the interaction therefore provides some information to the user which they can easily interpret, validate and make an appropriate, timely, response. If the user were repeatedly presented with the wrong genre of music, however, their need to understand the underlying algorithmic process and constraints will become more important.
We believe there is desire and benefit to using conversational agents for natural and shared human-machine reasoning in applications for which the interpretation of responses are high risk and high consequence, such as critical decision making environments. However, there are significant diferences between the requirements for this and typical conversational agents, which must be considered in design.
Consider the example where a user wishes to perform analysis
of a certain entity and explore it’s associations with another entity
through a conversational agent. The agent can provide responses
and a simple explanation. This interaction can be an example of
shared reasoning, where the user is directing the conversation based
upon their own thoughts and the agent is interpreting the user’s
intentions and objects of interest, before making inferences to extract
data to include in it’s response. There are dangers present where
actions which are informed by shared reasoning are high risk and
high consequence. For example, if through shared reasoning the
user incorrectly confirms their hypothesis and directs a subsequent
action, such as arresting an innocent person or launching an
unnecessary ofensive operation. Some specific risks include; the way the
agent interprets subtleties in the intention of the user request, the
introduction of bias, the way the agent explains a complex series of
connections, the way it translates the uncertainty involved in those
connections and exposes missing data, and the propagation of
uncertainty along the conclusion pathway. Additionally, the algorithm
selected by the agent to explore data influences which pathway is
described, this needs explaining to a user. Errors in mitigating any
of these risks could lead to a mistaken, or deliberately manipulated,
action with adverse efects. Unlike typical conversational agent
applications, the user does not necessarily have an expectation or an
easy way to validate results. They also need to access, understand
and interpret the evidence underpinning any response. A simple
chat with summarised text responses cannot fully address the risks
noted above and therefore lacks transparency and a mechanism that
supports situation awareness, rigour, visible reasoning, and sense
making. Wong et al. [
To date, research into conversational agents has looked to improve the agent itself, by making it human-like or its responses more contextual. This paper however considers the vulnerabilities of shared human-machine reasoning and the requirements for visibility of interactions, identifying key considerations. A framework is presented, with input from experienced military intelligence analysts, of foundational research areas for developing shared humanmachine reasoning applications, such as conversational agents, for evidence based critical decision making environments.
In situations where a user aims to retrieve information or data,
particularly if they do not already know how to access it or what
they are looking for, a conversation can be the prefered way to
reach their desired outcome. The conversational agent provides a
gateway to the information they seek, extracting the users
intentions through a two way dialogue, then translating these intentions
into query language and describing the results back. For many users
this is a far more intuitive approach to retrieve information [
There are a variety of approaches to developing conversational
agents, however the neural models which power most of the
commercially available smart assistants lack a sense of context and
grounding which their human interlocutors possess [
Studies to date have focused upon the development of methods
and technologies for conversational agents to deliver believable and
contextual conversations. While potentially extremely helpful, this
paper proposes that the use of conversational agents to interpret
intelligence observations through semantic knowledge graphs can
introduce risks due to a loss of situational awareness (SA). SA plays
a vital role in dynamic decision making environments [
We propose that there are some critical vulnerabilities in the field of intelligence analysis and other evidence based decision making environments. These are magnified by the use of applications which share reasoning ability between both human and machine, such as conversational agents.
Research is required to reach an understanding of how machine reasoning can be introduced alongside human reasoning in a way which mitigates vulnerabilities, whilst still exploiting the significant benefits of more natural and powerful interactions between humans and data. This paper delivers a framework for providing algorithmic transparency, with associated research areas which are the foundation to exploring how applications can be designed to deliver shared human-machine reasoning. We examine the example of conversational agents used in conjunction with semantic knowledge graphs. The research is specifically tailored to an evidence based decision making scenario (intelligence analysis) informed by semi-structured interviews with analysts who have experience working in intelligence environments. The framework helps segment key considerations and vulnerabilities for agent design and identifies challenges and areas for further research. 2
The framework proposed in this paper links various research topics which are each significant in their own right. We take a broad look at previous work on one example of an application technology which provides shared human-machine reasoning, that of conversational agents for querying semantic knowledge graphs. There are important aspects of our framework which have not received attention in research to date, specifically an understanding of how to make machine reasoning more visible to a user when intertwined with human reasoning. This is crucial when agents are used in decision making environments and is a central question for our research. 2.1
The desire for humans to be able to speak with machines through
human-like language has been around for some time, with relevant
research published as early as the 1950’s [
Early chat interfaces, notably ELIZA [
One area where conversational agents have been applied to
higher risk and more uncertain environments is in health care. It
is dangerous in decision making environments if a conversational
agent is able to bias a decision, or influence the decision maker.
Robertson et al. [
The issues of capturing context, managing inconsistency in responses, providing trust and confidence, and removing bias informed by training data, can be mitigated if we provide the agent with a foundation for knowledge from which it can extract meaning and content deterministically. This is the case if we allow the conversational agent to interact with a semantic knowledge graph to perform specific search and reasoning tasks. 2.2
Hoppe et al. [
Semantic knowledge graphs can be complex to query,
particularly with more advanced graph traversal and query methods. To
query an RDF graph, for example, we can use the SPARQL query
language [
The power of semantic knowledge graphs has led them to
become crucial to supporting many AI applications, including
question and answer systems [
Conversational interfaces require the ability to identify a users
intention and intention definition is therefore an important
consideration, as these will trigger the relevant action in response to a
query. Intentions may be domain specific, for example an
intelligence analyst may wish to perform particular tasks which do not
translate to other environments. To identify possible tasks we may
look to use existing work, such as the task taxonomy for graph
visualisation presented by Lee et al. [
Intelligence analysts are crucial to military decision making
because they provide situational awareness (SA) to commanders. A
key method applied by military analysts is situational logic, this
underpins much of an analysts standard process to achieve an
understanding of a situation. Heuer [
Looking to ACH allows us to understand critical aspects which feature in an analysts thinking. Such as, the evidence which underpins hypotheses and the related strengths and weaknesses, the propagation of weaknesses in a fused evidence picture, the ability to compare the strength of multiple alternative hypotheses, the relative impact of removing pieces of evidence upon hypotheses, and the relative influence of diferent hypotheses and evidence upon possible narratives.
While the principles of ACH are sound, in practice it is flawed.
ACH is typically a matrix table approach and the table display itself
is limited in the amount of information which can be clearly
articulated, so the text is summarised and lacking in surrounding context.
Additionally, it introduces an arbitrary structure to hypotheses and
evidence. This can produce adverse cognitive efects where the way,
and order, in which hypotheses are listed can afect how much they
are considered. Additionally, if analysts rely on their experience
alone when assessing possible hypotheses they are prone to bias.
“Psychological research into how people go about generating
hypotheses shows that people are actually rather poor at thinking of
all the possibilities. If a person does not even generate the correct
hypothesis for consideration, obviously he or she will not get the
correct answer.”[
As Wong and Varga [
In time sensitive scenarios, for example if an analyst is tasked
to understand a situation prior to an imminent military action
with little lead time, situational awareness is particularly
important. Thomas and Cook [
While traditional interfaces to provide this are complex and lack lfuidity, a natural language approach to interactions could be the solution. An analyst can easily interact in natural language with a conversational agent, in a timely fashion, to explore the graph before concluding with a plausible narrative and achieving SA much more quickly. Crucial to providing true SA, however, is that an analyst can easily and visibly understand how and why a conversational agent has provided the responses they have. In a sense, this requires visualisation of the agents conclusion pathway. To date, as far as we are aware, there has not been research conducted to understand the vulnerabilities to introducing conversational agents in the field of intelligence analysis, nor design steps which could help mitigate risks. 3
We have produced a research framework to identify the design requirements for applications which involve shared human-machine reasoning for use in decision making fields, such as intelligence analysis. The framework is underpinned by an exploration of existing literature and unstructured interviews with a small selection of experienced military intelligence analysts. We provide an example aid to help demonstrate the ‘visibility’ aspect of the framework.
Approaches to date which develop machine reasoning through
conversational agents, including [
The framework diagram presents the relationship between
machine reasoning, shown as a ‘black box’, and human reasoning.
There has been much work and discussion describing machine
learning methods as a ‘black box’[
The human user requires explainability of the cognition taking
place within a ‘black box’ (XAI). XAI has received a large amount of
attention in recent years, with a focus upon understanding machine
learning classifications. The meaning of explainability is key to how
it is designed into interfaces. Current XAI research, as reviewed by
Gilpin et al. [
For applications which allow fluidity and rigour in shared
humanmachine reasoning, it is not enough to merely provide explainability
of the internal workings of a system through result metrics. There
needs to also be visibility of what reasoning the machine is doing
and why, how it’s reasoning fits within the fluidity and rigour model
[
This paper presents a simple scenario to demonstrate how the visibility of machine reasoning can be designed within an application alongside human reasoning. The example considers a conversational agent query system for a semantic knowledge graph. 3.1
Example: Conversational agents for graph exploration 3.1.1 Analyst Interviews. In order to understand what requirements exist for visibility of machine reasoning in conversational agent responses, and to map functional relationships against the goals and constraints, we first need to understand what visibility means in the context of intelligence analysis. Much work has been done to understand the general thought processes applied by analysts, however to date research has not considered the interaction between an analyst and conversational agent and the impact of shared reasoning. We begin this discussion by conducting interviews with a small number of experienced intelligence analysts, to understand for what tasks conversational agents could help, any vulnerabilities which exist, and for each task what visibility means to an analyst.
The analysts interviewed for this study identified areas where a natural language interaction with data could be extremely beneficial. For example, when performing situational logic analysis analysts apply a process of hypothesis creation, testing, and comparison, related to real world entities. Analysts formulate hypotheses, often linked to future strategy, impact, events, and activities i.e. ‘that Person X and Person Y are travelling to Location C for Event A, which will have impact Z’. A key requirement is to understand the connections between these entities and the surrounding context, in particular related to the key points of connection. Considerations need to take into account the provenance and certainty of observations and the impact of data changes, including unexpected observations, upon the analysts hypothesis. For example, if by accounting for uncertain data an alternative hypothesis presents itself as most likely, or if additional data is included after an update in the situation which changes the overall picture.
A semantic knowledge graph approach can provide much needed persistence of data, with rigour in the capture of contextual information. However, a graph increases in complexity and scale as it evolves over time and it becomes increasingly dificult for an analyst to assess their hypotheses against it. The analysts interviewed in this study identified that current analytics tools to explore graph data are often over-complicated, with significant learning required to understand how to perform functional interactions for filtering and configuration. Resulting visualisations are then overloaded with too much information. Additionally, there is insuficient explanation of the meaning and constraints of functionality where analysts are interested in “function rather than mechanics”. This leads to a barrier to analysts using tools because they find them of-putting and unnatural. A conversational approach could provide access to powerful functionality, but with less complication and learning required, and greater understanding of methods through two way dialogue. This can help an analyst to explain dificult concepts, such as their level of risk aversion when considering appropriate evidence across a conclusion pathway.
Analysts identified a number of other benefits to using conversational agents, beyond being more natural to interact with. When exploring data they can allow for timely, coherent, and regular searches, where an ongoing conversation is maintained and the agents memory can be accessed and utilised. This capability would be useful for analysts who want to ask questions such as, “have there been any more visits to Location X?”. A key feature of conversational agents, identified by the analysts, is the ability to clearly articulate an audit trail to explain how information was found in the process of an investigation. This trail helps provide ethical accountability. Each interaction with the agent provides a time-stamped message, including associated information found and the state of the graph data at that point in time. Analysts can be influenced by bias and by capturing their line of questioning an agent could aid an analyst to consider alternative possibilities. A conversational agent can allow for a deeper explanation of findings, including feedback and suggestions for selected alternative inquiries which is based upon a knowledge of the underlying and surrounding data. To provide a raw picture of all this data to an analyst would be too voluminous and complex for them to digest manually. In this way an agent can aid an analyst to identify alternative hypotheses which are not restricted to their own experiences or assumptions.
This reasoning aspect of conversational agents goes beyond a simple query tool, to incorporate elements of sense making and inferencing, and there are vulnerabilities to doing so. Analysts identified several risks when using conversational agents. These represent important problems which require mitigation to confidently apply conversational agents in the area of intelligence analysis. If an agent is able to guide an analyst by refuting and suggesting alternative hypotheses then there is potential for the analyst to be mislead. An agent could guide an analyst towards inaccurate conclusions in a way which is dificult for the analyst to refute, given the complexity of the underlying graph and the fact that it is not visible to the analyst. If an analyst is interested in key connections in a path they are vulnerable to the agents choice of path, where there is an adverse impact if non-relevant connections are identified as key. Within the conversation text itself it is dificult to describe the provenance and certainty of information as well as the key information, particularly for many connecting observations. This leads to textual responses which are either hard to interpret and overload the analyst with too much information, or are summarised so that important information can be missing.
To help mitigate some of these issues the analysts interviewed described what requirements for visibility in conversational agents they have, in association with their goals for a system. Analysts felt an understanding of the underlying processes and algorithms applied by conversational agents should focus upon the functional meaning, in light of the intelligence analysis task, rather than any mathematical method. Specifically, analysts were interested in ‘how’ and ‘where’ a conversational agent was exploring the graph and ‘why’ it deemed information to be interesting, including the specifics of the sub-graph extracted, such as the provenance, history and confidence of observations. Analysts emphasised the need for a balance between identifying the ‘key’ underlying data observations or entities, while not overwhelming the user, and also providing a contextual understanding of what ‘key’ means. Analysts were particularly interested in allowing for human reasoning of more intangible observations, alongside the deductive rigour of the machine. This would include an understanding of weaknesses, missing data, and ability to apply intuition. Additionally, visibility of past conversations, the current state of the conversation, and the state and evolution of the graph at each stage, is important for auditing purposes and ethical accountability. 3.1.2 Example Case: Scenario. A conversational agent’s interpretation of a user’s intention will inform which thought process, or algorithm, is applied to deliver a response. For example, a user’s query to find relationships between two diferent entities may invoke a ‘find connections’ intention. This approach to match query to intention is typically performed using machine learning techniques for classification. Accurate conversational responses therefore begin with an assessment of possible intentions and an accurate machine learning intention classifier, and later involve the accuracy of entity and relationship extraction, the building of knowledge graph query syntax, subjectivity in which algorithms meet which intentions, variability and constraints of heuristic methods, and the reliability and completeness of the knowledge graph itself. There are, therefore, uncertainties which need to be addressed within a user interface. For example, what is the impact upon an analysts decision if an agent interprets a subtly diferent intention, with diferent goals and constraints, and employs a diferent algorithm? How many intentions should an agent allow? How distinct do they need to be to mitigate uncertainties? Fundamentally we need to understand how ‘visibility’ of agent thinking can be provided to an analyst.
In intelligence analysis it is crucially important that analysts can fully interpret the information and evidence which is guiding their decisions. Without visibility of their ‘conclusion pathway’, i.e. the pieces of information which are informing their acceptance or rejection of hypotheses, they are vulnerable to mistakes, personal and experiential bias, and deception. The use of conversational agents presents challenges to the visibility of thought processes, by handing over some of this processing to an agent. The nature of chat bot interfaces, where a user types a message and receives a text reply can encourage a narrow focus for investigation. A user will typically receive responses based upon their questions, with little awareness of data observations which lie on the periphery of their line of questioning (reduced SA). The bigger picture is hidden and opportunities for deception are increased.
Potential vulnerabilities are best explained with an example, as described in Table 1. All of these queries relate to a straightforward ‘connections’ intent which finds a path between pairs of entities. We are provided with information akin to ‘explainability’ in the Algorithmic Transparency Framework (Figure 2), where the internals of the system (the graph connections which are found between our entities of interest) are described in natural language.
Scenario: An information request is received by an analyst to understand the supply of equipment and ingredients to produce a weapon (‘X’) to ‘Organisation X’. It is suspected that an individual with access to the necessary equipment (a scientist) is supplying the goods.
The conversational agent has identified that Person A is a scientist. It has also identified that Person A is connected to Organisation X, therefore the agent can perform deductive reasoning to find that Organisation X is linked to a scientist. Furthermore, Person A appears a good candidate to suspect in supplying Organisation X with weapon X. We can see that Person A is connected to both organisation and weapon, and we have an explanation for how. However, any uncertainty or alternative narratives within the data are not presented to us and the response is narrowly framed by our line of questioning. There is a lot more information needed by an analyst in order to understand these connections. This is because the explanation does not take into account the requirements for ‘visibility’, including how the conversational agent maps it’s reasoning to the analyst’s goal, or an understanding of the constraints present in the reasoning approach. In this case, the analyst wishes to test their hypothesis and to allow for reasoning about alternative possibilities. The system, however, is constrained to apply a single shortest path algorithm which traverses the graph and returns data to the analyst. Observations which lie on longer paths are ignored and an analysts true situational awareness is reduced. Even worse, data can be introduced to mislead an algorithm, and thus manipulate the results presented to a user. This vulnerability ties closely with confirmation bias, where by understanding how the algorithm will look across data points it may be possible to introduce data to reinforce bias.
This is a simplistic example, but it helps demonstrate some of the pitfalls with chat bots and data filtering algorithms which reduce situational awareness. This is particularly the case in more realistic scenarios or if advanced graph traversal algorithms such as probabilistic methods, heuristic approaches to explore multiple paths, or pattern matching methods are applied. As a situation becomes more complex, for example with observations which arise from diferent sources and demonstrate varying levels of confidence and reliability, designing for visibility of machine reasoning becomes critical to providing a clear picture which empowers an analyst to perform human reasoning. Much research has looked to develop explainability of machine learning algorithms which present the user with mathematical representations, for example, of how features in the model relate to classification results. Little has been done, however, to understand how visibility should be provided for these models within the context of their use, nor for how knowledge graph traversal algorithms are applied and explained to a user in tandem with conversational agents. These are key areas requiring further work. 3.1.3 Example Case: Visibility. The example visual aid shown in Figure 3 revisits the scenario described earlier in Table 1 and accompanies the textual responses. Figure 3 displays the path found by the agent in addition to other nodes which are close by (within a single edge from the path). There is a key for the colour of nodes provided in the interface which is linked to their semantic class. The path found is akin to the agents conclusion pathway as this traces the series of observations which connect the two entities of interest. The extra relationships which are not on the path provide additional context and better mapping between the algorithm functions and analyst goals for the system. By providing this addition to supplement the text in the visual form of a sub-graph network, analysts can better understand the conclusion pathway taken by the agent and this gives them greater visibility of the context in which deductions are made by the agent. For example, the agent has deduced that Organisation X is linked to a scientist, Person A, through Person B. However, this deduction ignores other possibilities for a relationship between Person A and Person B which do not involve Organisation X, for example membership at the same gym. Additionally, the agent’s reasoning does not consider other entities which have similar access to equipment as scientists, for example university students. An analyst, when faced with this graph, could perform more intuitive abductive reasoning to question the role of the university. The systems constraints are more obvious, where an analyst can identify paths which have not been explored and nodes which could be relevant but have been missed. Without visibility of the machine thought processes, the human cannot interpret, critique, nor build upon machine reasoning.
The additional visual aid is helpful for an analyst to sense check the agents thinking, providing some ability to verify findings by comparing alternative hypotheses which may arise from the inclusion of close nodes. Take the first query, for example, where the agent finds a connection between Organisation X and a scientist (Person A). The agent can explain that the most critical node in the path is Person B and an analyst therefore must be confident in the association between Person A and Person B to be confident in the path as a whole. By considering the surrounding context we see that both people are members of the same weights gym. There is a plausible association which is not related to Organisation X. Likewise, if we take a look at the most critical node linking Person A to Weapon X, which is Equipment A. Person A has purchased Equipment A which is used to produce Weapon X, however it is also used to produce Wood Glue and Person A has participated in a woodwork training course. Again, they have a plausible reason for making this purchase which is not related to Weapon X and the conversational agent has ignored this.
The graph in Figure 3 provides the additional context that
Equipment A is also owned by University X. If an analyst explores this
connection they will see the display shown in Figure 4. Figure 4
shows just the sub-graph for the agents response to “is there a path
between University X and Organisation X?”. We find that there
is indeed and again Person B is the most critical node, Person D
and Person C (a student at the university) are also important. The
addition of the visual aid to the conversational responses helps
to overcome some of the issues identified by analysts, specifically
by providing visibility of the conclusion pathway, additional
context and algorithm constraints, and by highlighting key points of
connection within the graph. The conversation text itself can also
pull out key vulnerabilities, for example where a node is
particularly important to a path based upon its betweenness centrality
score [
The example visual aid is a helpful start, however it is flawed in
many ways. An important issue facing analysts is information
overload. In the simple example provided it is easy for an analyst to
understand the graph visualisation, however in a more realistic
scenario the complexity and scale of the graph would present a
significant challenge. To tackle this problem more advanced
traversal algorithms are required in addition to utilising the reasoning
power of semantic knowledge graphs. Approaches such as concept
lattice analysis [
A more realistic scenario would require a smarter extraction of
the surrounding graph context beyond that shown here, including
uncertainties in the data and better definition of what we mean by
‘close’ to the path. Rather than simply displaying additional
connections along a conclusion pathway, we need a method to display the
important alternative connections which if considered could afect
our hypotheses. To do this requires an understanding of how
analysts make inferences across graphs. Wong and Varga [
A greater understanding is needed of what ‘visibility’ means in the context of intelligence analysis tasks, goals and constraints, therefore more detailed studies should look to explore this concept with analysts.
The framework proposed in this paper has wider implications for the design of shared human-machine reasoning applications beyond the conversational agent example discussed. Future work should therefore also look to see how this framework can be applied to the design of other applications which provide shared human-machine reasoning, for example applications which include reasoning through machine learning.
There is a place for conversational agents in the field of intelligence analysis and, if designed carefully, they could deliver significant advantages to analysts compared to current practices and analytics tools. There are, however, risks to using them in a decision making environment where visibility of the reasoning, evidence, goals and constraints which underpin analysis is crucial, in addition to the explainability of a result. We provide a design framework which highlights important research areas to explore when looking to develop applications for shared human-machine reasoning, in fields which require evidence based decision making. Future work should look to apply the ‘Algorithmic Transparency Framework’ to the design of applications in real world scenarios and to tackle the challenges identified in this paper.
This research was assisted by experienced military intelligence analysts who work for the Defence Science Technology Laboratory (Dstl).