The adoption of complex artificial intelligence (AI) systems in environments that involve high risk and high consequence decision making is severely hampered by critical design issues. These issues include system transparency which covers (i) the explainability of results and (ii) the ability of a user to inspect and verify system goals and constraints. We present a novel approach to designing a transparent conversational agent (CA) AI system for information retrieval to support criminal investigations. Our method draws from Cognitive Task Analysis (CTA) interviews to inform the system architecture, and Emergent Themes Analysis (ETA) of questions about an investigation scenario, to understand the explanation needs of diferent system components. Furthermore, we implement our design approach to develop a preliminary prototype CA, named Pan, which demonstrates transparency provision. We propose to use Pan for exploring system requirements further in the future. Our approach enables complex AI systems, such as Pan, to be used in sensitive environments introducing capabilities which otherwise would not be available.
• Computer systems organization puting.
Artificial intelligence (AI) based conversational agent (CA)
technologies are complex systems which are increasing in popularity
[
However, for complex applications to be used in high risk and
high consequence domains, transparency is crucial. If an analyst
misinterprets system processes and information caveats when
retrieving information in a live investigation then the impacts can be
serious, for example leading to errors such as directing resources
to the wrong location, or failing to find a vulnerable victim.
Misinterpretation is a particular risk where there are subjectivities,
such as when using a CA to interpret human intentions. We define
transparency as the ease with which a user can (i) explain results
provided by a system, in addition to (ii) being able to inspect and
verify the goals and constraints of the system within context [
The domain of intelligence analysis is broad and diverse,
therefore we have focused upon a narrow spectrum of criminal
intelligence analysis and information retrieval tasks. To develop the
prototype we first gathered and analysed data from CTA interviews
with operational police analysts to identify the way they recognise,
construct, and develop their questioning strategies in an
investigation. We captured important attributes within the interview data
linked to the Recognition-Primed Decision (RPD) model [
We also present findings from scenario based interviews with a diferent set of operational analysts. In these we looked to identify what information is required in explanations of the various components that form a CA. The interview data is distilled to distinct statements made by the analysts and further refined using Emergent Themes Analysis (ETA), to form an explanation framework covering CA system components (contribution 2). We describe a novel CA prototype, named Pan, (contribution 3) designed to address transparency issues, using our findings from the two sets of interviews.
The work discussed in this paper provides a preliminary investigation of transparency issues for information retrieval with complex systems, in the specific domain of criminal investigations. In future work we plan to probe further and to evaluate the prototype through experimentation with intelligence analysts. 2
Analysts play an important role in criminal investigations, as the
results of their analysis underpins decision making by police
commanders. For example, intelligence analysis directs the prioritisation
of lines of inquiry in an investigation and assessments of key
suspects. The process of intelligence analysis involves repetitive and
intellectually non-trivial information retrieval tasks where “each
piece of insight leads to intense periods of manual information
gathering”[
Manual formulation of query syntax or interactions with traditional analysis tools can be cumbersome and time consuming. A more natural interaction, which removes the requirement for analysts to translate their questions into restrictive syntax or structures, could speed up this process significantly. If an analyst were able to communicate with their data in the same way as they do with their colleagues, through natural language, then they could achieve significant time savings and speed up investigations.
We define typical CAs as being able to understand users by
matching their input pattern to a particular task category
(intention), for example through ‘Artificial Intelligence Markup Language’
(AIML) [
Some research to date has touched on the need for a CA to be
able to explain its responses. Preece et al. describe the ability to
ask a CA ‘why’ they have provided a particular response, so an
analyst can obtain the agent’s rationale. An explanation could be
“a summary of some reasoning or provenance for facts”[
Intelligence analysis is a field where analysts operate in complex,
subjective, uncertain and ambiguous environments, and a simple
explanation of the data or a model which defines a response is not
enough to satisfy their needs for understanding. For example, if
the method applied by the system presents significant constraints
of which the analyst is not aware. Previous research has looked
at this issue and developed a design framework for algorithmic
transparency [
In a policing scenario, when an analyst is presented with a situation
they immediately look to make sense of it. They apply experience
to recognise aspects of the situation and construct a plausible
narrative explanation with supporting evidence. Klein [
We desire a CA that can recognise situations in a similar fashion and respond to analyst questions appropriately. We also need analysts to recognise the behaviour of a CA in each situation when it attempts to understand and respond to the analyst. We propose that the RPD model provides a useful foundation to designing CA intentions so a CA can recognise analyst inputs, in addition to an explanation structure so that its behaviour can also be recognised and understood by the analyst. 3 3.1
We conducted Cognitive Task Analysis (CTA) interviews, applying
the Critical Decision Method [
For each interview we attempted to understand how analysts
identified what was happening and the information they needed to
advance their investigations. Critical to this process is how analysts
recognise and respond to situations. We analysed analyst interview
statements, structuring them against the Recognition-Primed
Decision (RPD) model [
the motive, the risk to the victim, and possible suspects
Does the victim have any previous convic- Find convictions? tions
missing.
Thought he had been kidnapped due to witness report. known to be vulnerable
Why? a concise and clear representation of an analyst’s behaviour when retrieving information, and thus can be used to give an explanation structure for their intentions. We can design system processes that mirror this representation.
In Table 1, we also show how we extract individual questions asked by analysts from interview statements and can structure them against the RPD model. Furthermore, we can interpret the RPD attributes more generically to suit multiple questions of the same type. During the interviews each analyst provided many examples of their information needs and the questions that they asked when performing an investigation. For example, one analyst stated that “I looked through every database for the victim’s name, custody records, PNC (Police National Computer), stop and search, vehicles he drove, to see if he had been stopped and searched with other people in the vehicle and if they had been named.” [CTA: A1, 15:00]. From this statement, we can extract a number of questions posed by the analyst that could be directed towards a CA, including “how many vehicles have travelled to the victims address?” To answer this question the analyst provides cues for ‘vehicles’, ‘travelled’ and ‘victims address’. Their goal is to retrieve summary information i.e. ‘how many’, and they are interested in finding a specific pattern of data in the database, which connect the cues. Table 2 provides a diferent example, with generic RPD attributes.
In this paper, we present how RPD attributes can be used to dynamically model analyst intentions for searching and retrieving information, through Formal Concept Analysis (FCA). FCA is an
Why?
tim Name, Activity) Present confirmation Expected that input details and pattern exist Perform adjacent information search for entities extracted Retrieve list for further exploration.
To find new lines of inquiry.
analysis approach which is efective at knowledge discovery and
provides intuitive visualisations of hidden meaning in data [
The lattice, as shown in Figure 1, presents distinct object
groupings. The final layer of concept circles are complete concept
intentions, where all parts of the RPD are considered. The circles are
sized based upon the number of associated questions. We can see
that three questions in our set can be answered by combining the
highlighted attributes. These attributes can answer the question,
‘how many vehicles are in our database?’, with ‘vehicle’ as a cue.
The CA looks for adjacent information i.e. where there are instances
of the class ‘vehicles’, presents a summary count, and retrieves a list.
To provide transparency we propose we can simply present what
attributes, and therefore functional processes, underpin a concept
with their descriptions. Our model-agnostic and modular approach
is akin to what Molnar [
We interviewed four intelligence analysts with more than 10 years operational experience, from a diferent organisation to those interviewed previously. We aimed to explore their requirements for understanding the responses and processes of a CA in the context of a criminal investigation. Each interview lasted an hour and we presented interviewees with a series of questions and corresponding CA responses with two explanation conditions, switching the order of presentation. For one condition, responses described the data alone (1) and, in the other condition, the data and system processes (2). We were not attempting to test the diferences between conditions, rather we used them as a starting point from which we could explore additional needs. Throughout interviews a single researcher took extensive notes from which individual statements were extracted. In total there were 114 distinct statements extracted, with counts for each analyst ranging from 24 to 34. 4.2
To analyse the statements we used an approach called Emergent
Themes Analysis (ETA), as described by Wong and Blandford [
An analyst’s ability to have clarification, verification and
justification of system processes is crucially important, as identified by
all analysts interviewed. This finding supports the framework for
providing algorithmic transparency presented by Hepenstal et. al
[
In Table 4, we display the framework areas and related
subthemes that emerged from ETA. Specific areas in the explanation
framework can be linked to existing models for sensemaking, such
as the Data Frame Model [
We have developed an initial prototype CA application called Pan,
which uses FCA to define the diferent intention concepts to which
it can respond. The objects (questions) which are attached to a
concept are used as training data for machine learning text
classification, so that a user’s question can be matched to an appropriate
intention. Each intention concept has associated attributes and
we have developed methods to handle these as individual models,
which create query syntax and interact with the database. In this
way, FCA can combine multiple distinct combinations of attribute
models flexibly to meet diferent analyst intentions. We propose
that by combining model-based attributes with FCA to define
intention concepts we provide a highly flexible approach to developing
CA intentions. The objects and corresponding RPD attributes are
critical for providing visibility to an analyst for the responses given
by a CA and are akin to explainability scenarios i.e. “narratives of
possible use that seek to answer the questions: who will use this
system and what might they need to know to be able to make sense
of its outputs?” [
Our work to identify the core understanding needs for CA components has helped to inform the design of explanations for diferent parts of the system, for example, when the CA matches user input to an intention concept, triggers associated attribute models, and responds. The explanation provides information required for an analyst to understand the CA component themes of ‘Data’, ‘Extracted Entities’, and ‘Response’. As an analyst types their query and entities are extracted, they are provided with identifier information where possible. We have also designed for the ability for an analyst to inspect and verify system goals and constraints. In our prototype we allow the user to step into the intention concept which has been triggered through a dialog window, so they can inspect and verify clear textual descriptions with our explanation structure, of the cues, goals, actions, expectancies and purpose of the intention. For example, when a concept triggers the action for ifnding single shortest path connections between instances, the analyst is presented with a description that includes any constraints to be wary of. Specifically, that it will not find longer paths or consider multiple routes. These caveats will impact how the analyst considers any information returned or how to rephrase their question. The attribute descriptions for each RPD module hang together as a narrative, akin to explainability scenarios. We intend to run experiments with Pan and operational intelligence analysts to validate our understanding of explanation needs and our RPD explanation structure for CA intentions. 6
In order for AI systems to be used for high risk and high consequence decision making they must provide transparency of their reasoning. As put by one analyst, “[the principal analyst] said none of my analysts would stand up in court where the beginning point of their evidence is an algorithm.” [CTA: A4, 32:30] and that “You have to be able to trace it (your reasoning) all the way back to evidentially explain why you did each part... an analyst always has to justify what they have done, so does a system.” [CTA: A4, 35:00] We believe that Pan addresses these issues by providing algorithmic transparency of its reasoning, using an architecture that aids recognition and explanations that meet our explanation framework. Early feedback from analysts on our approach is positive, opening routes for Pan to be tested in high risk and high consequence application domains where traditional CAs would not be deployed. 7
In this paper, we describe our approach to capture and model analyst thought when retrieving information in a criminal investigation. We also present analysis to understand their needs for explanations from a complex CA system. Finally, we describe a prototype CA which incorporates FCA and RPD to build intention concepts and is therefore, we believe, transparent by design. We plan to evaluate the transparency impacts of our approach to intention concept design, gather additional requirements, and to validate our explanation framework through experimentation with operational analysts. To date we have not explored how a CA should present its responses to an analyst. Thus, we will look to explore how explanations are communicated, such as the specific textual or visual method.
The role of investigation scope was prominent in CTA interviews with analysts, where their questions were framed by the initial scope, thus introducing the risk that important information beyond the scope is missed. We will consider how CAs can help mitigate the constraints of investigation scope, through machine reasoning for example. Analysts expressed the desire to avoid obvious follow up questions, so it would be helpful for a CA to predict and explore additional questions autonomously. One approach for this is to model investigation paths as a Bayesian network. Transparency is a critical issue in autonomous systems and our explanation structure could help understanding by aiding the explanations of system behaviours across model states.
This research was assisted by experienced military and police analysts who work for the Defence Science Technology Laboratory (Dstl) and the National Crime Agency (NCA) in the United Kingdom.