In general, threats can be loosely divided into two categories known threats and unknown threats. Traditional threat detection systems are limited to the identification of known threats that have been previously encountered and labeled by a security expert. These supervised learning systems are able to learn to detect and identify known threats but are unable to react to unknown threats. To this end, we have developed an unsupervised learning anomaly detection system to identify anomalous behavior without training data. Our system's interactive interface supports human-machine teaming to classify these identified anomalies as threats or benign events; however, system transparency is required to enhance operator trust and improve their feedback into the system. Transparency in this case is particularly challenging as our anomaly detection framework is based on algorithms which are inherently hard to explain (neural networks). In this paper, we introduce a realworld task and system that requires transparency, and we propose explanation methods for increasing the transparency of our threat detection tool alongside a user study for evaluating these explanations.
• Human-centered computing~Human computer interaction (HCI) • Human-centered computing~HCI design and evaluation methods • Human-centered computing~Interactive systems and tools
IUI Workshops’19, March 20, 2019, Los Angeles, USA Copyright © 2019 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
Forward deployed military installations face unique challenges when automating threat detection in security monitoring systems. In particular, development of a general framework for identifying emerging security threats poses two technical obstacles: (1) the security framework must be robust to an environment where “normal” activities are initially unknown and (2) the framework must support collaboration with operators to determine what types of anomalous behavior constitute an actual threat. To this end, we have developed an unsupervised anomaly detection system, DAART, (for Detection of Anomalous Activity in Real Time), to identify anomalous behavior without training data. DAART’s interactive interface supports human-machine teaming to classify these identified anomalies as threats or benign events.
Traditional threat detection systems are limited to the identification of known threats that have been previously encountered and labeled by a security expert, such as a man wielding a gun or an intruder in a restricted location. The unsupervised nature of the DAART system additionally supports identification and action on unknown threats, which is necessary to adapt to ever changing environments. A by-product of this, however, is an initial trend towards recall over precision, meaning many benign activities may be alerted to the user. DAART’s Active Learning component learns from operator feedback in the form of accepting or rejecting alerts (alert-level feedback) to better distinguish benign anomalous behavior from threats. A human-inthe-loop system, such as this, requires system transparency to improve operator trust, accelerate operator workflow, and better enable operators to provide the valuable feedback required to improve the system’s threat classifications.
A threat detection system may err in two distinct ways: (1) false
positives in which benign behavior is predicted to be a threat and
(2) false negatives in which a threat is considered benign (and
therefore not alerted to the user). Operator trust is negatively
affected if a system produces many false positives without
explanation or if the operator cannot confirm whether the system
produces false negatives. System transparency in the form of
alertlevel and system-level explanations therefore enhances trust,
because users can better understand when and why a system makes
mistakes as well as to ensure the system doesn’t miss any potential
threat behavior, respectively [
Not all anomalies are threats and not all threats are equally
important. System transparency accelerates operator workflow by
providing the evidence needed to quickly and accurately prioritize
and determine the validity of threats. Finally, the DAART system
improves with user feedback, so the goal is to get the best feedback
as possible from users while minimizing the time and effort to
provide it. System transparency enhances the feedback process
because users’ feedback is improved when they have an
understanding of how the system works and why an alert is
considered anomalous. Furthermore, users’ time and effort are
minimized when providing feedback through the same
visualizations presented to them for explanation purposes, as these
are already familiar [
In this paper we present the DAART system for identifying anomalous behavior without training data, and we propose interactive explanation methods for improved operator trust, accelerated workflow, and enhanced operator feedback through system transparency. In particular, we propose methods for determining and displaying explanation information, such as multimodal localization (or attention) and normalcy exemplar, and an interactive explanation interface to present these and other simple explanation types (system confidence, alternate classifications, features) to users for promoting transparency and providing a means for user feedback. We additionally propose a user study to evaluate these various interactive explanation methods for the DAART system.
Detecting anomalies in sensor data requires a standardized feature
representation of the incoming data. Traditionally, these features
are defined by expert scientists who specialize in particular sensor
modalities. More recently, supervised machine learning models
have been able to outperform expert-defined features in their
descriptiveness about the original sensor data [
Interactive machine learning systems incorporate end-user
feedback to re-train underlying algorithms and improve their
output. Users may provide this feedback in the form of interactively
labeling data [
The system we present here builds on interactive machine
learning techniques, such as accepting and rejecting system’s output
[
There is growing interest in system transparency, or explainable
artificial intelligence (XAI), driven in part by both DARPA’s XAI
initiative [
System transparency can be provided through explanations or
visualizations that provide insight into what the system is doing and
why it is doing it (see [
We generate discriminative features and perform anomaly
detection using an approach based on Generative Adversarial
Networks (GANs) [
Generative Adversarial Networks (GANs) have demonstrated
the ability to generate images from a random noise vector that are
able to fool another model attempting to determine if the image is
real or fake. A GAN consists of two competing models. A Generator
(G) model learns how to transform random noise into a fake image.
A Discriminator (D) model then tries to determine if the fake image
is real or fake. Over time both models are trained until the fake
images are indistinguishable from the real images. An Adversarial
Learned Inference (ALI) [
GANomaly extends ALI to perform anomaly detection as part of the feature extraction process. This approach consists of three subnetworks: A Generator (G), Encoder (E), and Discriminator (D). During the training process, the GANomaly model is trained only on normal images, in essence learning a model of normalcy. The images (video frames, or vectors from other sensor types) are fed into the Generator, which learns two things: (1) a lower dimensional mapping (z), and (2) to reconstruct the image (xʹ). The reconstructed image is then fed into both the Encoder and Discriminator. The Encoder learns a second lower dimensional mapping of the reconstructed image (zʹ), and the Discriminator learns to tell the difference between real and fake images. During the training process, the Generator learns by minimizing the loss between the original image and the fake image (x - xʹ). The Encoder learns by minimizing the loss between the first and second lower dimensional spaces (z - zʹ).
Once trained, GANomaly is used in live operations to test each input vector (e.g. video frame) to compute an anomaly score. Since the model was trained only on normal behavior, an anomaly score that determines whether an input is anomalous or not can be computed based on the L1-normalized Euclidean difference between the first lower dimensional mapping learnt by the encoder and the lower dimensional mapping learnt from the reconstructed image (z - zʹ).
Scopes define specific filters on sensor and facility conditions that the user wants the anomaly detection system to be restricted to when discovering anomalies. When a Scope is specified, all data that matches the defined filters will be processed by the anomaly detection algorithm, which calculates an n-dimensional probability distribution of the data over the features learned during feature extraction. The result is a baseline normalcy model defining what sensor data is considered normal activity.
This baseline normalcy model is incrementally updated each
time new sensor data is ingested into the DAART system. We use
this normalcy model to compute a strangeness metric proposed in
prior work [
When the DAART system identifies anomalous activity, it alerts the user. An example of the DAART system upon identifying anomalous activity is shown in Figure 2.
Figure 2 (right) shows the alert, which includes a video clip of the anomalous activity and a timestamp at which the activity occurs. Users interact with an alert to either specify that it is “Not a threat” or provide a threat class for it. In addition to viewing the clip, the system also explains the anomaly using a timeseries chart showing the strangeness score of the anomaly compared to prior readings as shown by Figure 2 (left). Users can additionally modify the strangeness threshold above which an anomaly is alerted. We propose additional methods for explanation and user feedback in the following section.
For operators to better understand the anomalies and threats that DAART alerts them to, it would be ideal to be able to isolate which part of the sensor reading was anomalous. In the case of EO video, this could mean showing the user a bounding box which identifies where in the video stream the anomalous activity is occurring. This type of functionality is extremely valuable in helping the operator decide what threat label to assign to new unknown threats, and to help them better determine what course of action is reasonable in response to a threat.
Because of the fully unsupervised GAN-based approach DAART uses for anomaly detection, localization of the anomalous activity in sensor readings is non-trivial. Unlike many supervised approaches, in which the detection of specific objects or actions are triggers for anomaly or threat alerts, the current GANomaly-based unsupervised approach uses a more context-oriented approach which examines the entire sensor reading at once.
Recently, however, because of the popularity of GAN-based
techniques for unsupervised machine learning, approaches have
been developed for fully unsupervised object detection and
localization using GANs [
We propose to integrate this introspection-based approach into its unsupervised GAN-based feature learning, allowing anomalies detected using those learned features to be localized. The operatorfacing DAART explanation interface will then be updated to show which parts of an anomalous sensor reading are most responsible for the reading being considered anomalous.
We propose to generate normalcy exemplars that can be displayed to operators to compare against detected threats. These normalcy explanations allow the system to describe what the situation typically looks like to help explain why a new instance is deemed anomalous or threatening.
We propose two techniques for generating normalcy exemplars for this comparison. The first technique is to simply determine the existing normal exemplar (non-anomalous prior reading) that is most similar to the anomalous input using the features space. A side-by-side view displays that exemplar sensor reading against the detected anomalous reading for comparison. Furthermore, bounding boxes can be added to highlight differences in the anomalous input by utilizing the localization information. One limitation to this technique is that not all differences between the normal and anomalous scenes are important. The second, and more complex, proposed technique accounts for this limitation by generating a synthetic “normal” feature vector that is similar to the anomalous reading, but without the features that make it an anomaly. The GAN then generates a synthetic sensor reading from feature vector. In this case, the only difference between the two displayed exemplars are the elements of the input that make it anomalous.
We can similarly use the GAN to determine normalcy exemplars for other data types, such as audio and acoustic sensor data, but a challenge of this task will be determining appropriate ways to expose this information to operators. Audio can be handled similarly to imagery, for example, by providing two audio clips the operator can listen to for comparison. However, for the other data types, we will work with operators to determine what view of each modality fits best into their existing threat detection workflow as part of this task.
In addition to the explanation information discussed in prior
sections, prior work has introduced simple explanation types [
Normal Exemplar
Detected Anomaly
System Confidence(Anomaly) =
99.71% Threat Class “Smoke Bomb” Alternative Classes smoke bomb fire fog
Localization (Attention)
Threat Cluster Graph fog smoke bomb fire
These explanation presentations additionally provide an
intuitive means for user feedback [
We outline a user study of our proposed explanation interface to evaluate the effects of varied explanation information on trust, feedback quality, and overall human-machine team performance.
The goal of the proposed study is to answer the following research
questions:
Q1: Which explanation information yields the optimal
humanmachine team?
Similar to [
Q2: How is trust affected by varied explanation information? As trust is particularly important to the adoption of a system such as ours in the military domain, we intend to evaluate the effect of varied explanation information on system trust 5.2
To support examination of the identified research questions, we propose a crowdsourced user study. We will identify a dataset and specific task that is representative of real system usage, but also approachable to non-security experts. This might include video on a street corner for which the human-machine team is tasked with identifying suspicious behavior or video replay from a tower defense-style game2 for which the human-machine team is tasked with identifying aggressive behavior towards a base. In the study, we will hold all aspects of the DAART system constant, but simply vary the explanation information shown to the user during the task. After the task we will evaluate the human-machine team performance, as well as ask users to score the system in terms of trust, frustration, complexity. In this way, we can study the effects of the various explanations on user experience.
In this paper we present an unsupervised anomaly detection system, DAART, that identifies anomalies from normal behavior and classifies those anomalies as threats through interaction with operators. We additionally propose and explanation interface towards the goal of a DAART system that the user not only understands and trusts but is maximally accurate due to increased, improved user feedback. Our proposed user study aims to evaluate the explanation interface to increase effectiveness and reduce complexity.
This work was supported by AFRL contract FA8650-18-P-1628 and advised by Dr. Olga Mendoza-Schrock and Mr. Todd Rovito. This publication was cleared for public release via 88ABW-2019-0665.