Unmanned Aerial Vehicles (UAV) operators must maintain high levels of situation awareness on their area of operation. To achieve this, they use the Command and control (C2) map, which are shared among forces, and is regularly overloaded with data that is irrelevant to their mission. UAV operators' missions require distilled information at the right timing. Yet, the existing filtering mechanisms of C2 maps are layer-based and insufficient. We propose a new approach to automatically and dynamically filter information items on the map based on environmental and mission context. To achieve this, we introduce a three-tier artificial intelligence (AI)-based algorithm (GiCoMAF), where we delineate the use of machine learning (ML) models to support UAV missions. For the GiCoMAF development, tagged data was collected in simulated experimental runs with professional UAS operators. Different types of ML models were evaluated and fitted into the algorithm. The models achieved a relatively high accuracy at modeling human preference and area of interest. The approach presented in this study can be further implemented to support other operators in time-critical spatial-temporal problems.
February 2010, Afghanistan, an American helicopter fired
on three suspected trucks, killing 23 innocent civilians, and
wounding 12. The attack was approved based on
information provided by operators of an unmanned aerial vehicle
(UAV) who did not report the presence of civilians in the
trucks
The use of UAVs in the military domain is increasing, due
to their ability to perform missions without risking human
operators
For decluttering, most C2 maps are based on information
layers. Layers can manually or automatically (via a set of
rules) be hidden, shown or dimmed. The layer mechanism
reduces the information overload problem by hiding or
dimming layers, yet, at the same time, it may cause information
deprivation due to an inherent tradeoff; any action performed
on a layer affects it entirely. Thus, it is impossible to hide
irrelevant information items in a layer, while showing
relevant ones of the same layer. Therefore, aspiring to solve the
C2 map information overload tradeoff,
Two guidelines led the development of the automatic and
dynamic algorithm. First, working at an information item
level requires advanced techniques. Those techniques rise
from the artificial intelligence (AI) domain. Second, as the
problem is both spatial and temporal, the solution should
adopt perceptual concepts inspired by
The military domain is seeking for techniques to
incorporate AI into C2 systems and Machine Learning (ML)
models have been used in UAV, GIS and C2 domains
There are other ML models that can be used for the
purpose of this study. For example, Long Short-Term
Memory (LSTM), is a common deep learning technique for time
series data, that can be used for multi-stream data as well
Acknowledging UAV operators’ C2 map needs, this study aims to introduce the GiCoMAF solution for distilling the information presented on the map to show mission relevant important information items, while minimizing or hiding less important or distracting items. The first goal is to develop an algorithm that automatically and dynamically fil
ters the information items on the C2 map. The automatic part of the algorithm, addresses the filtering at the information items’ level. The dynamic part of the algorithm addresses the evolving environmental context of the area of operation. The second goal is to delineate the use of ML models in the construction of the algorithm by demonstrating how its construction can be achieved using ML models. This goal is attained using importance and relevance labelled data collected empirically from UAV operators. Their inputs enable the ML models to learn the operators’ contextual information needs, and to showcase the algorithm’s feasibility. This paper describes the process of developing the GiCoMAF algorithm using ML models. A third goal, not described in this paper, is to evaluate the update rate of the GiCoMAF empirically, exploring its effect on operators’ mental workload, situation awareness and perception of the experience.
In the following Sections the definition of the GiCoMAF algorithm is first outlined. Then, the process of acquiring the ML models constructing the algorithm’s tiers is detailed, including the data collection, manipulation, and models evaluation. Lastly, the discussion Section discusses how to combine all tiers into an operating filter algorithm.
2
The GiCoMAF – Gibsonian Command and Control Map AI Filter algorithm, consists of two tiers, each answers a different research question. The integration of the tiers creates the filter rule, and incorporates the outcomes of these questions into the workflow of the operators. Tier I aims at the information item level, and answers the question what is the perceived importance of each information item? Tier II aims at the map as a whole, and answers the question how can the operator’s area of interest be modeled on the map? Tier III aims at the dynamicity property of the algorithm, and answers the question how often should the automatic filter be updated?
The construction of the GiCoMAF is illustrated in
Figure 2. The distinction between Tiers I and II is
important. Tier I predicts each information item’s importance. The
scale is inclusive, i.e., it provides an indication of how
important it is to show the information item on the map, and an
indication of how important it is not to show the item. The
incentive behind this logic is that some non-important
information items may be harmless and operators will be
oblivious to their presence, while others may be disturbing.
Moreover, predicted information item’s importance should not be
handled in the same way throughout the map space, and this
is where Tier II comes into play. Consider an information
item with a neutral predicted importance, i.e., not important
but not disturbing. While the item per se is not considered
disturbing, possibly, if it is within the operators’ area of
interest, they may be more sensitive to disruptions, and the
item can inadvertently cause clutter or quickly become
disturbing. To avoid such cases, it may be better to filter out the
item. Outside the area of interest, however, leaving a neutral
item may be a good strategy, as its importance may rise as
the mission evolves. Hence non-important items for the
immediate context, may be valuable to foresee future
evolvement of the situation and prepare for it. Therefore, Tier I of
predicting the information items’ importance is not enough,
and the algorithm should model the operators’ area of
interest as derived in Tier II. The final decision rule, hence, is
a combination of these two tiers. Tier I and II of the
algorithm are based on ML models, thus, by learning examples
from UAV operators, the algorithm predicts and executes an
automatic filter for new operators and in new unknown
scenarios. ML models require tagged examples to be learned
from. Therefore, an experiment which emulated the work
of UAV operator in operational scenarios was designed and
conducted to collect tagged data from UAV operators
Tier I aims to model information items’ importance as perceived by the operators. An information item’s importance may vary based on environmental context, mission, and characteristics. Generally, operators want important information items to be shown on the map. If an information item is not perceived as important, its presence can be disturbing and then operators would prefer that it will not to be shown, or not disturbing and operators may be impartial to its presence on the map. Therefore, Tier I attempts to predict and classify information items’ importance level, into a four tics scale, based on the context: Positive importance represents important (1) and very important (2) information items. Zero importance represent non-important information items, that operators have no preference to whether they should be shown or not. Negative importance represents information items that disturb and distract operators from their mission context.
The prediction is done using a ML model. The model, once trained, has the ability to deduce item importance from insightful environmental and mission related measures that describe the context. For example, the average distance of an information item from the UAV payload may describe its mission related context. The density of information items around an information item, for example, may reflect the environmental context. Higher density raises the probability of some operational event happening at that location. A ML model can find the relationship between an information item and the route, and then predict the importance of the information item; and it can determine that as the environment is denser, the probability of showing non-important items increases. These examples of relations between derived measures like distance and density and the predicted value are given for simplifying purposes, and the real relationship that emerge from the ML model may be more complicated. Furthermore, selecting the most suitable ML model (Table 1) was determined empirically as detailed in Section 3. 2.2
Tier II of the algorithm addresses the areas of interest for the
operators in the environment. It is essentially a spatial
problem that can be represented using geospatial measures on the
map (e.g. polygons, heatmaps, etc.). The ’area of interest’ is
not necessarily a singular area, and can be phrased as ’areas
of interest’, where each area has a different level of interest.
For example, Figure 3 illustrates multilevel areas of interest,
where in the center of the polygons occurs the mission, and
therefore the smallest area around this focus has the highest
interest. The surroundings do withhold interest to the
operator since they may affect the mission’s center. Their
relevance decreases as they get farther than the mission’s center.
The ’area of interest’ is modeled using the concept of field
of relevance, an adaptation of
Similar to Tier I, prediction in this tier is done using a ML model, deducing from insightful environmental and mission related measures that describe the context. In this tier those measures are in respect to a spatial element (e.g., a square of 10 meters2), without relating to any particular information item within the area of interest. For example, a mission related measure can be the average time a spatial element is in the UAV payload’s field of view, assuming higher average time may indicate higher relevance of that element. An environmental measure can be the density of information items around a certain element. Assuming higher density around a spatial element indicates upon the probability of some operational event happening at that location, and in turn higher relevance. Similar to Tier I, the examples of relations are for simplifying purposes, and the exact ML model (Table 1) was determined through an empirical process detailed in Section 3.
In the construction of the filter, an experiment emulating the work of UAV operators in the military domain was executed. Participants, professional military UAV operators, were asked to perform a mission of supporting a ground battalion in urban battlefield scenarios. The data collected using the feedbacks they provided during and after the experiment was used to construct the ML models of Tiers I and II. The process is illustrated in Figure 2. The research was approved by the Institutional Review Board at Ben Gurion University. Informed consent was obtained from each participant.
Data for Tiers I and II were collected in a set of experimental runs, detailed in Zak, Parmet, and Oron-Gilad 2019a and 2019b. The experiment aimed to emulate the work of UAV operator in the military domain. Using a designated system developed for this task (UCES – UAV Command and Control Experiment System), a battlefield scenario was developed by subject matter experts with a UAV mission to assist a ground battalion in conquering an urban neighborhood. The ‘five paragraph order’, a common military standard of writing a fight plan of the mission, was outlined on the C2 map (Figure 4) and programmed in the VR-Forces simulation engine. The mission was 12 minutes long, representing a sequence of events that in real life settings may take several hours. Thirteen professional military UAV operators performed a reconnaissance mission as if they were acting in a real-world battlefield. The UCES allowed them to control the vehicle’s payload, observe the battlefield from an aerial point of view, and get real-time information from a C2 map. Occasionally at specific points, the scenario paused, and a scoring session had started. They were asked to label two types of information using the UCES map: tagging individual information items’ importance; and drawing their current contextual area of interest as polygons on the map. There were 88 scoring sessions in total, an average of 6.5 sessions for each experimental run. The data collected from the runs was put together into two datasets. The dataset containing the information item’s importance tags was the input for Tier I, and the dataset containing the area of interest polygons was the input for Tier II. Then, ML models for the two tiers were developed. The processes and outcomes are detailed in Sections 3.2 and 3.3. 3.2
The data collected in the experiment was divided into two tables. The first table was an event diary, where each row represents one event in one experimental run (e.g., UAV movement, forces movement, cross-fire event, etc.). This table contained approximately 90,000 rows. The second table is a tagging table, where each row represents an information item in a scoring session in an experimental run and its perceived importance as was reported by the participant. This table contained 1,203 rows, where each information item has inherent attributes like its location, type, the time it was last modified, etc. However, item attributes are insufficient for introducing mission context to the ML models. Therefore, the two tables had to be joined into one training dataset. For that, the event diary had to be manipulated into insightful variables. Thirty-five derived measures were calculated from the event diary to describe the environment and the mission. The objective of the ML model was to classify information items’ importance. Misclassifying a non-important information item as important does not have the same implication as misclassifying an important information item as disturbing. Therefore, the error function could not be merely classification accuracy, and weights were given as penalty for misclassifications based on the sensitivity and specificity of the misclassification.
All four techniques described in Table 1 were tested in
a classification configuration. Model technique parameters
were optimized using the k-fold cross validation technique,
where k = 10. During this process, 22,945 models were
built in total for all four techniques. Then, data were divided
by participant; 10 randomly chosen participants as the
trainLR
NN
RF
XGBoost
ing set and the 3 remaining as the validation set. The final
model for each technique was built using the derived
optimal parameters set. Figure 5 illustrates the results for two
types of errors: (a) weighted error, the average k-fold result
for the optimal parameters set, the training set, and the
validation set; and (b) sign error (percent of misclassifications
of important/very important as disturbing, and vice versa),
for all three sets. From Figure 5 it is evident that RF shows
patterns of overfitting, and multinomial LR and NN perform
better than XGBoost in terms of weighted error. However,
in terms of sign error XGBoost outperforms the others.
Figure 6 delves into the differences among the models by
illustrating a multiclass receiver operating characteristic (ROC)
graphs
Similar to Tier I, the data was divided into two tables; an event diary and a table with the reported feedback from the participants. In this tier, each row in the reported feedback table represents coordinates of the area of interest’s polygon of a scoring session in an experimental run. Each scoring session consisted of one polygon. Subtracting two cases where the participant forgot to draw an area of interest, this table had 86 rows. Also, the events diary was manipulated into 20 derived measures.
The objective of the model was to define the field of relevance by predicting the probability of each point on the map to be in the area of interest. To facilitate the model construction, the map was divided into a grid of cells, where each cell represents an area of 25x25 meters on the ground, 9,016 cells in total. Thus, each row in the dataset represents a cell in a scoring session of an experimental run, with the predicted label of (1) if the cell was in the drawn area of interest in that scoring session, and (0) otherwise. The derived measures were calculated with respect to each cell (e.g. the average distance of a cell from the UAV route, etc.).
The same four ML techniques that were used in Tier I were used in the regression configuration, but the predicted value was logistic (a number in [0,1]). Model performance was measured using root mean square error (RMSE). Due to the large scale of data and computing resource limitations, 408 models were built in total in four k-fold cross validation processes, where k = 3. Figure 7 illustrates the average k-fold RMSE results for the optimal parameters set, and the RMSE of the training and validation sets. While logistic LR and NN have better results on the validation set in terms of RMSE, both XGBoost and RF perform better on both the training set and k-fold results. Figure 8 delves into the results and illustrates a ’rotated confusion matrix’. The x-axis represents the predicted value, binned in intervals of 0.01. The y-axis represents the average original value of the rows corresponding to the bins of the x-axis. The size of the points represents the relative number of records in each bin. A perfect model would provide points aligning with the diagonal of the plot. According to Figure 8, both LR and NN collapse and do not provide good differentiation. RF provides something close to binary results, which may conflict with the field of relevance concept. Therefore, although not perfectly aligning with the diagonal, XGBoost performs better than the other three models. An illustration of XGBoost’s performance relative to the runner-up RF model, in two scoring session of two different participants is given in Figure 9. Figure 9 demonstrates how each operator marked the field of relevance, at the same stage showing that the structure of the field of relevance depends on the environment, the mission, and operators’ individual characteristics and preferences. The XGBoost model seems to handle these characteristics well.
4
Operators of UAVs work in uncertain and dynamic
environments. Their main focus is on managing their vehicle’s
payload (e.g., camera video-feed) while they are required to
maintain orientation and awareness to their current location
and its surroundings, and plan ahead. In order to develop
the essential SA, they continuously and constantly interact
with a C2 map (see
The data tagging experiment was designed to collect labelled data towards the construction of the ML models of Tiers I and II; predicting information items’ importance and predicting the field of relevance, respectively. Using the collected data, four different models were optimized and developed for each tier. In Tier I both XGBoost and RF had good results on the validation set, with RF on the upper hand.
However, since the RF had extremely low training error, for
both error types, it may be overfitted. Further study on the
effects of each model on UAV operators’ mission performance
is needed to examine if the effect of overfitting is
negligible. Therefore, to be cautious, the XGBoost model was
chosen for further development. The XGBoost model, however,
had an accuracy sign error of 25.9%, and its decisions were
hanging on the thread of an (Table 2). This magnitude of
the error is reasonable when modeling human preferences
and performance
The two tiers delineate the use of ML models in the GiCoMAF algorithm . Yet, is not clear cut how to combine Tiers I and II into the GiCoMAF algorithm. Several options for how the tier combination protocols are being discussed in the following section. 4.1
The exact combination of Tiers I and II into the GiCoMAF algorithm may depend on the mission, the environment, and even the organization that the operators are part of. This section suggests three possible combination approaches; however, the final setup is not limited to these approaches and each organization/user/system case may dictate the development of a tailored solution. These approaches provide a show/no-show decision rule based on relevance and importance thresholds. The optimal approach used in the algorithm, as well as its thresholds, should be continuously evaluated through a process of experiments.
Positive Correlation – This approach assumes that there should be a positive correlation between the field of relevance and the information density. Thus, the more relevant a spatial element is, the more information should be presented around it. To avoid distraction, the farther away from the spatial element, the less information should be shown. Figure 10a illustrates this approach where there is a negative relation between the field of relevance and the information items’ importance. In order to use this approach, an adaptive show/no-show threshold should be set based on information items’ importance, where the threshold is rather low at points with high relevance (implying more information items would be shown), and rather high at points with low relevance (only very important information items would be shown).
Negative Correlation – In contrast to the positive correlation approach, this approach assumes that there should be a negative correlation between the field of relevance and information density. I.e., the more relevant a spatial element is, the more attention operators direct to it, and therefore fewer information items should be shown to avoid disruption. Thus, as illustrated in Figure 10b, there should be a negative relation between the field of relevance and information items’ importance, in similar but opposite approach to the relation described in the positive correlation.
Binary Relevance Decision – This approach assumes that only important information items should be shown on the map, and only in areas which are currently most relevant to the operators. Therefore, there are only two constant thresholds in the algorithm – one to decide the minimum importance an information item should have in order to be shown, and one to decide the minimum relative relevance a spatial point should have in order to set the importance threshold into motion. Figure 10c illustrates the approach.
For a further study, we propose to examine these three approaches and choose the fittest approach for the context that was addressed in this paper (i.e., the same mission type and profile of participants). For that a new multistage scenario similar in nature to the experimental scenarios detailed in this paper is required. Using the ML models developed in Tiers I and II, each scenario can be run in one of the three combination approaches, as well as running with no filter at all. In the first experiment, participants will be introduced to all of the approaches and different thresholds setups would be tested. Then, using the optimal threshold for each approach, new operators would participate in a two-stages crossover experiment with eight possible treatments – the three combination approaches and no filter at all.
This study aimed to introduce a solution to the information overload of C2 maps used by UAV operators. By solving operators’ need for distilled information at the right time and in the right place, it is expected that operators will benefit more from the C2 map at lower efforts, and overall mission performance will improve. The study had met its goals. First, a solution was achieved by introducing the three-tier Gibsonian Command and Control Map AI Filter algorithm (GiCoMAF). Then, ML models were developed and evaluated using tagged data collected in an experiment. The results of the experiment are encouraging. The ML models that emerged from the first experiment were satisfying, and indicated high accuracy and usability of the algorithm, a step towards a solution to the information overload problem, and high workload of UAV operators. The combination of the tiers into a single algorithm is yet to be fully defined, and should be determined by an additional set of experiments we recommend performing in future studies.
The contribution of this study lays in various aspects. First, this study targets a long-neglected field of improving the use of C2 maps by operators. Second, by using ML models in the construction of the GiCoMAF algorithm, this study utilizes AI concepts and techniques to lay the foundation of an algorithm that is targeted for improving human performance and human cognitive aspects of workload and SA. And third, successful construction of the algorithm can improve mission performance and enhance the cognitive abilities of operators to perform spatial-temporal tasks beyond the UAV domain. Algorithms of this form can be further implemented in any domains where an overloaded spatial and temporal information has to be filtered, e.g., emergency dispatching systems, information guided surgeries, search and rescue missions, air traffic control, etc.
6
This research is partially funded by the ”Negev” scholarship, and by the George Shrut Chair in Human Performance Management at Ben-Gurion University of the Negev. Corresponding author – Yuval Zak