February 2010, Afghanistan, an American helicopter fired on three suspected trucks, killing 23 innocent civilians, and wounding 12. The attack was approved based on information provided by operators of an unmanned aerial vehicle (UAV) who did not report the presence of civilians in the trucks (Filkins 2010). In a later news report, Shanker and Richtel (2011) cite Army officials claiming that the leading cause for the tragic incident was information overload that lead to poor Situation Awareness (SA). SA can be defined as one's perception of the environment around him or her at any given point in time (Endsley 1988). Thus, although there were evidences for children in the trucks, the UAV operators "did not adequately focus on them amid the swirl of data".
The use of UAVs in the military domain is increasing, due to their ability to perform missions without risking human operators (Izzetoglu et al. 2015). UAV operators monitor the payload, often a camera sending video feed, in various missions (e.g., reconnaissance, guidance of forces; Marusich et al. 2016), in addition to multiple tasks (e.g., navigation and orientation, flying the vehicle, radio communication; Everaerts 2008). A command and control (C2) map, often in a different display, is used for orienting and making sense of the payload's outputs. The C2 map shows mission critical information and intelligence-related information items such as markings of potential targets, location of allied forces and so forth. A cognitive work analysis of UAV operators workflows, emphasizes frequent and continuous use of the C2 map during missions (Back et al. 2019). The C2 map, however, being shared among military elements, is showcasing information that is irrelevant to the UAV operators. It has been indicated (Endsley 2000;Sandom 2000) that information overload is a contributor to poor Situation Awareness high workload and low overall performance. According to Back et al. (2019), the information clutter in C2 maps may often lead operators to neglect the map, and rely solely on the payload's feed, and may also lead to fatal results as the tragic incident of February 2010. Answering Adams' (2015) call for incorporating human factors limitations in the design of UAVs, it serves as an incentive to address the information overload problem. Some advanced solutions for improving UAV operators' SA had been suggested, e.g. using synthetic vision (Calhoun et al. 2005). Such solutions, however, may require costly adjustments of the vehicle's payload.
For decluttering, most C2 maps are based on information layers. Layers can manually or automatically (via a set of rules) be hidden, shown or dimmed. The layer mechanism reduces the information overload problem by hiding or dimming layers, yet, at the same time, it may cause information deprivation due to an inherent tradeoff; any action performed on a layer affects it entirely. Thus, it is impossible to hide irrelevant information items in a layer, while showing relevant ones of the same layer. Therefore, aspiring to solve the C2 map information overload tradeoff, Zak, Oron-Gilad, and Parmet (2018) called for 'breaking' the layer mechanism and addressing the information at the information items' level. Thus, instead of filtering layers of information items, the filter will handle each information item individually (as Figure 1: Left. An illustration of a layer-based filter, where entire layers can be turned on/off. Right. An information-item level based filter, where each single information item can be considered as a separate layer and handled individually. illustrated in Figure 1). This can be achieved by an algorithm that dynamically and automatically filters information items on the C2 map by their importance and relevance to UAV operator's current mission. Automating this process, however, is challenging as it can inadvertently reduce operators' SA and performance, especially regarding items chosen not to be shown (Endsley and Kiris 1995).
Two guidelines led the development of the automatic and dynamic algorithm. First, working at an information item level requires advanced techniques. Those techniques rise from the artificial intelligence (AI) domain. Second, as the problem is both spatial and temporal, the solution should adopt perceptual concepts inspired by Gibson and Crooks' (1938) field of safe travel. The field of safe travel, was defined as the spatial field where it was safe to steer a car. It was dependent on driver state, vehicle and environment, and it was constantly changing as the vehicle moved and the context of driving changed. Gibson and Crooks referred to the field's intuitiveness as affordance. In Section 2 we use these guidelines for the development of the GiCoMAF (Gibsonian Command and Control Map AI Filter algorithm) but first, in Sections 1.1 and 1.2, we review Machine Learning (ML) models that were considered for the AI implementation and delineate the research goals.
The military domain is seeking for techniques to incorporate AI into C2 systems and Machine Learning (ML) mod-els have been used in UAV, GIS and C2 domains (e.g. Azak and Bayrak 2008;Bao 2016;Choi and Cha 2019;Dzieciuch et al. 2017;Noh and Jeong 2010;Rapaport 2015), but to our knowledge, not as the interface between the C2 map and the operators. For our research, we looked for models that can be used for classification (to classify information item's importance) or logistic regression (for defining the field of relevance, as detailed later). There are other ML models that can be used for the purpose of this study. For example, Long Short-Term Memory (LSTM), is a common deep learning technique for time series data, that can be used for multi-stream data as well (Behera, Keidel, and Debnath 2019;Bouaziz et al. 2017). Given that each information item in the C2 map can be perceived as a standalone time series, this model was considered for this study too. However, the exact number of parallel streams (i.e. the number of the information items) is constantly changing as more information items are added to the map. Therefore, the LSTM model was neglected.
Acknowledging UAV operators' C2 map needs, this study aims to introduce the GiCoMAF solution for distilling the information presented on the map to show mission relevant important information items, while minimizing or hiding less important or distracting items. The first goal is to develop an algorithm that automatically and dynamically fil- In Generalized Linear Models (GLM) the relationship between the predicting variables and the independent variable is a linear function. It is mostly used to predict a single continuous numeric value, but variations allow it to be used for classification (multinomial linear regression; Greene 2012) or for predicting a probability (logistic regression; Walker and Duncan 1967). Lasso is a method of performing feature selection alongside GLM by introducing a coefficients size penalty as a constraint. The effect of the penalty on the regression is controlled by a user defined λ parameter (Tibshirani 1996;Tibshirani et al. 2012).
NN is a composite of input, output, and middle layers (a.k.a. 'hidden layers') presuming to mimic the work of neurons in the human brain. Each layer consists of an undefined number of neurons, which sum the input received from the previous layer using an activation function f , and 'fire' the result to the next layer. NNs can handle non-linear problems (Kanevski et al. 2004).
Decision tree is a prediction for non-linear problems where each node of the graph represents a predicting variable. The returned value is the final node the decision led to (Quinlan 1986). The number of possible results is limited by the number of possible values of the label, and by the depth of the tree (maximum number of nodes a branch can have). Increasing the number of possible results can be done by increasing the tree's depth, but it may result in undesired patterns of overfitting. Random Forest overcomes this limitation by producing multiple trees, each one is trained on a randomly selected subset of the same dataset. The returned value is an average of the predictions of all trees for a regression type model, and the highest frequent value for the a classification type model (Breiman 2001).
Gradient boosting is an iterative ensemble of trees model, where in each iteration a decision tree is learned using the residuals of the previous iteration (Friedman 2002).
XGBoost is an open source package implementing an efficient scalable tree boosting system, incorporating a regularized model to prevent overfitting (Chen and Guestrin 2016).
ters the information items on the C2 map. The automatic part of the algorithm, addresses the filtering at the information items' level. The dynamic part of the algorithm addresses the evolving environmental context of the area of operation.
The second goal is to delineate the use of ML models in the construction of the algorithm by demonstrating how its construction can be achieved using ML models. This goal is attained using importance and relevance labelled data collected empirically from UAV operators. Their inputs enable the ML models to learn the operators' contextual information needs, and to showcase the algorithm's feasibility. This paper describes the process of developing the GiCoMAF algorithm using ML models. A third goal, not described in this paper, is to evaluate the update rate of the GiCoMAF empirically, exploring its effect on operators' mental workload, situation awareness and perception of the experience.
In the following Sections the definition of the GiCoMAF algorithm is first outlined. Then, the process of acquiring the ML models constructing the algorithm's tiers is detailed, including the data collection, manipulation, and models evaluation. Lastly, the discussion Section discusses how to combine all tiers into an operating filter algorithm.
The GiCoMAF -Gibsonian Command and Control Map AI Filter algorithm, consists of two tiers, each answers a different research question. The integration of the tiers creates the filter rule, and incorporates the outcomes of these questions into the workflow of the operators. Tier I aims at the information item level, and answers the question what is the perceived importance of each information item? Tier II aims at the map as a whole, and answers the question how can the operator's area of interest be modeled on the map? Tier III aims at the dynamicity property of the algorithm, and answers the question how often should the automatic filter be updated?
The construction of the GiCoMAF is illustrated in Figure 2. The distinction between Tiers I and II is important. Tier I predicts each information item's importance. The scale is inclusive, i.e., it provides an indication of how important it is to show the information item on the map, and an indication of how important it is not to show the item. The incentive behind this logic is that some non-important information items may be harmless and operators will be oblivious to their presence, while others may be disturbing. Moreover, predicted information item's importance should not be handled in the same way throughout the map space, and this is where Tier II comes into play. Consider an information item with a neutral predicted importance, i.e., not important but not disturbing. While the item per se is not considered disturbing, possibly, if it is within the operators' area of interest, they may be more sensitive to disruptions, and the item can inadvertently cause clutter or quickly become disturbing. To avoid such cases, it may be better to filter out the item. Outside the area of interest, however, leaving a neutral item may be a good strategy, as its importance may rise as the mission evolves. Hence non-important items for the immediate context, may be valuable to foresee future evolvement of the situation and prepare for it. Therefore, Tier I of predicting the information items' importance is not enough, and the algorithm should model the operators' area of interest as derived in Tier II. The final decision rule, hence, is a combination of these two tiers. Tier I and II of the algorithm are based on ML models, thus, by learning examples from UAV operators, the algorithm predicts and executes an automatic filter for new operators and in new unknown scenarios. ML models require tagged examples to be learned from. Therefore, an experiment which emulated the work of UAV operator in operational scenarios was designed and conducted to collect tagged data from UAV operators (Zak, Parmet, andOron-Gilad 2019a, 2019b). Tier III defines the rate at which the filter rule of Tiers I and II should be applied. At this stage of development Tier III cannot be based on ML models, and represent a pure cognitive issue. Since the scope of this study was to delineate the construction of the GiCoMAF algorithm using ML models, the process of studying the cognitive effect of various filter update rates and setting the optimal rate is due to future research.
Tier I aims to model information items' importance as perceived by the operators. An information item's importance may vary based on environmental context, mission, and characteristics. Generally, operators want important information items to be shown on the map. If an information item is not perceived as important, its presence can be disturbing and then operators would prefer that it will not to be shown, or not disturbing and operators may be impartial to its presence on the map. Therefore, Tier I attempts to predict and classify information items' importance level, into a four tics scale, based on the context: Positive importance represents important (1) and very important (2) information items. Zero importance represent non-important information items, that operators have no preference to whether they should be shown or not. Negative importance represents information items that disturb and distract operators from their mission context.
The prediction is done using a ML model. The model, once trained, has the ability to deduce item importance from insightful environmental and mission related measures that describe the context. For example, the average distance of an information item from the UAV payload may describe its mission related context. The density of information items around an information item, for example, may reflect the environmental context. Higher density raises the probability of some operational event happening at that location. A ML model can find the relationship between an information item and the route, and then predict the importance of the information item; and it can determine that as the environment is denser, the probability of showing non-important items increases. These examples of relations between derived measures like distance and density and the predicted value are given for simplifying purposes, and the real relationship that emerge from the ML model may be more complicated. Furthermore, selecting the most suitable ML model (Table 1) was determined empirically as detailed in Section 3.
Tier II of the algorithm addresses the areas of interest for the operators in the environment. It is essentially a spatial problem that can be represented using geospatial measures on the map (e.g. polygons, heatmaps, etc.). The 'area of interest' is not necessarily a singular area, and can be phrased as 'areas of interest', where each area has a different level of interest. For example, Figure 3 illustrates multilevel areas of interest, where in the center of the polygons occurs the mission, and therefore the smallest area around this focus has the highest interest. The surroundings do withhold interest to the operator since they may affect the mission's center. Their relevance decreases as they get farther than the mission's center. The 'area of interest' is modeled using the concept of field of relevance, an adaptation of Gibson and Crooks (1938). The field of relevance depicts operators' areas of interest based on the environment, mission, operators' behavior, etc. Moreover, it corresponds with the affordance property as the field of relevance highlights areas that are intuitively more focused upon by the operators. Due to the probabilistic characteristic of ML models, it was decided to adopt the quality map approach of Morse, Engh, and Goodrich 2010, and to model the field of relevance as a pseudo-Gaussian heatmap, where each spatial element on the map gets a value between 0 (no relevance) and 1 (high relevance) corresponding to the probability of that spatial point to be in the operator's area of interest.
Similar to Tier I, prediction in this tier is done using a ML model, deducing from insightful environmental and mission related measures that describe the context. In this tier those measures are in respect to a spatial element (e.g., a square of 10 meters 2 ), without relating to any particular information item within the area of interest. For example, a mission related measure can be the average time a spatial element is in the UAV payload's field of view, assuming higher average time may indicate higher relevance of that element. An environmental measure can be the density of information items around a certain element. Assuming higher density around a spatial element indicates upon the probability of some operational event happening at that location, and in turn higher relevance. Similar to Tier I, the examples of relations are for simplifying purposes, and the exact ML model (Table 1) was determined through an empirical process detailed in Section 3.
In the construction of the filter, an experiment emulating the work of UAV operators in the military domain was executed. Participants, professional military UAV operators, were asked to perform a mission of supporting a ground battalion in urban battlefield scenarios. The data collected using the feedbacks they provided during and after the experiment was used to construct the ML models of Tiers I and II. The process is illustrated in Figure 2. The research was approved by the Institutional Review Board at Ben Gurion University. Informed consent was obtained from each participant.
Data for Tiers I and II were collected in a set of experimental runs, detailed in Zak, Parmet, and Oron-Gilad 2019a and 2019b. The experiment aimed to emulate the work of UAV operator in the military domain. Using a designated system developed for this task (UCES -UAV Command and Control Experiment System), a battlefield scenario was developed by subject matter experts with a UAV mission to assist a ground battalion in conquering an urban neighborhood. The 'five paragraph order', a common military standard of writing a fight plan of the mission, was outlined on the C2 map (Figure 4) and programmed in the VR-Forces simulation engine. The mission was 12 minutes long, representing a sequence of events that in real life settings may take several hours. Thirteen professional military UAV operators performed a reconnaissance mission as if they were acting in a real-world battlefield. The UCES allowed them to control the vehicle's payload, observe the battlefield from an aerial point of view, and get real-time information from a C2 map. Occasionally at specific points, the scenario paused, and a scoring session had started. They were asked to label two types of information using the UCES map: tagging individual information items' importance; and drawing their current contextual area of interest as polygons on the map. There were 88 scoring sessions in total, an average of 6.5 sessions for each experimental run. The data collected from the runs was put together into two datasets. The dataset containing the information item's importance tags was the input for Tier I, and the dataset containing the area of interest polygons was the input for Tier II. Then, ML models for the two tiers were developed. The processes and outcomes are detailed in Sections 3.2 and 3.3.
The data collected in the experiment was divided into two tables. The first table was an event diary, where each row represents one event in one experimental run (e.g., UAV movement, forces movement, cross-fire event, etc.). This table contained approximately 90,000 rows. The second table is a tagging table, where each row represents an information item in a scoring session in an experimental run and its perceived importance as was reported by the participant. This table contained 1,203 rows, where each information item has inherent attributes like its location, type, the time it was last modified, etc. However, item attributes are insufficient for introducing mission context to the ML models. Therefore, the two tables had to be joined into one training dataset. For that, the event diary had to be manipulated into insightful variables. Thirty-five derived measures were calculated from the event diary to describe the environment and the mission.
The objective of the ML model was to classify information items' importance. Misclassifying a non-important information item as important does not have the same implication as misclassifying an important information item as disturbing. Therefore, the error function could not be merely classification accuracy, and weights were given as penalty for misclassifications based on the sensitivity and specificity of the misclassification.
All four techniques described in Table 1 were tested in a classification configuration. Model technique parameters were optimized using the k-fold cross validation technique, where k = 10. During this process, 22,945 models were built in total for all four techniques. Then, data were divided by participant; 10 randomly chosen participants as the train-
The objective of the model was to define the field of relevance by predicting the probability of each point on the map to be in the area of interest. To facilitate the model construction, the C2 map was divided into a grid of cells, where each cell represents an area of 25x25 meters on the ground, 9,016 cells in total. Each row in the dataset represents one cell in a scoring session of an experimental run, with the predicted label of (1) if the cell was in the drawn area of interest in that scoring session, and (0) otherwise. The derived measures were calculated with respect to each cell (e.g. the average distance of a cell from the UAV route, etc.). Similar to Tier I, the data was divided into two tables; an event diary and a table with the reported feedback from the participants. In this tier, each row in the reported feedback table represents coordinates of the area of interest's polygon of a scoring session in an experimental run. Each scoring session consisted of one polygon. Subtracting two cases where the participant forgot to draw an area of interest, this table had 86 rows. Also, the events diary was manipulated into 20 derived measures.
The objective of the model was to define the field of relevance by predicting the probability of each point on the map to be in the area of interest. To facilitate the model construction, the map was divided into a grid of cells, where each cell represents an area of 25x25 meters on the ground, 9,016 cells in total. Thus, each row in the dataset represents a cell in a scoring session of an experimental run, with the predicted label of (1) if the cell was in the drawn area of interest in that scoring session, and (0) otherwise. The derived measures were calculated with respect to each cell (e.g. the average distance of a cell from the UAV route, etc.).
The same four ML techniques that were used in Tier I were used in the regression configuration, but the predicted value was logistic (a number in [0,1]). Model performance was measured using root mean square error (RMSE). Due to the large scale of data and computing resource limitations, 408 models were built in total in four k-fold cross validation processes, where k = 3. Figure 7 illustrates the average k-fold RMSE results for the optimal parameters set, and the RMSE of the training and validation sets. While logistic LR and NN have better results on the validation set in terms of RMSE, both XGBoost and RF perform better on both the training set and k-fold results. Figure 8 delves into the results and illustrates a 'rotated confusion matrix'. The x-axis represents the predicted value, binned in intervals of 0.01. The y-axis represents the average original value of the rows corresponding to the bins of the x-axis. The size of the points represents the relative number of records in each bin. A perfect model would provide points aligning with the diagonal of the plot. According to Figure 8, both LR and NN collapse and do not provide good differentiation. RF provides something close to binary results, which may conflict with the field of relevance concept. Therefore, although not perfectly aligning with the diagonal, XGBoost performs better than the other three models. An illustration of XGBoost's performance relative to the runner-up RF model, in two scoring session of two different participants is given in Figure 9. Figure 9 demonstrates how each operator marked the field of relevance, at the same stage showing that the structure of the field of relevance depends on the environment, the mission, and operators' individual characteristics and preferences. The XGBoost model seems to handle these characteristics well.
Operators of UAVs work in uncertain and dynamic environments. Their main focus is on managing their vehicle's payload (e.g., camera video-feed) while they are required to maintain orientation and awareness to their current location and its surroundings, and plan ahead. In order to develop the essential SA, they continuously and constantly interact with a C2 map (see Back et al. (2019) for a Cognitive Work Analysis). The map, is often overloaded with data that is irrelevant to their mission, and like the operational environment things change rapidly. The existing layer-based map filter mechanism causes a tradeoff dilemma for the operator -showing an entire layer with its relevant and irrelevant information, or hiding an entire layer and losing important information. Furthermore, as noted in Back et al. the existing filtering mechanisms require operators to manually interact with the interface, highlight or hide layers of information at the busiest times of their mission. Thus, currently, obtaining information from C2 maps pose high demands on operators, and therefore, they often cope with impaired SA. This study proposes a solution to dynamically and automatically adjust the C2 map display to operators' needs by introducing an AI-based dynamic and automatic algorithm that filters the information on the C2 map at the individual items level (as opposed to layers), as described in section 2, and laying the foundation for the algorithm by delineating the use of ML models in its construction as detailed in Figure 2.
The data tagging experiment was designed to collect labelled data towards the construction of the ML models of Tiers I and II; predicting information items' importance and predicting the field of relevance, respectively. Using the collected data, four different models were optimized and developed for each tier. In Tier I both XGBoost and RF had good results on the validation set, with RF on the upper hand. However, since the RF had extremely low training error, for both error types, it may be overfitted. Further study on the effects of each model on UAV operators' mission performance is needed to examine if the effect of overfitting is negligible. Therefore, to be cautious, the XGBoost model was chosen for further development. The XGBoost model, however, had an accuracy sign error of 25.9%, and its decisions were hanging on the thread of an (Table 2). This magnitude of the error is reasonable when modeling human preferences and performance (as seen in previous studies, e.g. Agichtein et al. 2016;Guimerà et al. 2012;Liu, Bian, and Agichtein 2008), especially when having a limited number of observations. The XGBoost results of Tier II were visually evaluated and provided the best and most accurate pseudo-Gaussian heatmap for the field of relevance. as illustrated in Figures 8 and 9. It should be noted that accuracy can be further improved by testing other ML models and parameters. Indeed, due to resource limitations, not all suitable models could be tested. Future study can attempt to improve the results of this study by applying additional ML models and techniques.
The two tiers delineate the use of ML models in the GiCo-MAF algorithm . Yet, is not clear cut how to combine Tiers I and II into the GiCoMAF algorithm. Several options for how the tier combination protocols are being discussed in the following section.
The exact combination of Tiers I and II into the GiCoMAF algorithm may depend on the mission, the environment, and even the organization that the operators are part of. This section suggests three possible combination approaches; however, the final setup is not limited to these approaches and each organization/user/system case may dictate the development of a tailored solution. These approaches provide a show/no-show decision rule based on relevance and importance thresholds. The optimal approach used in the algorithm, as well as its thresholds, should be continuously evaluated through a process of experiments.
Positive Correlation -This approach assumes that there should be a positive correlation between the field of relevance and the information density. Thus, the more relevant a spatial element is, the more information should be presented around it. To avoid distraction, the farther away from the spatial element, the less information should be shown. Figure 10a illustrates this approach where there is a negative relation between the field of relevance and the information items' importance. In order to use this approach, an adaptive show/no-show threshold should be set based on information items' importance, where the threshold is rather low at points with high relevance (implying more information items would be shown), and rather high at points with low relevance (only very important information items would be shown).
Negative Correlation -In contrast to the positive correlation approach, this approach assumes that there should be a negative correlation between the field of relevance and information density. I.e., the more relevant a spatial element is, the more attention operators direct to it, and therefore fewer information items should be shown to avoid disruption. Thus, as illustrated in Figure 10b, there should be a negative relation between the field of relevance and information items' importance, in similar but opposite approach to the relation described in the positive correlation.
Binary Relevance Decision -This approach assumes that only important information items should be shown on the map, and only in areas which are currently most relevant to the operators. Therefore, there are only two constant thresholds in the algorithm -one to decide the minimum importance an information item should have in order to be shown, and one to decide the minimum relative relevance a spatial point should have in order to set the importance threshold into motion. Figure 10c illustrates the approach.
For a further study, we propose to examine these three approaches and choose the fittest approach for the context that was addressed in this paper (i.e., the same mission type and profile of participants). For that a new multistage scenario similar in nature to the experimental scenarios detailed in this paper is required. Using the ML models developed in Tiers I and II, each scenario can be run in one of the three combination approaches, as well as running with no filter at all. In the first experiment, participants will be introduced to all of the approaches and different thresholds setups would be tested. Then, using the optimal threshold for each approach, new operators would participate in a two-stages crossover experiment with eight possible treatments -the three combination approaches and no filter at all.
This study aimed to introduce a solution to the information overload of C2 maps used by UAV operators. By solving operators' need for distilled information at the right time and in the right place, it is expected that operators will benefit more from the C2 map at lower efforts, and overall mission performance will improve. The study had met its goals. First, a solution was achieved by introducing the three-tier Gibsonian Command and Control Map AI Filter algorithm (GiCo-MAF). Then, ML models were developed and evaluated using tagged data collected in an experiment. The results of the experiment are encouraging. The ML models that emerged from the first experiment were satisfying, and indicated high accuracy and usability of the algorithm, a step towards a solution to the information overload problem, and high workload of UAV operators. The combination of the tiers into a single algorithm is yet to be fully defined, and should be determined by an additional set of experiments we recommend performing in future studies.
The contribution of this study lays in various aspects. First, this study targets a long-neglected field of improving the use of C2 maps by operators. Second, by using ML models in the construction of the GiCoMAF algorithm, this study utilizes AI concepts and techniques to lay the foundation of an algorithm that is targeted for improving human performance and human cognitive aspects of workload and SA. And third, successful construction of the algorithm can improve mission performance and enhance the cognitive abilities of operators to perform spatial-temporal tasks beyond the UAV domain. Algorithms of this form can be further implemented in any domains where an overloaded spatial and temporal information has to be filtered, e.g., emergency dispatching systems, information guided surgeries, search and rescue missions, air traffic control, etc.
This research is partially funded by the "Negev" scholarship, and by the George Shrut Chair in Human Performance Management at Ben-Gurion University of the Negev. Corresponding author -Yuval Zak