=Paper=
{{Paper
|id=Vol-2819/session2paper1
|storemode=property
|title=Predictive Analytics in the Naval Maritime Domain
|pdfUrl=https://ceur-ws.org/Vol-2819/session2paper1.pdf
|volume=Vol-2819
|authors=Bonnie Johnson
}}
==Predictive Analytics in the Naval Maritime Domain==
https://ceur-ws.org/Vol-2819/session2paper1.pdf
Predictive Analytics in the Naval Maritime Domain
Bonnie Johnson
Naval Postgraduate School Systems Engineering Department
bwjohnson@nps.edu
Abstract decision-maker’s choices act upon the world, causing
Predictive analytics offers a potential game changing capa- changes in outcome probabilities (Cox 2015). Yet, this cru-
bility for Naval tactical decision superiority. Tactical opera- cial concept of causal efficacy is seldom developed in detail
tions could take a significant leap in progress with the aid of in decision analysis, and the fact that formal probability the-
a real-time automated predictive analytics capability that pro- ory applies only to events rather than to actions and their
vides predictions of second and third order effects of possible consequences is seldom emphasized (Pearl 2008). The use
courses of action. This future capability would accompany
current developments in the use of artificial intelligence and
of Bayesian Networks (BN) and causal graph models may
data analytics to improve battlespace knowledge and offer provide a solution to predict probabilities of outputs given
automated battle management aids to the tactical warfighter. inputs and observations. These types of models can be used
As the automated battle management aids develop tactical to build quantitative representations of complex dynamic
course of action options the predictive analytics capability situations. Dynamic BN models and BN-learning algo-
could predict how the adversary might respond to each course rithms can learn from data to create an adaptive capability
of action option. The predictive analytics capability could that can predict outcomes in a changing environment.
continue to “wargame” possible blue force/red force actions
and responses—generating predictions of second and third Using methods of machine learning to process and ana-
order effects. These predictions offer the tactical warfighter a lyze large amounts of heterogeneous data and information,
more strategic perspective in making tactical course of action artificial intelligence (AI) technology can make predictions
decisions. By performing this analysis using an automated about probable effects, outcomes, and responses. These PA
aid with artificial intelligence, it allows the capability to sup- and AI methods can provide a powerful capability for tacti-
port real-time decisions and to analyze great amounts of data cal decision-making. Armed with the knowledge of possible
(both sensor data and historical data) and handle highly com-
effects and adversary responses to courses of action, warf-
plex tactical environments This real-time wargaming trans-
lates into high order computations that would be impossible ighters can leap ahead in terms of applying longer-term
to be performed manually in the short reaction times given. strategy to near-term warfare decisions. A critical enabler of
This paper discusses the results of a study of predictive ana- developing an executable model of blue forces and red
lytic capabilities in the naval maritime domain. forces is the incorporation of the correct metrics, premises,
and assumptions (Talbot and Ellis 2015).
This paper begins (in Section II) with a description of the
I. Introduction authors’ concept for a future predictive analytics capability
A predictive analytics (PA) capability – that can take into that could support a real-time operational automated deci-
account possible consequences and effects into the process sion aid. Section III discusses data concepts required to sup-
of decision-making – is key to enabling decision superiority port such a future PA capability. Section IV contains an
for naval forces. The PA capability, based on automated data overview of AI and game theoretic methods that show prom-
analytics, would support battle management aids (BMA) by ise for enabling an automated PA decision aid. Finally, Sec-
developing “what-if” and “if-then” predictive scenarios to tion V contains the conclusion.
shape the synthesis of future intelligent decisions and adap-
tive capabilities. This conceptual capability would inform
decisions concerning courses of action (COA) based on
II. A Conceptual Naval Maritime Predictive
what the longer-term effects are projected to be. It would Analytics Capability
enable short-term and long-term objectives to be weighed as
tactical decisions are made. In essence, a PA capability is a
The ability to perform predictive analytics in support of
critical step in enabling a real-time wargaming capability for maritime operations, such as planning and tactical warfare,
naval operations. requires a set of analytic capabilities that study the available
The goal of decision making is to change the probabilities data, develop COA options, and make predictions concern-
of outcomes to make preferred outcomes more likely. The
This will certify that all author(s) of the above article/paper are employees of the U.S. Government and performed this work as part of their employment, and
that the article/paper is therefore not subject to U.S. copyright protection. No copyright. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0). In: Proceedings of AAAI Symposium on the 2nd Workshop on Deep Models and Artificial Intelligence for Defense Applica-
tions: Potentials, Theories, Practices, Tools, and Risks, November 11-12, 2020, Virtual, published at http://ceur-ws.org
ing their effects for the purpose of selecting options with de- Each of the 1st order effects is then analyzed (step 4),
sired effects. Figure 1 illustrates a conceptual framework for based again on the red force model, to estimate a set of pos-
a PA capability for the naval maritime domain. sible adversarial responses. These constitute 2nd order ef-
Required inputs to this capability are shown as self- fects. Each 1st order effect may map into one or more possi-
awareness and situational awareness knowledge. Self- ble 2nd order effects.
awareness amounts to the development of a blue force The 2nd order effects, which may now contain a signifi-
model which keeps track of the location, status, and capabil- cant number of possibilities, are analyzed (step 5) using
ities of the blue force resources or warfighting assets. Situa- knowledge of our blue forces (contained in the blue force
Figure 1 – Conceptual Framework for Predictive Analytics Capability
tional awareness consists of real-time sensor data feeds that model) to predict the 3rd order effects. The 3rd order effects
are fused and analyzed to provide an understanding of the are a set of predicted blue force states that result from the
battlespace or operational environment. From the blue force possible adversary responsive actions. Thus, there is a one-
and situational awareness models, a set of possible COAs to-one mapping of possible 2nd order effects to possible 3rd
(shown as step one) are developed that represent a contin- order effects.
uum of possible blue force actions that can be taken at any The set of 3rd order effects are evaluated (step 6) to iden-
moment in time. These include, as examples, the placement tify undesirable outcomes. Any undesired 3rd order effects
or movement of assets, sensor tasking, weapon engagement can be used to feedback into the set of blue force COA op-
decisions, the and the use of countermeasures. tions and eliminate undesired COAs. Thus, the conceptual
The capability requires a model of the red force, or adver- PA capability is an analysis tool to provide a deeper under-
sary, that estimates what is known about the adversary as standing of the COA options in terms of their possible causal
well as predicts what the adversary knows about the blue effects and expected consequences.
force and the situation. The PA capability evaluates (step 2) Each step in the PA capability can include an estimate of
each COA option in terms of our knowledge of the red the certainty of the analysis, providing a level of confidence
forces to predict (step 3) the effects of each option on the in the predictions. This would add even greater refinement
adversary. Steps 2 and 3 produce a set of predicted 1st order in terms of evaluating the desirability or undesirability of
effects. Each blue force COA option has a direct mapping to 3rd order effects, and consequently blue force COA options.
its predicted 1st order red force effect.
The conceptual PA capability can enhance future auto- about the status, location, and readiness of the blue force
mated tactical decision aids. Figure 2 illustrates a tactical warfighting resources or assets. The model would assess the
decision aid, showing how capabilities for PA and readiness of each resource as well as the overall force read-
knowledge discovery would interact with the tactical re- iness. The model would contain an assessment or prediction
sources (shown along the bottom row) as well as the “deci- of each resource’s capability to perform an assigned COA.
sion engine.” The conceptual resource management capabil- Examples include probability of kill, probability of detec-
ity would assess and prioritize missions and use those results tion, probability of jamming, etc. The model could also pre-
to develop the COAs, which the PA capability would eval- dict overall force capability given a particular threat envi-
uate based on predicted 1st, 2nd, and 3rd order effects. ronment.
The red force model is envisioned as an estimated predic-
tion of what is known about the adversary based on data and
intelligence available. This model would estimate what
types of capabilities the red force possesses and approximate
the overall red force readiness. The model would predict the
adversary’s intent, tactics, and strategies for the purpose of
predicting how the adversary might act in different situa-
tions or respond to blue force actions. The model could
make an educated guess as to what the adversary knows
about the situation and about the blue forces. This prediction
would be based on an assessment of the blue force’s possible
visibility to the red force based on what is known about the
adversary’s location and surveillance capabilities. The red
force model would become the Navy’s prediction of what is
known about the adversary.
Figure 2 – Predictive Analytics as a Core Capabil-
ity for an Automated Tactical Decision Aid
III. Data and Knowledge Concepts for Predic-
tive Analytics
A naval tactical “decision-maker is not interested in data
or big data as such; but the knowledge it provides. (Zhao,
Kendall, and Young, 2015, p. 22).” They are interested in
actionable knowledge required to gain and maintain the tac-
tical advantage. Gaining and maintaining knowledge of the
maritime domain is not only a required capability that ena-
bles the conceptual PA capability, but it has a direct impact
on the accuracy of the predictions made. The levels of com- Figure 3 – Three Knowledge Models to Support
pleteness and accuracy dictate how good the internal models Predictive Analytics
are as well as the predicted 1st, 2nd, and 3rd order effects. Figure 3 – Conceptual Computer-Aided Models as
Three categories of tactical maritime knowledge are illus- Enablers for a Predictive Analytics Capability
trated in Figure 3 as knowledge of the blue forces,
knowledge of the red forces, and knowledge of the opera- The operational situation model would constitute the cur-
tional situation. Conceptually, computer-aided models of rent situational or maritime awareness. This model, based
each could be created to support real-time naval operations primarily on real-time sensor data, would contain the under-
as well as the PA capability. Each model could be developed standing of the battlespace in terms of the weather, combat
and continuously updated based on the input data that is con- identification, and threat assessment and tracking. It would
stantly changing to reflect the changes in the states of the be comprised of information on the location, kinematics,
blue forces, the red forces, and the environment. and identification of all objects (friendly, neutral or foe) in
The blue force model would represent all that is known the area of interest. The model’s completeness, accuracy,
about the blue forces at any given time. It would provide the and up-to-date-ness would depend entirely on the data col-
Navy with self-awareness by containing what is known lected. The operational situation model would also contain
predictions of projected future states of the area of interest. A. Predictive Analytics as a Data-Driven and Auto-
Examples of this could include projected impact points of mated Process
threat missiles, projected locations of enemy aircraft and Abbott (2014) describes predictive analytics as a data-
ships, and future weather and environmental conditions. driven process of discovering interesting and meaningful
Developing and maintaining these changing and action- patterns and inducing models from the data, rather than bas-
able models depends on a number of data collection, fusion, ing results on assumptions made by the analyst. PA is a pro-
security, and management capabilities. The naval tactical cess that results in discovering variables to be included in
domain has data architectures in place for collecting, pro- the model, parameters that define the model, weights or co-
cessing, and fusing sensor data for developing situational efficients in the model, and also the very form of the model
awareness of the battlespace. This data supports combat itself. These models can then be used to build predictions.
identification, threat identification and tracking, and projec- PA, as described by Abbott (2014), does not do anything
tions of kinematic objects in the area of interest. that a human analyst could not accomplish manually given
In order to develop an internal model of the blue force, enough time. The reason to automate the process is because
the Navy would also need to collect data concerning blue the number of variables and permutations can quickly result
force asset status, location, and capability (Johnson, 2019, in thousands of computations. Automated algorithms can
Rowe 2019). Brown (2019) developed a conceptual archi- sift through the many potential combinations of data to iden-
tecture for collecting blue force asset data to support the de- tify patterns and interesting results.
termination of force readiness. Self-awareness data could One aspect of the conceptual PA capability that is beyond
also be used to determine individual resource readiness and human capability is the ability to store or “remember” all of
general blue force self-awareness. the numerous COA options, permutations, effects, and out-
In order to develop an internal model of the red force, the comes. In a tactical combat situation, these options and ef-
Navy needs to analyze the situational awareness data along fects would be changing continuously as the environment
with information from intelligence sources to make infer- changes, creating an even more complex memory challenge.
ences about the capabilities, location, and readiness of red And for the envisioned PA capability, the 1st, 2nd, and 3rd
force assets. The use of intelligence sources could be used order effects and permutations needs to be stored and easily
to model likely red force intent, tactics, and strategies. The accessed for evaluation.
combination of predicted red force asset knowledge with Automation also plays an important role in providing data
knowledge of our blue force assets can be used to make in- fusion, correlation and analytics for developing and contin-
ferences about the adversary’s knowledge of the blue force. ually updating the blue force, red force, and operational sit-
Maintaining knowledge of the real-world is a critical part uation models required for the conceptual PA capability.
of implementing a PA capability for the tactical Navy. The
models provide a “belief state” that become the basis for B. Statistical Methods
making predictions about the consequences of COAs. Rus- Statistical methods are widely used to perform a confirm-
sell and Norvig (2010, p. 480) write that the belief state is a atory analysis concerning a hypothesis involving a relation-
“representation of the set of all possible worlds” that a sys- ship between inputs and outputs (Abbott 2014). The analysis
tem may exist in. The belief state is then used to generate
confirms or denies the causal relationship and quantifies the
COA options and corresponding possible outcomes and
degree of that confirmation or denial.
consequences, and to evaluate these options.
Regression analysis is a statistical method for analyzing
and modeling the relationship between a continuous de-
IV. Artificial Intelligence and Game Theoretic pendent variable and an independent variable to build a
Methods for Predictive Analytics model for making predictions. The first step is to identify
and explain the best model that represents the relationship
A number of data analytic methods exist that can support between the dependent and independent variables. The sec-
the many different types of estimation and predictive capa- ond step is to use this model to predict future values of the
bilities that have been described up until this point. For ex- dependent variable given specific values of the independent
ample, Kalman filters have been widely used for projecting variable. (Kalaian and Kasim 2017)
the future kinematic states of moving objects in the bat- Discriminant analysis is a statistical technique that uses
tlespace. This is a form of computational prediction. Data the information from a set of independent variables to pre-
fusion analytics are used to combine and assess heterogene- dict the value of a discrete (categorical) dependent variable,
ous data from different types of sensor to enhance our ability which represents the mutually exclusive groups in the pre-
to identify and understand combat objects in the battlespace. dictive model (Kalaian and Kasim 2017). Discriminant anal-
This section focuses on AI and game theoretic methods that ysis can be used to identify the best combination of inde-
can potentially be used to evaluate the COA options by pre- pendent variables or predictors, that provide the best dis-
dicting 1st, 2nd, and 3rd order effects. crimination between groups in an effort to accurately predict
a membership in a particular group. This technique can be A Bayesian network representing the probabilistic varia-
used for threat identification (as friendly, neutral, or foe) to bles and their relationships for the combat identification of
match a tracked object’s characteristics to the appropriate an air object is shown in Figure 4. This network shows fac-
group’s predictive model. tors involved in determining the combat identification of an
Table 1 lists differences between using statistical methods airborne object based on what information is known about
and PA methods. Statistical methods can apply to small data the object and its environment. Factors, such as what is
sets and rely heavily on ensuring the models are built known about the object’s kinematics, the object’s proximity
properly and are typically linear; whereas PA methods draw to the airport and the level of turbulence in the near environ-
heavily on machine learning and AI, require lots of data, and ment, are shown as variables (or nodes) in the network. The
have no provable optimum solution. network also contains nodes representing how the object is
identified, such as by intelligence, interrogation friend or foe
Table 1 – Statistics vs. Predictive Analytics (Source: Abbott (IFF) or by electronic surveillance means (ESM). It can be
2014) noted that the directions of the arrows can be used to show
Statistics Predictive Analytics the causal relationship between the actual identity (in the
Models based on theory; Models often based on non- real world) and how it will affect the factors that allow it to
there is an optimum. parametric algorithms; no be identified. The arrow directions can also be reversed (as
guaranteed optimum. in Figure 4) to show that given a variety of information
Models typically linear. Models typically nonlinear. sources, they can be used to support the identification of the
Data typically smaller; algo- Scales to big data; algorithms object.
rithms often geared toward not as efficient or stable for
accuracy with small data. small data.
The model is king. Data is king.
C. Graph Theory Methods
Bayesian networks (Bayes network, belief network, deci-
sion network, Bayes model, or probabilistic directed acyclic
graphical model) are a category of statistical models that
represent a set of variables and their conditional dependen-
cies in the form of graphs. Bayesian networks offer a sys-
tematic way to represent relationships between variables
and their dependencies explicitly and concisely—greatly
simplifying the process of specifying probabilities for the
large numbers of variables that may exist (Russell and
Figure 4 - An Example Bayesian Network for Combat
Norvig 2010). Identification (Source: van Gosliga and Jansen, 2003)
Bayesian networks are ideal for taking an event that oc-
curred and predicting the likelihood that any one of several D. Decision Theory
possible known causes was the contributing factor. For ex- Decision theory provides methods for selecting among
ample, a Bayesian network could represent the probabilistic actions based on the desirability of outcomes, often in situ-
relationships between diseases and symptoms. Given symp- ations that are only partially understood. Russell and Norvig
toms, the network could be used to compute the probabilities (2010, p. 610) describe these situations as “nondeterministic
of the presence of various diseases. Efficient algorithms can partially observable environments.” Thus, the AI system
perform inference and learning in Bayesian networks. may not know the current state completely, so a random var-
Bayesian networks that model sequences of variables are iable is used to represent the possible outcome states. The
called dynamic Bayesian networks. Generalizations of decision-maker’s preferences are represented by a utility
Bayesian networks that can represent and solve decision function which assigns values corresponding to the desira-
problems under uncertainty are called influence diagrams. bility of the possible outcome states.
The network’s nodes represent observable quantities, hy- Automation can support the development and application
potheses, or unknown parameters. The edges represent con- of utility functions. The utility functions become complex
ditional dependencies. Each node is associated with a prob- for complex decision spaces, such as military tactical oper-
ability function whose input is a particular set of values rep- ations. Such problem spaces are characterized by many pos-
resenting the node’s parent variables and whose output is the sible outcomes and many possible factors affecting each
probability of the variable represented by the node. outcome. They also introduce uncertainty and dependences
among the variables representing factors. Automated sys- E. Learning-Driven Methods
tems can develop probability models reflecting the stochas- Learning in terms of AI systems is defined as “the capa-
tic processes that generate outcomes. The systems must also bility of drawing intelligent decisions by self-adapting to the
model the error in the utility estimates that may be intro- dynamics of the environment, taking into account the expe-
duced by unknowns, incomplete knowledge, and bias. The rience gained in past and present system states, and using
use of multi-attribute theory along with the models of ex- long term benefit estimations” (Kim 2018, p. 222). Imple-
pected utilities and associated error can provide an auto- menting learning algorithms requires large amounts of train-
mated aid for making decisions. ing data. Kim (2018) explains that progress is being made in
Decision theory can be thought of as the combination of learning algorithmic game theory which lies in the intersec-
probability theory and utility theory. The use of a decision tion of game theory and AI learning algorithms. These meth-
network, also called an influence diagram, combines Bayes- ods show potential for the military domain by implementing
ian networks with node types for actions and utilities. Deci- many iterations of a wargame and training the learning al-
sion networks provide a useful framework to aid AI in mak- gorithms to identify the best COAs and blue force strategies
ing complex decisions involving multi-attributes, multi-var- based on desired game outputs.
iables, many possible outcomes, and knowledge uncer- Supervised learning (also referred to as predictive model-
tainty. Figure 5 shows an example of an influence diagram ing) is a method that uses a “supervisor” target variable to
with a military application. The oval nodes, referred to as represent the answer to a question of interest or a value that
chance nodes, represent random variables. The rectangle is unknown but could support decision-making if known.
nodes, called decision nodes, represent decision points Supervised learning uses “ground truth” to train the AI sys-
where there is a choice among actions. The hexagonal nodes tem using prior knowledge. The goal is to learn a function
are the utility nodes which represent the AI system’s utility given some input data and desired outputs that best approx-
function. imates the relationship between the input and desired output.
Figure 5 – An Example Decision Network for Military Actions (Adapted from Campos and Ji 2008)
This function can then be used to classify target variables or first employing unsupervised learning to explore unlabeled
perform regression on continuous target variables. datasets to cluster and classify information in order to con-
AI machine learning methods can be used to support pre- struct a supervised classification model. Therefore, the AI
dictions based on comparing real-time data with “best mod- system is, in a sense, training itself in an automated fashion.
els.” Figure 6 shows a process of first training the system to Most game-learning algorithms are designed to improve
find a best model by running many iterations allowing su- a program based on watching or playing against knowledge-
pervised learning to occur. The best model can then be used able opponent players. Although it is certainly important to
in the operational system (in the second row) as a standard understand how a program (or player) could learn from good
by which to compare incoming real-time data. As data be- players, it is equally important to know how those good
gins to match the model, future state predictions can be in- players became good in the first place. Kim (2018) explains
ferred. that some learning work has considered how programs
might become strong players while relying neither on active
analysis nor on experience with experts. Most of these ap-
proaches can be considered as self-play, in which either a
single player or a population of players evolves during com-
petition on large numbers of contests.
F. Game Theoretic Methods
Game theory methods encompass a wide range of behav-
ioral relations among players and is an umbrella term for the
science of logical decision-making in humans and comput-
ers. Several game theoretic methods can support the naval
tactical predictive analytics application. These include de-
Figure 6 – Example of Supervised Learning for Predictive scriptive interpretation, normative (or prescriptive) interpre-
Analytics tation, counterfactuals, and regret minimization.
Descriptive interpretation is a way of viewing game the-
Unsupervised learning (or descriptive modeling), has no ory that attempts to predict how an adversary will act and
target variable or desired output. The goal of unsupervised respond in different strategic settings. This ability was in-
learning is to infer the natural structure present within a set cluded as part of the conceptual PA capability. Descriptive
of data points. Input data is analyzed and grouped together interpretation suggests that game theory can successfully
based on the proximity of input values to each other. The predict how an adversary will make decisions given a set of
groups are then segmented and labeled. Unsupervised learn- circumstances. This method assumes that the game players
ing is useful for exploratory analysis and for dimensionality are rational and will act to maximize their payoffs. While
reduction. Ontanon, Montana, and Gonzalez (2014) de- this method provides insights, it will be limited by the im-
scribe the process as “learning from observation” (LFO) and perfect knowledge held by both the blue force and the red
explain that the process discovers a “mapping” from the per- force.
ceived state of the environment and actions. Normative or prescriptive interpretation is a game theory
Machine learning has made continued progress in devel- method of selecting the best COAs for players. It is prescrip-
oping methods that can generalize from data, adapt to tive in that this method determines what the player “should
changing environments, and improve performance with ex- do,” rather than actually predicting what a player might do.
perience, as well as progress in understanding fundamental Normative (prescriptive) interpretation is a fundamental ap-
underlying issues. By integrating over the distribution of op- proach to the conceptual PA capability proposed in this pa-
ponent strategies rather than taking a simple empirical aver- per. This method attempts to determine the best blue force
age, insights from game theory can be used to derive novel COA based on what is established as the best outcome, or
learning algorithms (Blum 2008). Applying machine learn- the most desired 3rd order effect. A Nash equilibrium (an
ing to game theory may shed light on possible opponent important game theory concept) constitutes a player’s best
strategies by improving a program by playing a game many response to the actions of other players. Thus, the concep-
times against a knowledgeable opponent player. Unsuper- tual PA capability could provide an analytical way to deter-
vised learning enables modeling of the real-world and red mine when the blue force’s COA constitutes a Nash equilib-
forces when sensor data is not matching known (supervised rium. However, it is important to note that there are situa-
learning) constructs. It can support the classification of sen- tions in which it is best to play a non-equilibrium strategy if
sor observations and predict some inferential knowledge. one expects the red force to do so, or if blue force assets
Doherty et. al. (2016) propose using a multi-step process of need to be conserved for a longer-term mission.
Counterfactuals (another game theory concept) are claims not fixed and sequential as they are in a chess game. In naval
or hypotheses that are contrary to the facts. A counterfactual operations, the resources, tactics, and COAs can be vast, dy-
can be thought of as a hypothetical state of the world used namic, changing over time, can occur at any time, and are
to assess the impact of action. Counterfactuals are often largely unknown to the opponent. Therefore, the game the-
written as conditional statements in which the conditional ory approach must function with incomplete information
clause is false—imagining hypothetically what could have and large numbers of instances. Zinkevich et. al. (2007) de-
happened. Counterfactual thinking can support the concep- scribe a process of implementing regret minimization in
tual PA capability by considering as many possible hypo- games with incomplete information to determine a Nash
thetical future states as possible and analyzing them to elim- equilibrium for very large instances to minimize counterfac-
inate undesired COAs. tual regret which minimizes overall regret. In this process, a
A related game theory method is regret minimization. framework creates an abstraction of a particular decision
This is a method of running many possible counterfactual point to approximate the behavior of the CFR. These ap-
hypotheses and carefully altering different COA decisions proximations are then mapped back into the full game.
in each run (or game) to see if this has a positive or negative Brown et. al. (2019) explain that this CFR abstraction
effect on the outcome. Regret refers to how much better a method can be manual and domain specific and may miss
player would have done if they had made one decision over strategic nuances of the game. They describe the use of Deep
another at a specific decision point in the game. CFR which uses deep neural networks instead of the CFR
Zinkevich et. al. (2007) describe counterfactual regret abstraction to approximate the behavior of CFR in the full
minimization (CFR) as a self-play algorithm that learns to game. Deep CFR shows promise as a game theoretic method
play a game by repeatedly playing against itself. It starts for PA but requires significant computational power.
with a strategy that is uniformly random, where it will play
every action at every decision point with equal probability.
It simulates playing games against itself and after every V. Conclusion
game, it revisits decisions and finds ways to improve its This paper presented a conceptual framework for apply-
strategy. It repeats this process for all combinations of ing PA to naval tactical decisions as an automated battle
games (which can amount to millions or billions of runs), management aid. It described the need to develop and main-
improving its strategy each time. As it plays, it gets closer tain knowledge models of the blue force, red force, and op-
and closer towards an optimal strategy for the game: a strat- erational situation and described how these models are re-
egy that can do no worse than tie against any opponent. The quired for a future PA capability. The paper evaluated AI
way it improves over time is by summing the total amount and game theoretic methods, describing how a combination
of regret it has for each action at each decision point and of statistical, graph theory, decision theory, learning-driven,
selecting the combination with the least amount of regret. and game theory methods can be applied to enable a future
Positive regret for a particular COA means that the blue PA capability.
force would have done better if they had taken that action The payoff for implementing a PA capability as part of an
more often. Negative regret means that the blue force would automated battle management aid is predicting 1st, 2nd, and
have done better by not taking that action at all. 3rd order effects of possible COAs in order to make the most
After each game in CFR with the program playing against effective tactical decisions. This real-time wargaming capa-
itself, it computes and adds in the new regret values for all bility would enable short-term and long-term objectives to
of the decisions it just made. It then recomputes its strategy be weighed, contributing to ensuring that preferred out-
so that it takes actions with probabilities proportional to their comes are more likely. A PA capability could bridge the gap
positive regret. If an action would have been good in the between tactical and strategic thinking, emphasizing causal
past, then it will choose it more often in the future. It repeats efficacy – or consequences of actions. In the decades ahead,
this process for billions of games. Therefore, CFR produces the Navy will need to maintain maritime decision superior-
a long sequence of strategies that it was using on each game. ity by incorporating strategic thinking into naval tactical de-
Counter-intuitively, that sequence of strategies does not nec- cisions – this can be accomplished with predictive analytics.
essarily converge to anything useful. However, if you com-
pute the average strategy over those billions of strategies in
the sequence, then that average strategy will converge to- References
wards a Nash equilibrium for the game. In a chess-like Abbott, Dean. 2014. Applied Predictive Analytics: Principles and
game, this average strategy can then be used against any op- Techniques for the Professional Data Analyst. Hoboken, NJ: Wiley
ponent. & Sons.
However, naval tactical situations are vastly more com- Bilusich, Daniel, Fred Bowden, and Svetoslav Gaidow. 2011. “Ap-
plex than chess-like games. The numbers and types of blue plying Influence Diagrams to Support Collective C2 in Multina-
force and red force warfare resources, tactics, and COAs are
tional Civil-Military Operations.” Presented at the 16th Interna- Zhao, Ying, Anthony Kendall, Bonnie Young, and W. Baumgart-
tional Command and Control Research and Technology Sympo- ner. 2015. “Big Data Architecture and Analytics for Improving
sium. Combat Identification and the Common Tactical Air Picture.” Na-
Blum, Avrim and Yishay Mansour. 2007. “Learning, Regret Min- val Research Program Research Report, Naval Postgraduate
imization, and Equilibria.” Chapter 4 (pp. 79-102) in book, Algo- School, Monterey, CA.
rithmic Game Theory, Edited by Noam Nisan, Tim Roughgarden Zinkevich, Martin, Michael Johanson, Michael Bowling, and Car-
Eva Tardos, and Vijay Vazirani, Cambridge University Press. melo Piccione. 2007. “Regret Minimization in Games with Incom-
Brown, Noam, and Tuomas Sandholm. 2018. “Superhuman AI for plete Information.” Technical Report for Department of Compu-
Head-up No-limit Poker: Libratus Beats Top Professionals.” Sci- ting Science, University of Alberta.
ence Vol 359, pp. 418-424, January 2018.
Brown, Noam, Adam Lerer, Sam Gross, and Tuomas Sandholm.
2019. “Deep Counterfactual Regret Minimization.” Proceedings of
the 36th International Conference on Machine Learning (ICML),
Long Beach, CA.
Campos, Cassio, and Qiang Ji. 2008. “Strategy Selection in Influ-
ence Diagrams Using Imprecise Probabilities.” Proceedings of the
24th Conference on Uncertainty in Artificial Intelligence. Helsinki,
Finland, July 2008.
Cox, Louis. 2015. Breakthroughs in Decision Science and Risk
Analysis. Hoboken, New Jersey: Wiley & Sons.
De Campos, Cassio, and Qiang Ji. 2008. “Strategy Selection in In-
fluence Diagrams using Imprecise Probabilities.” Presented at the
24th Conference on Uncertainty in Artificial Intelligence.
Doherty, Conor, Steven Camina, Kevin White, and Gary Oren-
stein. 2016. The Path to Predictive Analytics and Machine Learn-
ing. Sebastopol, CA: O’Reilly.
Bowling, Michael, Neil Burch, Michael Johanson, and Oskari
Tammelin. 2015. “Heads-Up Limit Hold’em Poker is Solved.” Sci-
ence Vol 347, Issue 6218, pp. 145-149. 09 January 2015.
Kalaian, Sema, and Rafav Kasim. 2017. Decision Management:
Concepts, Methodologies, Tools, and Applications. Information
Resources Management Association (IRMA).
Kim, Sungwook. 2014. Game Theory Applications in Network De-
sign. Information Science Reference.
Kim, Sungwook. 2018. “Basic Concepts for Game Theory.” Chap-
ter 10 in the book, Game Theory: Breakthroughs in Research and
Practice. IGI Global: Hershey, PA.
Kuhn, Max, and Kjell Johnson. 2016. Applied Predictive Model-
ing. New York, New York: Springer.
Moravcik, Matej, Martin Schmid, Neil Burch, Viliam Lisy, Dustin
Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael John-
son, and Michael Bowling. 2017. “DeepStack: Expert-Level Arti-
ficial Intelligence in Headd-Up No-Limit Poker.” Science, Vol
356, Issue 6337, pp. 508-513. 01 March 2017.
Ontanon, Santiago, Jose Montana, and Avelino Gonzalez. 2014.
“A Dynamic-Bayesian Network Framework for Modeling and
Evaluating Learning from Observation.” Expert Systems with Ap-
plications, Vol 41, Issue 11, pp. 5212-5226, September 2014.
Pearl, Judea. 2008. “Causal Inference.” JMLR Workshop and Con-
ference Proceedings. 6:39-58.
Russell, Stuart, and Peter Norvig. 2010. Artificial Intelligence, A
Modern Approach, 3rd edition. Upper Saddle, NJ: Pearson.
Talbot, Patrick, and Dennis Ellis. 2015. Applications of Artificial
Intelligence for Decision-Making. Middletown, DE.
Van Gosliga, Ing Sicco Pier, and Ir Henk Jansen. 2003. “A Bayes-
ian Network for Combat Identification.” Paper presented at the
RTO IST Symposium on Military Data and Information Fusion and
published in RTO-MP-IST-040, Prague, Czech Republic, 20-22
October 2003.