=Paper=
{{Paper
|id=Vol-1788/STIDS2016_T04
|storemode=property
|title=An Extended Maritime Domain Awareness Probabilistic Ontology Derived from Human-aided Multi-Entity Bayesian Networks Learning
|pdfUrl=https://ceur-ws.org/Vol-1788/STIDS_2016_T04_Park_etal.pdf
|volume=Vol-1788
|authors=Cheol Young Park,Kathryn Blackmond Laskey,Paulo C. G. Costa
|dblpUrl=https://dblp.org/rec/conf/stids/ParkLC16
}}
==An Extended Maritime Domain Awareness Probabilistic Ontology Derived from Human-aided Multi-Entity Bayesian Networks Learning==
https://ceur-ws.org/Vol-1788/STIDS_2016_T04_Park_etal.pdf
An Extended Maritime Domain Awareness
Probabilistic Ontology Derived from Human-aided
Multi-Entity Bayesian Networks Learning
Cheol Young Park, Kathryn Blackmond Laskey, Paulo C. G. Costa
The Sensor Fusion Lab & Center of Excellence in C4I
George Mason University, MS 4B5
Fairfax, VA 22030-4444 U.S.A.
cparkf@masonlive.gmu.edu, [klaskey, pcosta]@gmu.edu
Abstract— Ontologies have been commonly associated with semantics based on Multi-Entity Bayesian Networks (MEBN),
representing a domain using deterministic information. a Bayesian probabilistic logic [1]. PR-OWL has been extended
Probabilistic Ontologies extend this capability by incorporating to PR-OWL 2 [14], which provides a tighter link between the
formal probabilistic semantics. PR-OWL is a language that deterministic and probabilistic aspects of the Ontologies.
extends OWL with semantics based on Multi-Entity Bayesian MEBN is flexible enough to represent a variety of complex and
Networks (MEBN), a Bayesian probabilistic logic. Developing uncertain situations. MEBN has been applied to systems
probabilistic ontologies can be greatly facilitated by the use of a [2][3][4][5][6][7] for Predictive Situation Awareness (PSAW),
modeling framework such as the Uncertainty Modeling Process providing the ability to estimate and predict aspects of a
for Semantic Technology (UMP-ST). An example of using UMP-
temporally evolving situation.
ST was the development of a probabilistic ontology to support
PROGNOS (PRobabilistic OntoloGies for Net-Centric Developing probabilistic ontologies can be greatly
Operational Systems), a system that supports Maritime Domain facilitated by the use of a modeling framework such as the
Awareness (MDA). The PROGNOS probabilistic ontology UMP-ST, a modeling process for constructing a probabilistic
provides semantically aware uncertainty management to support ontology [13]. The UMP-ST consists of four main disciplines:
fusion of heterogeneous input and probabilistic assessment of (1) Requirement, (2) Analysis & Design, (3) Implementation,
situations to improve MDA. However, manually developing and and (4) Test. UMP-ST was used to develop a probabilistic
maintaining a probabilistic ontology is a labor-intensive and ontology to support PROGNOS (PRobabilistic OntoloGies for
insufficiently agile process. Greater automation through a
Net-Centric Operational Systems), a system to support
combination of reference models and machine learning methods
may enhance agility in probabilistic situation awareness (PSAW)
Maritime Domain Awareness (MDA). The existing system for
systems. For this reason, a process for Human-aided MEBN MDA (e.g., US Navy's Net-Centric infrastructure, FORCENet)
Learning in PSAW (HMLP) was suggested. In previous work, we is used to fuse lower-level multi-sensor data, analyze the fused
used UMP-ST to develop the PROGNOS probabilistic ontology. data by human analysts, and support decision-making for naval
This paper presents an extended PROGNOS probabilistic operations. However, the era of big data requires greater
ontology developed using HMLP. The contribution of this automation. The PROGNOS probabilistic ontology [7]
research is to introduce the extended PROGNOS probabilistic supports ingestion of lower-level data, fusion of heterogeneous
ontology and present a comparison between two processes (UMP- input, and probabilistic assessment of situations to improve
ST and HMLP). MDA. PROGNOS is a prototype system that aims especially to
identify threatening targets (e.g., terrorists and terrorist-ships).
Keywords—Probabilistic Ontology; Maritime Domain
Awareness; Predictive Situation Awareness; Bayesian Networks; Manually developing and maintaining a probabilistic
Multi-Entity Bayesian Networks; Uncertainty Modeling Process for ontology is a labor-intensive and insufficiently agile process.
Semantic Technology; Human-aided Machine Learning Furthermore, it is important to make use of data when available.
Therefore, greater automation through a combination of
I. INTRODUCTION reference models and machine learning methods has the
In information science, integration of heterogeneous, potential to enhance agility and effectiveness in modeling a
distributed, and unstructured information is a difficult and probabilistic ontology for PSAW. For this reason, a process for
complex challenge. A major issue is ensuring information Human-aided MEBN Learning in PSAW (HMLP) has been
compatibility, for which ontologies have become a standard suggested [20]. HMLP contains three supporting
solution [18]. Traditional ontologies are limited to methodologies, MEBN-RM [10], a reference MEBN model for
deterministic knowledge. Probabilistic Ontologies (POs) move PSAW [8], and MEBN learning algorithms [9][10]. These
beyond this limitation by incorporating formal probabilistic component methodologies enable efficient and effective
semantics. Probabilistic OWL (PR-OWL) [19] is a modeling. MEBN-RM and the reference model are introduced
probabilistic ontology language that extends OWL with in Section 2 below.
STIDS 2016 Proceedings Page 28
In previous work, we used UMP-ST to develop the resident node in MEBN, an MFrag mapping between a relation
PROGNOS PO. This paper presents an extended PROGNOS in RM and an MFrag in MEBN, and an MTheory mapping
PO developed using HMLP. In the following sections, the between an RM and an MTheory. An MTheory can be
paper (1) provides background information, (2) introduces the constructed automatically from a relational database by using
original PROGNOS PO derived from UMP-ST, (3) presents mapping rules in MEBN-RM. Therefore, MEBN-RM can
the extended PROGNOS PO derived from HMLP, and (4) support a MEBN learning algorithm, which develops an
compares two processes. MTheory from a dataset, or an MTheory developer, who aims
to develop an MTheory using domain knowledge and MEBN
II. BACKGROUND knowledge. HMLP exploits MEBN-RM for efficient
This section introduces (1) MEBN, (2) MEBN-RM development of an MTheory.
Mapping Model, (3) A Reference MEBN Model for PSAW, (4) C. A Reference MEBN Model for PSAW
Uncertainty Modeling Process for Semantic Technology
(UMP-ST), and (5) Human-aided MEBN learning in PSAW A reference model is an abstract framework to which a
(HMLP). HMLP assumes input data based on the relational developer refers in order to develop a specific model. A
model (RM) as its data schema. We choose RM because it is reference MEBN model for PSAW is a reference model for a
the most popular database model and has the necessary PSAW-MTheory which specifies references for MFrags, RVs,
expressive power to represent entities and their relationships. It relationships of RVs, and entities. The reference MEBN model
is necessary to define how to convert elements of RM to for PSAW can support the design of a PSAW-MTheory and
elements of MEBN, so a mapping rule between MEBN and improve the quality of the PSAW-MTheory. The references for
RM, called MEBN-RM, was developed. Also, we introduce a entity are classified into five categories (Time entity T,
reference MEBN model for PSAW which provides a set of Observer entity OR, Sensor entity SR, Target entity TR, and
basic templates to support the design of a MEBN model for Reported target entity RT). Entities derived from these
PSAW. HMLP is a modification of UMP-ST, so UMP-ST is categories describe a situation in which an observer OR
introduced in this section. Some of the following background observes a target TR and interprets it as a reported target RT
summaries are taken from [20]. using a sensor SR at a certain time T [20]. The reference
MEBN model for PSAW provides some referring random
A. MULTI-ENTITY BAYESIAN NETWORKS variables (RV), called PSAW-RVs. PSAW-RVs are classified
MEBN is a compact model combining Bayesian networks into five categories (Observing condition RV, Reported object
(BN) with First-order logic (FOL) to represent repeated RV, Target object RV, Situation RV, and Context RV). These
structures in a joint distribution representing domain PSAW-RVs are defined in five types of MFrags (Observing
knowledge. MEBN is a highly expressive model for treating condition MFrag, Report MFrag, Target MFrag, Situation
uncertainty and complex forms of data and information. A MFrag, and Context MFrag). An observing condition RV
MEBN model, called an MTheory, is composed of fragments, defined in an observing condition MFrag represents
called MFrags. An MFrag consists of a set of resident nodes, a probabilistic knowledge about conditions of a sensor (e.g.,
set of context nodes, a set of input nodes, an acyclic directed maintenance conditions for a sensor). A reported object RV
graph for the nodes, and a set of class local distributions (CLD) defined in a report MFrag represents probabilistic knowledge
for the nodes. A resident node is a random variable which is about a relation or an attribute of observed targets (e.g., a
associated with a function or predicate of FOL and whose class reported target size). A target object RV defined in a target
local distribution is resident in an MFrag. A context node is MFrag represents probabilistic knowledge about a relation or
derived from a resident node and determines conditions under an attribute for actual targets (e.g., an actual target size). A
which the class local distribution defined in the MFrag is valid. situation RV defined in a situation MFrag represents
An input node has its distribution defined elsewhere and probabilistic knowledge about situations of targets (e.g., a
conditions the class local distribution defined in the MFrag. collaborating situation for targets). A context RV defined in a
Nodes for an acyclic directed graph are associated with context MFrag represents probabilistic knowledge about
resident and input nodes. An FOL function or predicate of a conditions under which the class local distribution defined in
resident node contains ordinary variables, which can be the MFrag is valid. For example, an RV Predecessor(pre_t, t)
replaced with entity identifiers to generate multiple instances of can be a context RV. The context RV Predecessor(pre_t, t)
the RVs. MFrags in an MTheory are used to generate instances means that the time interval pre_t occurs immediately before
of fragments of BN. The fragments of BN are combined to the time interval t. More specific information for the reference
form a Bayesian network, called a situation-specific Bayesian MEBN model for PSAW can be found in [20].
Network (SSBN). An MTheory can be used to generate an D. Uncertainty Modeling Process for Semantic Technology
unbounded number of different SSBNs. Further information
(UMP-ST)
about MEBN can be found in [1].
UMP-ST is a framework to support the design of a
B. MEBN-RM Mapping Model probabilistic ontology [13]. The PROGNOS probabilistic
MEBN-RM [10] is a mapping model which provides a ontology was developed using UMP-ST. UMP-ST provides
specification for how to convert relational databases [11][12] processes for constructing a probabilistic ontology through four
to MTheories [1]. The relational model (RM) is the most disciplines: (1) Requirement, (2) Analysis & Design, (3)
popular database model. MEBN-RM provides an entity Implementation, and (4) Test.
mapping between a relation in RM and an entity in MEBN, a
resident node mapping between an attribute in RM and a
STIDS 2016 Proceedings Page 29
In the Requirement discipline, requirement statements are Fig. 1. Process for Human-Aided MEBN Learning (This figure was taken
defined. The requirement statements can contain goals, queries, from [20] and was modified)
and evidence for a probabilistic ontology. Objectives to be
achieved by reasoning with the probabilistic ontology are
specified by statements for goals (e.g., detect a ship of interest).
To achieve the objectives, specific query statements are
specified in this discipline (e.g., what is the type of a ship?). To
support the queries, evidence associating with the queries is
determined in this discipline (e.g., an appearance of a ship). In
the Analysis & Design discipline, entities, attributes,
relationships, and probabilistic rules are defined. These are
used to support the goals, queries, and evidence. For example, Stakeholders who request the development of a reasoning
we are developing a probabilistic ontology, which aims to model or a probabilistic ontology provide needs and/or
detect a ship of interest (the goals). The goal is achieved by missions as inputs of HMLP. An output from the end of HMLP
identifying the type of a ship (the queries) given information is a reasoning model (in our case, a probabilistic ontology for
about the appearance of the ship (the evidence). For this PSAW). The followings describe the four steps in HMLP. (1)
situation, a ship entity is required. Also, type and appearance In the Analyze Requirements step, requirements which contain
attributes for the ship entity are required. Suppose that the goals to be achieved, queries to answer, and evidence to be
appearance attribute may depend on the type attribute. This is used in answering queries are defined. Also, the requirements
specified by a probabilistic rule. In the Implementation include performance criteria, which are used in the Test
discipline, a probabilistic ontology is developed using results Reasoning Model step, to evaluate the probabilistic ontology.
from the previous disciplines. A probabilistic ontology based (2) In the Design World Model and Rules step, a world model
on MEBN is used to reason about uncertainty. Therefore, a and rules are developed using the requirements in the previous
probabilistic ontology contains OWL classes based on step. This step contains two sub-steps (Design World Model
elements from MEBN such as an MFrag, an MTheory, a node, step and Design Rules step). The Design World Model step
a probability distribution, and an entity. In this step, these defines the world model which may include entities, attributes,
OWL classes are defined. For example, the ship entity defined and relations (e.g., RM) using the requirements, domain
in the previous discipline is mapped to an entity type indicating knowledge and/or existing data schemas. The world model is
a ship in the probabilistic ontology. The attributes ship used to identify rules. In the Design Rules step, the rules or
appearance and ship type are mapped to random variables ship influencing relationships between attributes in the world model
appearance and ship type, respectively. The probabilistic rule are defined. (3) In the Construct Reasoning Model step, a
for the attributes ship appearance and ship type is converted to probabilistic ontology is constructed using a training dataset,
the joint probability for the random variables ship appearance the world model, and the rules. This step includes two sub-
and ship type. The random variables ship appearance and ship steps (Map to Reasoning Model step and Learn Reasoning
type may belong to an MFrag representing attributes of a ship. Model step). The Map to Reasoning step maps the world model
The MFrag ship and other MFrags related with a maritime and rules to a candidate probabilistic ontology. The Learn
domain may integrate into an MTheory representing a Reasoning Model uses a MEBN learning method to learn the
maritime situation. The Test discipline is used to assess the probabilistic ontology from a training dataset. (4) The Test
probabilistic ontology developed in the Implementation Reasoning Model step evaluates the learned probabilistic
discipline. More specific information for UMP-ST can be ontology in the previous step to determine whether to accept it.
found in [13]. The accepted probabilistic ontology is a final result from
HMLP.
E. Human-aided MEBN learning in PSAW (HMLP)
III. PROGNOS PO VIA UMP-ST
HMLP is a framework which aims the development of a
probabilistic ontology in PSAW. HMLP provides specific To develop the PROGNOS PO, three iterations of the four
development steps and supporting methods (MEBN-RM, the steps in UMP-ST (Requirement, Analysis & Design,
reference MEBN model for PSAW, and MEBN learning). Implementation, and Test) were performed [14]. The following
HMLP improves MEBN learning by providing expert sub-sections summarize the four steps in UMP-ST to develop
knowledge which is used to limit the search space of the PROGNOS PO.
parameters, variables, and structures for a probabilistic
A. Requirements
ontology in PSAW.
The Requirement step identifies requirements containing
Similar to the four disciplines of UMP-ST, HMLP contains goals, queries, and evidence for a probabilistic ontology. The
four steps (Fig. 1): (1) Analyze Requirements, (2) Design requirements for the PROGNOS PO were developed gradually
World Model and Rules, (3) Construct Reasoning Model, and over the three iterations. In the first iteration, a simple
(4) Test Reasoning Model. (See a full discussion of HMLP in requirement regarding a ship of interest was identified [7]. In
[20]). A summary of HMLP is presented below. the second iteration, requirements for two types of terrorist-
ships were defined. In the third iteration, requirements for crew
members in a ship of interest were specified. The following list
shows part of the resulting requirements [14].
STIDS 2016 Proceedings Page 30
1. Identify if a ship is of interest, electronic devices such as Radio, Radar, and AIS (Automatic
1.1 Is the ship being used to exchange illicit cargo? Identification System). A ship can show behavior such as
1.1.1 Was the ship hijacked? Aggressive, Erratic, Evasive, and Normal. A ship can have a
1.1.2 Does the ship have a terrorist crew member? (terrorist) crewmember who may belong to a (terrorist)
1.1.2.1 Is the crew member associated with any terrorist organization. A ship can have a terrorist plan such as
organization? BombPort and ExchangeIllicitCargo.
... After developing the model semantics, conditional rules
1.2 Is the ship being used as a suicide ship to bomb a port? were identified. There were three iterations of this process. The
... following list shows a few of the conditional rules from [14].
The main goal was to identify a ship of interest (i.e., a
terrorist-ship). In this requirement, we assumed the ship of 1.(a) If a crew member is a member of a terrorist
interest may exchange illicit cargo and/or be used as a suicide organization, then it is more likely that he is a terrorist.
ship to bomb a port. To support this goal, we needed to identify 1.(b) If an organization has a terrorist member, it is more
the type of a crew member of a ship. If the type of a crew likely that it is a terrorist organization.
member is a terrorist, the ship is highly likely to be a terrorist- ...
ship. To identify whether a crew member is a terrorist, we can 4.(a) Research shows that if a crew member has a relationship
check whether the crew member is associated with any terrorist with terrorists, there is a 68% chance that he has a friend who
organization. is a terrorist.
...
B. Analysis & Design
These conditional rules were derived from extensive
This step defines the types of entities, their properties and research about terrorism [16] and from the knowledge provided
relationships, and the rules that apply to them, i.e., the by a domain expert. These rules were used to develop the
semantics of the domain model. The Unified Modeling PROGNOS PO.
Language (UML) diagrams can provide a convenient and
understandable visualization of the classes and relationships for C. Implementation
the model semantics. The requirements defined in the previous In the Implementation step, the PROGNOS PO was
step are used to develop the model semantics. Thus, entities, designed. The PROGNOS PO can be found in [14][15]. Fig. 3
attributes for the entities, and relationships between the entities shows the PROGNOS PO containing five groups of MFrags.
were identified. For example, from Requirement 1, an entity
Fig. 3. Original PROGNOS probabilistic ontology
was derived (i.e., a ship) and an attribute of the entity was
derived (i.e., the type of a ship). From Requirement 1.1.2, a
new entity was derived (i.e., a (terrorist) person) and a
relationship between the entities was derived (i.e., a ship has a
crew (terrorist) member). In the second iteration, Carvalho [14]
developed the model represented by UML as shown in Fig. 2.
Fig. 2. Entities, their attributes, and relations for the MDA model after the
second iteration (This figure provided by permission of Carvalho [14])
The first set of MFrags is for a ship of interest. It includes
nine MFrags Aggressive Behavior, Terrorist Plan, Evasive
Behavior, Erratic Behavior, Unusual Route, Bomb Port Plan,
Ship Of Interest, Electronics Status, and Exchange Illicit Cargo
Plan. These MFrags are used to reason about properties of a
ship (e.g., unusual behavior and an illegal plan). The second set
of MFrags is for a person of interest. It includes four MFrags
Person Communications, Person Background Influences,
Person Cluster Associations, and Person Relations. These
MFrags are used to identify a person who may communicate
with a terrorist, has a suspicious background and history, and
The classes and relationships form six natural groups (i.e., has a relationship with a terrorist. The third set of MFrags is for
Electronics, Behavior, Ship, Position, Plan, and Social information of relationships between two ships. It includes two
Network). The ship types are NavyShip, FishingShip, and MFrags, Radar and Meeting. These MFrags are used to
MerchantShip. Ship routes are UnusualRoute and UsualRoute. identify whether one ship is within radar range of another ship
Two ships can meet each other at a position. A ship can use
STIDS 2016 Proceedings Page 31
and whether two ships are meeting. The fourth set of MFrags is 84
85
]
[F: TerroristPlan_MFrag
for information about the relationship between a person and an 86
87
[C: isA(ship,Ship)]
[R: hasTerroristPlan(ship) [IP: hasTerroristCrew(ship)][IP: isHijacked(ship)]]
organization. It includes one MFrag Terrorist Person in which 88
89
]
[F: ElectronicsStatus_MFrag
a person who belongs to an organization is identified. The last 90
91
[C: isA(ship,Ship)]
[R: isElectronicsWorking(ship)]
set of MFrags is for information about a relationship between a 92
93
[R: hasResponsiveRadio(ship)
[IP: hasEvasiveBehavior(ship)][RP: isElectronicsWorking(ship)]]
person and a ship. It includes two MFrags Has Terrorist Crew 94
95
[R: hasResponsiveAIS(ship)
[IP: hasEvasiveBehavior(ship)][RP: isElectronicsWorking(ship)]]
and Ship Characteristics. These MFrags are used to link a 96 ]
97 [F: Radar_MFrag
person and a ship, and to identify whether a ship has a terrorist 98 [C: isA(ship1,Ship), isA(ship2,Ship)] [C: ( ¬ ( ship1 = ship2 ) )]
99 [R: isWithinRadarRange(ship1, ship2)]
crew member. 100 ]
101 [F: PersonClusterAssociations_MFrag
102 [C: isA(person,Person)]
The following list shows part of a partial PROGNOS PO 103 [R: hasOccupation(person) [RP: hasClusterPartition(person)]]
104 [R: hasEducationLevel(person) [RP: hasClusterPartition(person)]]
containing information about MFrags (F), context nodes (C), 105 [R: hasClusterPartition(person) [IP: isTerroristPerson(person)]]
resident nodes (R), resident parent nodes (RP), and input parent 106
107
[R: hasEconomicStanding(person) [RP: hasClusterPartition(person)]]
[R: hasNationality(person) [RP: hasClusterPartition(person)]]
nodes (IP). Note that a partial probabilistic ontology doesn't 108 ]
contain a class local distribution and domain information for a PO 1 shows the context nodes and the resident nodes in the
random variable. MFrags, and the relationship between the resident nodes. For
PO 1: Original PROGNOS probabilistic ontology example, the MFrag ErraticBehavior_MFrag (Line 1~6)
1
2
[F: ErraticBehavior_MFrag
[C: isA(ship,Ship)]
contains an isA context node and three resident nodes
3
4
[R: hasErraticBehavior(ship) [IP: hasExchangeIllicitCargoPartition(ship)]]
[R: hasEquipmentFailure(ship)]
hasErraticBehavior, hasEquipmentFailure, and isCrewVisible.
5
6 ]
[R: isCrewVisible(ship)[RP: hasErraticBehavior(ship)][RP: hasEquipmentFailure(ship)]] The resident node hasErraticBehavior is influenced by an
7
8
[F: TerroristPerson_MFrag
[C: isA(person,Person), isA(org,Organization)]
input node hasExchangeIllicitCargoPartition. The resident
9
10
[R: isTerroristOrganization(org)[RP: isTerroristPerson(person), isMemberOfOrganization(person, org)]]
[R: isTerroristPerson(person)][R: isMemberOfOrganization(person, org)]
node isCrewVisible is influenced by the resident nodes
11 ] hasErraticBehavior and hasEquipmentFailure. This
12 [F: ShipCharacteristics_MFrag
13 [C: isA(ship,Ship), isA(person,Person)] PROGNOS PO was tested in the next step.
14 [R: hasCrewMember(ship, person)] [R: hasTypeOfShip(ship)][R: isHijacked(ship)]
15 ]
16 [F: EvasiveBehavior_MFrag D. Test
17 [C: isA(ship,Ship)]
18
19 ]
[R: hasEvasiveBehavior(ship)[IP: hasExchangeIllicitCargoPartition(ship)]] In this step, the PROGNOS PO was evaluated to determine
20
21
[F: PersonCommunications_MFrag
[C: isA(person,Person)]
whether to accept it. To do this, the case-based evaluation, in
22
23
[R: communicatesWithTerrorist(person)[IP: isTerroristPerson(person)] ]
[R: usesChatroom(person) [RP: communicatesWithTerrorist(person)]]
which various scenarios were defined and used to examine the
24 [R: usesEmail(person) [RP: communicatesWithTerrorist(person)]] reasoning implications of the probabilistic ontology, was used.
25 [R: usesCellular(person) [RP: communicatesWithTerrorist(person)]]
26 [R: usesWeblog(person) [RP: communicatesWithTerrorist(person)]] For example, given a scenario which was developed by a
27 ]
28 [F: PersonBackgroundInfluences_MFrag subject matter expert (SME), some information (e.g., history of
29 [C: isA(person,Person)]
30 [R: hasInfluencePartition(person) [IP: isTerroristPerson(person)]] a target) from the scenario for a target was used as evidence for
31
32
[R: knowsPersonImprisionedInOIForOEF(person) RP: hasOIForOEFInfluence(person)]]
[R: hasFamilyStatus(person) [RP: hasInfluencePartition(person)]] inference of the PROGNOS PO to identify some properties
33
34
[R: hasOIForOEFInfluence(person) [RP: hasInfluencePartition(person)]]
[R: knowsPersonKilledInOIForOEF(person) [RP: hasOIForOEFInfluence(person)]] (e.g., whether the target is a terrorist) of the target. If the result
35
36
]
[F: AggressiveBehavior_MFrag
of inference coincided exactly with the scenario from SME, we
37
38
[C: isA(ship,Ship)]
[R: hasAggressiveBehavior(ship) [IP: hasBombPortPlan(ship), hasExchangeIllicitCargoPartition(ship)]]
could think that the probabilistic ontology was reasonable. For
39
40
[R: hasWeaponVisible(ship) [RP: hasAggressiveBehavior(ship)]]
[R: isJettisoningCargo(ship) [RP: hasAggressiveBehavior(ship)]]
this test, three qualitatively different scenarios were used [14].
41 [R: speedChange(ship) [RP: hasAggressiveBehavior(ship)]]
42
43
[R: turnRate(ship) [RP: hasAggressiveBehavior(ship)]]
[R: propellerTurnCount(ship) [RP: speedChange(ship)]]
After three iterations for UMP-ST, an overall test for the
44
45
[R: cavitation(ship) [RP: speedChange(ship)][RP: turnRate(ship)]]
[R: shipRCSchange(ship) [RP: turnRate(ship)]]
PROGNOS PO was performed using a simulation. In the real
46 ] world situation, it is very difficult to acquire a real dataset to
47 [F: ShipOfInterest_MFrag
48 [C: isA(ship,Ship)] [R: isShipOfInterest(ship) [IP: hasTerroristPlan(ship)]] develop such a probabilistic ontology which contains rare
49 ]
50 [F: ExchangeIllicitCargoPlan_MFrag events. For this reason, the simulation was used to produce a
51 [C: isA(ship,Ship)]
52 [R: hasExchangeIllicitCargoPlan(ship) [IP: hasTerroristPlan(ship)]] test dataset given different scenarios generated randomly.
53
54
[R: hasExchangeIllicitCargoPartition(ship)
[IP: hasTypeOfShip(ship)][RP: hasExchangeIllicitCargoPlan(ship)]] Carvalho [14] and Costa et al [15] introduced some results for
55
56
]
[F: PersonRelations_MFrag this test. In such a test, it is important that knowledge used to
57
58
[C: isA(person,Person)]
[R: hasKinshipToTerrorist(person) [RP: hasTerroristBeliefs(person)]]
develop a probabilistic ontology and knowledge used to
59
60
[R: hasFriendshipWithTerrorist(person) [RP: hasTerroristBeliefs(person)]]
[R: hasTerroristBeliefs(person) [IP: isTerroristPerson(person)]]
develop a simulation for testing the probabilistic ontology
61
62
]
[F: Meeting_MFrag
should not be same. If they are same, the test is meaningless,
63
64
[C: isA(ship1,Ship), isA(ship2,Ship)]
[C: ( ¬ ( ship1 = ship2 ) )]
because the probabilistic ontology and the simulation are same
65 [R: areMeeting(ship1, ship2) [IP: hasExchangeIllicitCargoPartition(ship1)]] models, but just in different forms.
66 [R: areMeetingReport(ship1, ship2) [RP: areMeeting(ship1, ship2)]]
67 ]
68 [F: BombPortPlan_MFrag IV. PROGNOS PO VIA HMLP
69 [C: isA(ship,Ship)] [R: hasBombPortPlan(ship) [IP: hasTerroristPlan(ship)]]
70
71
]
[F: HasTerroristCrew_MFrag In this section, we introduce an extended PROGNOS PO
72
73
[C: isA(ship,Ship), isA(person,Person)]
[C: hasCrewMember(ship,person)] derived from the HMLP process. The following shows how the
74
75 ]
[R: hasTerroristCrew(ship) [IP: isTerroristPerson(person)]]
development operates.
76 [F: UnusualRoute_MFrag
77 [C: isA(ship2,Ship), isA(ship1,Ship)]
78 [C: ( ¬ ( ship1 = ship2 ) )]
A. Analyze Requirements
79 [R: hasUnusualRoute(ship1)
80 [RP: hasNormalChangeInDestination(ship1)] This step is not much different from the requirement step in
81 [IP: hasBombPortPlan(ship1)][IP: areMeeting(ship1,ship2)]]
82 [R: hasUnusualRouteReport(ship1) [RP: hasUnusualRoute(ship1)]] UMP-ST. Therefore, we can reuse requirements developed
83 [R: hasNormalChangeInDestination(ship1) [IP: hasTypeOfShip(ship1)] ]
STIDS 2016 Proceedings Page 32
from the PROGNOS project. The full requirements can be evidence (e.g., reported objects) to be reported to estimate
found in [14]. However, the reference MEBN model for actual targets (e.g., target objects), so relations (i.e.,
PSAW can provide more items by which a PSAW modeler can Ship_Report, Person_Report, Organization_Report,
consider predefined entities, RVs, and MFrags for PSAW. Ship_Ship_Report, Person_Org_Report, Ship_Person_Report,
Recall the four MFrag groups from the reference model: and ReportedTarget) for the reported object group are added in
Reported Object, Observing Conditions, Target Object, and the world model for the extended PROGNOS PO.
Situation. The last of these, Situation, is of special note. In Observations may contain observation errors influenced by
PSAW, understanding a situation in which targets operate for observing conditions (e.g., weather). The observing condition
their own purposes is one of the important issues. Identifying group contains two relations Sensor and SensorProperty. In the
just the type of a target is an insufficient task for PSAW. The previous step, a requirement for the awareness for a situation
meaning of awareness is not to perceive and estimate actual was added. Therefore, we added a relation Field for the
properties of a target but is to understand, interpret, and explain situation group in Fig. 4. Relations (i.e., Location, SensorOf,
the relationships between targets. Kokar et al [17] stated: “The and ActualTarget) which are not classified in these groups are
main part of being aware is to be able to answer the question supporting relations used to join the relations in the four groups.
of “what’s going on?””. Awareness of a situation is subjective Fig. 4. Part of EER Model for a PROGNOS world model
according to an observer, who is aware of the situation. The
modeler, who is developing a probabilistic ontology to support
PSAW, should define what situation will be considered and
explained through all observation from the world. For the
awareness of the PROGNOS situation, we add the following
new requirement.
New Goal 1: Recognize emergency situation at sea
Query 1.1: How high is the potential terrorist
threat?
Evidence 1.1.1: Ship(s) of interest
Evidence 1.1.2: Crew member(s) of
interest
The new goal aims to alert a response team when the threat
reaches a certain level. This will be accomplished by
estimating potential terrorist attacks in the field given
estimation of terrorist ships and terrorist crew members.
In HMLP, a requirement can contain a performance
criterion specifying a measure of accuracy (e.g., the mean
squared error or the Brier score [26]). For example, we might
require that the mean squared error between ground truth and The reference model provides some rules or relationships
estimated results from the probabilistic ontology shall be less between these groups as shown in the arrows (Fig. 4). The
than a given threshold (e.g., a mean squared error < 0.1). observing conditions group and the target object group can
influence the reported object group. For example, the attribute
B. Design World Model and Rules sensorPerformance in the relation SensorProperty influenced
This step performs two sub-steps (Design World Model and the report attributes in the report relations Ship_Report,
Design Rules). The Design World Model step is to define a Person_Report, Organization_Report, Ship_Ship_Report,
world model for PROGNOS from the requirements defined in Person_Org_Report, and Ship_Person_Report. The arrows in
the previous step. Fig. 4 indicate these relationships. The following shows a few
of these rules.
In this step, the reference MEBN model for PSAW can be
used to identify possible entities, random variables, and Rule 1: causal ({hasErraticBehavior, sensorPerformance},
relationships between the random variables. Fig. 4 shows a hasErraticBehaviorRPT)
PROGNOS world model represented in an EER (enhanced
Rule 2: causal ({isShipOfInterest, isTerroristPerson},
entity–relationship) model. We develop the PROGNOS world
PotentialTerroristAttacks)
model using the requirements and the reference model.
...
The reference model suggests four groups: (1) Reported
Object, (2) Observing Condition, (3) Target Object, and (4) Rule 1 means that two attributes hasErraticBehavior and
Situation. A world model for the original PROGNOS PO sensorPerformance cause the attribute hasErraticBehaviorRPT.
included the seven relations (e.g., Target, Ship, Person, Rule 2 means that two attributes isShipOfInterest and
Organization, Person_Org, Ship_Person, and Ship_Ship). The isTerroristPerson cause the attribute PotentialTerroristAttacks.
original PROGNOS PO treated only the target object group. In
other words, it did not emphasize sensing. We would expect
STIDS 2016 Proceedings Page 33
C. Construct Reasoning Model
In PO 2, we added the ship report MFrag which can be used
This step performs two sub-steps (Map to Reasoning Model to reason about Rule 1. Also, we added the situation MFrag
and Learn Reasoning Model) to construct the PROGNOS PO. which can be used to reason about Rule 2.
MEBN-RM provides a converting rule from RM to a
probabilistic ontology. Entity relations which contain only one In the Learn Reasoning Model step, the extended
attribute for the primary key of the relation (e.g., ship and PROGNOS PO can be refined using a MEBN learning
person) can be defined as entity types in the probabilistic algorithm. The goal of MEBN learning is to learn an MTheory
ontology. Each of the attributes in the relations could be from a training dataset. A basic MEBN learning method for
mapped to a resident node in the probabilistic ontology using relational datasets was suggested [9][10]. This approach
MEBN-RM. For example, the attribute hasErraticBehavior of assumes that the training dataset is stored in a relational
the relation Ship became the resident node database based on RM. MEBN learning searches parameters,
hasErraticBehavior(ship). variables, and structures to find an MTheory that provides a
good fit to the training dataset. In our case, the structures are
Rules which are defined in the previous step are used to given by the above steps as suggested in the PSAW reference
develop relationships between resident nodes in the model. Therefore, only parameter learning is required. The
probabilistic ontology. For example, from Rule 1, we had a goal of parameter learning is to estimate the parameters of a
conditional dependence P(hasErraticBehaviorRPT(ship_report) class local distribution L given a training dataset D and the type
| hasErraticBehavior(ship), sensorPerformance(shipSensor, of distribution being learned, which fit well the training dataset
ship)). From Rule 2, we had a conditional dependence D.
P(PotentialTerroristAttacks(field) | isShipOfInterest(ship),
isTerroristPerson(person)). For a discrete random variable case, Dirichlet distribution is
commonly used because it is conjugate to the multinomial
We could model the extended PROGNOS PO as shown in distribution. With a Dirichlet prior distribution, the posterior
Fig. 5 using the resident nodes, the relationships between the predictive distribution has a simple form [21][22]. For
resident nodes, and the MFrag groups. continuous random variables, multiple regression can be used.
Fig. 5. Extended PROGNOS probabilistic ontology Park et al [9] introduced a basic MEBN parameter learning and
structure learning for a conditional Gaussian hybrid model in
which no discrete random variable may have a continuous
parent random variable.
For example, parameters for a conditional Gaussian
distribution can be estimated using multiple regression. The
following class local distribution (CLD) is an illustrative
example of a conditional linear Gaussian CLD for the node
Speed_RPT(rt, tr), which means a speed report rt for a target tr.
The CLD of the node is a continuous CLD with hybrid parents
(Sensor_Condition and Speed). In this case, we assume that the
discrete parent node Sensor_Condition(sr, tr), which means a
condition of a sensor sr for a target tr, has two states (Good and
Bad) and the node Speed (tr), which means an actual speed of a
target tr, is continuous.
Fig. 5 shows a set of MFrags in the extended PROGNOS CLD 1 [Conditional Linear Gaussian]: Speed_RPT(rt, tr)
PO. The list on the left indicates the four MFrag groups. Each 1 if some sr.tr have (Sensor_Condition = Good) [
2 Ɵ1.0 + Ɵ1.1*Speed + NormalDist(0, Ɵ1.2)
group is decomposed into sub-groups. For example, the target 3 ] else [
object group contains five sets of MFrags (Person MFrags, 4 Ɵ2.0 + Ɵ2.1*Speed + NormalDist(0, Ɵ2.2)
Ship MFrags, MFrags for the relationship between two ships, 5 ]
MFrags for the relationship between a person and a ship, and
MFrags for the relationship between a person and an Parameter learning for this CLD estimates the parameters
organization). The following list (PO 2) shows part of new (Ɵ1.0, Ɵ1.1, and Ɵ1.2) in Line 2 and the parameters (Ɵ2.0, Ɵ2.1,
MFrags added into the extended PROGNOS PO. and Ɵ2.2) in Line 4 using multiple regression.
PO 2: Part of New MFrags added into the original PROGNOS probabilistic ontology
1 [F: Orgainzation_Report_MFrag
D. Test Reasoning Model
2 [C: isA(sr,SENSOR), isA(tr ,ORGANIZATION), isA(rt,REPORTEDTARGET)]
3 [C: SensorOf(sr, tr), tr = ReportedTarget(rt)] This step performs two sub-steps (Experiment Reasoning
4 [R: isTerroristOrganizationRPT(rt)
5 [IP: isTerroristOrganization(tr)] Model and Evaluate Experimental Results) to evaluate the
6
7 ]
[IP: performance(sr, tr)]
extended PROGNOS PO from the test dataset. In the
8
9
]
[F: Situation_MFrag Experiment Reasoning Model step, the performance of
10
11
[C: isA(ship,SHIP), isA(person,PERSON), isA(field,FIELD)]
[C: field = Location(ship)]
estimation and prediction for the extended PROGNOS PO can
12
13
[C: hasCrewMember(ship, person)]
[R: PotentialTerroristAttacks(field) [IP: isShipOfInterest(ship), isTerroristPerson(person)]]
be assessed using a performance measure (e.g., the mean
14
15
]
...
squared error or the Brier score). Each experiment consists of
the following five steps. (1) The test dataset provides entity
STIDS 2016 Proceedings Page 34
information (e.g., ship1, person1, and field1) and ground truth number of entities in the extended model is ten (Field, Ship,
information (e.g., isShipOfInterest_ship1 = true, Person, Organization, ShipSensor, PersonSensor,
isTerroristPerson_person1 = true) to the extended PROGNOS OrganizationSensor, ReportedShip, ReportedPerson, and
PO. (2) Given these, the extended PROGNOS PO is used to ReportedOrganization). Table 1 shows that the feature of the
compute a marginal probability distribution (e.g., extended PROGNOS PO is more comprehensive than the
P(PotentialTerroristAttacks_field1 | isShipOfInterest_ship1 = feature of the original PROGNOS PO. The original
true, isTerroristPerson_person1 = true) in response to a query. PROGNOS PO contains 51 RVs, while the extended
(3) The test dataset provides ground truth data (e.g., PROGNOS PO contains 115 RVs. This means that the
PotentialTerroristAttacks_field1 = High). (4) Steps 1-3 are extended PROGNOS PO can answer more various questions.
repeated for all test cases. (5) Finally, for results for all cases, For example, a reasoning about potential terrorist attacks in a
the measures are calculated. field can be performed using the extended PROGNOS PO, but
the original PROGNOS PO can’t. Also, the extended
In the Evaluate Experimental Results step, we evaluate the PROGNOS PO contains observing conditions for sensors, so
measures using the performance criteria in the requirements this may enable us to perform more accurate reasoning.
defined in the Analyze Requirement step (e.g., a mean squared
error < 0.1). If the evaluation is not satisfied (e.g., a mean
TABLE 1. Comparison between the original PROGNOS probabilistic
squared error >= 0.1), we can return to the previous steps to ontology and the extended PROGNOS probabilistic ontology
improve the performance of the extended PROGNOS PO. We
can investigate the extended PROGNOS PO in the Construct Entities
Random
Relationships MFrags
Reasoning Model step. Unsatisfactory performance can be Variables
Original 3 51 53 18
caused by a training database of insufficient size. In this case, Extended 10 116 147 36
we may find more datasets and apply them to the learning
process. Also, it is possible that the MEBN learning algorithm
If we assume that there is a training dataset for MEBN
which we use is ineffective. In this case, the application of a
learning, the development period for the PROGNOS PO can be
more effective MEBN learning algorithm is required. The
reduced. Usually, to develop an RV and its parameter, we
world model in the Construct Reasoning Model step can be
study literature related to the RV and find possible parameters
incorrect. For this, we may need to conduct a further field
for the RV. Another way for the development of such an RV is
investigation and research to develop a more accurate world
model. The requirements in the Analyze Requirements step can to use domain expert knowledge. A subject matter expert
be impracticable or requires a too high standard to address it. In (SME) may provide values and parameters for the RV, and
this case, readjustments for the requirements can be performed relationships between RVs. In the PROGNOS project, to
by the stakeholders. develop one RV, we used the following steps: (1) an SME in
the maritime domain explained domain knowledge to an RV
V. COMPARING UMP-ST AND HMLP developer, (2) the RV developer developed the RV using the
HMLP is a modification of UMP-ST that specifies some MEBN/PR-OWL software [27], and (3) the RV in the
detailed sub-steps and uses two reference models (the reference MEBN/PR-OWL software was evaluated by the SME. These
MEBN model for PSAW and MEBN-RM). These reference steps were implemented with at least one day per RV. If we
models can support efficient modeling for a probabilistic assume that for each RV, one day may be required to develop it
ontology in PSAW. The first steps (Requirement) for both by one RV developer and one SME, then the original
processes are same. In the case of HMLP, the reference MEBN PROGNOS PO requires around 51 days. On the contrary, if we
model for PSAW provides some guidance on groups of assume that all datasets are available, the development with
entities to be defined (i.e., Reported Object, Observing MEBN learning may require around one day for setting the
Condition, Target Object, and Situation). In the second step of datasets and learning a PO using a MEBN learning algorithm.
HMLP, the reference model also supports developing a world VI. CONCLUSION
model in terms of PSAW by providing candidate entities (i.e.,
UMP-ST was applied for construction of a probabilistic
T, OR, SR, TR, and RT), attributes, and relationships. In the
ontology to support PROGNOS including the PROGNOS PO.
third step of HMLP, MEBN-RM supports the development of
The PROGNOS PO played an important role in the operation
entities, random variables, and MFrags from a relational
of PROGNOS. However, manually developing and
schema. HMLP also makes use of MEBN learning algorithms,
maintaining a probabilistic ontology is a labor-intensive and
so given a training dataset, a probabilistic ontology can be
insufficiently agile process. Therefore, HMLP containing the
efficiently constructed. The second and third steps are mainly
reference models and machine learning methods was
different with UMP-ST. These steps in HMLP can accelerate
introduced. In the previous work for PROGNOS, UMP-ST was
the modeling for probabilistic ontologies in PSAW and
applied to develop the PROGNOS PO. This paper applied
produce more comprehensive models.
HMLP to develop the extended PROGNOS PO which was
Table 1 shows feature comparison between the original more comprehensive than the original model and was
PROGNOS PO and the extended PROGNOS PO. Each developed more quickly.
number in the table means the number of the features (entities,
random variables, relationships between random variables, and The following summarizes future research issues. (1)
MFrags). For example, the number of entities in the original HMLP in this research was not fully applied with MEBN
model is three (Ship, Person, and Organization), while the learning from a training dataset. Evaluation of effectiveness
(i.e., reasoning accuracy) of reasoning models learned from
STIDS 2016 Proceedings Page 35
MEBN learning is required. (2) A probabilistic ontology can [13] Carvalho, R. N., Laskey, K. B., & Da Costa, P. C. (2016). Uncertainty
contain MFrags, context nodes, resident (or inputs) nodes, modeling process for semantic technology (No. e2045v1). PeerJ
Preprints.
graphs, FOL formula for nodes, and class local distributions for
[14] Carvalho, R. N. (2011). Probabilistic Ontology: Representation and
nodes. These elements can be subject to MEBN learning. Modeling Methodology. PhD Dissertation. George Mason University.
Especially, FOL formula learning in a probabilistic ontology is [15] Costa, Paulo. C. G., Laskey, Kathryn B., Chang, Kuo-Chu, Sun, Wei,
a difficult topic relative to the others. In our approach, a dataset Park, Cheol Y., & Matsumoto, Shou. (2012). High-Level Information
for learning is given from a relational database. Because we Fusion with Bayesian Semantics. Proceedings of the Nineth Bayesian
rely on MEBN-RM, we do not need to perform the Modelling Applications Workshop.
complicated task of FOL formula learning from text data. FOL [16] Sageman, M. (2004). Understanding terror networks. University of
formula learning in a probabilistic ontology can be supported Pennsylvania Press.
by Inductive Logic Programming [23][24] and Statistical [17] Kokar, M. M., Matheus, C. J., & Baclawski, K. (2009). Ontology-based
Natural Language Processing [25]. (3) Also, future steps for situation awareness. Information fusion, 10(1), 83-98.
the extended PROGNOS PO are to apply it to a realistic [18] Smith, B. (2003). Ontology. Retrieved September 2, 2016from
http://ontology.buffalo.edu/smith/articles/ontology_pic.pdf.
reasoning system for Maritime Domain Awareness.
[19] Paulo C. G Costa, Bayesian Semantics for the Semantic Web. PhD
ACKNOWLEDGMENTS Dissertation, George Mason University, July 2005. Brazilian Air Force.
[20] Park, C. Y., Laskey, K. B., Costa, P. C., & Matsumoto, S. (2016). A
We appreciate Dr. K. C. Chang, Dr. W. Sun, Dr. R. process for human-aided Multi-Entity Bayesian Networks learning in
Carvalho, Dr. R. Haberlin, Mr. S. Matsumoto, and Mr. A. Predictive Situation Awareness. In 19th International Conference on
Mugali for their efforts in the previous PROGNOS research. Information Fusion (FUSION).
The research was partially supported by the Office of Naval [21] Heckerman, D., Geiger, D., & Chickering, D. M. (1995). Learning
Research (ONR), under Contract#: N00173-09-C-4008. Bayesian networks: The combination of knowledge and statistical data.
Machine Learning, 20:197–243.
[22] Koller, D., & Friedman, N. (2009). Probabilistic Graphical Models:
Principles and Techniques. The MIT Press, 1 edition.
REFERENCES [23] Lavrac, N. & Dzeroski, S. (1994). Inductive Logic Programming:
Techniques and Applications. Ellis Horwood, New York.
[1] Laskey, K. B. (2008). MEBN: A Language for First-Order Bayesian [24] Muggleton, S., & De Raedt, L. (1994). Inductive logic programming:
Knowledge Bases. Artificial Intelligence, 172(2-3). Theory and methods. The Journal of Logic Programming, 19, 629-679.
[2] Laskey, K. B., D’Ambrosio, B., Levitt, T. S., & Mahoney, S. M. (2000). [25] Manning, C. D., & Schütze, H. (1999). Foundations of statistical natural
Limited Rationality in Action: Decision Support for Military Situation language processing (Vol. 999). Cambridge: MIT press.
Assessment. Minds and Machines, 10(1), 53-77. [26] Brier, G. W. (1950). Verification of forecasts expressed in terms of
[3] Wright, E., Mahoney, S. M., Laskey, K. B., Takikawa, M. & Levitt, T. probability. Monthly weather review, 78(1), 1-3.
(2002). Multi-Entity Bayesian NetworksforSituation Assessment. [27] Costa, P., Ladeira, M., Carvalho, R. N., Laskey, K., Santos, L., &
Proceedings of the Fifth International Conference on Information Fusion. Matsumoto,S. (2008). A first-order Bayesian tool for probabilistic
[4] Costa, P. C. G., Laskey, K. B., Takikawa, M., Pool, M., Fung, F., & ontologies. in Proceedings of the Twenty-First International Florida
Wright, E. J. (2005). MEBN Logic: A Key Enabler for Network Centric Artificial Intelligence Research Society Conference (FLAIRS 2008),
Warfare. Proceedings of the 10th ICCRTS. (Coconut Grove, FL,USA), pp. 631–636, AAAI Press.
[5] Suzic, R. (2005). A generic model of tactical plan recognition for threat
assessment. In Defense and Security (pp. 105-116). International Society
for Optics and Photonics.
[6] Costa, P. C. G., Laskey, K. B., & Chang, K. C. (2009). PROGNOS:
Applying Probabilistic Ontologies To Distributed Predictive Situation
Assessment In Naval Operations. Proceedings of the 14th ICCRTS.
[7] Carvalho, R. N., Costa, P. C. G., Laskey, K. B., & Chang, K. C. (2010).
PROGNOS: predictive situational awareness with probabilistic
ontologies. In Proceedings of the 13th International Conference on
Information Fusion.
[8] Park, C. Y., Laskey, K. B., Costa, P. C. G., & Matsumoto, S. (2014).
Predictive Situation Awareness Reference Model using Multi-Entity
Bayesian Networks. Proceedings of the 17th International Conference
on Information Fusion.
[9] Park, C. Y., Laskey, K. B., Costa, P. C. G., & Matsumoto, S. (2013).
Multi-Entity Bayesian Networks Learning For Hybrid Variables In
Situation Awareness. Proceedings of the 16th International Conference
on Information Fusion (Fusion 2013).
[10] Park, C. Y., Laskey, K. B., Costa, P. C. G., & Matsumoto, S. (2013).
Multi-Entity Bayesian Networks Learning In Predictive Situation
Awareness. Proceedings of the 18th International Command and Control
Technology and Research Symposium (ICCRTS 2013).
[11] Codd, E. F. (1970). A Relational Model of Data for Large Shared Data
Banks. Communications of the ACM.
[12] Codd, E. F. (1969). Derivability, Redundancy, and Consistency of
Relations Stored in Large Data Banks. IBM Research Report.
STIDS 2016 Proceedings Page 36