=Paper=
{{Paper
|id=None
|storemode=property
|title=SCUBA: An Agent-Based Ontology Creation and Alignment Method for Socio-Cultural Modeling
|pdfUrl=https://ceur-ws.org/Vol-808/STIDS2011_CR_T2_KretzEtAl.pdf
|volume=Vol-808
|dblpUrl=https://dblp.org/rec/conf/stids/KretzPP11
}}
==SCUBA: An Agent-Based Ontology Creation and Alignment Method for Socio-Cultural Modeling==
https://ceur-ws.org/Vol-808/STIDS2011_CR_T2_KretzEtAl.pdf
SCUBA: An Agent-Based Ontology Creation and
Alignment Method for Socio-Cultural Modeling
Donald R. Kretz William D. Phillips Bruce E. Peoples
Raytheon IIS Raytheon NCS Raytheon IIS
Garland, TX Largo, FL State College, PA
donald_r_kretz@raytheon.com bill.phillips@raytheon.com bruce_e_peoples@raytheon.com
Abstract -- An otherwise promising business, political, or military We apply ontology as our modeling method of choice. An
strategy can be crippled by an incomplete understanding of the ontology can conceptualize a complex domain in a way that
social-cultural factors that define and influence a region. Such both humans and machines can understand, but the use of
omissions are sometimes due to oversight, but often stem from a ontology in this context presents us with two important
fundamental lack of understanding of how to model such difficult
challenges. First, manual ontology creation is a time-
and unfamiliar concepts. The information required to generate
useful contextual models is typically available but vast, and consuming and highly subjective process, particularly when
manual interpretation of detailed text is time-consuming, highly attempting to model abstract social and cultural concepts.
subjective, and requires specialized skills. The SCUBA project While formal models are required to conform to strict rules
achieved a balanced human-computer modeling paradigm to 1) involving provable logic and model consistency, they will
automate the creation of social and cultural ontologies from always incorporate some amount of bias. Every human
selected source materials using previously-developed tools, 2) modeler will have a slightly different perspective of the same
apply a variety of nominal, semantic, structural, and statistical small part of the world, and will make different value
matching techniques to align multiple ontologies using an agent- judgments about what parts are important and how those parts
based multimodel, and 3) evaluate the effectiveness of the
interrelate. Striving for added richness by adding more
generation and alignment processes using precision, recall, and
various other measures of effectiveness. Preliminary results of information only complicates this problem and adds to the
our initial agent-based experiments were promising – by applying severity of the “knowledge acquisition bottleneck” [2]. We
ensembles of multiple matching techniques, we achieved believe, therefore, that by applying automated ontology
significant improvements in alignment F-scores and other generation against various corpora of domain-relevant
measures of performance while dramatically reducing the materials, we can generate a useful first approximation of a
amount of time required to manually produce coordinated, useful domain model. An automated generation process will “learn”
domain models. from the information it can “study”. The model it constructs
will, therefore, be representative of the “world” described in
Keywords-ontology; ontology alignment; social ontology; the input material it receives.
cultural ontology; ensemble alignment; agent-based alignment
The second challenge involves the alignment of multiple
I. INTRODUCTION models. Accommodating multiple domain ontologies is
Having an incomplete understanding of the social-cultural usually necessary to capture the complexities of domains
factors that define and influence a region can cripple an having socio-cultural dimensions and to leverage existing
otherwise promising business, political, or military strategy. models. There has been a loosely-associated body of work in
Too often, models that guide strategy development and this area that we collected under the general heading of
operational planning do not include critical social and cultural “ontology alignment theory”. Our interpretation of this theory
elements. These omissions can be blamed partly on oversight, is essentially built on the principle of approximation – because
but often stem from a fundamental lack of understanding of any ontology is an approximate representation of its real-world
how to model such difficult and unfamiliar concepts. The domain, generating and aligning multiple ontologies that all
information required to generate useful contextual models is represent the same domain yields a richer higher-order
typically available but is often distributed across vast approximation of the real world (i.e., removes some of the
repositories. Furthermore, the manual interpretation of detailed subjectivity or bias associated with applying a single model).
text is time-consuming, highly subjective, and requires
As described by Euzenat and Shvaiko [3], the matching
specialized skills. We believe that socio-cultural awareness is
operation accepts ontologies as inputs, and produces an
best achieved by a system that combines multiple information
ontology as its output (see Figure 1). The input ontologies (O1
sources using a variety of automated extraction, mediation, and
and O2) are independent domain ontologies, perhaps derived
analysis tools, but guided by a human knowledge engineer in
from different sources of information or developed by
an interactive paradigm called balanced cooperative modeling
different ontology engineers. Optionally, a third ontology (Ω)
[1].
may be included as input – this may be an upper ontology or
may be the composite ontology (or alignment) produced by a
The authors are grateful to have received funding for SCUBA under the 2010
Raytheon Innovation Challenge.
previous matching operation. The latter case suggests that Observation #2: Certain
ertain matching techniques produce
matching operations can be chained for continued refinement more useful alignments for certain
by feeding the output from one operation (i.e., an aligned ontology pairs than others
other
ontology) as input to the next matching process.
Observation #3: The
he selection of candidate ontology
pairings and matching techniques can
be guided by heuristics and aided by the
inspection of model metacharacteristics
metach
This paper presents SCUBA, an agent-based
agent framework
for ontology alignment based on the observations stated
above. We will describe the methodology we applied, as well
as provide some initial results.
II. METHODOLOGY
The objective of SCUBA was to develop a balanced
human-computer
computer modeling paradigm to 1) automate a the
creation of social and cultural ontologies from selected source
materials, 2) apply a variety of nominal, semantic, structural,
and statistical matching techniques to align multiple
Figure 1. Matchingg operation, from Euzenat and S
Shvaiko [3] ontologies, and 3) evaluate the effectiveness of the generation
and alignment processes. Since our work was mainly focused
In addition to the ontologies, a couple of additional inputs on the alignment framework, we will concentrate most of this
are provided. First, a set of rules directs the matcher to section on that effort.
perform certain types of comparisons (i.e., which entities or
attributes to compare,, what sort of comparison to make, etc.). A. Ontology Generation
These rules are derived from a set of basic matching In answering the challenge of ontology generation, we
techniques described later in this paper.. To accompany the relied on the groundbreaking work of a number of others,
rules, a set of parameters informs the matcherer what limits or particularly Maedche and Volz[4]] and Cimiano and Völker
constraints to impose on the rules. For example
example, a rule might [5]. We used the common academic ontology generator
cause a name similarity technique to be performed using a Text2Onto [6]] to generate ontologies from bodies of text we
fuzzy string comparison on a “name” property, but a obtained from various sources, including the Yale University
parameter might indicate that only values having a confidence Human Relations Area Files (HRAF) [7], [ Yale University's
valuee higher than 50% are to be considered a match. Finally, Outline of Cultural Materials (OCM) [8], the United Nations
the
he result of this operation is an ontology, referred to as Ω Development Programme – Human Development
De Reports [9]
prime,, that expresses the set of correspondences between the and others. Documents were clustered by geographic area, and
entities in O1 and O2. a separate ontology was generated for each area. The team
Considering the points made above, we set two primary also generated 95 separate ontologies utilizing the 54 Cultural
goals for our project: Eliminate the knowledge acquisition and 41 Social text files obtained from open source materials.
material
bottleneck through semantic parsing and extraction of domain All ontologies were created in the Web Ontology Language
concepts from data sources into multiple ontologies and (OWL) format.
contexts, and bridge the gap between multiple, heterogeneous Additionally, the
he team manually generated a set of "gold
ontology. In our
generated ontologies and a single domain ontology standard” ontologiess to compare with the automatically
initial phase, we chose to apply the previous work of others in generated models.. Seven ontologies were created based on the
generating ontologies from text using readily available tools Department of Defense (DoD) PMESII-PT
PMESII paradigm (Political,
(citations to follow). Our investigation,
tion, instead, focused on the Military, Economic, Social, Information,
Information Infrastructure,
effective alignment of ontologies through various techniques Physical Environment, Time) [10] [10 using Protégé [11]. An
of mapping. Here, too, we borrowed from the work of others ontology for Time was not created.
created To provide instance data
for specific techniques and algorithms (citations to follow)
follow). for the PMESII-P P ontologies, the team developed a method to
However, we began with the premise that ontologies have automate the merging of Yale HRAF instance data with Yale
characteristics that make them more or less suited for effective OCM codes in the developed ontologies,
onto saving weeks of
alignment with certain other ontologies. Rather than approach manual labor.
the problem using a single
ngle technique or by applying complex
n-way comparisons,, we formulated three key observations that B. Ontology Alignment
guided our efforts: 1) General Approach
Observation #1: Certain pairs of ontologies are more As stated, we focused most of our work on ontology
effectively
ffectively aligned with one another alignment. We again borrowed heavily from the body of prior
than with other ontologies research in specific ontology matching techniques, most of
which were collected and documented by Euzenat and • HA - Heuristic Agent:: determine which ontology pairs
Shvaiko [3]. ]. In order to investigate our own hypotheses, make good candidates for matching, which matching
however, we constructed a customized agent-based
based framework behaviors should
uld be applied, and manage the
using the Java Agent DEvelopment Framework (JADE) [12]. execution of selection and matching workflows.
We used agents to develop automated workflo
workflows for the two
• MA - Matching Agent:: creates mappings of the
main component processes: selecting the optimal set of
concepts and relationship types between two
alignment candidates and most promising match techniques
techniques,
ontologies.
and performing the matching operation by applying the rules
to the alignment candidates (Figure 2).. Specific match • SA - Similarity Agent:: calculates the similarity
simil
techniques were re encoded as composable sets of agent between concepts.
behaviors. An agent-oriented design allowed ed us to apply a
• UA - Utility Agent:: performs supporting tasks such as
technique known as “ensemble forecasting”, which is common
data and ontology storage/retrieval, job ID
in highly specialized domains such as weather prediction.
management, etc.
Yilmaz [14] refers to this idea as a multimodel
multimodel, or a set of
component models that, together, define the behavior of a Each matching algorithm or technique was implemented as
more complex process. Using ensemble forecasting or a behavior. In JADE parlance, a behavior is a set of actions to
multimodeling,, various combinations of matching algorithms be performed. Coding each set of actions in a separate
(“behaviors”) were re applied against concept pairs, then component, rather than in the agent itself, allowed
allow each agent
evaluated in order to determine the strength of the match. An to select and compose the behaviors it wishedwishe to use to
average, or ensemble mean,, of the different behaviors inspire
inspired complete an assigned task. There are many techniques for
greater confidence because it essentially smoothe
smoothed the performing the matching operation, and some are betterb at
performance peaks and troughs introduced by model matching certain types of entities and properties than others.
imperfections or context sensitivities.
ties. For example, the Furthermore, a better match might result in some cases if more
concepts “car” and “automobile” produce very low results for than one technique is applied at the same time (“matcher
all name-based
based match behaviors, but semantic match composition”).
behaviors rate them as nearly identical. Hence, while any one
technique for matching two concepts is inherently unreliabl
unreliable, • Name-based
based (“terminological”) techniques compute
an ensemble mean that accounts for the strengths and some measure of similarity based on strings
weaknesses of all match techniques yield yielded a higher- containing names, descriptions, comments, etc.
confidence correlation. The matches can be used to produce a Comparisons based on simple or fuzzy string
merged ontology in any format desired; e.g., a set of OWL comparisons would match “George Bush” with
assertions (i.e., “sameAs” or equivalentClass”) between “George Bush”, “George W. Bush”, Bush” or “G. W.
matched concepts, or Semantic Web Rule Language
anguage (SWRL) Bush”. Matching can also be performed perfor using
rules to bridge the aligned models. synonyms (“newspaper” matches “periodical”) or
other language-based
based methods like lemmatization,
which would match houses to house, mice to mouse,
etc.
• Semantic techniques rely on deductive methods to
Figure 2. Primary SCUBA workflows
justify their matching results. A semantic model mod
could contain a very rich set of relations, with
inferred associations between ontologies. For
2) Agents, Behaviors, and Ensembles example, “brain injury” and “head injury” might be
inferred to be synonymous based on the fact that a
In the SCUBA framework, a community of agents “brain” is “part-of”
of” a “head”.
interacts to perform the high-level
level operations of candidate • Structural techniques takee into account an entity’s
selection and ontology matching. Each agent determines the attributes or properties, as well as other related
types of behaviors it needs in order to perform its current task, entities, when performing a match. For example, a
and loads them dynamically. Agents serve in a vari
variety of roles: constraint-based
based rule would match “Book” and
“Volume” if each contained the key properties of
• OA - Ontology Agent:: perform as a proxy for an author, year, publisher, and title.
title Similarly, a graph-
ontology by mediating access to its concepts as well based rule would match “Book” and “Volume” if the
as responding to inquiries about its metacharacteristics two concepts had the same (or similar) subclasses,
(e.g., depth, breadth, number of concepts, etc.). like “Novel”, “Textbook”, and “Children’s”.
• EA - Evaluation Agent: make a judgment as to the • Extensional techniques are applied not to concepts,
relatedness of available ontologies along some but to instances. Typically applied when wh other
relevant dimension (e.g., domain relevance, semantic techniques contain little name or structure overlap,
similarity, etc.). these techniques entail matching two concepts based
on their membership; i.e., the objects that belong to
each particular class. For example, book titles are the roles of agents and behaviors in the candidate selection
unique enough that, with some estima
estimable probability, process.
two instances having same title or label are likely to
be the same object. If the object is classified
differently in two separate ontologies, a match
between concepts then becomes possible.
3) Workflow Heuristics
Heuristics are encoded inside ann Heuristic Agent and
govern the selection and matching workflows
workflows. Many such
heuristics can be encoded simultaneously in one or many
agents,, and a single HA can construct complex heuristic
workflows from multiple matching behaviors chosen from
different categories.. For instructional purposes, the following
example is used throughout the rest of this section to describe
what happens in each step of the process:
Agent: HA01
Behavior: MinDepth-MaxDepth Figure 3. Candidate selection workflow
Other agents: EA01, OA1-OAn, MA01, SA01SA01-SA03
Behavior Description: Inspect each ontology for its depth. 5) Workflow 2: Ontology Matching
As candidates, choose the ontology with the minimum
Once the candidate ontologies and techniques are chosen
depth to be matched with the ontology having the
for alignment, the matching process is carried out using the
t
maximum depth. Perform an alignment of the two
agents and behaviors described above. Figure 4 illustrates the
candidates using an average of all available name
name-based
ontology matching process.
and
nd semantic matching techniques. Evaluate the results
using an F-score statistic.
In the example, HA01 performs candidate selection by
directing EA01 to evaluate ontologies according to their depth.
EA01 requests a depth statistic from each of the OAs, an and
reports the results – the ontologies having the least and greatest
depths – back to HA01. The candidate ontologies have now
been identified, and the first phase is complete. HA01 then
moves into the ontology matching phase. The agent directs
MA01 to matchh the selected ontologies using all of the namename-
based and semantic behaviors. MA01 manages the next level of
orchestration, directing a set of SAs to perform an alignment,
assigning each to use one of the specific matching behaviors.
For example, if there are defined behaviors for Lev Levenschtein
distance (name-based), Jaro-Winkler
Winkler (name
(name-based), and
WordNet similarity (semantic), the MA tasks three SAs – one
per behavior - to align the concepts in the candidate ontologies Figure 4. Ontology alignment workflow
and record the results of their work. MA01 then computes the
ensemble mean and reports its result back to HA01. Once the 6) Scoring and Evaluating
match process is complete, other components can refer to the All of the concept matching results were recorded in a
scores in order to produce a number of possible outcomes: a database for later use. The entries included
include the two concepts
merged ontology, a set of rules mapping pairs of sim similar being matched and their ontologies of origin, the behavior used
concepts, ontology entries reflecting class equivalencies, etc. to perform the match, and the similarity score that was
4) Workflow 1: Candidate Selection normalized to range between -1.0
1.0 (known to be different) and
1.0 (known to be the same). A score of 0.0 indicated complete
All
ll available ontologies are evaluated and compared uncertainty. The scoring results were
we used to compute the
according to a subset of predetermined set of criteria (e.g., ensemble mean over each discrete concept match (i.e., by
depth, breadth, domain relevance, numberr of concepts, etc.). averaging the scores of all behaviors that were applied to each
From this observation, the most suitable pairs are selected for of the match pairs).
alignment. Additionally, matching techniques are chosen to
maximize the effectiveness of the alignment process for the C. Demonstration
types of ontologies chosen as candidates. Figure 3 describes
A military planning exercise was chosen as a scenario to contend, to include terrain and weather and any infrastructure
demonstrate SCUBA, since this is type of event is typically a and demographics in the area of operations (AO). IPB
time consuming, manual, and ad hoc process that can take evaluates the threat by analyzing current intelligence to
hours to days depending on size of the mission and echelon of determine how the threat normally organizes for combat and
command. War planners skim through available classified conducts operations under similar circumstances.
sources of information such as Signal Intelligence ((SIGINT),
Communications Intelligence (COMINT),, and Human
Intelligence (HUMINT),, but typically spend the majority of
their effort analyzing Open Source (OSINT OSINT) or simply
searching the Internet 1 . As a result, criticall information and
cross relationships between pieces of information are
commonly missed due to time constraints and the limits of
human processing ability.. Compounding the difficulty of the
research effort, the number of data sources is necessarily
limited by time and staff and not all information may be up up-to-
date.
1) Military Planning Scenario & Decision Making Model
The Military Decision Making Process (MDMP) Model
[13] is a standardized mission planning and decision making
model used by the US Army and combatant commands
(COCOMs) to support counterinsurgency operations (COIN).
The formal tactical planning process of counterinsurgency Figure 5. Military Decision Making Process Model
operations is performed by the commander’s
nder’s staff utilizing the
MDMP model. In plain language, MDMP identifies the c) PMESII Ontology Structure
problem, develops solutions, compares alternatives, and PMESII-PT,
PT, or for brevity PMESII, is a framework
recommends a best decision to the commander. used to describe and understand the operating environment
a) Mission Analysis [10].. PMESII provides structure to the IPB process, process and
facilitates the organization of facts and assumptions about
Mission analysis is crucial to the MDMP. It allows the actors operating in an AO. Each letter in the PMESII acronym
commander
nder to begin the battlefield visualization. The outcome corresponds to a specific variable of interest to the war planner:
of mission analysis is a tactical problem definition that feeds P – Political, M – Military, E – Economic,
Economic S – Social, I –
the process of determining feasible solutions. Mission Analysis Information, I – Infrastructure, P – Physical Environment, and
consists of 17 steps, not necessarily sequential, and results in a T – Time.
formal staff briefing to the commander. Figure 5 depicts the
breakdown of the MDMP model and green shad shading is used to 2) Military Planning Using SCUBA
highlight the relevant steps for the SCUBA demonstration. As illustrated in Figure 5,, the eighth variable, Time,
b) Initial Intelligence Preparation of the Battlefield (IPB) was not modeled in this scenario since the time element
el was
already embedded in the instance data populating the other
IPB is a systematic, continuous
nuous process of analyzing the variables. The DoD currently uses an expansion of the PMESII
threat and the effects of the environment on the unit. It model that includes about 60 sub-categories.
sub By merging
identifies facts and assumptions that determine likely threat PMESII with the Yale OCM model the SCUBA team extended
COAs. The IPB supports the commander and staff and is the level of fidelity
delity to approximately 900 super class and class
essential to estimates and decision making. It prov
provides the basis concepts providing much greater model fidelity.
fidelity An example of
for intelligence collection and synchronization to support COA this expansion for the Social PMESII variable can be seen in
development and analysis. Furthermore, itt is a dynamic process blue in Figure 6.
that continually integrates new intelligence information.
When SCUBA executes, it ingests data from text
IPB defines the battlefield or operational environment in documents, extractss domain relevant concepts, and links those
order to identify the characteristics of the environment that concepts both vertically within individual PMESII variables
influence friendly and threat operations, help determine the and horizontally across the PMESII model. Instance (source)
area of interest,, and identify gaps in current intelligence. IPB data is connected to each concept, which allows later review by
describes the battlefield’s effects, including the evaluation of the war planner or intelligence
nce analyst. The main advantage of
alll aspects of the environment with which both sides must this paradigm is that instead of a planning staff performing
manual keyword search queries across a variety of databases, a
1
single lookup within SCUBA will provide the analyst or
This process was described to the SCUBA team during a December 2010 operational planner with all relevant information
info on a historic,
visit to the Joint Operations Center at US Central Command Headquarters, social, or cultural topic of interest.
MacDill AFB, FL.
Figure 6. PMESII ontology structure
While the focus of SCUBA is in the socio
cio-cultural domain, combination of structural and semantic matching. The
an expanded PMESII model was created in order to remainder of the figure illustrates multiple concepts arising
demonstrate horizontal relationships between variables. The from a single paragraph: Military Organization, Districts, and
result is a major improvement over existing systems that are Police, as well as, additional instance data on each of those
highly specialized and restricted in scope.. Additionally, when concepts arising from other source material.
this same effort is performed manually, the PMESII variables
are commonly divided between staff officers. This practice III. RESULTS
produces information stovepipes and complicates the task of The team identified
ified dozens of possible evaluation metrics,
identifying cross-variable effects.. In contrast, SCUBA many of which were used in the candidate selection process.
facilitates such understanding. As an overall measure of effectiveness, however, we report
Continuing with the example in Figure 6,, the concept class our results in terms of F-scores using the formulas below.
hierarchy in green are those identified and created by SCUBA.
Notice that in addition to aligning similar concepts, SCUBA (1)
can create class hierarchies, merge similar concepts into a
single class, and
nd link original instance data to each relevant
concept. (2)
A small portion of the merged PMESII Ontology generated
by SCUBA using open source socio-cultural cultural information of
Afghanistan was displayed in Raytheon’s hyperbolic semantic (3)
graph tool and is shown in Figure 7.. Notice the equivalence
relationship identified between “Military Organization” and
“Militia”. Also, “Districts” in one ontology was aligned with The F-score is a measure of a test's accuracy which
“District” (no ‘s’) in another ontology. This was all performed considers both the precision (“exactness”) and recall
automatically by the SCUBA agents andd behaviors with no (“completeness”) of the test. In the models we chose for
human in-the-loop.
loop. In the case of Districts/District, the SCUBA testing, over
ver 60,000 comparisons were made between
heuristic relied on structural matching techniques. The match
occurring between Military Organization and Militia was a
Figure 7. PMESII ontology structure in hyperbolic browser
concepts. Using name-based
based comparison alone, precision was Structural MOPs
typically high (~90%), while recall was much lower (~20 - • Measure of Concept Count – Total number of
30%). Because the name-based
based approach suffered from a high concepts in the ontology.
number of false negatives, the F-scores
scores averaged only ~40% • Measure of Concept Instance Count – Number of
(see Figure 8). However, when semantic matching was linked paragraph instanceses over all concepts.
combined with name-based
based matching, there was a dramatic • Measure of Relationship Type Count – Total number
reduction of false negatives - this resulted in significant of unique relationships in ontology, i.e. ‘is a part of’,
increase in recall (~80%) and brought the average F F-score to ‘is equivalent to’, etc.
above 80% (all differences were significant) – see Figure 9. • Measure of Relation Instance Count – Number of
relationship links between concepts.
Even greater improvement is expected when additional
• Measure of Maximum Depth – Levels of concept
behaviors are added. Based on these results, we are hierarchy within the ontology.
encouraged by the prospect of evolving information alignment • Measure of Degree Centrality - Measure used often in
and interoperability from a manual,, costly chore to an social network theory - average number of
effective semi-automated process. relationships linked to each concept.
A. Measures of Performance
In order to determine whether the automated alignalign-and- Usability MOPs
merge methodology defined by SCUBA demonstrated any • User Recognition – Survey score indicating how
improvement over existing ontology generation tools alone, the similar ontology structure is with current models.
SCUBA merged ontology was compared against an ontology • Fitness for User – Survey score indicating how easy it
generated using Text2Onto [5][6].. In both cases, the same data is for the user to load and navigate among the
set and initial taxonomy were used. The comparison was made concepts in the ontology
across 12 measures of performance (MOPs) that fall within
three general measurement
asurement dimensions: Structural, Usability,
and Timeliness.
Figure 8. Results for string-only
only alignment Figure 9. Results for String + Semantic Alignment
Timeliness MOPs suited for cases where information is spread across numerous
• Speed to Build Ontology – Time to create the and small data sources, or in cases where narrowly specific
ontologies. ontologies are merged with broader, more general ones.
• Time to Perform Alignment – Time to perform
alignment between 2 or more ontologies. IV. SUMMARY
In this paper, we presented SCUBA, an agent-based
Figure 10 depicts these MOPs for each of the ontology creation and alignment framework developed to
ontologies. It is immediately clear that the Human Generated address the shortcomings of current socio-cultural modeling
ontology was most recognizable with information in a format efforts. SCUBA achieves a balanced human-computer
most easily used while the purely machine generated, modeling paradigm to 1) automate the creation of social and
Text2Onto ontology scored lowest in this area. SCUBA scored cultural ontologies from selected source materials, 2) apply a
well in this area because it was based on the same PMESII
variety of nominal, semantic, structural, and statistical
model used by military planners. Conversely, the Human
Generated ontology took longest to build and was much matching techniques to align multiple ontologies in a
smaller than the faster and larger generated Text2Onto multimodeling environment, and 3) evaluate the effectiveness
ontology. These results were in line with our expectations. of the generation and alignment processes. Preliminary results
What we intended to see was whether SCUBA could create of our initial agent-based experiments were promising – by
ontologies that were at least as large/deep as those created by applying ensembles of multiple matching techniques, we
software algorithms or humans, but were richer and more achieved significant improvements in alignment F-scores and
usable similar to those generated by humans. other evaluation measures while dramatically reducing the
amount of time required to manually produce coordinated,
When breaking down the Speed to Build by individual useful domain models.
algorithms, the SCUBA string-based matching agents and
ontology behaviors were executed on par with Text2Onto, ACKNOWLEDGMENT
while the semantic matching agents took considerably more The authors acknowledge the valuable contributions of the
time to execute. This is understandable because the semantic SCUBA development team at Raytheon IIS (State College, PA
algorithms are more complex with the purpose of determining and Garland, TX) and Raytheon NCS (Largo, FL).
additional positive matches through synonym, lemmatization,
and morphological comparisons. This significantly improved REFERENCES
the accuracy of the results, as boldly illustrated in Figure 9, [1] K. Morik. Balanced cooperative modeling. Machine Learning, 11:217–
however there is a corresponding increase in ontology 235, 1993.
generation run time. We believe this is reasonable (it is still [2] B. G. Buchanan and D. C. Wilkins (editors). Readings in Knowledge
significantly lower than the Human Generated ontology) and Acquisition and Learning: Automating the Construction and
can be further reduced by adding computing resources. Improvement of Expert Systems. Morgan Kaufmann, San Mateo, CA.,
1993.
Concept Count [3] J. Euzenat and P. Shvaiko, Ontology Matching. Springer-Verlag, Berlin
(hundreds) Heidelberg (DE), 2007.
100
Relation Type
90 Concept [4] A. Maedche and R. Volz, “The ontology extraction maintenance
80 Instance Count framework Text-To-Onto,” in Proc. ICDM’01 Workshop on Integrating
Count 70 (thousands)
60 Data Mining and Knowledge Management, 2001.
50
[5] P. Cimiano and J. Völker. Text2Onto - A Framework for Ontology
40
30
Learning and Data-driven Change Discovery. In Andres Montoyo,
20 Degree Rafael Munoz, Elisabeth Metais, Proc. 10th Int. Conf. on Applicat. of
Fitness for User
Centrality
10 Natural Language to Inform. Syst. (NLDB), volume 3513 of Lecture
0
Notes in Computer Science, pp. 227-238. Springer, Alicante, Spain, June
2005.
[6] http://code.google.com/p/text2onto/
User
Recognition
Maximum Depth [7] http://www.yale.edu/hraf/
[8] http://www.yale.edu/hraf/Short_OCM_List_121503.pdf
Human Generated
Relation [9] http://hdr.undp.org/en/
Speed to Build Text2Onto
Instance Count [10] Field Manual 2-0, Intelligence. 2010. HQ, Dept. of the Army
(1/minutes) SCUBA
(millions)
[11] http://protege.stanford.edu/
Figure 10. MOP comparison between Text2Onto and SCUBA [12] http://jade.tilab.com/
[13] Field Manual 101-5: Staff Organization and Operations. Dept. of the
Regarding the other MOPs, SCUBA either met or Army.
exceeded the performance of Text2Onto. For example, for [14] L. Yilmaz, A. Lim, S. Bowen, and T. “Requirements and Design
Concept Instance Count and Relation Instance Count, SCUBA Principles for Multisimulation with Multiresolution, Multistage
Multimodels” in Proceedings of the 2007 Winter Simulation Conference,
identified close to 50% more concept and relationship instances Washington, DC, 2007.
than Text2Onto. This is an indication that the strategy of
generating multiple smaller ontologies, and then aligning and
merging the results into a larger composite ontology can
improve information quality. Thus, SCUBA seems ideally