SCUBA: An Agent-Based Ontology Creation and Alignment Method for Socio-Cultural Modeling Donald R. Kretz William D. Phillips Bruce E. Peoples Raytheon IIS Raytheon NCS Raytheon IIS Garland, TX Largo, FL State College, PA donald_r_kretz@raytheon.com bill.phillips@raytheon.com bruce_e_peoples@raytheon.com Abstract -- An otherwise promising business, political, or military We apply ontology as our modeling method of choice. An strategy can be crippled by an incomplete understanding of the ontology can conceptualize a complex domain in a way that social-cultural factors that define and influence a region. Such both humans and machines can understand, but the use of omissions are sometimes due to oversight, but often stem from a ontology in this context presents us with two important fundamental lack of understanding of how to model such difficult challenges. First, manual ontology creation is a time- and unfamiliar concepts. The information required to generate useful contextual models is typically available but vast, and consuming and highly subjective process, particularly when manual interpretation of detailed text is time-consuming, highly attempting to model abstract social and cultural concepts. subjective, and requires specialized skills. The SCUBA project While formal models are required to conform to strict rules achieved a balanced human-computer modeling paradigm to 1) involving provable logic and model consistency, they will automate the creation of social and cultural ontologies from always incorporate some amount of bias. Every human selected source materials using previously-developed tools, 2) modeler will have a slightly different perspective of the same apply a variety of nominal, semantic, structural, and statistical small part of the world, and will make different value matching techniques to align multiple ontologies using an agent- judgments about what parts are important and how those parts based multimodel, and 3) evaluate the effectiveness of the interrelate. Striving for added richness by adding more generation and alignment processes using precision, recall, and various other measures of effectiveness. Preliminary results of information only complicates this problem and adds to the our initial agent-based experiments were promising – by applying severity of the “knowledge acquisition bottleneck” [2]. We ensembles of multiple matching techniques, we achieved believe, therefore, that by applying automated ontology significant improvements in alignment F-scores and other generation against various corpora of domain-relevant measures of performance while dramatically reducing the materials, we can generate a useful first approximation of a amount of time required to manually produce coordinated, useful domain model. An automated generation process will “learn” domain models. from the information it can “study”. The model it constructs will, therefore, be representative of the “world” described in Keywords-ontology; ontology alignment; social ontology; the input material it receives. cultural ontology; ensemble alignment; agent-based alignment The second challenge involves the alignment of multiple I. INTRODUCTION models. Accommodating multiple domain ontologies is Having an incomplete understanding of the social-cultural usually necessary to capture the complexities of domains factors that define and influence a region can cripple an having socio-cultural dimensions and to leverage existing otherwise promising business, political, or military strategy. models. There has been a loosely-associated body of work in Too often, models that guide strategy development and this area that we collected under the general heading of operational planning do not include critical social and cultural “ontology alignment theory”. Our interpretation of this theory elements. These omissions can be blamed partly on oversight, is essentially built on the principle of approximation – because but often stem from a fundamental lack of understanding of any ontology is an approximate representation of its real-world how to model such difficult and unfamiliar concepts. The domain, generating and aligning multiple ontologies that all information required to generate useful contextual models is represent the same domain yields a richer higher-order typically available but is often distributed across vast approximation of the real world (i.e., removes some of the repositories. Furthermore, the manual interpretation of detailed subjectivity or bias associated with applying a single model). text is time-consuming, highly subjective, and requires As described by Euzenat and Shvaiko [3], the matching specialized skills. We believe that socio-cultural awareness is operation accepts ontologies as inputs, and produces an best achieved by a system that combines multiple information ontology as its output (see Figure 1). The input ontologies (O1 sources using a variety of automated extraction, mediation, and and O2) are independent domain ontologies, perhaps derived analysis tools, but guided by a human knowledge engineer in from different sources of information or developed by an interactive paradigm called balanced cooperative modeling different ontology engineers. Optionally, a third ontology (Ω) [1]. may be included as input – this may be an upper ontology or may be the composite ontology (or alignment) produced by a The authors are grateful to have received funding for SCUBA under the 2010 Raytheon Innovation Challenge. previous matching operation. The latter case suggests that Observation #2: Certain ertain matching techniques produce matching operations can be chained for continued refinement more useful alignments for certain by feeding the output from one operation (i.e., an aligned ontology pairs than others other ontology) as input to the next matching process. Observation #3: The he selection of candidate ontology pairings and matching techniques can be guided by heuristics and aided by the inspection of model metacharacteristics metach This paper presents SCUBA, an agent-based agent framework for ontology alignment based on the observations stated above. We will describe the methodology we applied, as well as provide some initial results. II. METHODOLOGY The objective of SCUBA was to develop a balanced human-computer computer modeling paradigm to 1) automate a the creation of social and cultural ontologies from selected source materials, 2) apply a variety of nominal, semantic, structural, and statistical matching techniques to align multiple Figure 1. Matchingg operation, from Euzenat and S Shvaiko [3] ontologies, and 3) evaluate the effectiveness of the generation and alignment processes. Since our work was mainly focused In addition to the ontologies, a couple of additional inputs on the alignment framework, we will concentrate most of this are provided. First, a set of rules directs the matcher to section on that effort. perform certain types of comparisons (i.e., which entities or attributes to compare,, what sort of comparison to make, etc.). A. Ontology Generation These rules are derived from a set of basic matching In answering the challenge of ontology generation, we techniques described later in this paper.. To accompany the relied on the groundbreaking work of a number of others, rules, a set of parameters informs the matcherer what limits or particularly Maedche and Volz[4]] and Cimiano and Völker constraints to impose on the rules. For example example, a rule might [5]. We used the common academic ontology generator cause a name similarity technique to be performed using a Text2Onto [6]] to generate ontologies from bodies of text we fuzzy string comparison on a “name” property, but a obtained from various sources, including the Yale University parameter might indicate that only values having a confidence Human Relations Area Files (HRAF) [7], [ Yale University's valuee higher than 50% are to be considered a match. Finally, Outline of Cultural Materials (OCM) [8], the United Nations the he result of this operation is an ontology, referred to as Ω Development Programme – Human Development De Reports [9] prime,, that expresses the set of correspondences between the and others. Documents were clustered by geographic area, and entities in O1 and O2. a separate ontology was generated for each area. The team Considering the points made above, we set two primary also generated 95 separate ontologies utilizing the 54 Cultural goals for our project: Eliminate the knowledge acquisition and 41 Social text files obtained from open source materials. material bottleneck through semantic parsing and extraction of domain All ontologies were created in the Web Ontology Language concepts from data sources into multiple ontologies and (OWL) format. contexts, and bridge the gap between multiple, heterogeneous Additionally, the he team manually generated a set of "gold ontology. In our generated ontologies and a single domain ontology standard” ontologiess to compare with the automatically initial phase, we chose to apply the previous work of others in generated models.. Seven ontologies were created based on the generating ontologies from text using readily available tools Department of Defense (DoD) PMESII-PT PMESII paradigm (Political, (citations to follow). Our investigation, tion, instead, focused on the Military, Economic, Social, Information, Information Infrastructure, effective alignment of ontologies through various techniques Physical Environment, Time) [10] [10 using Protégé [11]. An of mapping. Here, too, we borrowed from the work of others ontology for Time was not created. created To provide instance data for specific techniques and algorithms (citations to follow) follow). for the PMESII-P P ontologies, the team developed a method to However, we began with the premise that ontologies have automate the merging of Yale HRAF instance data with Yale characteristics that make them more or less suited for effective OCM codes in the developed ontologies, onto saving weeks of alignment with certain other ontologies. Rather than approach manual labor. the problem using a single ngle technique or by applying complex n-way comparisons,, we formulated three key observations that B. Ontology Alignment guided our efforts: 1) General Approach Observation #1: Certain pairs of ontologies are more As stated, we focused most of our work on ontology effectively ffectively aligned with one another alignment. We again borrowed heavily from the body of prior than with other ontologies research in specific ontology matching techniques, most of which were collected and documented by Euzenat and • HA - Heuristic Agent:: determine which ontology pairs Shvaiko [3]. ]. In order to investigate our own hypotheses, make good candidates for matching, which matching however, we constructed a customized agent-based based framework behaviors should uld be applied, and manage the using the Java Agent DEvelopment Framework (JADE) [12]. execution of selection and matching workflows. We used agents to develop automated workflo workflows for the two • MA - Matching Agent:: creates mappings of the main component processes: selecting the optimal set of concepts and relationship types between two alignment candidates and most promising match techniques techniques, ontologies. and performing the matching operation by applying the rules to the alignment candidates (Figure 2).. Specific match • SA - Similarity Agent:: calculates the similarity simil techniques were re encoded as composable sets of agent between concepts. behaviors. An agent-oriented design allowed ed us to apply a • UA - Utility Agent:: performs supporting tasks such as technique known as “ensemble forecasting”, which is common data and ontology storage/retrieval, job ID in highly specialized domains such as weather prediction. management, etc. Yilmaz [14] refers to this idea as a multimodel multimodel, or a set of component models that, together, define the behavior of a Each matching algorithm or technique was implemented as more complex process. Using ensemble forecasting or a behavior. In JADE parlance, a behavior is a set of actions to multimodeling,, various combinations of matching algorithms be performed. Coding each set of actions in a separate (“behaviors”) were re applied against concept pairs, then component, rather than in the agent itself, allowed allow each agent evaluated in order to determine the strength of the match. An to select and compose the behaviors it wishedwishe to use to average, or ensemble mean,, of the different behaviors inspire inspired complete an assigned task. There are many techniques for greater confidence because it essentially smoothe smoothed the performing the matching operation, and some are betterb at performance peaks and troughs introduced by model matching certain types of entities and properties than others. imperfections or context sensitivities. ties. For example, the Furthermore, a better match might result in some cases if more concepts “car” and “automobile” produce very low results for than one technique is applied at the same time (“matcher all name-based based match behaviors, but semantic match composition”). behaviors rate them as nearly identical. Hence, while any one technique for matching two concepts is inherently unreliabl unreliable, • Name-based based (“terminological”) techniques compute an ensemble mean that accounts for the strengths and some measure of similarity based on strings weaknesses of all match techniques yield yielded a higher- containing names, descriptions, comments, etc. confidence correlation. The matches can be used to produce a Comparisons based on simple or fuzzy string merged ontology in any format desired; e.g., a set of OWL comparisons would match “George Bush” with assertions (i.e., “sameAs” or equivalentClass”) between “George Bush”, “George W. Bush”, Bush” or “G. W. matched concepts, or Semantic Web Rule Language anguage (SWRL) Bush”. Matching can also be performed perfor using rules to bridge the aligned models. synonyms (“newspaper” matches “periodical”) or other language-based based methods like lemmatization, which would match houses to house, mice to mouse, etc. • Semantic techniques rely on deductive methods to Figure 2. Primary SCUBA workflows justify their matching results. A semantic model mod could contain a very rich set of relations, with inferred associations between ontologies. For 2) Agents, Behaviors, and Ensembles example, “brain injury” and “head injury” might be inferred to be synonymous based on the fact that a In the SCUBA framework, a community of agents “brain” is “part-of” of” a “head”. interacts to perform the high-level level operations of candidate • Structural techniques takee into account an entity’s selection and ontology matching. Each agent determines the attributes or properties, as well as other related types of behaviors it needs in order to perform its current task, entities, when performing a match. For example, a and loads them dynamically. Agents serve in a vari variety of roles: constraint-based based rule would match “Book” and “Volume” if each contained the key properties of • OA - Ontology Agent:: perform as a proxy for an author, year, publisher, and title. title Similarly, a graph- ontology by mediating access to its concepts as well based rule would match “Book” and “Volume” if the as responding to inquiries about its metacharacteristics two concepts had the same (or similar) subclasses, (e.g., depth, breadth, number of concepts, etc.). like “Novel”, “Textbook”, and “Children’s”. • EA - Evaluation Agent: make a judgment as to the • Extensional techniques are applied not to concepts, relatedness of available ontologies along some but to instances. Typically applied when wh other relevant dimension (e.g., domain relevance, semantic techniques contain little name or structure overlap, similarity, etc.). these techniques entail matching two concepts based on their membership; i.e., the objects that belong to each particular class. For example, book titles are the roles of agents and behaviors in the candidate selection unique enough that, with some estima estimable probability, process. two instances having same title or label are likely to be the same object. If the object is classified differently in two separate ontologies, a match between concepts then becomes possible. 3) Workflow Heuristics Heuristics are encoded inside ann Heuristic Agent and govern the selection and matching workflows workflows. Many such heuristics can be encoded simultaneously in one or many agents,, and a single HA can construct complex heuristic workflows from multiple matching behaviors chosen from different categories.. For instructional purposes, the following example is used throughout the rest of this section to describe what happens in each step of the process: Agent: HA01 Behavior: MinDepth-MaxDepth Figure 3. Candidate selection workflow Other agents: EA01, OA1-OAn, MA01, SA01SA01-SA03 Behavior Description: Inspect each ontology for its depth. 5) Workflow 2: Ontology Matching As candidates, choose the ontology with the minimum Once the candidate ontologies and techniques are chosen depth to be matched with the ontology having the for alignment, the matching process is carried out using the t maximum depth. Perform an alignment of the two agents and behaviors described above. Figure 4 illustrates the candidates using an average of all available name name-based ontology matching process. and nd semantic matching techniques. Evaluate the results using an F-score statistic. In the example, HA01 performs candidate selection by directing EA01 to evaluate ontologies according to their depth. EA01 requests a depth statistic from each of the OAs, an and reports the results – the ontologies having the least and greatest depths – back to HA01. The candidate ontologies have now been identified, and the first phase is complete. HA01 then moves into the ontology matching phase. The agent directs MA01 to matchh the selected ontologies using all of the namename- based and semantic behaviors. MA01 manages the next level of orchestration, directing a set of SAs to perform an alignment, assigning each to use one of the specific matching behaviors. For example, if there are defined behaviors for Lev Levenschtein distance (name-based), Jaro-Winkler Winkler (name (name-based), and WordNet similarity (semantic), the MA tasks three SAs – one per behavior - to align the concepts in the candidate ontologies Figure 4. Ontology alignment workflow and record the results of their work. MA01 then computes the ensemble mean and reports its result back to HA01. Once the 6) Scoring and Evaluating match process is complete, other components can refer to the All of the concept matching results were recorded in a scores in order to produce a number of possible outcomes: a database for later use. The entries included include the two concepts merged ontology, a set of rules mapping pairs of sim similar being matched and their ontologies of origin, the behavior used concepts, ontology entries reflecting class equivalencies, etc. to perform the match, and the similarity score that was 4) Workflow 1: Candidate Selection normalized to range between -1.0 1.0 (known to be different) and 1.0 (known to be the same). A score of 0.0 indicated complete All ll available ontologies are evaluated and compared uncertainty. The scoring results were we used to compute the according to a subset of predetermined set of criteria (e.g., ensemble mean over each discrete concept match (i.e., by depth, breadth, domain relevance, numberr of concepts, etc.). averaging the scores of all behaviors that were applied to each From this observation, the most suitable pairs are selected for of the match pairs). alignment. Additionally, matching techniques are chosen to maximize the effectiveness of the alignment process for the C. Demonstration types of ontologies chosen as candidates. Figure 3 describes A military planning exercise was chosen as a scenario to contend, to include terrain and weather and any infrastructure demonstrate SCUBA, since this is type of event is typically a and demographics in the area of operations (AO). IPB time consuming, manual, and ad hoc process that can take evaluates the threat by analyzing current intelligence to hours to days depending on size of the mission and echelon of determine how the threat normally organizes for combat and command. War planners skim through available classified conducts operations under similar circumstances. sources of information such as Signal Intelligence ((SIGINT), Communications Intelligence (COMINT),, and Human Intelligence (HUMINT),, but typically spend the majority of their effort analyzing Open Source (OSINT OSINT) or simply searching the Internet 1 . As a result, criticall information and cross relationships between pieces of information are commonly missed due to time constraints and the limits of human processing ability.. Compounding the difficulty of the research effort, the number of data sources is necessarily limited by time and staff and not all information may be up up-to- date. 1) Military Planning Scenario & Decision Making Model The Military Decision Making Process (MDMP) Model [13] is a standardized mission planning and decision making model used by the US Army and combatant commands (COCOMs) to support counterinsurgency operations (COIN). The formal tactical planning process of counterinsurgency Figure 5. Military Decision Making Process Model operations is performed by the commander’s nder’s staff utilizing the MDMP model. In plain language, MDMP identifies the c) PMESII Ontology Structure problem, develops solutions, compares alternatives, and PMESII-PT, PT, or for brevity PMESII, is a framework recommends a best decision to the commander. used to describe and understand the operating environment a) Mission Analysis [10].. PMESII provides structure to the IPB process, process and facilitates the organization of facts and assumptions about Mission analysis is crucial to the MDMP. It allows the actors operating in an AO. Each letter in the PMESII acronym commander nder to begin the battlefield visualization. The outcome corresponds to a specific variable of interest to the war planner: of mission analysis is a tactical problem definition that feeds P – Political, M – Military, E – Economic, Economic S – Social, I – the process of determining feasible solutions. Mission Analysis Information, I – Infrastructure, P – Physical Environment, and consists of 17 steps, not necessarily sequential, and results in a T – Time. formal staff briefing to the commander. Figure 5 depicts the breakdown of the MDMP model and green shad shading is used to 2) Military Planning Using SCUBA highlight the relevant steps for the SCUBA demonstration. As illustrated in Figure 5,, the eighth variable, Time, b) Initial Intelligence Preparation of the Battlefield (IPB) was not modeled in this scenario since the time element el was already embedded in the instance data populating the other IPB is a systematic, continuous nuous process of analyzing the variables. The DoD currently uses an expansion of the PMESII threat and the effects of the environment on the unit. It model that includes about 60 sub-categories. sub By merging identifies facts and assumptions that determine likely threat PMESII with the Yale OCM model the SCUBA team extended COAs. The IPB supports the commander and staff and is the level of fidelity delity to approximately 900 super class and class essential to estimates and decision making. It prov provides the basis concepts providing much greater model fidelity. fidelity An example of for intelligence collection and synchronization to support COA this expansion for the Social PMESII variable can be seen in development and analysis. Furthermore, itt is a dynamic process blue in Figure 6. that continually integrates new intelligence information. When SCUBA executes, it ingests data from text IPB defines the battlefield or operational environment in documents, extractss domain relevant concepts, and links those order to identify the characteristics of the environment that concepts both vertically within individual PMESII variables influence friendly and threat operations, help determine the and horizontally across the PMESII model. Instance (source) area of interest,, and identify gaps in current intelligence. IPB data is connected to each concept, which allows later review by describes the battlefield’s effects, including the evaluation of the war planner or intelligence nce analyst. The main advantage of alll aspects of the environment with which both sides must this paradigm is that instead of a planning staff performing manual keyword search queries across a variety of databases, a 1 single lookup within SCUBA will provide the analyst or This process was described to the SCUBA team during a December 2010 operational planner with all relevant information info on a historic, visit to the Joint Operations Center at US Central Command Headquarters, social, or cultural topic of interest. MacDill AFB, FL. Figure 6. PMESII ontology structure While the focus of SCUBA is in the socio cio-cultural domain, combination of structural and semantic matching. The an expanded PMESII model was created in order to remainder of the figure illustrates multiple concepts arising demonstrate horizontal relationships between variables. The from a single paragraph: Military Organization, Districts, and result is a major improvement over existing systems that are Police, as well as, additional instance data on each of those highly specialized and restricted in scope.. Additionally, when concepts arising from other source material. this same effort is performed manually, the PMESII variables are commonly divided between staff officers. This practice III. RESULTS produces information stovepipes and complicates the task of The team identified ified dozens of possible evaluation metrics, identifying cross-variable effects.. In contrast, SCUBA many of which were used in the candidate selection process. facilitates such understanding. As an overall measure of effectiveness, however, we report Continuing with the example in Figure 6,, the concept class our results in terms of F-scores using the formulas below. hierarchy in green are those identified and created by SCUBA. Notice that in addition to aligning similar concepts, SCUBA (1) can create class hierarchies, merge similar concepts into a single class, and nd link original instance data to each relevant concept. (2) A small portion of the merged PMESII Ontology generated by SCUBA using open source socio-cultural cultural information of Afghanistan was displayed in Raytheon’s hyperbolic semantic (3) graph tool and is shown in Figure 7.. Notice the equivalence relationship identified between “Military Organization” and “Militia”. Also, “Districts” in one ontology was aligned with The F-score is a measure of a test's accuracy which “District” (no ‘s’) in another ontology. This was all performed considers both the precision (“exactness”) and recall automatically by the SCUBA agents andd behaviors with no (“completeness”) of the test. In the models we chose for human in-the-loop. loop. In the case of Districts/District, the SCUBA testing, over ver 60,000 comparisons were made between heuristic relied on structural matching techniques. The match occurring between Military Organization and Militia was a Figure 7. PMESII ontology structure in hyperbolic browser concepts. Using name-based based comparison alone, precision was Structural MOPs typically high (~90%), while recall was much lower (~20 - • Measure of Concept Count – Total number of 30%). Because the name-based based approach suffered from a high concepts in the ontology. number of false negatives, the F-scores scores averaged only ~40% • Measure of Concept Instance Count – Number of (see Figure 8). However, when semantic matching was linked paragraph instanceses over all concepts. combined with name-based based matching, there was a dramatic • Measure of Relationship Type Count – Total number reduction of false negatives - this resulted in significant of unique relationships in ontology, i.e. ‘is a part of’, increase in recall (~80%) and brought the average F F-score to ‘is equivalent to’, etc. above 80% (all differences were significant) – see Figure 9. • Measure of Relation Instance Count – Number of relationship links between concepts. Even greater improvement is expected when additional • Measure of Maximum Depth – Levels of concept behaviors are added. Based on these results, we are hierarchy within the ontology. encouraged by the prospect of evolving information alignment • Measure of Degree Centrality - Measure used often in and interoperability from a manual,, costly chore to an social network theory - average number of effective semi-automated process. relationships linked to each concept. A. Measures of Performance In order to determine whether the automated alignalign-and- Usability MOPs merge methodology defined by SCUBA demonstrated any • User Recognition – Survey score indicating how improvement over existing ontology generation tools alone, the similar ontology structure is with current models. SCUBA merged ontology was compared against an ontology • Fitness for User – Survey score indicating how easy it generated using Text2Onto [5][6].. In both cases, the same data is for the user to load and navigate among the set and initial taxonomy were used. The comparison was made concepts in the ontology across 12 measures of performance (MOPs) that fall within three general measurement asurement dimensions: Structural, Usability, and Timeliness. Figure 8. Results for string-only only alignment Figure 9. Results for String + Semantic Alignment Timeliness MOPs suited for cases where information is spread across numerous • Speed to Build Ontology – Time to create the and small data sources, or in cases where narrowly specific ontologies. ontologies are merged with broader, more general ones. • Time to Perform Alignment – Time to perform alignment between 2 or more ontologies. IV. SUMMARY In this paper, we presented SCUBA, an agent-based Figure 10 depicts these MOPs for each of the ontology creation and alignment framework developed to ontologies. It is immediately clear that the Human Generated address the shortcomings of current socio-cultural modeling ontology was most recognizable with information in a format efforts. SCUBA achieves a balanced human-computer most easily used while the purely machine generated, modeling paradigm to 1) automate the creation of social and Text2Onto ontology scored lowest in this area. SCUBA scored cultural ontologies from selected source materials, 2) apply a well in this area because it was based on the same PMESII variety of nominal, semantic, structural, and statistical model used by military planners. Conversely, the Human Generated ontology took longest to build and was much matching techniques to align multiple ontologies in a smaller than the faster and larger generated Text2Onto multimodeling environment, and 3) evaluate the effectiveness ontology. These results were in line with our expectations. of the generation and alignment processes. Preliminary results What we intended to see was whether SCUBA could create of our initial agent-based experiments were promising – by ontologies that were at least as large/deep as those created by applying ensembles of multiple matching techniques, we software algorithms or humans, but were richer and more achieved significant improvements in alignment F-scores and usable similar to those generated by humans. other evaluation measures while dramatically reducing the amount of time required to manually produce coordinated, When breaking down the Speed to Build by individual useful domain models. algorithms, the SCUBA string-based matching agents and ontology behaviors were executed on par with Text2Onto, ACKNOWLEDGMENT while the semantic matching agents took considerably more The authors acknowledge the valuable contributions of the time to execute. This is understandable because the semantic SCUBA development team at Raytheon IIS (State College, PA algorithms are more complex with the purpose of determining and Garland, TX) and Raytheon NCS (Largo, FL). additional positive matches through synonym, lemmatization, and morphological comparisons. This significantly improved REFERENCES the accuracy of the results, as boldly illustrated in Figure 9, [1] K. Morik. Balanced cooperative modeling. Machine Learning, 11:217– however there is a corresponding increase in ontology 235, 1993. generation run time. We believe this is reasonable (it is still [2] B. G. Buchanan and D. C. Wilkins (editors). Readings in Knowledge significantly lower than the Human Generated ontology) and Acquisition and Learning: Automating the Construction and can be further reduced by adding computing resources. Improvement of Expert Systems. Morgan Kaufmann, San Mateo, CA., 1993. Concept Count [3] J. Euzenat and P. Shvaiko, Ontology Matching. Springer-Verlag, Berlin (hundreds) Heidelberg (DE), 2007. 100 Relation Type 90 Concept [4] A. Maedche and R. Volz, “The ontology extraction maintenance 80 Instance Count framework Text-To-Onto,” in Proc. ICDM’01 Workshop on Integrating Count 70 (thousands) 60 Data Mining and Knowledge Management, 2001. 50 [5] P. Cimiano and J. Völker. Text2Onto - A Framework for Ontology 40 30 Learning and Data-driven Change Discovery. In Andres Montoyo, 20 Degree Rafael Munoz, Elisabeth Metais, Proc. 10th Int. Conf. on Applicat. of Fitness for User Centrality 10 Natural Language to Inform. Syst. (NLDB), volume 3513 of Lecture 0 Notes in Computer Science, pp. 227-238. Springer, Alicante, Spain, June 2005. [6] http://code.google.com/p/text2onto/ User Recognition Maximum Depth [7] http://www.yale.edu/hraf/ [8] http://www.yale.edu/hraf/Short_OCM_List_121503.pdf Human Generated Relation [9] http://hdr.undp.org/en/ Speed to Build Text2Onto Instance Count [10] Field Manual 2-0, Intelligence. 2010. HQ, Dept. of the Army (1/minutes) SCUBA (millions) [11] http://protege.stanford.edu/ Figure 10. MOP comparison between Text2Onto and SCUBA [12] http://jade.tilab.com/ [13] Field Manual 101-5: Staff Organization and Operations. Dept. of the Regarding the other MOPs, SCUBA either met or Army. exceeded the performance of Text2Onto. For example, for [14] L. Yilmaz, A. Lim, S. Bowen, and T. “Requirements and Design Concept Instance Count and Relation Instance Count, SCUBA Principles for Multisimulation with Multiresolution, Multistage Multimodels” in Proceedings of the 2007 Winter Simulation Conference, identified close to 50% more concept and relationship instances Washington, DC, 2007. than Text2Onto. This is an indication that the strategy of generating multiple smaller ontologies, and then aligning and merging the results into a larger composite ontology can improve information quality. Thus, SCUBA seems ideally