Automatic Support for Formative Ontology Evaluation ∗ Viktoria Pammer Chiara Ghidini Marco Rospocher Know-Center, Austria FBK-irst, Trento, Italy FBK-irst, Trento, Italy vpammer@know- ghidini@fbk.eu rospocher@fbk.eu center.at Luciano Serafini Stefanie Lindstaedt FBK-irst, Trento, Italy Know-Center and KMI TU serafini@fbk.eu Graz, Austria slind@know-center.at ABSTRACT into an ontology engineering tool, ontology evaluation can finally Just as testing is an integral part of software engineering, so is on- become formative, since feedback for potential improvement or tology evaluation an integral part of ontology engineering. We have review is given in the same “place” where ontology engineering implemented automated support for formative ontology evaluation happens. In this regard, formative ontology evaluation is inher- based on the two principles of i) checking for compliance with ently different from ontology evaluation metrics that aim to mea- modelling guidelines and ii) reviewing entailed statements in MoKi, sure an ontology’s characteristics only when it is regarded as “fin- a wiki based ontology engineering environment. These principles ished enough” to merit evaluation. exist in state of the art literature and good ontology engineering and evaluation practice, but have not so far been widely integrated into ontology engineering tools. 2. COMPLIANCE WITH GUIDELINES Modelling guidelines provide guidance to the modellers during the ontology construction process but do not impose strict constraints 1. INTRODUCTION on the ontology engineer. Hence, checking the compliance of an State of the art ontology evaluation practice relies on guidelines and ontology to modelling guidelines can be indicative only of poten- best practices in ontology engineering such as [7, 8], on ontology tial modelling errors. For instance, a typical modelling guideline is evaluation methodologies such as competency questions [12], and to verbally describe model elements (concepts, roles and to a cer- on reasoning to detect logical inconsistencies. The work we present tain extent also individuals) and document design decisions. While here follows up on such existing work by automatically checking it is impossible with the current state of the art to automatically an ontology in progress for compliance with modelling guidelines determine how good a description really is, it is possible to auto- to detect potential modelling errors and motivating ontology engi- matically check for model elements that are not documented at all. neers to review entailed statements throughout the modelling pro- In MoKi, a models checklist page (Fig. 1) lists modelling guide- cess in MoKi, a wiki based ontology engineering tool [6, 11] that lines, and for each guideline those model elements (concepts, prop- has recently been released as open-source. erties, individuals) that do not comply with the guideline. A quality Through integrating such support for ontology evaluation directly indicator visualises the “degree” to which a single model element complies to the whole set of modelling guidelines (Fig. 2). Such ∗The Know-Center is funded within the Austrian COMET Program a functionality is not available in comparable ontology engineering - Competence Centers for Excellent Technologies - under the aus- environments. pices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Interviews with ontology engineers who have used the a prior ver- Youth and by the State of Styria. COMET is managed by the Aus- sion of the models checklist to iteratively refine and improve their trian Research Promotion Agency FFG. The authors have also been ontologies indicate that such a functionality indeed supports the supported by APOSDLE (www.aposdle.org), which has been modelling activity. The models checklist was also deemed to be partially funded under grant 027023 in the IST work programme of helpful in evaluating the remaining amount of work by giving an the European Community. This paper was written while the sec- overview of the “status” of the model [1]. ond author was a Visiting Researcher in the Managing Complexity Theme at NICTA and she would like to thank the Centre for its hos- pitality. NICTA is funded by the Australian Government as repre- sented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program. Figure 1: Models checklist Figure 2: Quality indicator on a concept page. 3. REVIEWING LOGICAL ENTAILMENTS A key benefit of using a logically grounded language such as the Figure 3: Assertional effects are displayed on a concept page Web Ontology Language OWL [2] for specifying an ontology is after it has been edited (effects are enlarged on the picture). the possibility to automatically reason over such an ontology. The associated drawback is of course, that the larger and more complex the ontology, the more difficult it becomes for a single ontology en- 5. REFERENCES gineer to keep an overview over whether statements that logically [1] APOSDLE Deliverable 1.6. Integrated modelling follow from the ontology are true. methodology version 2, April 2009. State of the art ontology engineering tools such as Protégé and [2] B. Cuenca Grau, I. Horrocks, B. Motik, B. Parsia, the NeOn toolkit therefore contain the functionality to list entailed P. Patel-Schneider, and U. Sattler. OWL 2: The next step for statements provide explanations for them [4, 5]. A similar func- OWL. Journal of Web Semantics, 6(4):309–322, Nov. 2008. tionality in MoKi is called ontology questionnaire. While it does [3] S. Ghilardi, C. Lutz, and F. Wolter. Did I damage my not technically extend state of the art, its integration into MoKi’s ontology? A case for conservative extensions in description user interface puts an emphasis on motivating the ontology engi- logics. In Proceedings of KR2006: the 20th International neers to review logical entailments and act on them if the find they Conference on Principles of Knowledge Representation and do not agree with them. For instance, instead of “Entailed state- Reasoning, pages 187–197. AAAI Press, 2006. ments” or similar, the ontology questionnaire functionality is called [4] M. Horridge, B. Parsia, and U. Sattler. Explanation of OWL “Inferences - Do You Agree?”. The ontology questionnaire’s user entailments in protege 4. In International Semantic Web interface has been redesigned following the feedback on a prior ver- Conference (Posters & Demos), volume 401 of CEUR sion (not integrated in MoKi) described in [9]. Workshop Proceedings. CEUR-WS.org, 2008. It is also possible to consider the dynamics of an ontology, i.e. to [5] Q. Ji. RaDON - Repair and diagnosis in ontology networks, follow the changes made to an ontology and to feedback the logical http://www.neon-toolkit.org/wiki/RaDON. consequences of the changes to the ontology engineer. When only Last visited: 2010-07-28. the terminological axioms in an ontology are considered, such con- siderations are made under the name of “conservative extensions” [6] MoKi - The MOdelling WiKI. http://moki.fbk.eu. in description logics [3]. Analogously, it is possible to look for Last visited: 2010-07-28. consequences on data, i.e. to ask “If new statements about con- [7] N. F. Noy and D. L. McGuinness. Ontology development cepts and roles are added/removed, how does this affect individu- 101: A guide to creating your first ontology. Technical als in the ontology?” (assertional effects, see [10]). As an example, report, Stanford Knowledge Systems Laboratory and consider a knowledge base about the academic world. The knowl- Stanford Medical Informatics, 2001. edge base contains the fact that “EKAW 2010 is a conference”. An [8] Ontology Design Patterns. ontology engineer formalises the knowledge that conferences are a http://ontologydesignpatterns.org. Last particular kind of event, and that conferences produce proceedings. visited: 2010-07-28. (S)He adds the statements “Every conference is an event” and “Ev- [9] V. Pammer and S. Lindstaedt. Ontology evaluation through ery conference outputs only proceedings”. Assertional effects of assessment of inferred statements: Study of a prototypical these changes are the facts that “EKAW2010 is a conference” and implementation of an ontology questionnaire for OWL DL “EKAW2010 outputs only proceedings” (see Fig. 3 for how this is ontologies. In Knowledge Science, Engineering and displayed in MoKi). Such effects are displayed in MoKi directly Management, Third International Conference, KSEM 2009, after the ontology is changed. The assertional effects functionality number 5914 in LNAI, pages 394–405. Springer, 2009. in MoKi therefore makes ontology evaluation dynamic, by pointing [10] V. Pammer, L. Serafini, and S. Lindstaedt. Highlighting out potentially interesting inferences directly after they are gained assertional effects of ontology editing activities in OWL. In (or lost, when statements are removed). Such a functionality is not Proceedings of the 3rd International Workshop on Ontology available in comparable ontology engineering environments. Dynamics, (IWOD 2009), collocated with ISWC2009, volume 519. CEUR Workshop Proceedings, 2009. 4. CONCLUSION [11] M. Rospocher, C. Ghidini, V. Pammer, L. Serafini, and S. Lindstaedt. MoKi: The Modelling Wiki. In Proceedings of In this work we describe the integration of two state of the art prin- the Forth Semantic Wiki Workshop (SemWiki 2009), ciples for formative ontology evaluation into MoKi. Integration of co-located with ESWC 2009, volume 464 of CEUR ontology evaluation functionalities in ontology engineering tools Workshop Proceedings, pages 113–128, 2009. is, we believe, a prerequisite for ontology evaluation to become formative, which again is necessary for an ontology engineering [12] M. Uschold and M. Grüninger. Ontologies: Principles, process to become more iterative, more lively and thus more prone methods, applications. In Knowledge Engineering Review, to support the evolutionary engineering of ontologies. volume 11, pages 93–155. 1996.