=Paper=
{{Paper
|id=None
|storemode=property
|title=Automatic Support for Formative Ontology Evaluation
|pdfUrl=https://ceur-ws.org/Vol-674/Paper45.pdf
|volume=Vol-674
|dblpUrl=https://dblp.org/rec/conf/ekaw/PammerLGRS10
}}
==Automatic Support for Formative Ontology Evaluation==
https://ceur-ws.org/Vol-674/Paper45.pdf
Automatic Support for Formative Ontology Evaluation ∗
Viktoria Pammer Chiara Ghidini Marco Rospocher
Know-Center, Austria FBK-irst, Trento, Italy FBK-irst, Trento, Italy
vpammer@know- ghidini@fbk.eu rospocher@fbk.eu
center.at
Luciano Serafini Stefanie Lindstaedt
FBK-irst, Trento, Italy Know-Center and KMI TU
serafini@fbk.eu Graz, Austria
slind@know-center.at
ABSTRACT into an ontology engineering tool, ontology evaluation can finally
Just as testing is an integral part of software engineering, so is on- become formative, since feedback for potential improvement or
tology evaluation an integral part of ontology engineering. We have review is given in the same “place” where ontology engineering
implemented automated support for formative ontology evaluation happens. In this regard, formative ontology evaluation is inher-
based on the two principles of i) checking for compliance with ently different from ontology evaluation metrics that aim to mea-
modelling guidelines and ii) reviewing entailed statements in MoKi, sure an ontology’s characteristics only when it is regarded as “fin-
a wiki based ontology engineering environment. These principles ished enough” to merit evaluation.
exist in state of the art literature and good ontology engineering and
evaluation practice, but have not so far been widely integrated into
ontology engineering tools.
2. COMPLIANCE WITH GUIDELINES
Modelling guidelines provide guidance to the modellers during the
ontology construction process but do not impose strict constraints
1. INTRODUCTION on the ontology engineer. Hence, checking the compliance of an
State of the art ontology evaluation practice relies on guidelines and ontology to modelling guidelines can be indicative only of poten-
best practices in ontology engineering such as [7, 8], on ontology tial modelling errors. For instance, a typical modelling guideline is
evaluation methodologies such as competency questions [12], and to verbally describe model elements (concepts, roles and to a cer-
on reasoning to detect logical inconsistencies. The work we present tain extent also individuals) and document design decisions. While
here follows up on such existing work by automatically checking it is impossible with the current state of the art to automatically
an ontology in progress for compliance with modelling guidelines determine how good a description really is, it is possible to auto-
to detect potential modelling errors and motivating ontology engi- matically check for model elements that are not documented at all.
neers to review entailed statements throughout the modelling pro- In MoKi, a models checklist page (Fig. 1) lists modelling guide-
cess in MoKi, a wiki based ontology engineering tool [6, 11] that lines, and for each guideline those model elements (concepts, prop-
has recently been released as open-source. erties, individuals) that do not comply with the guideline. A quality
Through integrating such support for ontology evaluation directly indicator visualises the “degree” to which a single model element
complies to the whole set of modelling guidelines (Fig. 2). Such
∗The Know-Center is funded within the Austrian COMET Program
a functionality is not available in comparable ontology engineering
- Competence Centers for Excellent Technologies - under the aus- environments.
pices of the Austrian Federal Ministry of Transport, Innovation and
Technology, the Austrian Federal Ministry of Economy, Family and Interviews with ontology engineers who have used the a prior ver-
Youth and by the State of Styria. COMET is managed by the Aus- sion of the models checklist to iteratively refine and improve their
trian Research Promotion Agency FFG. The authors have also been ontologies indicate that such a functionality indeed supports the
supported by APOSDLE (www.aposdle.org), which has been modelling activity. The models checklist was also deemed to be
partially funded under grant 027023 in the IST work programme of helpful in evaluating the remaining amount of work by giving an
the European Community. This paper was written while the sec- overview of the “status” of the model [1].
ond author was a Visiting Researcher in the Managing Complexity
Theme at NICTA and she would like to thank the Centre for its hos-
pitality. NICTA is funded by the Australian Government as repre-
sented by the Department of Broadband, Communications and the
Digital Economy and the Australian Research Council through the
ICT Centre of Excellence program.
Figure 1: Models checklist
Figure 2: Quality indicator on a concept page.
3. REVIEWING LOGICAL ENTAILMENTS
A key benefit of using a logically grounded language such as the Figure 3: Assertional effects are displayed on a concept page
Web Ontology Language OWL [2] for specifying an ontology is after it has been edited (effects are enlarged on the picture).
the possibility to automatically reason over such an ontology. The
associated drawback is of course, that the larger and more complex
the ontology, the more difficult it becomes for a single ontology en- 5. REFERENCES
gineer to keep an overview over whether statements that logically [1] APOSDLE Deliverable 1.6. Integrated modelling
follow from the ontology are true. methodology version 2, April 2009.
State of the art ontology engineering tools such as Protégé and [2] B. Cuenca Grau, I. Horrocks, B. Motik, B. Parsia,
the NeOn toolkit therefore contain the functionality to list entailed P. Patel-Schneider, and U. Sattler. OWL 2: The next step for
statements provide explanations for them [4, 5]. A similar func- OWL. Journal of Web Semantics, 6(4):309–322, Nov. 2008.
tionality in MoKi is called ontology questionnaire. While it does [3] S. Ghilardi, C. Lutz, and F. Wolter. Did I damage my
not technically extend state of the art, its integration into MoKi’s ontology? A case for conservative extensions in description
user interface puts an emphasis on motivating the ontology engi- logics. In Proceedings of KR2006: the 20th International
neers to review logical entailments and act on them if the find they Conference on Principles of Knowledge Representation and
do not agree with them. For instance, instead of “Entailed state- Reasoning, pages 187–197. AAAI Press, 2006.
ments” or similar, the ontology questionnaire functionality is called
[4] M. Horridge, B. Parsia, and U. Sattler. Explanation of OWL
“Inferences - Do You Agree?”. The ontology questionnaire’s user
entailments in protege 4. In International Semantic Web
interface has been redesigned following the feedback on a prior ver-
Conference (Posters & Demos), volume 401 of CEUR
sion (not integrated in MoKi) described in [9].
Workshop Proceedings. CEUR-WS.org, 2008.
It is also possible to consider the dynamics of an ontology, i.e. to
[5] Q. Ji. RaDON - Repair and diagnosis in ontology networks,
follow the changes made to an ontology and to feedback the logical
http://www.neon-toolkit.org/wiki/RaDON.
consequences of the changes to the ontology engineer. When only
Last visited: 2010-07-28.
the terminological axioms in an ontology are considered, such con-
siderations are made under the name of “conservative extensions” [6] MoKi - The MOdelling WiKI. http://moki.fbk.eu.
in description logics [3]. Analogously, it is possible to look for Last visited: 2010-07-28.
consequences on data, i.e. to ask “If new statements about con- [7] N. F. Noy and D. L. McGuinness. Ontology development
cepts and roles are added/removed, how does this affect individu- 101: A guide to creating your first ontology. Technical
als in the ontology?” (assertional effects, see [10]). As an example, report, Stanford Knowledge Systems Laboratory and
consider a knowledge base about the academic world. The knowl- Stanford Medical Informatics, 2001.
edge base contains the fact that “EKAW 2010 is a conference”. An [8] Ontology Design Patterns.
ontology engineer formalises the knowledge that conferences are a http://ontologydesignpatterns.org. Last
particular kind of event, and that conferences produce proceedings. visited: 2010-07-28.
(S)He adds the statements “Every conference is an event” and “Ev- [9] V. Pammer and S. Lindstaedt. Ontology evaluation through
ery conference outputs only proceedings”. Assertional effects of assessment of inferred statements: Study of a prototypical
these changes are the facts that “EKAW2010 is a conference” and implementation of an ontology questionnaire for OWL DL
“EKAW2010 outputs only proceedings” (see Fig. 3 for how this is ontologies. In Knowledge Science, Engineering and
displayed in MoKi). Such effects are displayed in MoKi directly Management, Third International Conference, KSEM 2009,
after the ontology is changed. The assertional effects functionality number 5914 in LNAI, pages 394–405. Springer, 2009.
in MoKi therefore makes ontology evaluation dynamic, by pointing [10] V. Pammer, L. Serafini, and S. Lindstaedt. Highlighting
out potentially interesting inferences directly after they are gained assertional effects of ontology editing activities in OWL. In
(or lost, when statements are removed). Such a functionality is not Proceedings of the 3rd International Workshop on Ontology
available in comparable ontology engineering environments. Dynamics, (IWOD 2009), collocated with ISWC2009,
volume 519. CEUR Workshop Proceedings, 2009.
4. CONCLUSION [11] M. Rospocher, C. Ghidini, V. Pammer, L. Serafini, and
S. Lindstaedt. MoKi: The Modelling Wiki. In Proceedings of
In this work we describe the integration of two state of the art prin-
the Forth Semantic Wiki Workshop (SemWiki 2009),
ciples for formative ontology evaluation into MoKi. Integration of
co-located with ESWC 2009, volume 464 of CEUR
ontology evaluation functionalities in ontology engineering tools
Workshop Proceedings, pages 113–128, 2009.
is, we believe, a prerequisite for ontology evaluation to become
formative, which again is necessary for an ontology engineering [12] M. Uschold and M. Grüninger. Ontologies: Principles,
process to become more iterative, more lively and thus more prone methods, applications. In Knowledge Engineering Review,
to support the evolutionary engineering of ontologies. volume 11, pages 93–155. 1996.