=Paper=
{{Paper
|id=Vol-161/paper-19
|storemode=property
|title=Weighted Evaluation of Ontology Building Methods
|pdfUrl=https://ceur-ws.org/Vol-161/FORUM_18.pdf
|volume=Vol-161
|dblpUrl=https://dblp.org/rec/conf/caise/HakkarainenSHT05
}}
==Weighted Evaluation of Ontology Building Methods==
https://ceur-ws.org/Vol-161/FORUM_18.pdf
107
Weighted Evaluation of Ontology Building Methods
Sari Hakkarainen, Darijus Strasunskas, Lillian Hella, Stine Tuxen
Norwegian University of Science and Technology,
Sem Sælands vei 7-9, NO-7491 Trondheim, Norway
{sari, dstrasun, hella, stinemt}@idi.ntnu.no
Abstract. Ontologies are the core component in semantic Web applications.
The employment of an ontology building method affects the quality of ontology
and the applicability of ontology language. A weighted evaluation approach for
ontology building guidelines is presented in this paper. The evaluation criteria
are based on an existing classification scheme of a semiotic framework for
evaluating the quality of conceptual models. Directions for further refinement
of ontology building methods are discussed.
Keywords: semantic web, ontology building methods, quality evaluation
1 Introduction
The core components in the semantic Web [1] and its applications will be the ontolo-
gies. The quality of a semantic Web application will be highly dependent on the qual-
ity of its underlying ontology. The quality of the underlying ontology will in turn
depend on factors such as 1) the appropriateness of the language used to represent the
ontology, and 2) the quality of the engineering environment, including tool support
and method guidelines, as provided for creating the ontology by means of the lan-
guage.
There are also situated factors, such as the complexity of the specific task at hand
and the expertise of the persons involved. Method guidelines can thus be seen as an
important means to make ontology creation possible for a wider range of developers,
e.g., not only a few expert researchers in the ontology field but also companies want-
ing to develop semantic Web applications for internal or external use.
The objective is to inspect available method guidelines for semantic Web-based
ontology specification languages. The approach is to adapt the method classification
part of a model quality framework [12], and to define a computational framework for
the analytic evaluation of method guidelines.
The outline is as follows. Section 2 describes related work. Section 3 describes a
weighting method for seven categories in the classification framework. Section 4
describes weighting of requirements and computation of final score. Finally, Section 5
concludes the paper and suggests directions for future work and for further refinement
of ontology building methods.
Proceedings of the CAiSE'05 Forum - O. Belo, J. Eder, J. Falcão e Cunha, O. Pastor (Eds.)
© Faculdade de Engenharia da Universidade do Porto, Portugal 2005 - ISBN 972-752-078-2
108 Sari Hakkarainen, Darijus Strasunskas, Lillian Hella, Stine Tuxen
2 Ontology Building Support and Evaluation Methods
There exist several methodologies to guide the process of Web ontology building that
vary both in their level of generality and granularity. Some of the methodologies de-
scribe an overall ontology development process yet not the ontology creation itself.
Such methodologies are primarily intended to support the knowledge elicitation and
management of the ontologies in a basically centralised environment, for instance [6,
18, 19, 20]. These methodologies provide a life cycle in an overall ontology develop-
ment process as analysed in [2, 5, and 20], but only a few user guidelines for carrying
out the steps and for actually creating the ontology. A limited selection of method
guidelines were found for the Web ontology specification languages, which are at the
foci of this study, i.e. [3, 11, 14].
Our evaluation framework is based on the method classification part of the frame-
work of [12], it is most closely related to previous work using that same framework
[7, 9], and especially the evaluation of ontology languages and tools in [17]. In this
paper the framework is used for evaluating something different, namely method
guidelines for ontology building. Moreover, an interesting question is to which extent
it is suitable for this new evaluation task, so customizations to the framework are
suggested in order to improve its relevance for evaluating method guidelines in gen-
eral, and method guidelines for ontology building in particular.
3 Computation of Criteria Weight for Seven Semiotic Categories
A methodology classification framework consisting of seven semiotic categories of
modelling methodologies is described in [12]. We adapt the categories for classifica-
tion of the ontology building method guidelines [7] and suggest selection criteria and
coverage weight function for them. The principle modification here is that the concept
of application system (as the end product of the development process) is consequently
replaced by ontology (as the end product of applying the method guidelines). The
experiences from the case study [9] suggested that numerical values could be used for
the classification and thus qualify weighted selection techniques such as the [13]
PORE methodology. Therefore, we adapt PORE methodology here and define the
coverage weights -1, 1 and 2 for each category. The method guidelines are classified
accordingly in the next section.
Let CF be a classification framework such that CF has a fixed set Ç of categories
ç, where Ç = {ç1, ç2, ç3, ç4, ç5, ç6, ç7} and çi Ç. Each ç is a quadruple , where id is the name of the category, descriptor is a natural language
description, C is a set of selection criteria c, and cw defines a function of S that return
-1, 1, or 2 as coverage weight, where S is a set of satisfied elements c in the selection
criteria C of each category in Ç. Intuitively, we define a number of selection criteria
alongside an associated coverage weight function for each category in the classifica-
tion framework. The categories are as follows.
ç1 - Weltanschauung describes the underlying philosophy or view to the world. For a
method guideline we examine why the ontology construction is addressed in a par-
ticular way in a specific methodology. In accordance with the FRISCO report [4],
109
three views can be identified: the objectivistic view, i.e. reality exists independently
of any observer, where the relation between reality and the model is trivial or obvi-
ous, the constructivistic view, i.e. the reality exists independently of any observer,
where observer possesses only a restricted mental model and the relationship be-
tween reality and models of this reality are subject to negotiations among the
community of observers and evolve, and the mentalistic view, i.e. reality and the
relationship to any model is totally dependent on the observer we can only form
mental constructions of our perceptions. Weltanschauung can be ç1c1 – explicit, i.e.
stated in the document, ç1c2 – implicit, i.e. derivable from the documentation, or
ç1c3 – undefined, i.e. non derivable.
2, if ç1c1 S1 .
°
cw1 (S1 ) ® 1, if ç1c 2 S1 . (1)
°
¯1, if ç1c 3 S1 .
ç2 - Coverage in process concerns the method’s ability to address ç2c1 – planning for
changes, ç2c2 – single and co-operative development of ontology or aligned on-
tologies, which includes analysis, requirements specification, design, implementa-
tion and testing, ç2c3 – use and operations of ontologies, ç2c4 – maintaining and
evolution of ontologies, and ç2c5 – management of planning, development, opera-
tions and maintenance of ontologies.
1, if C 2 = 0.
°
cw 2 ( S 2 ) ® 1, if 0 < C 2 d 2. (2)
°̄ 2, if 2 < C 2 d 5.
ç3 - Coverage in product is described as how the method concerns planning, devel-
opment, usage and maintenance of and operate on ç3c1 – one single ontology, ç3c2
– a family of related ontologies, ç3c3 – a whole portfolio of ontologies in an organi-
zation, and ç3c4 – a totality of the goals, business process, people and technology
used within the organization.
1, if C3 0.
°
cw3 ( S3 ) ® 1, if 0 C3 d 2. (3)
°̄ 2, if 2 C3 d 4.
ç4 - Reuse of product and process support reuse of ontologies as products or reuse of
method as processes in order to avoid re-learning and recreation. There are six di-
mensions of reuse: ç4c1 – Reuse by motivation answers the question - why is reuse
done? Such rationales are for example productivity, timeliness, flexibility, quality,
and risk management goals. ç4c2 – Reuse by substance, answers the question –
what is the essence of the items to be reused? A product is the set of deliverables
that are produced during a project, such as models, documentation and test cases.
Reusing a development or maintenance method is process reuse. ç4c3 – Reuse by
development scope, answers the question – what is the coverage of the form and
the extent of reuse? The scope may be either external or internal to a project or or-
110 Sari Hakkarainen, Darijus Strasunskas, Lillian Hella, Stine Tuxen
ganization. ç4c4 – Reuse by management mode, answers the questions - how is re-
use conducted? Reuse may be planned in advance with existing guidelines and
procedures, or ad-hoc. ç4c5 – Reuse by technique answers the question - how is re-
use implemented? The reuse may be compositional and/or generative. ç4c6 – Reuse
by intentions, answers the question - what is the purpose of reused elements? There
are different degrees of intention. The elements may be used as they are, slightly
modified, used as a template or just used as an idea.
1, if 0 C4 d 2.
°
cw4 ( S 4 ) ® 1, if 2 < C4 d 4. (4)
°̄ 2, if 4 < C4 d 6.
ç5 - Stakeholder participation reflects the interests of different actors in the ontology
building activity. The stakeholders may be categorized into those ç5c1 – responsi-
ble for developing the method, those with ç5c2 – financial interest and those who
have ç5c3 – interest in its use. Further, there are different forms of participation. Di-
rect participation means every stakeholder has the opportunity to participate. Indi-
rect participation uses representatives, thus every stakeholder is represented
through other representatives that are supposed to look after their interests.
1, if C5 0.
°
cw5 ( S5 ) ® 1, if 0 C5 d 1. (5)
°̄ 2, if 1 C5 d 3.
ç6 - Representation of product and process can be based on linguistic and non-
linguistic data such as audio and video. Representation languages for both product
and process can be ç6c1 – informal, ç6c2 – semi-formal or ç6c3 – formal, having a
logical or executional semantics.
1, if C6 1.
°
cw6 ( S 6 ) ® 1, if C6 2. (6)
°̄ 2, if C6 3.
ç7 - Maturity is characterized on different levels of completion. Some methodologies
have been used for a long time; others are only described in theory and never tried
out in practice. Several conditions influence maturity of a method, namely if the
method is ç7c1 – fully described, if the method lends itself for ç7c2 – adaptation,
navigation and development, if the method is ç7c3 – used and updated through
practical applications, if it is ç7c4 – used by many organizations, and if the method
is ç7c5 – altered based on experience and scientific study of its use.
The selection criteria are exhaustive and mutually exclusive in the categories ç1, and
ç6, exhaustive in ç5, whereas the set of satisfied criteria S of the remaining categories
may also be the empty list ^`. The coverage weight cw is independent of any cate-
gory-wise prioritisation. Since the intervals are decisive for the coverage weight they
can be adjusted depending on preferences of the evaluator. However, when analysing
111
different evaluation occurrences the intervals need to be fixed in comparison, but may
be used as dependent variable.
1, if C7 0.
°
cw7 ( S7 ) ® 1, if 0 < C7 d 3. (7)
°̄ 2, if 3 < C7 d 5.
4 Computation of Importance Weight for Requirements
Requirements are categorized according to the categories of the classification frame-
work [12] and the importance weights are calculated according to Eq. 8 as follows.
Let R be a set of weighted requirements such that R has a fixed set RÇ of categories
rç, where categories in RÇ are the same as in the fixed set Ç of categories ç of the
classification framework CF, i.e. RÇ = Ç, and ç Ç, rç RÇ. rç is a triple , id is the name of the category, req_descriptor is a natural lan-
guage description of requirement, and iwrç defines a function of I that returns 1, 3, or
5 as importance weight based on priorities and policy of the company.
1, if rç may be satisfied, is optional,
°
iwr9 (I ) ®3, if rç should be satisfied, is recommended , (8)
°
¯5, if rç must be satisfied. is essential,
Finally, total coverage weights Twi for each ontology building guideline i are cal-
culated. Total weights calculated using equation (9) are used as overall feasibility rate
for supporting the choice of ontology building guidelines.
Tw i ¦ ( cw 9 u iw 9 )
9 Ç
r
(9)
5 Concluding Remarks
An evaluation of three method guidelines for semantic Web ontology building was
conducted using the [7, 12] framework. Evaluation of method guidelines was per-
formed in two steps, one general evaluation, i.e. their applicability for building on-
tologies in general, and one particular, i.e. how appropriate are they for ontology de-
velopment in a real world project - how applicable is the framework in practice. The
results of evaluation are presented in [8].
The main contribution of this paper is incorporation of numerical values and met-
rics to the classification framework for the classification and thus supporting qualifi-
cation of weighted selection to produce the more explicit evaluation results.
There are several interesting topics for future work, such as supplementing the
theoretical evaluations with empirical ones as larger scale semantic Web applications
112 Sari Hakkarainen, Darijus Strasunskas, Lillian Hella, Stine Tuxen
arise utilizing the empirical nature of [12], as well as evaluating more methods as they
emerge, e.g. [10, 15, 16].
References
1. Berners-Lee, T., Handler, J., Lassila, O.: The Semantic Web. Scientific American, May
(2001).
2. Corcho, O., Fernández-López, M., Gómez-Pérez, A.: Methodologies, tools and languages for
building ontologies: where is their meeting point?, Data & Knowledge Engineering, 46:1
(2003) pp 41 – 64.
3. Denker, G.: DAML+OIL Plug-in for Protége 2000 – User’s Guide. SRI Intl. AI Center Re-
port 7/8/03, (2003).
4. Falkenberg, E.D., Hesse, W., Lindgreen, P., Nilsson, B.E., Oei, H., Rolland, C., Stamper,
R.K., van Asssche, F.J.M., Verrjin-Stuart, A., Voss, K.: FRISCO - A Framework of
Information Systems Concepts. IFIP WG 8.1 Technical Report. December (1997).
5. Fernández-López, M.: Overview of Methodologies for Building Ontologies. Benjamins,
V.R., Chandrasekaran, B., Gómez-Pérez, A., Guarino, N., Uschold, M. (Eds.): Proc. of the
IJCAI-99 workshop on Ontologies and Problem-Solving Methods (KRR5) Stockholm, Swe-
den, (1999).
6. Fernándes-López, M., Gómez-Péres, A., Juriso, N.: METHONTOLOGY: From Ontological
Art Towards Ontological Engineering. Proc. of AAAI-97 Spring Symposium on Ontological
Engineering. Stanford University, (1997).
7. Hakkarainen, S., Hella, L., Tuxen, S.M., Sindre, G.: Evaluating the quality of web-based
ontology building methods: a framework and a case study. Proc. of 6th Intl. Baltic Conf. on
Databases and Information Systems (Baltic DBIS’04), Riga, Latvia, (2004).
8. Hakkarainen, S., Strasunskas, D., Hella, L., Tuxen, S.M.: Classification of Web-Based On-
tology Building Methods: a computational framework and a case study. Submitted (2005).
9. Hella, L., Tuxen, S.M.: An Evaluation of Ontology Building Methodologies - An analysis
and a case study, Information Systems Specialization Report, NTNU, Norway (2003).
10. Knublauch, H.: Protégé OWL Tutorial, 7th Intl. Protégé Conf., Bethesda, Maryland (2004).
11. Knublauch, H., Musen M.A., Noy N.F.: Creating Semantic Web (OWL) Ontologies with
Protégé. Tutorial at 2nd Intl. Semantic Web Conf. Sanibel Island Florida, USA (2003).
12. Krogstie, J.: Conceptual Modeling for Computerized Information System Support in Or-
ganizations, PhD Thesis 1995:87 NTH, Trondheim, Norway (1995).
13. Maiden, N.A.M., Ncube, C.: Acquiring COTS Software Selection Requirements. IEEE
Software March/April, pp 46-56 (1998).
14. Noy, N.F., McGuinness, D.L.: Ontology Development 101: A Guide to Creating Your First
Ontology. Stanford Knowledge Systems Laboratory Technical Report KSL-01-05 (2001).
15. Pepper, S.: The TAO of Topic Maps - Finding the Way in the Age of Infoglut, Ontopia AS,
Norway (2004). URL: http://www.ontopia.net/topicmaps/materials/tao.html
16. Smith, M. K., Welty, C., McGuinness, D. L.: OWL Web Ontology Language Guide, W3C
Recommendation 10 February, World Wide Web Consortium (2004).
17. Su, X., Ilebrekke L.: A comparative study of ontology languages and tools. Halpin, T., Siau,
K., Krogstie, J. (Eds.): Proc. of EMMSAD’02, Toronto, Canada (2002).
18. Sure, Y., Studer, R.: On-To-Knowledge Methodology – Final Version. Institute AIFB,
University of Karlsruhe (2002).
19. Swartout, B., Ramesh P., Knight, K., Russ, T.: Toward Distributed Use of Large-Scale
Ontologies. Symposium on Ontological Engineering of AAAI. Stanford, California. (1997).
20. Uschold M.: Building Ontologies: Towards a Unified methodology. Proc. of the 16th Conf.
of the British Computer Society Specialist Group on Expert Systems. Cambridge (1996).