Ontologies are the core component in semantic Web applications. The employment of an ontology building method affects the quality of ontology and the applicability of ontology language. A weighted evaluation approach for ontology building guidelines is presented in this paper. The evaluation criteria are based on an existing classification scheme of a semiotic framework for evaluating the quality of conceptual models. Directions for further refinement of ontology building methods are discussed.
The core components in the semantic Web [1] and its applications will be the ontologies. The quality of a semantic Web application will be highly dependent on the quality of its underlying ontology. The quality of the underlying ontology will in turn depend on factors such as 1) the appropriateness of the language used to represent the ontology, and 2) the quality of the engineering environment, including tool support and method guidelines, as provided for creating the ontology by means of the language.
There are also situated factors, such as the complexity of the specific task at hand and the expertise of the persons involved. Method guidelines can thus be seen as an important means to make ontology creation possible for a wider range of developers, e.g., not only a few expert researchers in the ontology field but also companies wanting to develop semantic Web applications for internal or external use.
The objective is to inspect available method guidelines for semantic Web-based ontology specification languages. The approach is to adapt the method classification part of a model quality framework [12], and to define a computational framework for the analytic evaluation of method guidelines.
The outline is as follows. Section 2 describes related work. Section 3 describes a weighting method for seven categories in the classification framework. Section 4 describes weighting of requirements and computation of final score. Finally, Section 5 concludes the paper and suggests directions for future work and for further refinement of ontology building methods.
There exist several methodologies to guide the process of Web ontology building that vary both in their level of generality and granularity. Some of the methodologies describe an overall ontology development process yet not the ontology creation itself. Such methodologies are primarily intended to support the knowledge elicitation and management of the ontologies in a basically centralised environment, for instance [6,18,19,20]. These methodologies provide a life cycle in an overall ontology development process as analysed in [2, 5, and 20], but only a few user guidelines for carrying out the steps and for actually creating the ontology. A limited selection of method guidelines were found for the Web ontology specification languages, which are at the foci of this study, i.e. [3,11,14].
Our evaluation framework is based on the method classification part of the framework of [12], it is most closely related to previous work using that same framework [7,9], and especially the evaluation of ontology languages and tools in [17]. In this paper the framework is used for evaluating something different, namely method guidelines for ontology building. Moreover, an interesting question is to which extent it is suitable for this new evaluation task, so customizations to the framework are suggested in order to improve its relevance for evaluating method guidelines in general, and method guidelines for ontology building in particular.
A methodology classification framework consisting of seven semiotic categories of modelling methodologies is described in [12]. We adapt the categories for classification of the ontology building method guidelines [7] and suggest selection criteria and coverage weight function for them. The principle modification here is that the concept of application system (as the end product of the development process) is consequently replaced by ontology (as the end product of applying the method guidelines). The experiences from the case study [9] suggested that numerical values could be used for the classification and thus qualify weighted selection techniques such as the [13] PORE methodology. Therefore, we adapt PORE methodology here and define the coverage weights -1, 1 and 2 for each category. The method guidelines are classified accordingly in the next section.
Let CF be a classification framework such that CF has a fixed set Ç of categories ç, where Ç = {ç 1 , ç 2 , ç 3 , ç 4 , ç 5 , ç 6 , ç 7 } and ç i Ç. Each ç is a quadruple <id, descriptor, C, cw>, where id is the name of the category, descriptor is a natural language description, C is a set of selection criteria c, and cw defines a function of S that return -1, 1, or 2 as coverage weight, where S is a set of satisfied elements c in the selection criteria C of each category in Ç. Intuitively, we define a number of selection criteria alongside an associated coverage weight function for each category in the classification framework. The categories are as follows. ç 1 -Weltanschauung describes the underlying philosophy or view to the world. For a method guideline we examine why the ontology construction is addressed in a particular way in a specific methodology. In accordance with the FRISCO report [4], three views can be identified: the objectivistic view, i.e. reality exists independently of any observer, where the relation between reality and the model is trivial or obvious, the constructivistic view, i.e. the reality exists independently of any observer, where observer possesses only a restricted mental model and the relationship between reality and models of this reality are subject to negotiations among the community of observers and evolve, and the mentalistic view, i.e. reality and the relationship to any model is totally dependent on the observer we can only form mental constructions of our perceptions. Weltanschauung can be ç 1 c 1 -explicit, i.e. stated in the document, ç 1 c 2 -implicit, i.e. derivable from the documentation, or ç 1 c 3 -undefined, i.e. non derivable.
(
ç 2 -Coverage in process concerns the method's ability to address ç 2 c 1 -planning for changes, ç 2 c 2 -single and co-operative development of ontology or aligned ontologies, which includes analysis, requirements specification, design, implementation and testing, ç 2 c 3 -use and operations of ontologies, ç 2 c 4 -maintaining and evolution of ontologies, and ç 2 c 5 -management of planning, development, operations and maintenance of ontologies.
(2) ç 3 -Coverage in product is described as how the method concerns planning, development, usage and maintenance of and operate on ç 3 c1 -one single ontology, ç 3 c 2 -a family of related ontologies, ç 3 c 3 -a whole portfolio of ontologies in an organization, and ç 3 c 4 -a totality of the goals, business process, people and technology used within the organization.
.
ç 4 -Reuse of product and process support reuse of ontologies as products or reuse of method as processes in order to avoid re-learning and recreation. There are six dimensions of reuse: ç 4 c 1 -Reuse by motivation answers the question -why is reuse done? Such rationales are for example productivity, timeliness, flexibility, quality, and risk management goals. ç 4 c 2 -Reuse by substance, answers the questionwhat is the essence of the items to be reused? A product is the set of deliverables that are produced during a project, such as models, documentation and test cases. Reusing a development or maintenance method is process reuse.
ç 5 -Stakeholder participation reflects the interests of different actors in the ontology building activity. The stakeholders may be categorized into those ç 5 c 1 -responsible for developing the method, those with ç 5 c 2 -financial interest and those who have ç 5 c 3 -interest in its use. Further, there are different forms of participation. Direct participation means every stakeholder has the opportunity to participate. Indirect participation uses representatives, thus every stakeholder is represented through other representatives that are supposed to look after their interests. .
ç 6 -Representation of product and process can be based on linguistic and nonlinguistic data such as audio and video. Representation languages for both product and process can be ç 6 c 1 -informal, ç 6 c 2 -semi-formal or ç 6 c 3 -formal, having a logical or executional semantics. .
ç 7 -Maturity is characterized on different levels of completion. Some methodologies have been used for a long time; others are only described in theory and never tried out in practice. Several conditions influence maturity of a method, namely if the method is ç 7 c 1 -fully described, if the method lends itself for ç 7 c 2 -adaptation, navigation and development, if the method is ç 7 c 3used and updated through practical applications, if it is ç 7 c 4 -used by many organizations, and if the method is ç 7 c 5 -altered based on experience and scientific study of its use.
The selection criteria are exhaustive and mutually exclusive in the categories ç 1 , and ç 6 , exhaustive in ç 5 , whereas the set of satisfied criteria S of the remaining categories may also be the empty list . The coverage weight cw is independent of any category-wise prioritisation. Since the intervals are decisive for the coverage weight they can be adjusted depending on preferences of the evaluator. However, when analysing different evaluation occurrences the intervals need to be fixed in comparison, but may be used as dependent variable.
Requirements are categorized according to the categories of the classification framework [12] and the importance weights are calculated according to Eq.
Finally, total coverage weights Tw i for each ontology building guideline i are calculated. Total weights calculated using equation ( 9) are used as overall feasibility rate for supporting the choice of ontology building guidelines.
An evaluation of three method guidelines for semantic Web ontology building was conducted using the [7,12] framework. Evaluation of method guidelines was performed in two steps, one general evaluation, i.e. their applicability for building ontologies in general, and one particular, i.e. how appropriate are they for ontology development in a real world project -how applicable is the framework in practice. The results of evaluation are presented in [8].
The main contribution of this paper is incorporation of numerical values and metrics to the classification framework for the classification and thus supporting qualification of weighted selection to produce the more explicit evaluation results.
There are several interesting topics for future work, such as supplementing the theoretical evaluations with empirical ones as larger scale semantic Web applications arise utilizing the empirical nature of [12], as well as evaluating more methods as they emerge, e.g. [10,15,16].