OntoMetrics: Application of on-line Ontology Metric Calculation

Abstract

A lot of work has been done regarding ontology evaluation in the recent years. Automatically calculated indicators are needed in order to assess ontology quality. Manual evaluation of ontologies would be very time consuming. While there is tool support for the detection modelling errors and the violation of ontology modelling guidelines, there is a lack of support for calculating ontology metrics. Many metrics have been proposed that correlate for example with ontology characteristics like Readability, Adaptability, and Reusability. However, no tools have been created or tools are no longer maintained and bound to certain ontology editors. OntoMetrics lls this gap by providing an on-line platform for ontology metric calculation.This paper presents the current status of OntoMetrics, use cases and planned future developments.

Keywords

• ontology
• ontology evaluation
• ontology metrics
• ontology quality

1 Introduction

Ontology evaluation is a task that has been subject to research for years. A strong focus has been laid on the selection of appropriate ontologies for reuse in ontology engineering. Considering existing ontologies as sources for the construction of new ones is part of accepted ontology engineering methods. Noy and McGuinnes for example have an explicit reuse-step in their method [ 1 ]. With the increasing number of existing ontologies and thus an increasing number of candidates for reuse in a certain domain, automated evaluation of ontologies is required. Swoogle1 as a search engine for ontologies calculates an ontology rank for the order of the presented search results [ 2 ]. Additionally, swoogle calculates some basic ontology-metrics that become part of the available ontology meta-data. Besides the reuse aspect, ontology quality should be monitored throughout the ontology life-cycle. This includes the creation of ontologies but also their maintenance. Again, there is a need for automated evaluation due to the complexity of ontologies and knowledgebases. A majority of the approaches suggests metric calculation in order to assess ontology characteristics (examples in [ 3, 4, 5, 6, 7, 8 ]).

Existing Ontology Development Environments like Protege provide only basic support for ontology evaluation. Plug-ins for ontology evaluation that have

1 http://swoogle.umbc.edu/

been developed are bound to a certain ontology editor. This comes with some disadvantages: (a) the user is forced to use the editor the plug-in is developed for, (b) plug-ins tend to be outdated if new editor versions evolve (see also [ 9 ]), and (c) discontinuation of editor development makes the plug-in unavailable for use. Furthermore, a number of approaches remained in the status of prototypes or just proposals. As a consequence, approaches to automated ontology evaluation are rarely available for empirical evaluation and practical use. So far, web based solutions seem to be the best at hand because of their public availability and their independence from ontology development environments. The OntoMetrics2 on-line platform has been developed as a consequence of this discussion by: 1. Providing a web-based platform for ontology metric calculation. 2. Supporting a standardized ontology format (OWL 2). 3. Collecting the theory behind the metrics and their validation. 4. Providing machine-readable XML-output for further analysis. After an introduction of the eld of ontology metrics in the next section, the remainder of this paper is dedicated to the OntoMetrics on-line platform. Section 3 describes the current platform functionality and the general usage scenarios. Examples of ontology metric calculation scneraios are provided in section 4. Finally, the future development roadmap is drawn in section 5.

2 Ontology Metrics for Ontology Evaluation

Similarly to Software Quality Metrics, Ontology Metrics are quanti ers that can be determined using a de ned measurement procedure that is applied to the ontology (software) [ 5 ]. A correlation between the calculated metrics and certain quality criteria is assumed or { even better { empirically validated. Publications on ontology metrics often see this validation as a problem ( see for example [ 10, 11, 8 ]). In order have a better validation of the proposed metrics, the metrics calculation needs to be made available, reproducible, and analysable. A platform like OntoMetrics addresses these issues.

Throughout the Ontology Life-cycle di erent aspects (Ontology Scopes) of the ontology have to be assessed regarding their quality. Furthermore, with each step in the life-cycle new ontology development artefacts (Ontology Layers) become available for evaluation. Thus, applicability of Ontology Evaluation Methods and the importance of Ontology Quality Criteria depend on the phases of the Ontology Life-cycle. As a results, ve dimensions for description of metric based ontology evaluation can be found: (I) Ontology Scopes, (II) Ontology Layers, (III) Ontology Life-cycle, (IV) Ontology Quality Criteria, and (V) Ontology Evaluation Methods. Pak and Zhou [ 12 ] de ne quite similar dimensions in their ontology evaluation framework.

Figure 1 illustrates the described dependencies throughout the Ontology Lifecycle. While the Vocabulary of an ontology is de ned during Conceptualization,

2 http://www.ontometrics.org

formal Metadata and ontology Taxonomy/Structure become available for analysis during Formalization. With the Integration of other ontologies, the Context of the ontology is set. Implementation then adds Population and the Application within an information system that uses the ontology. Operational data of the information system becomes available when the system is in use and the ontology is in the Maintenance phase. Except for Operation and Application layer (greyed in gure 1), data of all ontology layers is part of the OWL 2 speci cation and thus can be evaluated by an on-line platform like OntoMetrics. For the Ontology Scope it is depicted, when they can be addressed in the ontology life-cycle and at which phases they are most important (dark areas in gure 1): Domain Scope: How well does the ontology represent the real world? Generally, data driven evaluation methods can be applied here like presented by Alani et al. [ 13 ]. These methods either use text corpora of the represented domain or golden standard ontologies and compare the evaluated ontology to them. Ontology Quality Criteria that are evaluated in the Domain Scope (correctness of the ontology) are Accuracy, Completeness, Conciseness, and Consistency [ 14 ].

Conceptual Scope: What is the quality of the ontology in analogy to internal software quality characteristics? Generally, ontology structure based methods can be applied here. We divide between schema metrics suggested for example by Tartir et al. [ 8 ] that consider the special semantics of the ontology schema graph elements and graph-based metrics that calculate general graph characteristics like size and breadth for the taxonomical part of the ontology (for example Gangemi et al.[ 5 ]). Furthermore, Gangemi et al. suggest metrics based on annotations within the ontology. Ontology Quality Criteria that are evaluated in the Conceptual Scope are for example Computational E ciency, Adaptability, Clarity [ 14 ], Reusability [ 15 ], and Readability [ 8 ]. Application Scope: How well does the ontology in use as component of an ontology based information system (external software quality)? The third aspect of evaluation is the external quality of an ontology in conjunction with an information system. Thus, the characteristics of the used information system have an in uence on the on the measured quality. Taskbased methods as described by Porzel and Malaka [ 16 ] can be used here. Another possibility is the assessment of usage statistics [ 10 ]. However, little e ort has been spent on the development and validation of methods that evaluate ontologies within the Application Scope. A reason lies in the e ort of evaluating di erent ontologies for the use within the same information system or vice versa to evaluate the same ontology in di erent information systems in order to exclude the information system's in uence on the measurement. Nonetheless, monitoring changes in the metrics that are used to measure these criteria should provide information regarding ontology maintenance.Ontology Quality Criteria that can be evaluated in the Application Scope are E ciency, E ectivity, Accuracy and general Value (measured by Popularity). 3 OntoMetrics { Current State and Intended Use At its current state, the OntoMetrics platform has the following functional areas: { A web-interface to upload ontologies and to calculate a set of Ontology Quality

Metrics for them. { An XML-download of calculated Ontology Quality Metrics. { A wiki that explains the semantics and the calculation of the Ontology Quality

Metrics.

The web interface for metric calculation accepts ontologies represented in OWL or RDF using RDF-XML representation. The ontologies can either be uploaded by a le picker or copied into a text eld. Alternatively, an ontology URI can be used to specify which ontology has to be analysed. Prior to the metric calculation, the user can choose via a selector box, which kind of metrics should be calculated. The class metrics which have been adopted from Tartir et al. [ 8 ] cause a higher computational e ort. Here, special metrics for each named class within the ontology are calculated. When the calculation has been started by the `Start Calculation'-Button, the platform tries to retrieve all ontologies that are imported to the analysed ontology. If one of the imports is not available, a speci cation of the problem is provided. On the results page, the metrics can be directly analysed. A download of the results in XML-format is available as well.

Table 2 shows the metrics that can currently be calculated using OntoMetrics in addition to the standard OWL-API counting metrics. There are four general types of metrics. Schema Metrics take the special meaning of the OWL-Schemade nition constructs into account for the calculation of metrics on the ontology structure. Graph Metrics are metrics that can be generally applied to graphs (esp. trees). In the case of ontology evaluation, they are calculated for the taxonomy tree of the ontology. Knowledgebase Metrics do not only assess the type structure of an ontology but also instances that populate the ontology. At last, Class Metrics narrow the focus to single class evaluation.

For a explanation of the metrics, the reader can refer to the OntoMetricsWiki or to the given sources in the table. Although mainly two sources are given with Gangemi et al. [ 17 ] and Tartir et al. [ 8 ], most of these metrics have also been proposed by other authors or are basic graph metrics.

The table-head contains quality criteria as mentioned in section 2. For more comprehensive semantics of the criteria, refer to [ 12 ]. Those quality criteria have been selected, where a correlation to the calculated metrics has been proposed in the literature. Thus, for each of the quality criteria at least one metric is available in conjunction with the proposed direction of correlation. '+' means positive correlation and ' " negative correlation. As expected from the previous discussion, quality criteria that are relevant within the Domain and the Application Scope are under-represented. Furthermore, the Metric-Criteria-Matrix provided by table 2 is sparsely lled. Some of the metrics have just been proposed as an indicator for good ontology quality without explicitly naming quality criteria (e.g. in [ 8 ]). In consequence, there is a lot of room for further research on ontology metrics. The following usage scenarios in research and practice are seen for the OntoMetrics platform: 1. Empirical validation of proposed correlations between metrics and quality criteria regarding strength and signi cance. 2. Determination of in uences like the ontology usage context on these correlations. 3. Determination of of best practise metric pro les and values for certain domains and usage contexts. 4. Analysis of Domain Speci c Languages (DSL) and Models formulated in these languages if an OWL representation is available.3 5. Practical ontology quality assessment by calculating validated metrics. 6. Practical ontology quality assessment by monitoring anomalies in calculated metric values. 7. Proposal and validation of new metrics and their application by providing them on OntoMetrics. 8. Internal collection of evaluated ontologies for later calculation and validation of new metrics or theories regarding ontologies.

OntoMetrics supports these usage scenarios by providing easy, reliable and repeatable access to metric calculation. The XML-Representation of the calculation can be used for further automated processing of the results.Additionally, the wiki gives orientation regarding the application of already implemented metrics and also by providing room for the discussion of new ideas. Researchers are invited to contribute to the platform with their own proposals of quality metrics. This can be done by presenting a metric proposal for discussion in the wiki or by providing an implementation that can be included in the platform and thus would be available for validation and use on a broad scale.

4 Ontometrics-Usage Examples

In the following, two examples are provided that illustrate the usage scenarios presented in section 3. 4.1 Validation of Ontology Metrics for Ontology Design Patterns (ODP) The validation of of ontology metrics for Ontology Design Patterns (ODP) in [ 15 ] is an example for point 1 of the OntoMetrics usage scenarios in section 3. 3 For example Archimate models from the Enterprise Architecture Domain can be transformed to OWL using the toolset provided by the Timbus project: https: //opensourceprojects.eu/p/timbus/context-model/converters/wiki/Home/ + [ 8 ] y c n e i c E . p m o C s s e n e s i c n o

C

Ontology design patterns (ODP) have been proposed as encodings of best practices supporting ontology construction by facilitating reuse of proven solution principles. Di erent kinds of ODP have been proposed, like logical, transformation or content ODP, which represent di erent aspects of best practices (see [ 19, 20, 21 ]). This work focuses speci cally on Content ODP and on investigating the transferability of ontology quality metrics to Content ODP. The long term objective is to create an instrument for quality assurance of ODP. After a pre-selection of Ontology Metrics based on their general applicability to Content ODP, their correlation to Ontology Quality Criteria is investigated with respect to: { Clarity: Recognition of all concepts, relationships, and their correspondences. { Understandability: Comprehension of all concepts, relationships, their correspondences, and their meaning. { Adaptability to a given use case (The users got the task to adapt the respective pattern prior to evaluation). { Reusability: for example as a part of a larger pattern.

In an experiment (27 participants), nine ODP have been evaluated regarding these criteria. For the experiment, the participants have been provided with the ontology graph, the labels of used semantic relations, and the ontology annotations (Recognition annotations) on paper. In consequence, the role of attributes and attribute based metrics could not be evaluated in the experiments. Furthermore, the participants had to design and draw an ontology containing the evaluated ODP. This was done in order to foster the examination of ODP by the participants. The resulting ontologies have not been assessed. The ODP evaluation according to the four criteria (Clarity, Understandability, Adaptability, and Reusability) was based on a Likert-scale containing the values 1 (very good), 2 (good), 3 (satisfactory), 4 (fair), and 5 (unsatisfactory). The rating was done by the participants according to their perception. Figure 2 shows the average rating of each ODP with regard to the criteria. Afterwards, correlations between experiment-based evaluation results and calculated Ontology Metrics have been evaluated. This was based on the Pearson correlation. Two aspects have been evaluated: (1) the signi cance of the correlation and (2) the strength of the correlation. A correlation had been considered signi cant if the error probability (based on the Student distribution) was below 5%. For the correlation strength, a Pearson coe cient jrj 0:5 had been set as the threshold. Table 3 shows the results for those metrics that are available on OntoMetrics. Values matching the thresholds are marked green. Based on this assessment, metrics that seem to be appropriate for Content ODP evaluation are marked green as well.

At the time of the experiments, OntoMetrics was not available. Metrics have been calculated manually what increased e ort and presented a source for possible errors. On the base of OntoMetrics that also provides metrics that have not been used in the study at hand [ 15 ], an assessment of the applicability of these additional metrics to Content ODP would pose little e ort. The identi ed metrics could be used for (semi-)automated quality assurance of Content ODP. 4.2 Analysis of Enterprise Architecture Languages and Models The analysis of Enterprise Architecture (EA) languages and models is an example for the application of OntoMetrics to Domain Speci c Languages (DSL) as suggested in point 4 of the usage scenarios in section 3. Ontology evaluation can be used to assess the conceptual part of DSL speci cations. Quality Criteria like Coverage, Conciseness, Clarity, and Reusability can be evaluated. Antunes et al. present in [ 22 ] an ontology-based approach for Enterprise Architecture Analysis. Their approach comes with an ontology4 that represents the ArchiMate R language constructs and their relationships. ArchiMate R is a standard language for Enterprise Architecture Modelling that is maintained by the OpenGroup5. Furthermore, Antunes et al. provide a tool to convert ArchiMate models that are created with the Archi toolset6 to an OWL representation.

In consequence, OntoMetrics can be used to evaluate the provided ArchiMate R conceptualization as well as models that have been created with the Archi toolset. So far, there is not enough empirical data available in order to give a comprehensive assessment. However, some exemplary statements can be drawn from rst analysis results. For example, the ontology representing ArchiMate R has an Average Number of Paths of 1.0. This means that there exists in average 1 path from the root to each of the classes (language concepts) within the inheritance tree. In consequence, no multiple inheritance is used. Thus, a `good'Readability of the language concepts is assumed. Including model instances into the evaluation, for example the importance of certain language concepts can be assessed or vice versa the coverage of a model. Figure 3 shows the Class Importance of ArchiMate R concepts based on the Archisurance model that is based on a case study by the OpenGroup. The gure only contains classes that have more than 1 per cent of the total number of individuals in the model. These are 20 out of 55 concepts (Total Number of Classes) . The most important classes are BusinessActor and BusinessObject. Thus, a conclusion might be to provide specializations of these classes in order to improve Clarity since the semantics may be to unspeci c at present. However, this conclusion should only be drawn when a broad base of models is available for evaluation. Furthermore, it could be determined which ArchiMate R concepts are typically used in certain contexts an which ones are super uous. Vice versa, models for a certain context can be evaluated regarding the usage of typical concepts of that context.

5 Conclusion and Outlook

At its present state, OntoMetrics is a lightweight, handy tool for comparable, metric based ontology evaluation. In combination with the Quality-Metrics-andCriteria-Matrix (table 2) it can be used to assess ontologies using already accepted and suggested metrics. The main focus lies on the Conceptual Scope.

4 http://timbus.teco.edu/ontologies/DIO.owl

5 http://pubs.opengroup.org/architecture/archimate2-doc/toc.html

6 http://archimatetool.com/

However, Domain Scope and Application Scope are partially covered as described in section 2. The wiki provides information on the theoretical background of the calculated metrics. For the future development of the OntoMetrics platform, three main directions are planned: 1. Enhance the knowledgebase of OntoMetrics

The information in the wiki regarding the proposed metrics is planned to be extended by additional sources, use case descriptions and systematizations. 2. Provide additional quality metrics

New metrics of the introduced types can be added with little e ort via de ned programming interfaces. Furthermore, a support of data driven ontology metrics calculation is planned. 3. Extend the base functionality of OntoMetrics

In order to allow tool integration of OntoMetrics, a Web-Service that provides metric calculation functionality is planned.

References

1. Noy , N.F. , McGuinness , D.L. , et al.: Ontology development 101: A guide to creating your rst ontology ( 2001 )

2. Ding , L. , Finin , T. , Joshi , A. , Pan , R. , Cost , R.S. , Peng , Y. , Reddivari , P. , Doshi , V.C. , Sachs , J.: Swoogle: A semantic web search and metadata engine . In: Proc. 13th ACM Conf. on Information and Knowledge Management . Volume 304 . ( 2004 )

3. Alani , H. , Brewster , C. , Shadbolt , N.: Ranking ontologies with AKTiveRank . In: International Semantic Web Conference. ( 2006 ) 1 { 15

4. Burton-Jones , A. , Storey , V.C. , Sugumaran , V. , Ahluwalia , P.: A semiotic metrics suite for assessing the quality of ontologies . Data & Knowledge Engineering 55 ( 1 ) ( 2005 ) 84 { 102

5. Gangemi , A. , Catenacci , C. , Ciaramita , M. , Lehmann , J.: A theoretical framework for ontology evaluation and validation . In: SWAP . Volume 166 . ( 2005 )

6. Guarino , N. , Welty , C.A. : An overview of OntoClean . In: Handbook on ontologies. Springer ( 2009 ) 201 { 220

7. Lozano-Tello , A. , Gomez-Perez , A. : Ontometric: A method to choose the appropriate ontology . Journal of database management 2 ( 15 ) ( 2004 ) 1 { 18

8. Tartir , S. , Arpinar , I.B. , Moore , M. , Sheth , A.P. , Aleman-Meza , B. : OntoQA: Metric-based ontology quality analysis . In: IEEE Workshop on Knowledge Acquisition from Distributed , Autonomous, Semantically Heterogeneous Data and Knowledge Sources . Volume 9 . ( 2005 )

9. Poveda-Villalon , M. , Suarez-Figueroa , M.C. , Gomez-Perez , A. : Validating ontologies with oops! In: International Conference on Knowledge Engineering and Knowledge Management. ( 2012 ) 267 { 281

10. Gavrilova , T. , Gorovoy , V. , Bolotnikova , E.: New Ergonomic Metrics for Educational Ontology Design and Evaluation . In: SoMeT. ( 2012 ) 361 { 378

11. Fernandez , M. , Overbeeke , C. , Sabou , M. , Motta , E.: What makes a good ontology? A case-study in ne-grained knowledge reuse . In: Asian Semantic Web Conference . ( 2009 ) 61 { 75

12. Pak , J. , Zhou , L. : A Framework for Ontology Evaluation . In Sharman, R., Rao , H. , Raghu , T., eds.: Exploring the Grand Challenges for Next Generation E-Business . Volume 52 of Lecture Notes in Business Information Processing . Springer Berlin Heidelberg ( 2011 ) 10 { 18

13. Brewster , C. , Alani , H. , Dasmahapatra , S. , Wilks , Y. : Data driven ontology evaluation . ( 2004 )

14. Hlomani , H. , Stacey , D. : Approaches, methods, metrics, measures, and subjectivity in ontology evaluation: A survey . Semantic Web Journal ( 2014 ) 1{ 5

15. Lantow , B. , Sandkuhl , K. : An Analysis of Applicability using Quality Metrics for Ontologies on Ontology Design Patterns . Intelligent Systems in Accounting, Finance and Management 22 ( 1 ) ( 2015 ) 81 { 99

16. ECAI Workshop on Ontology Learning and Population, Valencia, Spain. ( 2004 )

17. Aldo Gangemi , Carola Catenacci, Massimiliano Ciaramita, Jos Lehmann: Ontology Evaluation and Validation: An integrated formal nmodel for the quality diagnostc task

18. Yao , H. , Orme , A.M. , Etzkorn , L. : Cohesion metrics for ontology design and application . Journal of Computer science 1(1) ( 2005 ) 107

19. Blomqvist , E. , Sandkuhl , K. : Patterns in Ontology Engineering: Classi cation of Ontology Patterns . In: ICEIS (3) . ( 2005 ) 413 { 416

20. Blomqvist , E. , Gangemi , A. , Presutti , V. : Experiments on pattern-based ontology design . In: K-CAP. ( 2009 ) 41 { 48

21. Presutti , V. , Gangemi , A. : Content Ontology Design Patterns as Practical Building Blocks for Web Ontologies . In: ER '08: Proceedings of the 27th International Conference on Conceptual Modeling . ( 2008 ) 128 { 141

22. Antunes , G. , Bakhshandeh , M. , Mayer , R. , Borbinha , J. , Caetano , A. : Using Ontologies for Enterprise Architecture Analysis . In: 2013 17th IEEE International Enterprise Distributed Object Computing Conference Workshops (EDOCW) . 361 { 368