As ontologies are the backbone of the Semantic Web, they attract much attention from researchers and engineers in many domains. This results in an increasing number of ontologies and semantic web applications. The number and complexity of such ontologies makes it hard for developers of ontologies and tools to decide which ontologies to use and reuse. To simplify the problem, a modularization algorithm can be used to partition ontologies into sets of modules. In order to evaluate the quality of modularization, we propose a new evaluation metric that quantifies the goodness of ontology modularization. In particular, we investigate the ontology module homogeneity, which assesses module cohesion, and the ontology module heterogeneity, which appraises module coupling. The experimental results demonstrate that the proposed metric is effective.
Ontologies represent the essential technology that enables and facilitates
interoperability at the semantic level, providing a formal conceptualization of the
data which can be shared, reused, and aligned. Therefore, ontologies have been
attracting much attention of researchers and engineers in many fields such as
knowledge management [
With this large number of existing ontologies it becomes obvious that use/reuse
of an existing ontology is preferable to building the ontology from scratch.
However, large-scale ontologies are difficult to reuse [
module integration are an effective way to build ontologies. Several tools for
ontology modularization have been proposed and users employ these tools to
prepare ontology-based systems [
The quality of an ontology (module) can be defined as the degree of
conformance to functional and non-functional requirements [
Therefore, in this paper, we propose a new ontology modularization
evaluation metric that can be used to assess the goodness of ontology modules. In
particular, we propose the module homogeneity (M OHO) as a metric of the
internal characteristics of module concepts, and the module heterogeneity (M OHE)
as an assessment of interdependency between ontology modules. The M OHO
metric will exploit semantic and structural characteristics of module concepts,
including concept names, the distances between concepts and the number of trees
generated from modularization. On the other hand, the M OHE metric will cover
different aspects of relations between modules. According to our investigation,
we discover that the proposed individual criteria accurately determine the
cohesion of a module as well as the coupling between modules. Firstly, the size of
module element is directly effected on cohesion, but we think it depends on
individual classes, so we count the lexical level of each module to a measure for the
module homogeneity. Secondly, the more connected concepts within the same
module show how the high module homogeneity, therefore we count the number
of connected concepts and namely the number of trees, since each tree shows
one one connected component of concepts. Thirdly, the depth of concepts is an
important aspect during the evaluation of the module homogeneity. The more
the concepts within the same module are close to each other, the more module
is cohesive. Lastly, one important issue when we partition one ontology is how
many edges will be cut. Therefore, in the current development, we consider these
individual and other evaluation metrics. We carried out a set of experiments to
validate the proposed metric and the experimental results show that is valid and
effective within the context of our partitioning tool [
Even if the main objective of the proposed metric is to evaluate the ontology modularization, it could be useful and can be used in many different scenarios. For example, it can be useful to ontology assessment to provide some insight to ontology developers to help them design ontologies, improve ontology quality, anticipate and reduce future maintenance requirements, as well as help ontology users choose the ontologies that best meet their needs.
The rest of the paper is organized as follows: we present a set of definitions and preliminaries used throughout the paper and a set of related work in Section 2. The proposed metric will be introduced in Section 3. Section 4 reports on the experimental results. Section 5 concludes the work. 2
2.1
This section is devoted first to present basic definitions that will be used throughout the paper, and then to present related work.
Let O be an ontology comprising of a set of axioms (classes, properties, relationships,) and P(O) represent the signature of the ontology constituting a set of entity names occurring in the axioms of O, i.e., its vocabulary [19]. In our implementation, each ontology is parsed and inferred by Apache Jena5 and then the corresponding concept graph is drawn by mapping the inferred result. We define a concept graph G = (C, R, L) as a labeled directed graph, where C = {c1, c2, ..., cn} is a finite set of nodes presenting the concepts of the ontology, i.e. its classes and data properties. R = {r1, r2, ..., rm} stands for a finite set of directed edges showing various relationships between concepts in the ontology O, such that rk ∈ R represents a directed relation between two adjacent concepts ci, cj ∈ C. L is a finite set of labels of graph nodes defining the properties of each concept, such as the names of concepts.
In general, ontology modularization covers the problem of identifying a
fragment or a set of fragments of an ontology. The process of identifying a fragment
of an ontology given a user input (request) is called ontology module
extraction [
The ontology modularization process (partitioning) can be defined as follows: given an ontology O represented as a concept graph G, the next step is to partition concepts, C, of each graph into a set of modules M1, M2, ..., Mk such that the cohesion (module homogeneity) of concepts in one module should be high, while the coupling (module heterogeneity) between any two modules is low. 2.2
Ontology evaluation is a crucial task in different domains. Therefore, several
approaches and metrics have been proposed and developed. Ma et.al [
of maximum concept subsumption of leaf concept. These metrics have been
integrated and implemented into an ontology measurement tool [
Metrics to measure the complexity of ontologies, emphasising on the problem of increasing the complexity of maintenance and management as ontologies evolve have been proposed in [24]. The defined metrics are composed of primitive metrics and complexity metrics. Primitive metrics assess the basic level of information, including the total number of classes, relations, and paths. While, complex metrics quantify the average relations per concept, the average paths per concept, and the ratio of maximum path length to average path length of the ontology. These metrics examine the concept aggregation and coherence of an ontology. Orme et.al. [18] focused on the number of externally defined referenced concepts. They proposed a set of coupling metrics for ontology-based systems represented in OWL, such as the number of external classes, the reference to external classes, and referenced includes.
A formal definition of some helpful metrics is provided to analyze the
coupling between classes in an ontology [
OntoQA [22] is a tool that implements a number of metrics such as richness, population, and cohesion. It proposes some schema metrics to measure the richness of schema relationships, attributes and schema inheritance. These metrics are focused on evaluating the ontology in general. Other proposed categories are class richness, average population, cohesion, the importance of a class, fullness of a class, class inheritance and class relationship richness, connectivity and readability. This work described two similar but not equal metrics. Class Relationship Richness is defined as the number of relationships that are being used by instances that belong to the class. On the other hand, the connectivity of a class is defined as the number of instances of other classes that are connected to instances of the selected class. The main differences are that these metrics take into account the instances belonging to the class instead of relations declared in the class. 3
To assess the goodness of the ontology modularization process, we introduce a
new evaluation metric. To decide on the goodness of an ontology modularization
approach, we need a set of metrics (evaluation criteria) that can be used to
evaluate the cohesion and the coupling of the partitioning result. To this end,
we propose the module homogeneity (M OHO )metric as a criterion for the
modularization coherence, and the module heterogeneity (M OHE) metric as
a criterion for the modularization coupling. In the following, we first describe
how to prepare ontology modules using our ontology partitioning tool [
There have been two major ways of modularizing ontologies: ontology
partitioning and ontology module extraction. In partitioning-based approaches, the
original ontology is usually divided into a number of modules, which are not
necessarily disjointed. To cope with the ontology partitioning problem, we proposed
and developed an ontology analysis and partitioning tool, called OAPT [
Each input/analyzed ontology is parsed and the corresponding concept graph is derived. After that, the approach starts to determine which nodes of the concept graph shall be selected as cluster (module) heads (CHs). To this end, we propose a ranking function that quantifies the importance of nodes inside the concept graph. The next step is to select which concepts represent the cluster heads, CH. If simply the nodes with the highest score are selected as the cluster heads, the distribution of these nodes within the concept graph would be disregarded. To avoid this problem, the distance between two cluster heads is measured, and among the highest score nodes, those with a specified distance from each other are selected as the cluster heads.
The final component of the SeeCOnt approach is to finalize the partitioning process. Once having decided upon the set of cluster heads (CHs), the SeeCOnt approach creates one cluster for each cluster head. Then, it places direct children in the corresponding cluster and finally, for remaining nodes, a membership function is used to determine the appropriate cluster of each node.
Similar to SeeCont, various modularization methods have been developed and used. Because they are based on different assumptions and techniques, modules created by these methods are different. Consequently, it is difficult to compare them. Thus, useful measurements to evaluate modules are required. 3.2
In software engineering, there are many metrics that evaluate software modules.
The most common two are coupling and cohesion, where cohesion represents the
functionality of module and coupling represents the interdependency between
pairs of modules [
SM H(Mi) = 1 m
X SemSim(CH, cj ) m − 1 j=2 (1) where SemSim is the semantic similarity between the cluster head and the other concepts of the module, assuming that the cluster head represents the central of the module. Since the name of concepts are always represented by nouns, in this implementation, we consider the semantic relations between nouns. In general, we use the four common semantic relations: hyponym/hypernym (is-a), part meronym/part holonym (part-of), member meronym/member holonym (member-of) and substance meronym/substance holonym (substance-of) between module concepts. – SrMH metric. We also evaluate the structural module homogeneity (SrM H) by measuring the distance between the cluster head and the other concepts within the module. It should be noted that this distance should be small where dist(CH, Ci) is the minimal path between CH and cj. – AvgDepth. It is also important to know what is the level of concepts in each module; checking how well module concepts are distributed. It can be used to measure the degree to which the semantic knowledge of an ontology to be measured is organized. Therefore, we propose the average depth of all concepts namely, AvgDepth which it shows in the following formula.
AvgDepth(Mi) =
Pjm=1 depth(cj) m where depth(Cj) is the path length of the concept cj to the root of the concept graph. – One more metric that can be used to validate the homogeneity of a module is the number of trees in each module, called NTree. It is a key issue that how many concepts are related to each other, or, are they separate from each other. So, we define one criterion to measure how many connected concepts exist in each module. If it is low, it shows the more cohesion. To compute this metric, we propose the following formula: and ideally should be 0. For that, we use the following formula to compute SrM H
SrM H(Mi) =
1 Xm 1 m − 1 j=2 dist(CH, cj) N T ree(Mi) = num of root concepts |Mi| where |Mi| is the total number of concepts in a module Mi.
By defining the individual metrics, we can define the combined module homogeneity metric (M OHO) as follows:
M OHO(Mi) = w1 × SM H(Mi) + w2 × SrM H(Mi) + w3 × AvgDepth(Mi) + w4 × N T ree(Mi) (5) 4 where wis are the weights to quantify each individual metric and Pi=1 wi = 1. Once computing the module homogeneity for each module, we can evaluate the homogeneity for the modularization process by defining the homogeneity of a modularization technique as follows:
MOHO(M) = 1 k
X |Mi| × MOHO(Mi) |O| i=1 where k is the number of modules, |Mi| is the number of concepts in module i, and |O| is the total number of concepts in the original ontology. (2) (3) (4) (6) Module Heterogeneity. To assess the module heterogeneity (M OHE), we quantify the interdependency between different modules. Therefore, we consider the following metrics: – Relative size (RS). The heterogeneity metric evaluates the coupling between ontology modules. The higher the coupling between modules the higher the relatedness between them. It is a desirable property to keep ontology modules loosely coupled in order to be independently used. Given an ontology O modularized into a set of modules {M1, M2, M3, .., Mk}. The first metric can be used to constitute M OHE metric is the relative size. By the relative size, we ensure that the ontology concepts are ”normally” distributed among the modules. To determine the relative size of modularization, we use the following formula: (7) (8) RS(M) =
1 kX−1 Xk ||Mi| − |Mj ||
k × |O| i=1 j=i+1
where |Mi| and |O| are the module and ontology sizes, respectively.
– DetachRel. Even if the relative size of concepts within ontology modules
can be consider as a good indicator of how these concepts are distributed
among the modules, however, we need another metric that quantifies the
relationships between modules. If there is a concept cj ∈ Mi such that all
the properties (relations) of the concepts are belonging to the same module,
then the module has a high cohesion and low coupling. However, in the
contrast, if concept cl ∈ Mj has some connected concepts in the other
modules, i.e. property(cl) ∈ ci ∈ Mg, g 6= j, it shows high coupling for
cl ∈ Mj . Therefore, we propose the deployment of the number of detached
relations between concepts (properties) as one of the coupling measure [
DetachRel(M) = 1 kX−1 Xk |rMi ∩ rMj | k i=1 j=i+1 |rMi ∪ rMj | where |rMi ∩ rMj | is the number of relations exist in both modules Mi and Mj . 4
We conducted a set of experiments to demonstrate that the proposed evaluation metrics are valid and effective within the context of our partitioning tool. To perform such evaluation, we collected a set of ontologies from the BioPortal repository6 and the ontology search using some generic keywords. The collected ontologies represent different domains and have different characteristics, as shown in Table 1. 6 http://bioportal.bioontology.org/
Ontology
GFO
BCO mouse anatomy nci anatomy
ENVO OBOOE
PW Delegation
Koala Dolce lite HarryPotter Conference Monetary Philosurfical To prepare ontology modules for evaluation, we apply our modularization approach to partition the ontologies listed in Table 1. The module homogeneity (M OHO) as well as the module heterogeneity (M OHE) have been recorded and reported.
Module Homogeneity. We carried out this set of experiments to evaluate the quality of module homogeneity metrics. We applied our modularization approach to the set of ontologies listed in Table 1. The optimal number of modules for each ontology generated using our partitioning tool has been identified and listed into the table, too. First, we get the results for each individual M OHO metric, such as SM H and the average module homogeneity has been computed using Eq.5. Results are reported in Figs. 2& 3. The results show, in general, that most of the tested ontologies have module homogeneity values greater than 0.2, which can be considered as a high value due to the involvement of several aspects to compute the module homogeneity. Secondly, ontologies partitioned only to one module have the higher M OHO values. Furthermore, we observe that ontologies belonging to the bio-domain have zero value for SM H, since we are currently using the WordNet dictionary, which is more generic dictionary and it fails semantics relationships between concepts from the bio-domain. Module Heterogeneity. In this set of experiments, we validate the module heterogeneity metric. We applied our modularization approach to the set of ontologies in Table 1 and computed both module heterogeneity metrics: relative size (RS) and DetachRel (Dat). Results are reported in Fig. 4. In general and as expected, ontologies partitioned only to one module have M OHE values of 0 for both metrics. The figure also shows that RS indicates that the ontology concepts are nearly equally distributed between ontology modules, except BCO,
Fig. 3: Combined MOHO.
Fig. 5: Comparing MOHO and MOHE.
Monetary, and Philosurfical ontologies. It also indicates that this modularization approach produces the minimum overlap (Detch) between ontology modules, i.e. less coupling, except the Philosurfical ontology.
Furthermore, we study the relationships between M OHO and M OHE metrics and the size of ontology (listed in Table 1). Results are reported in Fig. 5. In general, for small size ontologies, the modularization has higher M OHO (more coherent modules) and lower M OHE (less coupled modules) except for the Monetary ontology. As the ontology size increases the M OHO and M OHE values decrease and are nearly constant, respectively. 5
In this paper, we introduced a new ontology modularization evaluation metric. We offer the module homogeneity metric to assess the module cohesion, while we propose the module heterogeneity metric to quantify the coupling between modules. To validate, the proposed metrics, we carried out a set of experiments utilizing ontologies from different domains. The experimental results demonstrate that the metric is effective. In the future, we plan to exploit different and domain specific dictionaries during the evaluation of the semantic module homogeneity. Furthermore, we plan to investigate relationships between the evaluation metrics and different parameters of the modularization approach.
This work is partly funded by DFG in the INFRA1 project of CRC AquaDiva. 18. A. M. Orme, H. Yao, and L. H. Etzkorn. Coupling metrics for ontology-based systems. IEEE Software, 23(2):102–108, 2006. 19. J. Pathak, T. M. Johnson, and C. G. Chute. Survey of modular ontology techniques and their applications in the biomedical domain. Integrated Computer-Aided Engineering, 16(3):225–242, 2009. 20. Y. Qu and G. Cheng. Falcons concept search: A practical search engine for web ontologies. IEEE Transactions on Systems, Man, and Cybernetics, 41(4):810–816, 2011. 21. A. A. Romero, M. Kaminski, B. C. Grau, and I. Horrocks. Ontology module extraction via datalog reasoning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 1410–1416, 2015. 22. S. Tartir and I. B. Arpinar. Ontology evaluation and ranking using OntoQA. In
International Conference on Semantic Computing, pages 185–192, 2007. 23. C. D. Vescovo, D. Gessler, P. Klinov, B. Parsia, U. Sattler, T. Schneider, and A. Winget. Decomposition and modular structure of bioportal ontologies. In 10th International Semantic Web Conference ISWC 2011, pages 130–145, 2011. 24. Z. YANG, D. Zhang, and C. YE. Evaluation metrics for ontology complexity and evolution analysis. In ICEBE’06, pages 162–170, 2006.