=Paper=
{{Paper
|id=Vol-1605/paper4
|storemode=property
|title=A New Metric To Evaluate Ontology Modularization
|pdfUrl=https://ceur-ws.org/Vol-1605/paper4.pdf
|volume=Vol-1605
|authors=Alsayed Algergawy,Samira Babalou,Birgitta König-Ries
|dblpUrl=https://dblp.org/rec/conf/esws/AlgergawyBK16
}}
==A New Metric To Evaluate Ontology Modularization==
https://ceur-ws.org/Vol-1605/paper4.pdf
A New Metric To Evaluate Ontology
Modularization
Alsayed Algergawy1,2 , Samira Babalou3 , and Birgitta König-Ries1
1
Institute of Computer Science, Friedrich Schiller University of Jena, Germany
2
Department of Computer Engineering, Tanta University, Egypt
3
Department of Computer Engineering, University of Science and Culture, Iran
{firstname.lastname@uni-jena.de}
Abstract. As ontologies are the backbone of the Semantic Web, they
attract much attention from researchers and engineers in many domains.
This results in an increasing number of ontologies and semantic web
applications. The number and complexity of such ontologies makes it
hard for developers of ontologies and tools to decide which ontologies to
use and reuse. To simplify the problem, a modularization algorithm can
be used to partition ontologies into sets of modules. In order to evalu-
ate the quality of modularization, we propose a new evaluation metric
that quantifies the goodness of ontology modularization. In particular,
we investigate the ontology module homogeneity, which assesses module
cohesion, and the ontology module heterogeneity, which appraises mod-
ule coupling. The experimental results demonstrate that the proposed
metric is effective.
Keywords: Semantic Web, Ontology, Modularization, Evaluation met-
rics
1 Introduction
Ontologies represent the essential technology that enables and facilitates inter-
operability at the semantic level, providing a formal conceptualization of the
data which can be shared, reused, and aligned. Therefore, ontologies have been
attracting much attention of researchers and engineers in many fields such as
knowledge management [11,12], semantic search [20], etc. As a result, there is a
myriad of developed ontologies. For example, the National Center for Biomedi-
cal Ontology (NCBO) BioPortal 4 contains more than five hundred biomedical
ontologies and controlled vocabularies [7].
With this large number of existing ontologies it becomes obvious that use/reuse
of an existing ontology is preferable to building the ontology from scratch. How-
ever, large-scale ontologies are difficult to reuse [16]. Therefore, modularizing
ontologies and reusing only part(s) of the ontologies appropriate for a given
context are necessary approaches. Techniques for ontology modularization and
4
http://bioportal.bioontology.org/
module integration are an effective way to build ontologies. Several tools for
ontology modularization have been proposed and users employ these tools to
prepare ontology-based systems [4,6,19,23].
The quality of an ontology (module) can be defined as the degree of con-
formance to functional and non-functional requirements [7]. This degree should
be measurable. Current studies of the evaluation of modularization approaches
focus on modularization algorithms and the evaluation of the taxonomical struc-
ture of a created module [16]. According to [8], ontology evaluation determines
the quality and adequacy of an ontology for reuse in a specific context for a
specific goal. The evaluation of ontology is crucial in many fields, however, we
observe that the high cohesion and low coupling are the measures for the ontol-
ogy evaluation, but they do not have a unique or specific definition.
Therefore, in this paper, we propose a new ontology modularization evalua-
tion metric that can be used to assess the goodness of ontology modules. In par-
ticular, we propose the module homogeneity (M OHO) as a metric of the inter-
nal characteristics of module concepts, and the module heterogeneity (M OHE)
as an assessment of interdependency between ontology modules. The M OHO
metric will exploit semantic and structural characteristics of module concepts,
including concept names, the distances between concepts and the number of trees
generated from modularization. On the other hand, the M OHE metric will cover
different aspects of relations between modules. According to our investigation,
we discover that the proposed individual criteria accurately determine the co-
hesion of a module as well as the coupling between modules. Firstly, the size of
module element is directly effected on cohesion, but we think it depends on indi-
vidual classes, so we count the lexical level of each module to a measure for the
module homogeneity. Secondly, the more connected concepts within the same
module show how the high module homogeneity, therefore we count the number
of connected concepts and namely the number of trees, since each tree shows
one one connected component of concepts. Thirdly, the depth of concepts is an
important aspect during the evaluation of the module homogeneity. The more
the concepts within the same module are close to each other, the more module
is cohesive. Lastly, one important issue when we partition one ontology is how
many edges will be cut. Therefore, in the current development, we consider these
individual and other evaluation metrics. We carried out a set of experiments to
validate the proposed metric and the experimental results show that is valid and
effective within the context of our partitioning tool [2].
Even if the main objective of the proposed metric is to evaluate the ontology
modularization, it could be useful and can be used in many different scenarios.
For example, it can be useful to ontology assessment to provide some insight to
ontology developers to help them design ontologies, improve ontology quality,
anticipate and reduce future maintenance requirements, as well as help ontology
users choose the ontologies that best meet their needs.
The rest of the paper is organized as follows: we present a set of definitions
and preliminaries used throughout the paper and a set of related work in Section
2. The proposed metric will be introduced in Section 3. Section 4 reports on the
experimental results. Section 5 concludes the work.
2 Background
This section is devoted first to present basic definitions that will be used through-
out the paper, and then to present related work.
2.1 Preliminaries
Let O be an ontology P comprising of a set of axioms (classes, properties, re-
lationships,) and (O) represent the signature of the ontology constituting a
set of entity names occurring in the axioms of O, i.e., its vocabulary [19]. In
our implementation, each ontology is parsed and inferred by Apache Jena5 and
then the corresponding concept graph is drawn by mapping the inferred result.
We define a concept graph G = (C, R, L) as a labeled directed graph, where
C = {c1 , c2 , ..., cn } is a finite set of nodes presenting the concepts of the ontol-
ogy, i.e. its classes and data properties. R = {r1 , r2 , ..., rm } stands for a finite set
of directed edges showing various relationships between concepts in the ontology
O, such that rk ∈ R represents a directed relation between two adjacent con-
cepts ci , cj ∈ C. L is a finite set of labels of graph nodes defining the properties
of each concept, such as the names of concepts.
In general, ontology modularization covers the problem of identifying a frag-
ment or a set of fragments of an ontology. The process of identifying a fragment
of an ontology given a user input (request) is called ontology module extrac-
tion [10,21], while the process that partitions the ontology into a set of frag-
ments is called ontology partitioning
P P [2,3]. We define an ontology module Mi (O)
of an ontology O, with (Mi ) ⊆ (O), such that Mi (O) contains the same
information about the set of axioms of Mi as O. In this paper, we consider the
problem of evaluating the goodness of ontology partitioning techniques.
The ontology modularization process (partitioning) can be defined as follows:
given an ontology O represented as a concept graph G, the next step is to parti-
tion concepts, C, of each graph into a set of modules M1 , M2 , ..., Mk such that
the cohesion (module homogeneity) of concepts in one module should be high,
while the coupling (module heterogeneity) between any two modules is low.
2.2 Related work
Ontology evaluation is a crucial task in different domains. Therefore, several ap-
proaches and metrics have been proposed and developed. Ma et.al [13] proposed
a set of ontology metrics to measure ontology cohesion. These metrics are se-
lected as the criteria of ontology measurement for ontology based systems. They
include the number of ontology partitions, the number of minimally inconsistent
subsets, the average impact of intra-module relationships, and the average depth
5
https://jena.apache.org/
of maximum concept subsumption of leaf concept. These metrics have been in-
tegrated and implemented into an ontology measurement tool [14]. To evaluate
the ontology modules, the approach in [17] introduces cohesion and coupling
metrics based on the theory of software metrics.
Metrics to measure the complexity of ontologies, emphasising on the prob-
lem of increasing the complexity of maintenance and management as ontologies
evolve have been proposed in [24]. The defined metrics are composed of prim-
itive metrics and complexity metrics. Primitive metrics assess the basic level of
information, including the total number of classes, relations, and paths. While,
complex metrics quantify the average relations per concept, the average paths
per concept, and the ratio of maximum path length to average path length of the
ontology. These metrics examine the concept aggregation and coherence of an
ontology. Orme et.al. [18] focused on the number of externally defined referenced
concepts. They proposed a set of coupling metrics for ontology-based systems
represented in OWL, such as the number of external classes, the reference to
external classes, and referenced includes.
A formal definition of some helpful metrics is provided to analyze the cou-
pling between classes in an ontology [9]. These metrics include coupling between
entities (CBE) for ontologies with two possibilities. The CBE-out metric repre-
sents the coupling where the class belongs to the domain of the property, while
CBE-in represents the coupling where the class belongs to the range of the prop-
erty. Another evaluation framework has been proposed in [16], in which users can
analyze and compare modularization tools. To design a new evaluation frame-
work that enables the comparison of modularization tools, three perspectives
of tool evaluation dimensions are proposed: modularization performance, data
performance, and usability.
OntoQA [22] is a tool that implements a number of metrics such as richness,
population, and cohesion. It proposes some schema metrics to measure the rich-
ness of schema relationships, attributes and schema inheritance. These metrics
are focused on evaluating the ontology in general. Other proposed categories
are class richness, average population, cohesion, the importance of a class, full-
ness of a class, class inheritance and class relationship richness, connectivity and
readability. This work described two similar but not equal metrics. Class Rela-
tionship Richness is defined as the number of relationships that are being used
by instances that belong to the class. On the other hand, the connectivity of a
class is defined as the number of instances of other classes that are connected to
instances of the selected class. The main differences are that these metrics take
into account the instances belonging to the class instead of relations declared in
the class.
3 Proposed Metric
To assess the goodness of the ontology modularization process, we introduce a
new evaluation metric. To decide on the goodness of an ontology modularization
approach, we need a set of metrics (evaluation criteria) that can be used to
evaluate the cohesion and the coupling of the partitioning result. To this end,
Fig. 1: OAPT partitioning framework.
we propose the module homogeneity (M OHO )metric as a criterion for the
modularization coherence, and the module heterogeneity (M OHE) metric as
a criterion for the modularization coupling. In the following, we first describe
how to prepare ontology modules using our ontology partitioning tool [2] and
then how to assess the goodness of ontology modularization using the proposed
evaluation metrics.
3.1 Ontology modularization
There have been two major ways of modularizing ontologies: ontology parti-
tioning and ontology module extraction. In partitioning-based approaches, the
original ontology is usually divided into a number of modules, which are not nec-
essarily disjointed. To cope with the ontology partitioning problem, we proposed
and developed an ontology analysis and partitioning tool, called OAPT [2], as
shown in Fig.1. First, the input ontology is investigated and a set of ontology
features is collected in order to guide the user if this ontology is worth to be
modularized or not. After that the analyzed ontology will be partitioned using
the specified modularization algorithm, called SeeCOnt [1]. To make this paper
self-contained, we present a short description of the SeeCOnt approach. The
approach has three main components: preprocessing, ranking, and clustering.
Each input/analyzed ontology is parsed and the corresponding concept graph
is derived. After that, the approach starts to determine which nodes of the con-
cept graph shall be selected as cluster (module) heads (CHs). To this end, we
propose a ranking function that quantifies the importance of nodes inside the
concept graph. The next step is to select which concepts represent the cluster
heads, CH. If simply the nodes with the highest score are selected as the clus-
ter heads, the distribution of these nodes within the concept graph would be
disregarded. To avoid this problem, the distance between two cluster heads is
measured, and among the highest score nodes, those with a specified distance
from each other are selected as the cluster heads.
The final component of the SeeCOnt approach is to finalize the partitioning
process. Once having decided upon the set of cluster heads (CHs), the SeeCOnt
approach creates one cluster for each cluster head. Then, it places direct children
in the corresponding cluster and finally, for remaining nodes, a membership
function is used to determine the appropriate cluster of each node.
Similar to SeeCont, various modularization methods have been developed and
used. Because they are based on different assumptions and techniques, modules
created by these methods are different. Consequently, it is difficult to compare
them. Thus, useful measurements to evaluate modules are required.
3.2 Modularization Evaluation
In software engineering, there are many metrics that evaluate software modules.
The most common two are coupling and cohesion, where cohesion represents the
functionality of module and coupling represents the interdependency between
pairs of modules [5,17]. Similar to software module metrics, ontology module
metrics are designed to quantify ontology modules’ properties. To this end, we
propose module homogeneity (M OHO) to represent the module functionality
and module heterogeneity (M OHE) to represent the module interdependency.
Module Homogeneity. To quantify the module homogeneity we pursue to
evaluate semantic and structural characteristics of module concepts. Therefore,
we propose the M OHO metric to quantify the ontology modules homogeneity,
which include the following individual metrics:
– SMH metric. A module comprises a set of concepts that represents a spe-
cific part of the domain. This means that the module should have a high
cohesion. Semantic similarity measures can be used for different tasks such
as term disambiguation and checking ontology for consistency and coher-
ence [15]. Various lexical databases and dictionaries have been used to en-
hance the quality of these semantic metrics. We also make use of a semantic
measure to evaluate the semantic consistency of concepts within a module.
Given a module Mi = {CH, c2 , c3 , .., cm }, with m concepts, where CH is the
cluster head of the module, we use the following formula to compute the
semantic module homogeneity (SM H):
1 X
m
SM H(Mi ) = SemSim(CH, cj ) (1)
m − 1 j=2
where SemSim is the semantic similarity between the cluster head and the
other concepts of the module, assuming that the cluster head represents
the central of the module. Since the name of concepts are always repre-
sented by nouns, in this implementation, we consider the semantic rela-
tions between nouns. In general, we use the four common semantic relations:
hyponym/hypernym (is-a), part meronym/part holonym (part-of), member
meronym/member holonym (member-of) and substance meronym/substance
holonym (substance-of) between module concepts.
– SrMH metric. We also evaluate the structural module homogeneity (SrM H)
by measuring the distance between the cluster head and the other concepts
within the module. It should be noted that this distance should be small
and ideally should be 0. For that, we use the following formula to compute
SrM H
1 X
m
1
SrM H(Mi ) = (2)
m − 1 j=2 dist(CH, cj )
where dist(CH, Ci ) is the minimal path between CH and cj .
– AvgDepth. It is also important to know what is the level of concepts in
each module; checking how well module concepts are distributed. It can be
used to measure the degree to which the semantic knowledge of an ontology
to be measured is organized. Therefore, we propose the average depth of all
concepts namely, AvgDepth which it shows in the following formula.
Pm
j=1 depth(cj )
AvgDepth(Mi ) = (3)
m
where depth(Cj ) is the path length of the concept cj to the root of the
concept graph.
– One more metric that can be used to validate the homogeneity of a module
is the number of trees in each module, called NTree. It is a key issue that
how many concepts are related to each other, or, are they separate from each
other. So, we define one criterion to measure how many connected concepts
exist in each module. If it is low, it shows the more cohesion. To compute
this metric, we propose the following formula:
num of root concepts
N T ree(Mi ) = (4)
|Mi |
where |Mi | is the total number of concepts in a module Mi .
By defining the individual metrics, we can define the combined module ho-
mogeneity metric (M OHO) as follows:
M OHO(Mi ) = w1 × SM H(Mi ) + w2 × SrM H(Mi )
+ w3 × AvgDepth(Mi ) + w4 × N T ree(Mi ) (5)
P
where wi s are the weights to quantify each individual metric and 4i=1 wi = 1.
Once computing the module homogeneity for each module, we can evaluate the
homogeneity for the modularization process by defining the homogeneity of a
modularization technique as follows:
1 X
k
MOHO(M) = |Mi | × MOHO(Mi ) (6)
|O| i=1
where k is the number of modules, |Mi | is the number of concepts in module i,
and |O| is the total number of concepts in the original ontology.
Module Heterogeneity. To assess the module heterogeneity (M OHE), we
quantify the interdependency between different modules. Therefore, we consider
the following metrics:
– Relative size (RS). The heterogeneity metric evaluates the coupling be-
tween ontology modules. The higher the coupling between modules the higher
the relatedness between them. It is a desirable property to keep ontology
modules loosely coupled in order to be independently used. Given an ontol-
ogy O modularized into a set of modules {M1 , M2 , M3 , .., Mk }. The first
metric can be used to constitute M OHE metric is the relative size. By the
relative size, we ensure that the ontology concepts are ”normally” distributed
among the modules. To determine the relative size of modularization, we use
the following formula:
1 X X
k−1 k
RS(M) = ||Mi | − |Mj || (7)
k × |O| i=1 j=i+1
where |Mi | and |O| are the module and ontology sizes, respectively.
– DetachRel. Even if the relative size of concepts within ontology modules
can be consider as a good indicator of how these concepts are distributed
among the modules, however, we need another metric that quantifies the
relationships between modules. If there is a concept cj ∈ Mi such that all
the properties (relations) of the concepts are belonging to the same module,
then the module has a high cohesion and low coupling. However, in the
contrast, if concept cl ∈ Mj has some connected concepts in the other
modules, i.e. property(cl ) ∈ ci ∈ Mg , g 6= j, it shows high coupling for
cl ∈ Mj . Therefore, we propose the deployment of the number of detached
relations between concepts (properties) as one of the coupling measure [17].
To compute such a metric, we define DetachRel as given in the following
formula:
1 X X |rMi ∩ rMj |
k−1 k
DetachRel(M) = (8)
k i=1 j=i+1 |rMi ∪ rMj |
where |rMi ∩ rMj | is the number of relations exist in both modules Mi and
Mj .
4 Experimental Evaluation
We conducted a set of experiments to demonstrate that the proposed evalua-
tion metrics are valid and effective within the context of our partitioning tool.
To perform such evaluation, we collected a set of ontologies from the BioPor-
tal repository6 and the ontology search using some generic keywords. The col-
lected ontologies represent different domains and have different characteristics,
as shown in Table 1.
6
http://bioportal.bioontology.org/
Table 1: Data set specification.
Ontology Domain No. of class No. of properties No. of modules
GFO general 44 41 4
BCO biological 126 203 4
mouse anatomy anatomy 2746 1 12
nci anatomy anatomy 3304 2 25
ENVO environment 2159 4 16
OBOOE Observation 630 32 8
PW Pathway 2067 1 22
Delegation Grip computing 17 23 1
Koala human 19 6 1
Dolce lite Dolce 36 71 1
HarryPotter book 16 6 1
Conference conference 58 65 1
Monetary economic 38 28 4
Philosurfical philosophy 376 314 10
4.1 Experimental Results
To prepare ontology modules for evaluation, we apply our modularization ap-
proach to partition the ontologies listed in Table 1. The module homogeneity
(M OHO) as well as the module heterogeneity (M OHE) have been recorded
and reported.
Module Homogeneity. We carried out this set of experiments to evaluate the
quality of module homogeneity metrics. We applied our modularization approach
to the set of ontologies listed in Table 1. The optimal number of modules
for each ontology generated using our partitioning tool has been identified and
listed into the table, too. First, we get the results for each individual M OHO
metric, such as SM H and the average module homogeneity has been computed
using Eq.5. Results are reported in Figs. 2& 3. The results show, in general,
that most of the tested ontologies have module homogeneity values greater than
0.2, which can be considered as a high value due to the involvement of several
aspects to compute the module homogeneity. Secondly, ontologies partitioned
only to one module have the higher M OHO values. Furthermore, we observe
that ontologies belonging to the bio-domain have zero value for SM H, since we
are currently using the WordNet dictionary, which is more generic dictionary
and it fails semantics relationships between concepts from the bio-domain.
Module Heterogeneity. In this set of experiments, we validate the module
heterogeneity metric. We applied our modularization approach to the set of
ontologies in Table 1 and computed both module heterogeneity metrics: relative
size (RS) and DetachRel (Dat). Results are reported in Fig. 4. In general and
as expected, ontologies partitioned only to one module have M OHE values of
0 for both metrics. The figure also shows that RS indicates that the ontology
concepts are nearly equally distributed between ontology modules, except BCO,
Fig. 2: Individual MOHO metrics. Fig. 3: Combined MOHO.
Fig. 4: DataechRel and RS. Fig. 5: Comparing MOHO and MOHE.
Monetary, and Philosurfical ontologies. It also indicates that this modularization
approach produces the minimum overlap (Detch) between ontology modules, i.e.
less coupling, except the Philosurfical ontology.
Furthermore, we study the relationships between M OHO and M OHE met-
rics and the size of ontology (listed in Table 1). Results are reported in Fig.
5. In general, for small size ontologies, the modularization has higher M OHO
(more coherent modules) and lower M OHE (less coupled modules) except for
the Monetary ontology. As the ontology size increases the M OHO and M OHE
values decrease and are nearly constant, respectively.
5 Conclusion
In this paper, we introduced a new ontology modularization evaluation metric.
We offer the module homogeneity metric to assess the module cohesion, while
we propose the module heterogeneity metric to quantify the coupling between
modules. To validate, the proposed metrics, we carried out a set of experiments
utilizing ontologies from different domains. The experimental results demon-
strate that the metric is effective. In the future, we plan to exploit different and
domain specific dictionaries during the evaluation of the semantic module homo-
geneity. Furthermore, we plan to investigate relationships between the evaluation
metrics and different parameters of the modularization approach.
Acknowledgments
This work is partly funded by DFG in the INFRA1 project of CRC AquaDiva.
References
1. A. Algergawy, S. Babalou, M. J. Kargar, and S. H. Davarpanah. Seecont: A new
seeding-based clustering approach for ontology matching. In ADBIS, pages 245–
258, 2015.
2. A. Algergawy, S. Babalou, F. Klan, and B. Koenig-Ries. OAPT: A tool for ontology
analysis and partitioning. In 19th International Conference on Extending Database
Technology, EDBT 2016,, 2016.
3. F. Amato, A. D. Santo, V. Moscato, F. Persia, A. Picariello, and S.R.Poccia.
Partitioning of ontologies driven by a structure-based approach. In 2015 IEEE
International Conference on Semantic Computing, pages 320–323, 2015.
4. C. Bezerra, F. L. G. de Freitas, A. Zimmermann, and J. Euzenat. ModOnto: A
tool for modularizing ontologies. In Proceedings of the 3rd Workshop on Ontologies
and their Applications, 2008.
5. H. Chae, Y. Kwon, and D. Bae. A cohesion measure for object-oriented classes.
Software Pract Exp, 30:1405–1431, 2000.
6. M. d’Aquin, A. Schlicht, H. Stuckenschmidt, and M. Sabou. Ontology modular-
ization for knowledge selection: Experiments and evaluations. In 18th Interna-
tional Conference Database and Expert Systems Applications,DEXA, pages 874–
883, 2007.
7. A. D. et. al. Evaluation of the oquare framework for ontology quality. Expert Syst.
Appl., 40(7):2696–2703, 2013.
8. M. Fernandez, I. Cantador, and P. Castells. Core: A tool for collaborative ontology
reuse and evaluation. In 4th International Workshop on Evaluation of Ontologies
for the Web (EON 2006), 2006.
9. J. Garcı́a, F. J. Garcı́a-Fernández, and R. Therón. Defining coupling metrics among
classes in an OWL ontology. In IEA/AIE 2010, pages 12–17, 2010.
10. B. C. Grau, I. Horrocks, Y. Kazakov, and U. Sattler. Just the right amount:
extracting modules from ontologies. In WWW, pages 717–726, 2007.
11. Q. Guo and M. Zhang. Question answering based on pervasive agent ontology and
semantic web. Knowl.-Based Syst., 22(6):443–448, 2009.
12. Y. Ma, K. Lu, Y. Zhang, and B. Jin. Measuring ontology information by rules
based transformation. Knowl.-Based Syst., 50:234–245, 2013.
13. Y. Ma, H. Wu, X. Ma, B. Jin, T. Huang, and J. Wei. Stable cohesion metrics for
evolving ontologies. Journal of Software Maintenance, 23(5):343–359, 2011.
14. Y. Ma, X. Zhang, B. Jin, and K. Lu. A generic implementation framework for
measuring ontology-based information. Int. J. Computational Intelligence Systems,
7(1):136–146, 2014.
15. L. Meng, R. Huang, and J. Gu. A review of semantic similarity measures in
wordnet. International Journal of Hybrid Information Technology, 6(1), 2013.
16. S. Oh and H. Y. Yeom. A comprehensive framework for the evaluation of ontology
modularization. Expert Syst. Appl., 39(10):8547–8556, 2012.
17. S. Oh, H. Y. Yeom, and J. Ahn. Cohesion and coupling metrics for ontology
modules. Information Technology and Management, 12(2):81–96, 2011.
18. A. M. Orme, H. Yao, and L. H. Etzkorn. Coupling metrics for ontology-based
systems. IEEE Software, 23(2):102–108, 2006.
19. J. Pathak, T. M. Johnson, and C. G. Chute. Survey of modular ontology tech-
niques and their applications in the biomedical domain. Integrated Computer-Aided
Engineering, 16(3):225–242, 2009.
20. Y. Qu and G. Cheng. Falcons concept search: A practical search engine for web
ontologies. IEEE Transactions on Systems, Man, and Cybernetics, 41(4):810–816,
2011.
21. A. A. Romero, M. Kaminski, B. C. Grau, and I. Horrocks. Ontology module
extraction via datalog reasoning. In Twenty-Ninth AAAI Conference on Artificial
Intelligence, pages 1410–1416, 2015.
22. S. Tartir and I. B. Arpinar. Ontology evaluation and ranking using OntoQA. In
International Conference on Semantic Computing, pages 185–192, 2007.
23. C. D. Vescovo, D. Gessler, P. Klinov, B. Parsia, U. Sattler, T. Schneider, and
A. Winget. Decomposition and modular structure of bioportal ontologies. In 10th
International Semantic Web Conference ISWC 2011, pages 130–145, 2011.
24. Z. YANG, D. Zhang, and C. YE. Evaluation metrics for ontology complexity and
evolution analysis. In ICEBE’06, pages 162–170, 2006.