Modularity has been increasingly used as a solution to assist with the use and maintenance of large ontologies. However, there is lack of evaluation methods available for determining the quality of an ontology module. While initial work has been done on producing a comprehensive list of evaluation characteristics that can be used to check the quality of a module, certain aspects are still unclear. For instance, the initial list has not been structured into different groups for different types of evaluation criteria. It is also unclear on how to apply these characteristics to a particular module, and the metrics have not been experimentally validated for some types of ontology modules. In this paper, we structure the comprehensive evaluation criteria into groups, provide practical examples on how to use the evaluation characteristics to assess a module, and validate the evaluation characteristics with a set of ontology modules.
Over the last few years, there has been a growth in using modularity to assist with maintaining and using large ontologies. The general concept of modularity refers to dividing and separating the components of a large system such that modules can be recombined. Modularity is used to simplify and downsize an ontology for the task at hand; to modularise a large ontology into smaller manageable ontologies. Modularity has been successfully applied to a number of different ontologies to improve usability and assist with complexity. Examples include the myExperiment ontology [16], which is a collaborative environment where scientists publish and share their work-flows and experiment plans among groups, the Semantic Sensor Net ontology where there are various modules to describe sensors and observations [9], and BioTop ontologies for life sciences in which the principle of modularisation have been applied [22].
An issue concerning ontology modules is the lack of evaluation metrics. The existing works on evaluation metrics focus on only some metrics that suit the modularisation technique [20,21,26], and there is not always a quantitative approach to calculate them. Overall, the metrics are not comprehensive enough to apply to a variety of modules. It is therefore not clear on how to determine whether a module is of good quality.
We have already published initial work done on determining whether a module is a good or bad module with evaluation metrics [10]. The initial study [10] revealed 13 criteria from the literature, and 3 new ones, some of which were short of a metric for quantitative evaluation that have now been devised. In this work, we categorise the 16 criteria into 5 broader categories to structure them so that ontology developers can easily identify which ones to use for an application. We also demonstrate usage of the criteria with practical examples. Lastly, we perform an experimentation evaluation on a set of ontology modules using the evaluation criteria revealing how the modules measure in terms of quality.
The remainder of the paper is structured as follows. We discuss related works in Section 2. This is followed by a section on the evaluation criteria in Section 3. In Section 4 we discuss the types of modules we are going to evaluate. The experimental evaluation using the evaluation metrics is performed in Section 5. Finally we conclude in Section 6.
d’Aquin et. al [
Tartir et. al [23] propose richness criteria to measure the quality of ontologies. This criteria is based on how rich the ontology is with regard to attributes and subclasses. For cohesion metrics, Yao et. al propose metrics such as the number of root classes, number of leaf classes, and average depth of inheritance tree of all leaf node [26]. These metrics, however, does not reveal how the entities are related in a module as compared to the original ontology. The work done by Oh et. al on cohesion metrics, however, does measure the strength of the relations in a module [18]. To measure the coupling of an ontology, researchers propose metrics based on the number of externally defined referenced concepts [19]. This, however, does not take into account external links that different modules share. Oh and Ahn [17] have improved on this to consider the external links between different modules based on whether the link is hierarchical or relational. However, their metric is simply a sum value of the number of each type of links between modules, which does not measure the complete interdependence of a module since it only considers one type of variable in the module.
In previous work, we already defined a list of evaluation criteria alongside their mathematical metrics for modularity [10]. We now group them into categories of criteria, and provide some examples demonstrating how to use them. For the set of criteria, let O be an ontology with a corresponding set of axioms, Axioms(O), and M be a module with a corresponding set of axioms, Axioms(M). Let i, j be arbitrary entities in an ontology.
Structural criteria are calculated based on the structural and hierarchical properties of the module. These criteria are calculated by inspecting the syntax of the ontology. It is usually based on counting components of the ontology such as axioms, entities, etc., and is a numerical value. Calculating structural criteria involves evaluating the size, relations, and placement of entities within a module. We now list the structural criteria, alongside practical examples.
Size: Size is the number of entities in a module (the number of classes, object properties,
data properties, and individuals in a module) [
Relative size: The relative size is the size of the module, i.e., the number of entities in a module compared to the original ontology [12].
Relative size(M) = jMj
O j j Atomic Size(M) = jAxiomj jAtomj
atomic size is hence 67 = 1:17. This tells us that there is an average of 1.17 axioms per atom for the example.
als, 41 object properties, and 0 data properties. The source ontology, GFO, contains 78 classes, 0 individuals, 67 object properties, and 0 data properties. Hence the relative size is 7487++00++6471++00 = 0.61.
Appropriateness: Appropriateness is measured by mapping the size of an ontology module to some appropriateness function value between 0 and 1 to reflect the defect density [21]. (1) (2) (3) Appropriate(x) =
cos(x: where x is the number of axioms in the module. The authors propose the function with 250 axioms as it is based on the fact that the optimal size of software lines in a code is 200-300 lines [21].
tions and situations ontology [15] has 435 axioms. Therefore, based on the
equation for calculating appropriateness, the value is 12 21 cos(435: 2p50 ) = 0.16.
Atomic size: The atomic size of a module is the average size of a group of
interdependent axioms in a module. This is based on the notion of atoms in modules,
which are defined as a group of axioms with dependencies between each other
[24]. Dependent axioms are those that describe the same named entity.
Intra-module distance: The intra-module distance in a module is the distance between
entities in a module [
Relative intra-module distance(M) =
(5)
in Figure 2, and the farness values in Table 1. Using the farness values in the intramodule distance equation, we calculated the intra-module distance of the source ontology O to be 32 and of the module M to be 16. The relative intra-module distance is 1362 = 2, hence the module entities are twice as close as the original ontology.
Cohesion: Cohesion refers to the extent to which entities in a module are related to
each other. [
A B C D E
Farness, O A B - 1 1 1 0 4 3 4 3 (7) Correctness(M) = (true f alse i f Axioms(M) otherwise
tains those axioms which exist in the GFO ontology. Thus, the GFO-Abstract-Top ontology is logically correct. The GFO-Basic ontology, however, is a smaller module based on the GFO ontology but it also contains new axioms that do not exist in the GFO ontology. For instance, the entity Processual Structure exists in the GFO
Completeness: Completeness is when the meaning of every entity in a module is
preserved as in the source ontology [
is 10 as shown in Table 1. The number of entities in M is 5. Hence the cohesion value is as follows: Cohesion = 51(40) = 0.5.
By definition, an ontology is: “a logical theory accounting for the intended meaning of
a formal vocabulary, i.e. its ontological commitment to a particular conceptualisation of
the world” [
jMj i f å Axioms(Entityi(M)) j= Axioms(Entityi(O))
i otherwise Example 7 Consider a source ontology, DOLCE-Lite [15], where the endurant entity is defined as follows: endurant v 8 part.endurant endurant v spatio-temporal-particular endurant v 9 participant-in.perdurant endurant v 8 speci c-constant-constituent.endurant endurant v : quality endurant v : perdurant endurant v : abstract
branch of Endurant entities, with the removal of perdurant entities called DOLCEendurants, then the endurant entity is defined as follows: endurant v 8 part.endurant endurant v spatio-temporal-particular endurant v 9 participant-in.perdurant
iom endurant v 8 speci c-constant-constituent.endurant existed in the original ontology but not in the module. Therefore the DOLCE-endurants module has a false value for the completeness metric.
Relational criteria deal with the relations and behaviour that modules exhibit with other modules in a system of interrelated modules. We now list the relational criteria, alongside practical examples.
Inter-module distance: The inter-module distance in a set of modules has been described
as the number of modules that have to be considered to relate two entities [
Inter-module-distance(M)= 8<Ei;E j2å(Mi;;Mn) j(Mi;::;MNn)Mj((j(EMi;iE;:j:);Mn)j 1) j(Mi; ::; Mn)j > 1 :1 (9) where NM(Ei; E j) is the number of modules to consider to relate entities i and j. The product of j(Mi; ::; Mn)j(j(Mi; ::; Mn)j 1) represents the number of possible relations between entities in a set of modules Mi; :; Mn.
tity pair, we have the number of modules, NM, that have to be considered to
relate them in Table 2. The sum of NM is 126. The number of entities is 9, hence
the j(Mi; ::; Mn)j(j(Mi; ::; Mn)j 1) value = 9(8). Thus the inter-module distance is
91(182) = 1:75. For the set, it takes 1.75 modules to relate two entities in the set.
Coupling: A measure of the degree of interdependence of a module [
ure 3. The number of external links that have to be considered to relate M1 to other modules in the set, is 2. The number of possible external link between a module M1 and the other modules in the system is calculated as follows: jM1j(jM2j) + jM1j(jM3j) = 18. Hence the coupling(M1) = 128 = 0:11 which indicates a low interdependence toward other modules in the system.
Redundancy: Redundancy is the duplication of axioms within a set of ontology modules [21]. To measure redundancy in a set of modules, we calculate the fraction of duplicated axioms. k where å ni is the total number of axioms and n is the number of distinct axioms i=1 in a module. The resulting fraction is a value of redundancy.
no inter-related links that have been partitioned from a food ontology as shown in Figure 4. There are 3 ontology modules: Fruit, Vegetable, and Meat. Axioms that have been repeated more than once (redundant axioms) are shown in blocks. From the three modules, there is a total of 21 axioms, i.e., the Axt value is 21. There are 15 distinct axioms that exist in the set of modules (these axioms exist at most once and are those that are not blocks), hence Axd is 15. The redundancy of the set of partitioned modules is thus 212115 = 0:29. Hence, 29% of the axioms in the set of modules are redundant.
Ontology modules sometimes deal with hiding aspects of the source ontology from the module for privacy and simplification reasons. Information hiding within modules assesses whether the module encapsulates all the information in the module such that the privacy is preserved for each module. We now list the information hiding criteria, alongside practical examples.
Encapsulation: Encapsulation is a metric that holds when “a module can be easily
exchanged for another, or internally modified, without side-effects on the application
can be a good indication of the quality of the module” [
n Encapsulation(M i) = 1 (12)
Example 10. We calculate the encapsulation of the Fruit module as follows. There are 7 axioms in the Fruit module, i.e., the Axi = 7. In the Vegetable module, there are 3 overlapping axioms, i.e., they also exist in the Fruit module. In the Meat module, there are 2 overlapping axioms, i.e., they also exist in the Fruit module. Hence, the Encapsulation(Fruit) is calculated as 1 37 +3 72 = 0:76. Thus, 0.76 (76%), or a large amount of the domain knowledge is encapsulated in the Fruit module but the complete privacy of the Fruit module is not preserved.
Independence: Independence evaluates whether a module is self-contained and can be
updated and reused separately [
Ind(M i) = (true
Encapsulation(Mi) = 1 and Coupling(Mi) = 0 f alse otherwise (13) where jMij is the number of entities in the current module and jM jj is the number of entities in a related module in the set of n modules.
worked out the encapsulation value for the Fruit module in Example 11 as 0.76.
The richness or amount of information in an ontology is designed as one aspect to measure the quality of an ontology. For modules, this is important to measure in cases where abstraction is employed to compare the granularity of the source ontology to that of the module. Tartir et al. [23] propose measurable richness schema metrics. We now list the richness criteria, alongside practical examples.
Attribute richness: Attribute richness is defined as the average number of attributes per class [23]. where att is measured by the number of data properties in the module jDPj and jCj is the number of classes in the module. In an ontology, an attribute is used to describe an entity and each attribute, or data type, has a name and value.
Inheritance richness: Inheritance richness is defined as the number of subclasses per class in an ontology [23].
AR(M) = jattj
jCj å jHC(C1;Ci)j IRS(M) = Ci2C (14) (15) where jHC(C1;Ci)j is the number of subclasses per class and jCj is the total number of classes in the ontology.
B with 2 subclasses. Hence the sum of these subclasses is 2 + 2 = 4. There are 5 classes in total in M. The inheritance richness value for M is thus 45 = 0:8. Using the same method we work out the inheritance richness value for ontology O, which is 67 = 0:85.
To assist with the lack of evaluation metrics and corresponding formulae in ontology modules, we presented a comprehensive list of evaluation criteria for modules together with examples on how to operationalise them for ontology modules. To put context to the values for the metrics, experimentation was performed in other work stating which values are appropriate for the metrics for particular module types [12]. In Section 5, we show how each metric is used and what some expected values are.
In this work, we focus on evaluating certain types of ontology modules, i.e., those that are lacking from an existing ontology modularisation framework and experiment [11]. The modules were generated using the NOMSA modularisation tool 2. We briefly describe each module type here with its corresponding abbreviation which we will use to describe them in the remainder of the paper.
Axiom abstraction (AxAbs): This is a module containing hierarchical relations between entities, i.e., other relations are removed resulting in a bare taxonomy. Vocabulary abstraction (VocAbs): This is a module where certain types of entities are removed, for instance, the object properties or data properties of an ontology. High-level abstraction (HLAbs): This is a module where entities at a higher level in the hierarchy have precedence over the others.
Weighted abstraction (WeiAbs): This is a module where weighting is assigned to entities that are referenced by axioms more than others.
Feature expressiveness (FeatExp): This is a module where some axioms of the ontology are removed based on its language features.
In order to uncover information about how evaluation metrics relate to ontology modules, we use the Tool for Ontology Modularity Metrics (TOMM) software tool [12] which encompasses all the evaluation metrics described in Section 3. In this experiment, we evaluate the module types from Section 4, in terms of its quality.
2http://www.thezfiles.co.za/modularisation/
2. Run the TOMM metrics tool [12] for the modules to acquire module metrics. 3. Conduct an analysis from the metrics for each module.
The materials used for the experiment were as follows: TOMM metrics tool [12],
and a set of 128 ontologies that had been modularised using NOMSA ontology
modularisation tool 2 but derived from the set of ontologies described elsewhere [
Each of the modules were evaluated using the metrics that were implemented in the TOMM tool and the results for the numerical metrics are shown in Table 3. The atomic sizes of the modules indicates that there are on average between 2.34- 3.80 axioms that are grouped together in an atom for the modules. The appropriateness, which maps the size of the ontology module to a function is less than 0.3 for all the modules. Seeing that 1.0 is the optimal value for appropriateness, all the modules perform poorly for this metric. For a module to have an optimal value, it must have a value close to 250 axioms. The intra-module distance values which indicate the distance between entities in a module differ considerably, with the HLAbs modules having the lowest value of 142 698.4 and the AxAbs having the highest value of 866 354.60. The cohesion of a module indicates how closely related its entities are to each other, with higher values having a large number of relations among entities. The cohesion is small for all the modules in the set (less than 0.07). Most of the modules in this set do not contain attributes, as the attribute richness is less than 1 for all the modules. The number of subclasses per class is between 2.72 to 4.84 as indicated by the inheritance richness for the modules. In this set, all the modules were considerably smaller than the original ontologies, as indicated by relative size values that are less than 1. WeiAbs modules are the smallest at 0.26 (26% the size of the original ontology) while VocAbs modules are 0.85 (85% the size of the original ontology). It is also important to determine whether the entities in the modules have moved closer to each other in a module compared to the original ontology. AxAbs modules, VocAbs modules, and FeatExp modules have values less than 1 for the relative intra-module distance, which indicate that their entities are further in relation to the entities of the original ontology. While in HLAbs and WeiAbs the intra-module distance is larger, indicating that for these modules, the entities have moved closer as compared to the original ontology.
We have discussed the evaluation metric values for the set of ontology modules in isolation, i.e., without an indication of whether the modules in question appear to be good or of poor quality. To check if the modules are of good quality we refer to an existing benchmark dependency between the proposed ontology metrics and various types of ontology modules. The notion here is that for each module type, there is a pattern or dependencies between type and some set of metrics. This is shown in Table 4 with expected values for each type of module. The values that are underlined were not met for this experiment. For instance, reading Table 4, it states that for WeiAbs modules the cohesion value of range 0-0.25 and the relative size value of range 0.26-0.5 is met. However, for AxAbs, the cohesion value is met but the correctness value is not met.
We now examine the failed metrics. For appropriateness, it appears that the number of axioms in a module need to be between 167- 333 to have a value of between 0.751. In some cases, the original ontologies were already less than 167 axioms in size, so there could never reach an appropriate value. For the case of ontology modules, which by definition are a reduction of an ontology, this appropriateness metric needs to be redesigned to include modules that are less than 167 axioms. The next failed metric is the correctness value, which needs to be true, but tested false for some modules, meaning that some new axioms were added to the modules, i.e., axioms that were not from the original ontology. An inspection of the log files for the metrics revealed that OWL enumeration was used to declare individuals in an original ontology. In resulting modules, when the collection of individuals were broken up, they were re-represented as named instances using class identifiers. This means that new knowledge was not added to the module, but because of language syntax, it appears so. The ontology metrics tool therefore needs to recognise that this is not a new axiom to measure the correctness value accurately.
The evaluation criteria for ontology modules were already presented in previous work [10]. However, there was limited support for operationalising these metrics. In this paper, we provide practical examples on how the metrics can be applied to an ontology module to measure them. We also structured and grouped together the list of evaluation criteria to various higher level categories to differentiate those that examine, say structural aspects of a module (structural criteria) to those that examine how well the modules hide certain information of the source ontology (information hiding criteria). This could aid ontology developers in selecting the relevant evaluation criteria for their use-case.
We performed an experiment using various types of modules that were generated using NOMSA modularisation tool. The modules were evaluated with TOMM to reveal their metrics. The metrics indicate important information such as, the rate at which entities in a module move closer to each other (using the relative intra-module distance metrics), whether a module is rich with attributes, at what rate the size of the module differs as compared to the source ontology. These metrics are important for gauging how well the modules would fare with visualisation and comprehension tools or if there would be some performance issues due to redundancy, among other cases.
Initial work on evaluation criteria was refined by grouping the evaluation criteria list into categories to structure them to aid ontology developers. We summarised the list of evaluation criteria, together with practical examples on how to use them. An experimental evaluation revealed how the metrics can be used to evaluate modules and uncover information about how well certain modules fare for certain applications. We also identified a problem with the appropriateness value, which cannot be applied in some cases if a module has a small axiom size which needs to be changed. Another problem that was identified is that some knowledge is recognised as new knowledge by the evaluation metrics tool due to OWL representation issues.