In this paper, we discuss our experience in testing the HMatch matchmaking system by means of the tracks proposed in the ontology alignment evaluation initiative of 2006 1. HMatch is a system conceived for the goal of ontology matching in open and distributed systems. It is based on linguistic and structural matching techniques for the evaluation of affinity considering concept names and concept contexts. The paper discusses the results that have been obtained and the possible improvements of the matching techniques in ongoing and future work.
1.1
With respect to the different purposes of matching, the state of HMatch is the following:
– Schema matching. In developing HMatch, we started from the schema matching
functionalities of Artemis integration system [
Given two concepts, HMatch calculates their semantic affinity value as the linear
combination of a linguistic affinity value and a contextual affinity value. The basic techniques
used in HMatch are linguistic and structure-based techniques that are applied to concept
names and contexts. For a more detailed classification of these and other techniques the
reader can refer to [
Linguistic-based affinity techniques. Linguistic techniques consider names of
ontology elements and their meaning. To capture the meaning of names for ontology
matching, a thesaurus of terms and weighted terminological relationships is exploited. In
HMatch, the thesaurus is automatically derived from the lexical system WordNet [
A(t, t0) = maxi=1...k {Wt→int0 } if k ≥ 1 0 otherwise (1) where: k is the number of paths between t and t0 in T h; t →in t0 denotes the ith path of length n ≥ 1; Wt→int0 = W1tr · W2tr · . . . · Wntr is the weight associated with the ith path, and Wjtr , j = 1, 2, . . . , n denotes the weight associated with the jth terminological relationship in the path.
Structure-based affinity techniques. Structure-based techniques consider properties
and concepts directly related to a concept c through a semantic relation in an ontology.
Given a concept c, we denote by P (c) the set of properties of c, and by C(c) the set of
concepts that participate in a semantic relation with c (namely, its adjacents). The
context of a concept in HMatch is defined as the union of the properties and of the adjacents
of c, that is, Ctx(c) = P (c) ∪ C(c). Also contextual features are weighted in HMatch.
In particular, we associate a weight Wsp to strong properties, and a weight Wwp to weak
properties, with Wsp ≥ Wwp to capture the different importance they have in describing
the concept. In fact, strong properties are mandatory properties related to a concept and
they are considered more relevant in contributing to concept description. Weak
properties are optional for the concept in describing its structure, and, as such, are given less
importance. Each semantic relation has associated a weight Wsr which expresses the
strength of the connection expressed by the relation on the involved concepts.
Considering the semantic relations of OWL, we have the weights Wequivalence ≥ WsubClassOf .
The greater the weight associated with a semantic relation, the higher the strength of
the semantic connection between concepts. Given two elements e and e0 in the contexts
of c and c0, respectively, their affinity is calculated according to the following function
C(e, e0) → [
C(e, e0) = A(ne, ne0 ) · (1− | We, We0 | (2) where ne and ne0 denote the names of e and e0, respectively, while We and We0 denotes the weights associated with e and e0. As an example, suppose that we compare two concept contexts Ctx(c) and Ctx(c0) both containing the property author that is a strong property (i.e., featured by a minimum cardinality greater than or equal to 1) in the first context and a weak property (i.e., an optional property) in the second context. Thus, by using a weight equal to 1.0 for strong properties and equal to 0.5 for weak properties, we obtain:
C(authorCtx(c), authorCtx(c0)) = A(author, author) · (1− | 1.0 − 0.5 |) = 0.5 since A(author, author) = 1.0.
Given two concepts c and c0, the comprehensive semantic affinity SA(c, c0) is calculated as the weighted sum between their linguistic affinity LA(c, c0) and their contextual affinity CA(c, c0), as follows:
SA(c, c0) = Wla · LA(c, c0) + (1 − Wla) · CA(c, c0)
(3)
where Wla ∈ [
Matching models. Four matching models have been conceived to span from surface to intensive matching, with the goal of providing a wide spectrum of metrics suited for dealing with many different matching scenarios that can be encountered in comparing real ontologies, such as OWL ontologies. The main difference among the four matching models is the composition of the context. In the surface model, the context is not considered limiting to linguistic affinity. In the shallow model, only properties and property restrictions are considered for concept context. In the deep model, we consider both properties and semantic relations, such as taxonomic relations. Finally, in the intensive model we consider the whole context, by taking into account also the property ranges and values. For all the models the linguistic affinity LA(c, c0) between two concepts c and c0 is calculated to be equal to the function A(nc, nc0 ), where nc and nc0 denote the names of c and c0, respectively. For the contextual affinity evaluation, we provide two main strategies, namely the standard strategy and the Dice coefficient strategy. The standard strategy produces a nons-ymmetric contextual affinity measure. For each element e in the source concept context Ctx(c), we search for the best matching element e0 in the target concept context Ctx(c0) by exploiting the function C(e, e0) described above. Given the best matching value me found for e with respect to the elements in the context of c0, the comprehensive contextual affinity is calculated as follows: CA(c, c0) =
P ei∈Ctx(c) mei | Ctx(c) | where | Ctx(c) | denotes the number of elements in c.
According to the Dice coefficient strategy, the contextual affinity is calculated as follows:
CA(c, c0) = | x ∈ Ctx(c) ∩ Ctx(c0) |
| x ∈ Ctx(c) ∪ Ctx(c0) | where | x ∈ Ctx(c) ∩ Ctx(c0) | denotes the number of matching elements in Ctx(c) and in Ctx(c0), that is the number of elements having a value of C(e, e0) higher than a given matching threshold. 1.3
Since HMatch has been developed with the goal of achieving a high level of flexibility and configurability of the matching process, a matching policy P has be set, which is defined as follows:
P = hWla, T, M, C, I, S, Ei
where: Wla is the weight associated with the linguistic affinity; T ∈ [
For the purposes of the OAEI 2006 initiative, we adopted the standard implementation
of HMatch as a Prote´ge´2 plugin. This version adopts the Prote´ge´ OWL API3 and is fully
integrated into the Prote´ge´ framework. We only introduce a command line version in
order to use HMatch as an independent tool, especially for the benchmark. We have
implemented two main extensions specifically conceived for the contest. The first
extension is the support for the output Alignment format required by the organizers in
addition to the native HMatch results format. The second extension regards the evaluation
of the linguistic affinity. We introduced a new facility of HMatch that performs
linguistic affinity evaluation using a n-gram algorithm [
The HMatch implementation used for the contest together with the policy used for configuration and the results can be find at: http://islab.dico.unimi.it/OAEI2006/islab results.html. 2
All the results have been obtained by configuring HMatch with the following policy: Wla T M C I E S
Policy Value Linguistic affinity weight 0.5
Threshold 0.6 Matching model Deep Matching type One-to-One
Inheritance True
Empty context strategy Neutral
Contextual affinity strategy Dice coefficient 2 http://protege.stanford.edu/ 3 http://protege.stanford.edu/plugins/owl/api/ In particular, the most relevant parameters are i) the matching model, since the deep model forces HMatch to consider both properties and semantic relations in the concept contexts, ii) the weight for linguistic affinity, since the value 0.5 determines a perfect balance between the linguistic affinity evaluation and the contextual affinity evaluation, and iii) the threshold, which is used for cutting off the results that are not considered relevant in the matching case. We have tested several possible configurations of HMatch on the benchmark. In some matching cases there are other configuration policies that produce better results in terms of precision and recall than the one we have chosen. The actual choice was motivated by the fact that we considered the various tasks proposed in the contest with the goal of configuring HMatch with a policy that could guarantee a generally satisfactory behavior of the system in the different matching cases. In particular, we have tested HMatch on all the test cases provided in the contest, with the goal of receiving a feedback about the application of the system to different and highly heterogeneous matching cases. 2.1
Obtained results on the proposed benchmark are strongly affected by the fundamental role that the ontology linguistic features play in the HMatch matching process. In fact, we obtained an average precision value of 0.84 and an average recall value of 0.55. These results are influenced by the fact that we obtained poor result for the ontology cases where the concept and property labels were substituted with randomly generated strings of characters. The difference between precision and recall values when we consider all the cases is due to the fact that, in some of the randomly generated ontologies (e.g., case 259), there is a property which maintains the original name (i.e., lastName). This matching is retrieved by HMatch and it increases the precision of the results. Another issue that affects the results quality, from the linguistic point of view, is the presence of matching cases where the concept and property labels are french terms. In these cases, since some of the properties match, we obtained precision values about 0.4 and recall values about 0.2. The benchmark results are also useful to suggest possible improvements of HMatch, with the goal of addressing also the anomalous cases where the linguistic information is completely missing due to the design choices. 2.2
With the anatomy track, obtained results suggest the following considerations. Due to the domain specific terminology used in the ontologies, either using the WordNet thesaurus or a string matching technique, the results are affected by the fact that the concepts are labeled with long strings describing specific terms. In the case of a domain specific terminology, the linguistic matching would benefit from the availability of specific thesauri. Given the large amount of data in the two compared ontologies, the string-matching procedure for linguistic affinity is more suitable, while affecting the capability of the system to capture the semantics of the terms used in the two ontologies that would instead be possible using the thesaurus. Moreover, the openGalen ontology has a anomalous OWL structure, since OWL classes are used as meta-classes, while individuals represent the domain concepts. For this reason we needed a wrapper to compare the FMA concepts with the concepts of openGalen, and only the linguistic comparison was possible. 2.3
The directory matching is a new task for HMatch, which was not originally designed for dealing with peculiar features of directory repositories. In particular, two main characteristics of directory taxonomies require specific support not directly provided by HMatch. The terminology used for labeling the directories is often affected by the structure of the taxonomy itself more than by the subject of the directory. Examples of this terminology is given by terms like A-H that is referred to the alphabetic order more than to the subject of the directory, or African 2 where the name of the directory is associated with information about the number of equivalent directories in the taxonomy. A second problem is given by the taxonomy itself. In fact, HMatch gives the is-a semantics to the OWL sub-class relations as in formal ontologies. Although, the sub-directory relations represented as OWL sub-class relations have in fact different meanings. For example, we have a sub-class relation between Animal Webcams and Space and Science that denotes a generic positive relation between the two concepts rather than an is-a relation. Another example is given by the sub-class relation between California and United States that denotes a geographical part-of relation. 2.4
The food track requires to match two XML thesauri. We developed a wrapper from the SKOS XML format to OWL in order to match the thesauri with HMatch. The track requires also to recognize different kind of mapping relations between the source and the target, i.e., exactMatch, broadMatch, and narrowMatch. Using HMatch, we provide a measure of the semantic affinity between two concept, that is a measure of the fact the the two terms have the same meaning. Because of this reason, we provided only an evaluation of the exactMatch mapping between the two ontologies. In order to evaluate the broad and narrow matching relations, the thesaurus component of HMatch could be exploited, but this has not been done due to the contest requirement of using the same techniques for all different cases. 3
One of the main issues in the field of ontology matching is the need of flexible algorithms and tools, capable to adapt to different domains and also to different interpretation of the notions of alignment and similarity. Some of these differences depends on the concept descriptions provided by the ontologies to be compared with their specific level of semantic complexity. The choice of the best approach or the best combination of approaches depends on the specific matching case we are dealing with and on the domain of the ontologies. For example, formal ontologies can benefit from a logic approach, while thesauri and dictionaries require a deep linguistic analysis; finally, structure affinity is suitable for directories and repositories. The domain affects also the kind of techniques that are used as well as the matchmaking utilities (e.g., thesauri, external sources, type of mapping relations) that are involved in the matching process. A good example is given by the anatomy track of the contest. In this domain, we work with a specific and domain dependent terminology that requires a specific linguistic analysis. A second example is given by the matching of directories or also by the matching of spatial or temporal ontologies. In this cases, in fact, some properties or relations should be matched by using specific matching operators. For example, the property author and the property below have a different role on concept definition when used in a spatial domain, even if they are represented by means of the same language construct. The matching should take into account all these specific requirements by adapting the matching process and the matching operations to the specific domain that is taken into account. 3.1
The results obtained in the OAEI tasks show how HMatch can provide a good balance in the results between precision and recall with a fully automated matching that does not require any specific external source neither in terms of a training set of results nor in terms of domain specific thesauri. Although, if on a side this characteristic means that HMatch can be used in several different scenarios, on the other side, it shows a limitation of the system in working either with very specific domain ontologies or with ontologies in which the linguistic information is missing. Some other limitations regard the scalability of the linguistic techniques adopted by the system is the case of large ontologies. To overcome this limitation, we have implemented for the purpose of the contest a new string matching functionality. The main considerations that we can make based on matching cases and obtained results experienced are the following. 1. Linguistic features: the terminology used for naming and labeling concepts and properties is an important aspect of ontologies and provides information to conclude the similarity between the ontology elements. We are conscious that, In many cases, it is not sufficient alone, also because they embed a subjectivity element, deriving from who has been designed the ontology. However, the linguistic features are undoubtedly an important starting point also for deriving a first set of mappings to be refined by exploiting other kinds of matchings. 2. Structural features: concepts can be similar also in terms of their structure. The structure is seen in terms of the links that connect different concepts and also as the number and type of properties that characterize each concept. It is important to note that the structure evaluation does not refer to the semantics of the concept relations and properties. For example, in the directory taxonomies the semantics of the sub-category relation is not ever well defined and can denote many different real relations among categories, e.g., containment, is-a, part-of. In this case, the structure of the taxonomies that are considered is the key feature for detecting the similarity of the concepts, more than the relation semantics. 3. Logical features: from the logical point of view, the ontology matching should consider the formal semantics of the ontologies to be compared in order to i) evaluate the consistency between the mappings and the concept descriptions, ii) apply deductive reasoning to retrieve new mappings starting from an initial set of mappings (e.g., manually provided or retrieved by means of other techniques), iii) provide an interpretation of the resulting mappings. 3.2
By analyzing the results obtained in the different tracks, together with the general comments discussed in the previous section, a first improvement that can be introduced in HMatch is to emphasize the distinction among the linguistic, structural, and logic approaches to ontology matching. HMatch is based mainly on linguistic features. We believe that linguistic matching is a fundamental component for a semantic matchmaker, but we noticed that, in some cases, structure and logics of the ontologies to be compared should be considered with no reference to the ontology element names. Another important direction for improving HMatch is to emphasize the need of different metrics in order to take into account the specific features of the different ontology domains. HMatch provides four different matching models to address the fact that different ontologies can vary with respect to their semantic complexity and with respect to their structure. A further improvement in this direction is to support specific relations in the matching process, such as spatial or temporal relations. 3.3
The OAEI 2006 procedure is well suited to give to matching researchers a complete feedback about their work. Although, we believe that the requirement of using only one set of parameters for the whole contest was a strong limitation, especially because some of the test cases (i.e., anatomy, food) have peculiar features that would benefit from a more flexible configuration. We believe that the capability of matching algorithms to be configured in order to deal with different scenarios is a key feature for ontology matching, but the flexibility cannot be appreciated using the same configuration. If the goal is to test generic-purpose algorithms, the test cases should be more homogeneous with respect to the ontology type and domain. Otherwise, it should be possible to modify the algorithms configuration for the different cases. 3.4
The only comment we have is that, at the end of the evaluation phase, would be useful to have the expected results also for the blind tests, in order to improve the algorithms used where required. 3.5
The traditional precision and recall measures seem to be the most suitable for the matching result evaluation. Although, these measures should be calculated in a flexible way. For example, we should allow the algorithms to provide mappings also among external elements that are imported in the ontologies.
A simple suggestion for new measures is referred to the need of taking into account the time of computation in the matching evaluation. The idea is to combine the computation time with precision and recall, in order to measure the trade-off between time performances of the algorithms and quality of the results. 4
The experience of the OAEI 2006 contest was extremely useful as a feedback about the design and implementation of the current version of HMatch. We had some confirmation of the results obtained in the previous tests, but we had also some new helpful tip about possible improvements of the approach and related techniques. In particular, our future work will be devoted to: i) study new matching techniques that could be used in combination with the linguistic techniques of HMatch, in order to improve the flexibility of the system with respect to different matching scenarios; ii) address new purposes of the matching, such as directory of ontology version matching, by studying specific metrics and techniques for these cases; iii) implement and test a new version of HMatch in the context of the BOEMIE project, where our matchmaking system is used for the purpose of ontology evolution.