Ontology matching, the task of determining relations that hold among terms of two di erent ontologies, is a key issue in the Semantic Web and other related elds. In order to compare the behaviour of di erent ontology matching systems, the Ontology Alignment Evaluation Initiative (OAEI) has established a periodical controlled evaluation that comes in a yearly event. We present here our participation in the 2008 initiative. Our schema-based alignment algorithm compares each pair of ontology terms by, rstly, extracting their ontological contexts up to a certain depth (enriched by using transitive entailment) and, secondly, combining di erent elementary ontology matching techniques (e.g., lexical distances and vector space modelling). Benchmark results show a very good behaviour in terms of precision, while preserving an acceptable recall. Based on our experience, we have also included some remarks about the nature of benchmark test cases that, in our opinion, could help improving the OAEI tests in the future.
service is called CIDER (Context and Inference baseD alignER), which is the
subject of this study. It relies on a modi ed version of the semantic similarity
measure described in [
According to the high level classi cation given in [
For this study we have generalized the method, to admit any two ontologies, and a threshold value, as input. Comparisons among all pairs of ontology terms (not only the ones that could refer to a same user keyword) are established, producing as output an RDF document with the obtained alignments. 1.2
Our alignment process takes as basis a modi ed version of the semantic similarity
measure described in [
After that, a matrix M with all similarities is obtained. The nal alignment A is then extracted, nding the highest rated one-to-one relationships among terms, and ltering out the ones that are below the given threshold.
In terms of implementation, CIDER prototype has been developed in Java,
extending the Alignment API [
As the benchmark test does not consider mappings between instances, we have not computed instance alignment for this particular test. No other adaptations have been needed. 1.4
The version of CIDER used for this evaluation can be found at http://sid.cps.unizar.es/SEMANTICWEB/ALIGNMENT/OAEI08/ 1.5
The obtained alignments for the contest can be found at http://sid.cps.unizar.es/SEMANTICWEB/ALIGNMENT/OAEI08/results/CIDER.zip
The following subsections describe the participation of our system in two tracks
of the contest: benchmark and directory. Some remarks speci c to each test are
described, as well as a tentative explanation of the obtained results. Further
information about the whole results of the contest can be found at [
The target of this experiment is the alignment of bibliographic ontologies. A reference ontology is proposed, and many comparisons with other ontologies of the same domain are performed. The tests are systematically generated, modifying di erently the reference ontology in order to evaluate how the algorithm behaves when the aligned ontologies di er in some particular aspects. A total of 111 test cases have to be evaluated. They are grouped in three sets: 1. Concept test (cases 1xx: 101, 102, ...), that explore comparisons between the reference ontology and itself, described with di erent expressivity levels. 2. Systematic (cases 2xx). It alters systematically the reference ontology to compare di erent modi cations or di erent missing information. 3. Real ontology (cases 3xx), where comparisons with other \real world" bibliographic ontologies are explored.
We cannot provide results for benchmark cases 202 and 248-266, because our system does not deal with ontologies in which syntax is not signi cant at all (these cases present a total absence or randomization of labels and comments). Consequently, we expect a result with a low recall in this experiment, as the benchmark test unfavours methods that are not based on graph structure analysis (or similar techniques).
In Table 1 we show the obtained results, grouped by type of cases. We have obtained a very high precision (97%), which is in the top-three best values obtained in the contest (out of 13 participants), while recall has been lower (62%), due to the above mentioned reason. The extended results for the complete dataset has been published separately by the organizers2.
Alternatively to the o cial results, we have computed the precision and recall of the benchmark test excluding the cases 202 and 248-266 (and their variations 2 http://oaei.ontologymatching.org/2008/results/benchmarks.html 248-2, 248-4, etc.), in which ontology terms are described with non expressive texts. This is an \internal" exercise, which does not let us direct comparisons with other methods in the contest, but gives us another point of view (more accurate, according to the nal usage of our system) of the behaviour of our method. Results are given in Table 2. The objective of this experiment is to match terms from plain hierarchies, extracted from web directories. It consist of more than 4 thousand elementary alignments. We consider that our method cannot show all its strengths in this, because the available information is extremely sparse, lacking in semantic descriptions beyond hierarchical relationships (no instances, no properties, no comments, no synonyms, ...).
Results have been: 60% precision, 38% recall and 47% F-measure, which has been the second best result in this year competition (out of seven participants). A detailed comparison has been published by organizers3. We see that, even directory alignment is not the target of our system, it behaves reasonably well when matching plain hierarchies. 3
The following subsections contain some remarks and comments about the results obtained, as well as about the test cases and evaluation process. 3.1
As expected, we obtained better precision than recall in the benchmark test (due to the reasons mentioned in Section 2.1). Also in the directory experiment precision was higher than recall. However, it is consistent with the fact that our alignment is targeted to be used in an automatic way, minimizing human intervention. In this conditions, precision have to be promoted over recall. That is, maybe our system does not discover all correspondences, but we have to be sure that, in case it discovers an equivalence mapping between two terms, they 3 http://www.disi.unitn.it/ pane/OAEI/2008/directory/result/ are most likely referring to the same meaning. Otherwise their later integration would be erroneous, and the mistake would eventually be propagated to the other steps of the system. 3.2
Our method does not consider extensional information when comparing concepts, focusing only on the semantic description of the terms in the corresponding ontologies. Its inclusion could improve results in some cases where this information is available.
Additionally, although our system considers many features of ontologies, their
richness vary a lot from one case to another. We consider that the addition of
mechanisms to auto-adjust weights to the characteristics of ontologies (as they
do in [
Finally, although our similarity measure has been much optimized, in terms of time response, the overall alignment process can still be subject of further improvement. 3.3
We have found the benchmark test very useful as a guideline for our internal improvements of the method, as well as to establish a certain degree of comparisons with other existing methods.
On the other hand, we have missed some important issues that are not taken
into account in the systematic benchmark series:
1. Benchmark tests only consider positive matchings, not measuring the ability
of di erent methods to avoid links among barely related ontologies (only
case 102 of benchmark goes in that direction).
2. For our purposes, we try to emulate the human behaviour when mapping
ontological terms. As human experts cannot properly identify mappings
between ontologies with scrambled texts, neither does our system. However,
reference alignments provided in the benchmark evaluation for cases 202
and 248-266, do not follow this intuition. We hope this bias will be reduced
in future contests.
3. Related to the latter, cases in which equal topologies, but containing di erent
semantics, lead to false positives, are not explicitly taken into account in the
benchmark.
4. How ambiguities can a ect the method is not considered either in the test
cases. It is a consequence of using ontologies belonging to the same domain.
For example, it would be interesting to evaluate how \ lm" in an ontology
about movies, is mapped to \ lm" as a \thin layer" in another ontology.
Therefore it is di cult to evaluate the bene ts of including certain
disambiguation techniques in ontology matching [
We have presented here some results of our rst participation in the OAEI 2008 contest. We have limited to two tracks: benchmark and directory, but we hope to extend our participation in the future.
Our schema-based alignment algorithm compares the ontological contexts of each pair of terms (enriched with transitive inference) by combining di erent elementary ontology matching techniques (comparing vocabulary, taxonomies, relations,...). Benchmark results show a very good behaviour of our system in terms of precision, while keeping an acceptable recall. It con rms the validity of the measure we have conceived, and its suitability to be applied in ontology matching tasks. It encourages us to tackle further improvements, and to extend the scope and applicability of our techniques.
We have also included, based on our experience, some considerations about the nature of benchmark test cases that, in our opinion, could help improving future contests.
Acknowledgments. This work is supported by the CICYT project TIN200768091-C02-02.