Implementing Semantic Precision and Recall

Abstract

The systematic evaluation of ontology alignments still faces a number of problems. One is the argued inadequacy of traditional quality measures adopted from the field of information retrieval. In previous work, Euzenat and others have proposed notions of semantic precision and recall that are supposed to better reflect the true quality of an alignment by considering the deductive closure of a mapping rather than the explicitly stated correspondences. So far, these measures have been mostly investigated in theory. In this paper, we present the first implementation of a restricted version of semantic precision and recall as well as experiments in using it, we conducted on the results of the 2008 OAEI campaign.

Introduction

In this work, we treat alignments as sets of correspondences whereas correspondences give a relation between two entities from different ontologies. To evaluate alignments, we use the notion of aligned ontologies. An aligned ontology is made of the two ontologies which are referenced by an alignment and the correspondences contained in this alignment added into the aligned ontology as axioms. To convert correspondences into axioms, we use semantics as the natural and pragmatic semantics given by Meilicke and Stuckenschmidt [ 3 ]. The basis of our work is the work of Euzenat [ 2 ] which we adapted to our different understanding of alignment semantics. The basic notion given by Euzenat and used here is the notion of -consequences. These are correspondences which are implied by an aligned ontology given specific semantics. For ontologies O1 and O2, a corresponding alignment A and reductionistic semantics S, we say A SO1;O2 c if c is an -consequence.

Applying this definition to complete alignments instead of single correspondences, we get the closure of an alignment which resembles the sets of -correspondences used by Euzenat. For given ontologies O1, O2 and a reductionistic semantics S the closure Cn of an alignment A is given by CnSO1;O2 (A) = fc j A SO1;O2 cg.

We introduce a restricted variant of ideal semantic precision and recall which does not suffer from the problems of the ideal semantic precision and recall mentioned by Euzenat [ 2 ] and also prevent problems examined by David and Euzenat [ 1 ]. For this purpose, we call alignments non-complex if they contain only correspondences whose entities refer to single atomic concepts of the ontologies. 3

First Results

0.2 0.5 0.7 We applied the meaMatcher Semantics P R P R P R sures to two different test ASMOV nnaotunreal 00..3492 00..6492 00..871 00..1286 01..801 00..0195 sets taken from the OAEI pragmatic 0.49 0.74 0.85 0.23 1.0 0.13 test sets. In the following, DSSim nnaotunreal 00..1459 00..8532 00..4195 00..8532 00..1459 00..5823 we only present the repragmatic 0.23 0.88 0.23 0.88 0.23 0.88 sults generated for the conLily nnaotunreal 00..455 00..3466 00..6554 00..2214 00..6764 00..0097 ference test set of the OAEI pragmatic 0.48 0.51 0.66 0.22 0.65 0.07 2008. We evaluated the alignments provided by the depTaarbinleg1c.laAssgigcraelgparteecdispiroenciasniodnre(Pca)laln(ndoresceamllan(Rti)csr)e,snualttsuroafl caonndfeprreangcmeatteisctpsreetccisoimon- velopers of the ontology and recall; top-most line gives minimum confidence value (threshold) to consider a matchers. Aggregated recorrespondence sults for the conference set are presented in Table 1. The aggregation is done using the average of all values for a specific measure which are neither an error entry nor have the value ,,nan”. 4

Conclusion

Our results show that taking the semantics of the model into account can make a difference in judging the quality of matching systems not only in theory but also in practice. So far, this effect is rather limited, which is mainly due to the fact that most generated alignments as well as reference alignments only consist of equivalence statements. It is clear, however, that future work will also strongly focus on generating mappings other than equivalence mappings. Further, there is an ongoing effort to extend existing reference alignments with subsumption correspondences. In such an extended setting, the effect of the semantic measures will be even higher and our system will show its real potential for improving ontology mapping evaluation.

References

1. Je´roˆme David and Je´roˆme Euzenat. On fixing semantic alignment evaluation measures . In Proceedings of the ISWC 2008 Workshop on Ontology Matching , pages 25 - 36 , 2008 .

2. Je´roˆme Euzenat. Semantic precision and recall for ontology alignment evaluation . In Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI) , pages 348 - 353 , 2007 .

3. Christian Meilicke and Heiner Stuckenschmidt . An Efficient Method for Computing a Local Optimal Alignment Diagnosis . Technical report , University of Mannheim, 2009 .