Implementing Semantic Precision and Recall Daniel Fleischhacker and Heiner Stuckenschmidt University of Mannheim, Mannheim, Germany dfleisch@mail.uni-mannheim.de, heiner@informatik.uni-mannheim.de 1 Introduction The systematic evaluation of ontology alignments still faces a number of problems. One is the argued inadequacy of traditional quality measures adopted from the field of information retrieval. In previous work, Euzenat and others have proposed notions of semantic precision and recall that are supposed to better reflect the true quality of an alignment by considering the deductive closure of a mapping rather than the explicitly stated correspondences. So far, these measures have been mostly investigated in theory. In this paper, we present the first implementation of a restricted version of semantic precision and recall as well as experiments in using it, we conducted on the results of the 2008 OAEI campaign. 2 Restricted Semantic Precision and Recall In this work, we treat alignments as sets of correspondences whereas correspondences give a relation between two entities from different ontologies. To evaluate alignments, we use the notion of aligned ontologies. An aligned ontology is made of the two ontolo- gies which are referenced by an alignment and the correspondences contained in this alignment added into the aligned ontology as axioms. To convert correspondences into axioms, we use semantics as the natural and pragmatic semantics given by Meilicke and Stuckenschmidt [3]. The basis of our work is the work of Euzenat [2] which we adapted to our different understanding of alignment semantics. The basic notion given by Eu- zenat and used here is the notion of α-consequences. These are correspondences which are implied by an aligned ontology given specific semantics. For ontologies O1 and O2 , a corresponding alignment A and reductionistic semantics S, we say A SO1 ,O2 c if c is an α-consequence. Applying this definition to complete alignments instead of single correspondences, we get the closure of an alignment which resembles the sets of α-correspondences used by Euzenat. For given ontologies O1 , O2 and a reductionistic semantics S the closure Cn of an alignment A is given by CnSO1 ,O2 (A) = {c | A SO1 ,O2 c}. We introduce a restricted variant of ideal semantic precision and recall which does not suffer from the problems of the ideal semantic precision and recall mentioned by Euzenat [2] and also prevent problems examined by David and Euzenat [1]. For this purpose, we call alignments non-complex if they contain only correspondences whose entities refer to single atomic concepts of the ontologies. Definition 1 (Restricted Semantic Precision and Recall). Given consistent ontolo- gies O1 and O2 , two non-complex alignments between these two ontologies, namely the reference alignment R and the alignment A which is to be evaluated, and a reduc- tionistic semantics S. Further, let the aligned ontologies of the two ontologies with A resp. R be consistent. Restricted semantic precision and recall are defined as |CnS S O1 ,O2 (A) ∩ CnO1 ,O2 (R)| |CnS S O1 ,O2 (A) ∩ CnO1 ,O2 (R)| Pr (A, R) = S resp. Rr (A, R) = S |CnO1 ,O2 (A)| |CnO1 ,O2 (R)| 3 First Results 0.2 0.5 0.7 We applied the mea- Matcher Semantics P R P R P R sures to two different test none 0.42 0.42 0.7 0.18 0.81 0.09 ASMOV natural 0.39 0.69 0.81 0.26 1.0 0.15 sets taken from the OAEI pragmatic 0.49 0.74 0.85 0.23 1.0 0.13 test sets. In the following, none 0.49 0.52 0.49 0.52 0.49 0.52 DSSim natural 0.15 0.83 0.15 0.83 0.15 0.83 we only present the re- pragmatic 0.23 0.88 0.23 0.88 0.23 0.88 sults generated for the con- none 0.5 0.36 0.54 0.21 0.66 0.07 ference test set of the OAEI Lily natural 0.45 0.46 0.65 0.24 0.74 0.09 pragmatic 0.48 0.51 0.66 0.22 0.65 0.07 2008. We evaluated the align- ments provided by the de- Table 1. Aggregated precision (P) and recall (R) results of conference test set com- velopers of the ontology paring classical precision and recall (no semantics), natural and pragmatic precision and recall; top-most line gives minimum confidence value (threshold) to consider a matchers. Aggregated re- correspondence sults for the conference set are presented in Table 1. The aggregation is done using the average of all values for a specific measure which are neither an error entry nor have the value ,,nan”. 4 Conclusion Our results show that taking the semantics of the model into account can make a differ- ence in judging the quality of matching systems not only in theory but also in practice. So far, this effect is rather limited, which is mainly due to the fact that most generated alignments as well as reference alignments only consist of equivalence statements. It is clear, however, that future work will also strongly focus on generating mappings other than equivalence mappings. Further, there is an ongoing effort to extend existing ref- erence alignments with subsumption correspondences. In such an extended setting, the effect of the semantic measures will be even higher and our system will show its real potential for improving ontology mapping evaluation. References 1. Jérôme David and Jérôme Euzenat. On fixing semantic alignment evaluation measures. In Proceedings of the ISWC 2008 Workshop on Ontology Matching, pages 25–36, 2008. 2. Jérôme Euzenat. Semantic precision and recall for ontology alignment evaluation. In Pro- ceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI), pages 348–353, 2007. 3. Christian Meilicke and Heiner Stuckenschmidt. An Efficient Method for Computing a Local Optimal Alignment Diagnosis. Technical report, University of Mannheim, 2009.