Is my ontology matching system similar to yours? ⋆ Ernesto Jiménez-Ruiz, Bernardo Cuenca Grau, Ian Horrocks Department of Computer Science, University of Oxford, Oxford UK Abstract. In this paper we extend the evaluation of the OAEI 2012 Large BioMed track, which involves the matching of the semantically rich ontologies FMA, NCI and SNOMED CT. Concretely, we report about the differences and similarities among the mappings computed by the participant ontology matching systems. 1 Introduction The quality of the mappings computed by an ontology matching system in the Ontology Alignment Evaluation Initiative (OAEI) [2, 1] is typically measured in terms of preci- sion and recall with respect to a reference set of mappings. Additionally, the OAEI also evaluates the coherence of the computed mappings [1]. However, the differences and similarities among the mappings computed by differ- ent systems have often been neglected in the OAEI.1 In this paper we provide a more fine-grained comparison among the matching systems participating in the OAEI 2012 Large BioMed track;2 concretely (i) we have harmonised (i.e. voted) the computed map- ping sets, and (ii) we provide a graphical representation of the similarity of these sets. 2 Mapping harmonization We have considered the mappings voted (i.e. included in the output) by at least one ontology matching system. Figure 1 shows the harmonization (i.e. voting) results for the FMA-NCI and FMA-SNOMED matching problems. Mappings have received at most 11 and 8 votes (i.e. number of participating systems3 ), respectively. For example, in the FMA-NCI matching problem, 3,719 mappings have been voted by at least 2 systems. Figure 1 also shows the evolution of F-score, Precision and Recall for the different harmonized mapping sets. As expected the maximum recall (respectively precision) is reached with the minimum (respectively maximum) number of votes. For example, the maximum recall in the FMA-SNOMED problem is 0.81, which shows the difficulty of identifying correct mappings in this matching problem. The harmonized mapping sets with the best trade-off between precision and recall have been selected as the representative mapping sets of the participating ontology matching systems. For the FMA-NCI matching problem we have selected the mappings sets with (at least) 3, 4 and 5 votes, while in the FMA-SNOMED matching problem we have selected the sets with (at least) 2 and 3 votes (see dark-grey bars in Figure 1). ⋆ This research was financed by the Optique project with the grant agreement FP7-318338. 1 As far as we know, only in the 2007 Anatomy track some effort was done in this line: http: //oaei.ontologymatching.org/2007/results/anatomy/ 2 Results available at: http://www.cs.ox.ac.uk/isg/projects/SEALS/oaei/ 3 Systems with several variants have only been considered once in the voting process. Size Size 1.2 Precision 1.2 Precision Recall Recall F-measure F-measure 17,020 1 11,160 1 Number of mappings Number of mappings 0.8 0.8 Score Score 0.6 0.6 7,997 0.4 0.4 6,592 3,719 3,074 2,862 4,218 2,739 2,563 2,420 2,292 0.2 2,077 0.2 2,846 1,630 2,052 1,610 812 1,002 0 0 1 2 3 4 5 5 7 8 9 10 11 1 2 3 4 5 5 7 8 Number of votes Number of votes Fig. 1: Harmonisation in the FMA-NCI (left) and FMA-SNOMED (right) problems 0.60 0.40 GOMMA-Bk LogMap 0.50 LogMap-noe 0.30 UMLSL 0.40 GOMMA UMLSA UMLS Jaccard distance Jaccard distance ServOMapL YAM++ ServOMap 0.30 0.20 Vote5 Vote4 UMLSL Vote3 Vote2 UMLSA GOMMA 0.20 UMLS GOMMA-Bk YAM++ 0.10 ServOMapL Vote3 0.10 ServOMap LogMap-noe LogMap LogMapLt LogMapLt 0 0 0 0.10 0.20 0.30 0.40 0 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 Jaccard distance Jaccard distance Fig. 2: Mapping similarity in the FMA-NCI (left) and FMA-SNOMED (right) problems 3 Mapping similarity among systems We have compared the similarity among (i) the representative mapping sets from the harmonisation (see Section 2), (ii) the UMLS-based reference mappings of the track, and (iii) the mapping sets computed by the top-8 ontology matching systems in the FMA-NCI and FMA-SNOMED matching problems [1]. To this end we have calculated the jaccard distance (|MA ∪ MB | − |MA ∩ MB |)/|MA ∪ MB |, which ranges from 0 (the same) to 1 (different), between each pair (MA and MB ) of the mapping sets from (i)-(iii), and represented such distances in a two-dimensional scatterplot (see Figure 2). System names which are distant to each other indicate that their computed mappings differ to a large degree. For example, in Figure 2 (right), the mappings computed by LogMapLt and GOMMA are very different with respect to the mappings computed by other systems, as well as with respect to the harmonized and reference mapping sets. References 1. Aguirre, J., et al.: Results of the Ontology Alignment Evaluation Initiative 2012. In: Ontology Matching Workshop. Vol-946 of CEUR Workshop Proceedings (2012) 2. Euzenat, J., Meilicke, C., Stuckenschmidt, H., Shvaiko, P., Trojahn, C.: Ontology alignment evaluation initiative: Six years of experience. J. Data Sem. 15, 158–192 (2011)