The Ontology Alignment Evaluation Initiative1 (OAEI) is an international campaign for the systematic evaluation of ontology matching systems -software programs capable of finding correspondences (called alignments) between the vocabularies of a given set of input ontologies [22,7,9,23]. The matching problems in the OAEI are organised in several tracks, with each track involving different kinds of test ontologies [7]. The ontologies in the largest test case in the OAEI 2011 contain only 2,000-3,000 classes; however, ontology matching tools have significantly improved in the last few years and there is a need for more challenging and realistic matching problems for which suitable reference alignments exist [22,7].
UMLS-Metathesaurus (UMLS) [1] is currently the most comprehensive effort for integrating medical thesauri and ontologies, including the National Cancer Institute Thesaurus (NCI) [12,11], the Foundational Model of Anatomy (FMA) [19] and the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) [24], which are large-scale and semantically rich ontologies. NCI, FMA and SNOMED CT are gradually superseding the existing medical classifications and are becoming core platforms for accessing, gathering, and sharing biomedical knowledge and data. Hence, matching such large ontologies represents a very interesting challenge for the OAEI initiative.
In this paper we describe how the UMLS correspondences between NCI, FMA and SNOMED CT have been used as reference alignments for the new Large BioMed track2 in the OAEI initiative. Furthermore we present the results obtained in the OAEI 2011.5 campaign for this track and we propose a new reference alignment based on the harmonisation of the outputs of the participating ontology matching systems. Ontology alignments are often conceptualised as tuples with the form id, e 1 , e 2 , n, ρ , where id is a unique identifier for the mapping, e 1 , e 2 are entities in the vocabulary of the integrated ontologies, n is a numeric confidence measure between 0 and 1, and ρ is a relation between e 1 and e 2 , typically subsumption (i.e., e 1 is more specific than e 2 ) and equivalence (i.e., e 1 and e 2 are synonyms) [8]. The OAEI initiative uses an RDF format to represent the alignments3 [6] containing the aforementioned elements. Alternatively, OAEI alignments are also represented as OWL 2 subclass and equivalence axioms with the mapping identifier (id) and confidence (n) added as OWL 2 annotation axioms [4].
Although the standard UMLS distribution does not directly provide sets of alignments (in the OAEI sense) between the integrated ontologies, it is relatively straightforward to extract alignment sets from the information provided in the distribution files [15]. Concretely, we have processed the MRCONSO4 file, which contains every entity in UMLS together with its concept unique identifier (CUI), its source vocabulary (e.g. FMA), its language (e.g. English), and other attributes not relevant for the OAEI. Table 1 shows an excerpt from the MRCONSO file associated to the notion of "Joint".
It follows from Table 1 that the notion of "Joint" is shared by FMA, SNOMED CT and NCI. In particular, FMA contains the entities Joint and Set of joints, NCI the entities Articulation and Joint, and SNOMED CT only the entity Joint structure. All these entities have been annotated with the same CUI C0022417 and therefore, according to UMLS's intended meaning, they are synonyms. Then, for each pair of entities e1 and e2 from different sources and annotated with the same CUI, we have generated the corresponding (equivalence) UMLS-based alignments with a confidence value of 1.0 (see Table 2). The integration of new resources in UMLS combines expert assessment and sophisticated auditing protocols [1,3,10]. However, it has been noticed that UMLS-based alignments lead to a large number of unsatisfiable classes if they are represented as OWL 2 axioms and integrated with the input ontologies [15,14]. For example the integration of SNOMED CT and NCI via UMLS-based alignments leads to more than 20,000 unsatisfiable classes. To address this problem, we have presented in [14] a refinement of the (original) UMLS-based alignments that do not lead to (many) unsatisfiable classes (see Table 3). This refinement is based on the alignment repair module of the ontology matching system LogMap [14,16].
3 Results of the Large BioMed track in the OAEI 2011.5
In this section we briefly present the obtained results in the Large BioMed track of the OAEI 2011.5 campaign. 5 We have only evaluated the FMA-NCI matching problem, where the used versions of FMA and NCI contains 78,989 and 66,724 classes, respectively. The original and refined UMLS-based alignments (see Table 3) has been used as reference to evaluate the efficiency of participating ontology matching systems.
Table 4 summarizes the obtained results where systems has been ordered according to the F-measure against the refined UMLS-based reference alignment. LogMapLt -a simple ontology matcher-has been used as a base-line. Besides precision (P), recall (R), F-measure (F) and runtimes we have also evaluated the coherence of the alignments when reasoning together with the input ontologies. 6 Note that we have evaluated GOMMA [17] with two different configurations. GOMMA bk uses UMLS-based background knowledge, while GOMMA nobk has this feature deactivated.
GOMMA (with its two configurations) and LogMap are a bit ahead in terms of Fmeasure with respect to Aroma [5], CSA [25] and MapSSS [2], which could not top the results of the base-line LogMapLt. GOMMA bk obtained the best results in terms of recall, while LogMap provided the best results in terms of precision and F-measure. The use of the original UMLS-based reference alignment did not imply important variations. Since the original set contains more mappings, precision and recall slightly increases and decreases, respectively. It is worth mentioning, however, that GOMMA bk improves its results when comparing with the original UMLS-based reference alignment and provides the best F-measure.
Regarding mapping coherence, only LogMap generated an 'almost' clean output in all three tasks. Although GOMMA nobk also provides highly precise output correspondences, they lead to a huge amount of unsatisfiable classes.
The original UMLS-based reference alignment, as shown in Section 2, contains errors (i.e. lead to large number of unsatisfiable classes when integrated with the input ontologies). On the other hand, the refined UMLS-based reference alignment is based on the (incomplete) alignment repair techniques of the ontology matching systems LogMap [14,16], which may fail to detect and discard the appropriate alignments. Thus, in order to turn the extracted UMLS-based reference alignments into an agreed-upon gold standard expert assessment would be needed, which is almost unfeasible for large alignment sets. We have opted to move towards a silver standard by harmonising the outputs of different matching tools over the relevant ontologies. Similar silver standards have been developed for named entity recognition problems [21,13].
We have harmonised the outputs of the systems participating in the OAEI 2011.5 FMA-NCI matching problem. Each system has been associated a weighted vote based on its precision w.r.t. the refined UMLS-based reference alignment (see Table 4). For example, LogMap and MapSSS have been associated the weights 0.868 and 0.426, respectively. Note that systems participating with two versions (e.g. GOMMA and LogMap) have been only considered once in the voting process.
Figure 1 summarises the evolution of the F-measure, Precision and Recall for the harmonised alignment depending on the minimum required votes. For example the harmonised alignment set requiring 4.0 points of weighted votes has a precision of 0.971 and a recall of 0.369 w.r.t. the refined UMLS-based reference alignment. As expected precision increases and recall decreases as the required votes increase.
We have selected the harmonised alignment set with the highest F-measure (0.91) as the "first" silver standard of the FMA-NCI matching problem. This set contains 2,890 alignments that have been "at least" voted by two systems with weight 0.90. Note that this harmonised alignment has not been yet refined and it is known to lead to more than 14,000 unsatisfiable classes when integrated with FMA and NCI.
In the OAEI 2012 campaign 7 we also intend to evaluate the SNOMED-NCI and FMA-SNOMED matching problems using the correspondent UMLS-based reference alignments (see Table 3). We will also create harmonised silver standards alignments and we will evaluate the participating systems against them. This comparison will be very useful to analyse how different a system is with respect to the others.
Finally, we also intend to combine different reasoning and diagnosis tools such as ALCOMO8 [18] to generate error-free refinements of both the UMLS-based reference alignments and the harmonised silver standards.
This work was supported by the Royal Society, the EU FP7 project SEALS and by the EPSRC projects ConDOR, ExODA, LogMap and Score!. We also thank the organisers of the OAEI evaluation campaigns for providing test data and infrastructure.