In this paper we describe how the UMLS Metathesaurus-the most comprehensive effort for integrating medical thesauri and ontologies-is being used within the context of the Ontology Alignment Evaluation Initiative (OAEI). We also present the obtained results in the Large BioMed track of the OAEI 2011.5 campaign where the reference alignments are based on UMLS. Finally, we propose a new reference alignment based on the harmonisation of the outputs of the systems participating in the OAEI Large BioMed track.
The Ontology Alignment Evaluation Initiative1 (OAEI) is an international campaign for
the systematic evaluation of ontology matching systems —software programs capable
of finding correspondences (called alignments) between the vocabularies of a given set
of input ontologies [
UMLS-Metathesaurus (UMLS) [
In this paper we describe how the UMLS correspondences between NCI, FMA and SNOMED CT have been used as reference alignments for the new Large BioMed track2 in the OAEI initiative. Furthermore we present the results obtained in the OAEI 2011.5 campaign for this track and we propose a new reference alignment based on the harmonisation of the outputs of the participating ontology matching systems.
Ontology alignments are often conceptualised as tuples with the form hid; e1; e2; n; i,
where id is a unique identifier for the mapping, e1; e2 are entities in the vocabulary of
the integrated ontologies, n is a numeric confidence measure between 0 and 1, and is a
relation between e1 and e2, typically subsumption (i.e., e1 is more specific than e2) and
equivalence (i.e., e1 and e2 are synonyms) [
Although the standard UMLS distribution does not directly provide sets of
alignments (in the OAEI sense) between the integrated ontologies, it is relatively
straightforward to extract alignment sets from the information provided in the distribution files
[
It follows from Table 1 that the notion of “Joint” is shared by FMA, SNOMED CT and NCI. In particular, FMA contains the entities J oint and Set of joints, NCI the entities Articulation and J oint, and SNOMED CT only the entity J oint structure. All these entities have been annotated with the same CUI C0022417 and therefore, according to UMLS’s intended meaning, they are synonyms. Then, for each pair of entities e1 and e2 from different sources and annotated with the same CUI, we have
generated the corresponding (equivalence) UMLS-based alignments with a confidence value of 1:0 (see Table 2).
The integration of new resources in UMLS combines expert assessment and
sophisticated auditing protocols [
In this section we briefly present the obtained results in the Large BioMed track of the OAEI 2011.5 campaign.5 We have only evaluated the FMA-NCI matching problem, where the used versions of FMA and NCI contains 78,989 and 66,724 classes, respectively. The original and refined UMLS-based alignments (see Table 3) has been used as reference to evaluate the efficiency of participating ontology matching systems.
Table 4 summarizes the obtained results where systems has been ordered according to the F-measure against the refined UMLS-based reference alignment. LogMapLt —a simple ontology matcher—has been used as a base-line. Besides precision (P), recall (R), F-measure (F) and runtimes we have also evaluated the coherence of the alignments when reasoning together with the input ontologies.6 Note that we have evaluated
6 We have used the OWL 2 reasoner HermiT [
GOMMA (with its two configurations) and LogMap are a bit ahead in terms of
Fmeasure with respect to Aroma [
Regarding mapping coherence, only LogMap generated an ‘almost’ clean output in all three tasks. Although GOMMAnobk also provides highly precise output correspondences, they lead to a huge amount of unsatisfiable classes. 4
The original UMLS-based reference alignment, as shown in Section 2, contains errors
(i.e. lead to large number of unsatisfiable classes when integrated with the input
ontologies). On the other hand, the refined UMLS-based reference alignment is based on the
(incomplete) alignment repair techniques of the ontology matching systems LogMap
[
We have harmonised the outputs of the systems participating in the OAEI 2011.5 FMA-NCI matching problem. Each system has been associated a weighted vote based on its precision w.r.t. the refined UMLS-based reference alignment (see Table 4). For example, LogMap and MapSSS have been associated the weights 0.868 and 0.426, respectively. Note that systems participating with two versions (e.g. GOMMA and LogMap) have been only considered once in the voting process.
Figure 1 summarises the evolution of the F-measure, Precision and Recall for the harmonised alignment depending on the minimum required votes. For example the harmonised alignment set requiring 4.0 points of weighted votes has a precision of 0.971 and a recall of 0.369 w.r.t. the refined UMLS-based reference alignment. As expected precision increases and recall decreases as the required votes increase.
We have selected the harmonised alignment set with the highest F-measure (0.91) as the “first” silver standard of the FMA-NCI matching problem. This set contains 2,890 alignments that have been ”at least” voted by two systems with weight 0.90. Note that this harmonised alignment has not been yet refined and it is known to lead to more than 14,000 unsatisfiable classes when integrated with FMA and NCI. 5
In the OAEI 2012 campaign7 we also intend to evaluate the SNOMED-NCI and FMASNOMED matching problems using the correspondent UMLS-based reference alignments (see Table 3). We will also create harmonised silver standards alignments and we will evaluate the participating systems against them. This comparison will be very useful to analyse how different a system is with respect to the others.
Finally, we also intend to combine different reasoning and diagnosis tools such as
ALCOMO8 [
This work was supported by the Royal Society, the EU FP7 project SEALS and by the EPSRC projects ConDOR, ExODA, LogMap and Score!. We also thank the organisers of the OAEI evaluation campaigns for providing test data and infrastructure.