AgreementMakerLight (AML) is an automated ontology matching framework based on element-level matching and the use of external resources as background knowledge. This paper describes the configuration of AML for the OAEI 2013 competition and discusses its results. Being a newly developed and still incomplete system, our focus in this year's OAEI were the anatomy and large biomedical ontologies tracks, wherein background knowledge plays a critical role. Nevertheless, AML was fairly successful in other tracks as well, showing that in many ontology matching tasks, a lightweight approach based solely on element-level matching can compete with more complex approaches.
1.1
AgreementMakerLight (AML) is an automated ontology matching framework derived
from the AgreementMaker system [
The key design principles of AML were efficiency and simplicity, although
flexibility and extensibility—which are key features of AgreementMaker—were also high on
the list [
AML is primarily focused on lexically rich ontologies in general and on life sciences ontologies in particular, although it can be adapted to many other ontology matching tasks, thanks to its flexible and extensible framework. However, due to its short development time (eight months), it does not include components for instance matching or translation yet, and thus cannot handle all ontology matching tasks. 1.2
The AML workflow for the OAEI 2013 can be divided into six steps, as shown in Fig. 1: ontology loading, baseline matching and profiling, background knowledge matching (optional), extension matching and selection, property matching (conditional), and repair (optional).
Ontology Loading In the ontology loading step, AML reads and processes each of the input ontologies and stores the information necessary for the subsequent steps in its own data structures.
First, AML reads the localName, labels and synonym properties of all classes,
normalizes them, and enters them into the Lexicon [
Note that AML currently does not store or use comments, definitions, or instances.
Baseline Matching and Profiling In the baseline matching and profiling step, AML
employs an efficient weighted string-equivalence algorithm, the Lexical Matcher [
Regarding size, AML divides matching problems into three size categories (small, medium or large), which will affect decisions and thresholds during the background knowledge matching and the extension matching and selection steps.
Regarding cardinality, AML also considers three categories (near-one, medium and high), which will determine how selection is performed during the extension matching and selection step.
As for the property/class ratio, it determines whether AML will match properties during the property matching step.
Background Knowledge Matching For the OAEI 2013, AML employs three sources
of background knowledge: Uberon [
The UMLS Matcher uses a pre-processed version of the MRCONSO table from the UMLS Metathesaurus (in a text file). Each input ontology is matched against the whole UMLS table, then AML decides whether to use a single UMLS source (by comparing the coverage of all sources) or the whole table. When UMLS is a good fit for the matching problem, its alignment is used exclusively, and the extension matching and selection step is skipped. Otherwise, if it is a reasonable fit, its alignment is merged with the baseline alignment.
The WordNet Matcher queries the WordNet database for synonyms of each name in
the Lexicons of the input ontologies, using the Jaws API CITATION. These synonyms
are used to create temporary extended Lexicons, which are matched with the Lexical
Matcher. Because WordNet is prone to induce errors, AML uses it only to extend the
baseline alignment, meaning that it matches only previously unmatched classes.
Extension Matching and Selection The extension matching and selection step
comprises two matching sub-steps that alternate with two selection sub-steps. First, AML
employs a word-based similarity algorithm, the Word Matcher [
When the matching problem is profiled as ’large’, the Word Matcher is skipped because
it is too memory intensive to be used globally, and its local use is subsumed by that of
the Parametric String Matcher [
In the interactive matching track, AML employs an interactive selection algorithm,
which asks the user for feedback about mappings in case of conflict or below a given
similarity threshold, until a given number of negative answers is reached.
Property Matching In the property matching step, AML matches the ontology
properties. AML compares the properties’ types, domains and ranges, looking for mappings
in the class alignment when the domains/ranges are classes. Then, if the properties have
attributes in common, AML measures the word-based similarity between their names
(as per the Word Matcher [
Repair In the repair step, AML employs a heuristic repair algorithm [
The AML system and the alignments it produced for the OAEI 2013 are available at the SOMER project page (http://somer.fc.ul.pt/). 2 2.1
AML had a very high precision (100%) but a fairly low recall (40%) in the Benchmark track, returning empty alignments in several of the tests. This is a consequence of AML’s simple framework, which is exclusively based on element-level matching and does not handle instances. Nevertheless, it was interesting to note that AML had the highest F-measure/time ratio, which attests to its efficiency. 2.2
The AML Anatomy results are shown in Table 1. AML ran in this track both with and without background knowledge (AML-BK and AML respectively). In the case of this track, AML-BK selects Uberon exclusively as the source of background knowledge, and uses it for Lexicon extension. Thus, the only difference between AML and AMLBK is that the latter has Lexicons enriched with Uberon synonyms.
The results of AML-BK were very good, with a fairly high precision, and the highest recall, F-measure and recall+ in this year’s evaluation. However, the AML results without background knowledge were also good, ranking fourth overall in F-measure, and second if we exclude the systems using Uberon. In fact, we believe that the AML results are near-optimal for a strategy based solely on element-level matching, and that background knowledge is required to obtain substantial improvements. The impact and quality of the Uberon cross-references is clear when we note that AML-BK gained 10% recall over AML without any loss in precision. Finally, it is also noteworthy that AML was one of only two systems to produce coherent alignments. The AML Conference results with reference alignment 1 are shown in Table 2 (the results with reference alignment 2 are slightly worst for all systems, but do not affect their ranking). AML ran in this track with and without background knowledge, with AML-BK using WordNet as the only source of background knowledge (to match both classes and properties).
The results of both AML-BK and AML were good, having the highest precision of this year’s evaluation and ranking second and tied for third in terms of F-measure, respectively. An important part of the success of AML in this task was the property matching algorithm, which found 9 and 11 property mappings with 100% precision, with and without background knowledge respectively. 2.4
As we expected, the performance of AML in the Multifarm track was poor, with Fmeasures of only 4% and 3% when comparing different ontologies and the same ontologies respectively. Participation in this track was beyond our scope, as AML does not handle translations or employ structural-level matching, which are essential for success in this track. 2.5
The results of AML in the Library track were reasonable, as it ranked 4th in terms of F-measure (with 73%) and had the second highest recall of this year’s OAEI (87.7%). Nevertheless, there is clearly room for improvement regarding precision, which was significantly lower than that of other top systems (62.5%) likely due to the fact that AML does not take the language of labels into account. Indeed, the results of AML were very similar to the MatcherAllLabels benchmark. 2.6
The AML Interactive Matching results are shown in Table 3. AML ran with the same configurations used in the Conference track, except that in this track the selection algorithm employed is interactive, rather than automatic.
The results show that AML’s interactive selection algorithm was effective, gaining both precision and recall in comparison with the conference results. Nevertheless, this algorithm is far from optimized, and it should be possible to reduce the number of user interactions without sacrificing F-measure. 2.7
The AML Large Biomedical Ontologies results are shown in Table 4. AML ran in this track with six different configurations: without background knowledge (AML); with background knowledge (AML-BK); with specialized background knowledge (AMLSBK); and in all three cases with (-R) and without repair. AML-BK selects Uberon in all six tasks of this track (although never for Lexicon extension) and selects WordNet only in the SNOMED-NCI small task. AML-SBK is given access to UMLS, and selects it exclusively for all six tasks.
Our goals in testing all these configurations were: to assess the impact of using domain background knowledge both unrelated (Uberon) and directly related (UMLS) to the reference alignments; to assess the effect of using repair on the quality of the results; and to contribute to improve the quality of the reference alignments.
The results of AML-SBK were very good, with a marked advantage over all other systems in this year’s evaluation. This is unsurprising given that AML-SBK derived its alignments from UMLS using an automatic strategy that is likely analogous to that used to build the reference alignments in the first place. This evidently gives AMLSBK an advantage over systems that do not use UMLS. Note, however, that the strategy employed by AML is a general-purpose strategy for reusing preexisting mappings and cross-references, which is used for both UMLS and Uberon. The only issue is that the reference alignments were also automatically derived from UMLS, which makes the evaluation of AML-SBK positively biased.
The results of AML-BK were also good, ranking second overall in recall and F-measure
if we exclude the systems that used UMLS. However, in this case the evaluation of
AML-BK is negatively biased by the reference alignments. The reason for this is that
AML-BK uses Uberon, and many of the mappings derived from Uberon are not present
in UMLS despite being correct. This is particularly evident in the FMA-NCI matching
problem with whole ontologies, where the contribution of Uberon (based on
crossreferences which are manually curated) was approximately neutral, decreasing the
precision as substantially as it increased the recall (in relation to AML). Perhaps extending
the reference alignments by compiling mappings from multiple reliable data sources
such as Uberon could enable a fairer evaluation of the systems competing in this track,
and make the tasks less trivial for systems using background knowledge.
The use of repair led to clearly more coherent alignments, as all AML configurations
with repair obtained very low degrees of unsatisfiability. However, in terms of quality
of the results, the use of repair led to a minor increase in F-measure in some cases, but
a substantial decrease in others, and thus had a negative effect overall. This is tied to
yet another bias in the reference alignments, caused by the fact that they were
automatically repaired [
On the whole, the results of AML (without background knowledge) were interesting, and show that, for many ontology matching tasks, a lightweight approach based solely on element-level matching can compete with more complex approaches. It is worth highlighting that AML was among the quickest systems in all tracks, and thus had a consistently high F-measure/time ratio in all tracks except for Multifarm. However, the results in the Multifarm track, and to a lesser degree those in the Benchmark track, remind us that AML is still a system in development.
The results of AML-BK (and SBK) show that using suitable background knowledge is critical in specialized domains such as the biomedical, but can be advantageous even for more typical matching problems (such as those in the Conference track). 3.2
Implementing efficient and effective structural-level matching algorithms will be critical to improve the performance of AML overall. Language handling and translation will also be important to expand the scope of AML, and allow it to tackle tasks such as those in the Multifarm track. Finally, the inclusion of more sources of background knowledge will undoubtedly contribute to improve the performance of AML in tasks beyond the biomedical domain. 4
The participation of AML in the OAEI 2013 was a success overall, with very good results in the Anatomy, Conference, Interactive Matching and Biomedical Ontologies tracks, and reasonable results in the Library track. These results validate the background knowledge paradigm of AML, and demonstrate the effectiveness of a lightweight ontology matching strategy based solely on element-level matching. Nevertheless, it is also clear from the results that AML is not a complete ontology matching system yet, and that it can benefit from the addition of new tools to its base strategy.
Regarding its namesake, AML was able to build upon the success AgreementMaker had in the Anatomy track in previous OAEI competitions, and was able to transpose this success to the Large Biomedical Ontologies track.
DF, CP, ES and FMC were funded by the Portuguese FCT through the SOMER project (PTDC/EIA-EIA/119119/2010) and the multi-annual funding program to LASIGE. CP was also funded by the FLAD-NSF 2013 PORTUGAL-U.S. Research Networks Program through the project “Turning Big Data into Smart Data”. The research of IFC was partially supported by NSF Awards IIS-0812258, IIS-1143926, IIS-1213013, and CCF-1331800, by a UIC Area of Excellence Award, and by a IPCE Civic Engagement Research Fund Award.