AgreementMakerLight (AML) is an ontology matching system designed with scalability, extensibility and satisfiability as its primary guidelines, as well as an emphasis on the ability to incorporate external knowledge. In OAEI 2019, AML's development focused mainly on expanding its range of complex matching algorithms, but there were also improvements on its instance matching pipeline and ontology parsing algorithm. AML remains the system with the broadest coverage of OAEI tracks, and among the top performing systems overall.
1.1
OAEI 2017 and 2018, due to the unique characteristics of these matching tasks and to the unavailability of the TBox assertions in the HOBBIT datasets. 1.2
This section describes only the features of AML that are new for the OAEI 2019. It also
describes AMLC, a variant of AML tailored to complex matching. For further
information on AML’s simple matching strategy, please consult AML’s original paper [
We made a few extensions to AML’s ontology parser to enable it to infer the types of ontology properties declared only as rdf:property (which the OWL API interprets as annotation properties by default). There were critical to correctly interpret and match the datasets for the Knowledge Graph track.
We refined AML’s instance matching pipeline to more adequately distinguish between cases where lexical matching should be the primary strategy complemented by propertybased matching, and cases where property-based matching should be the primary strategy, by using the ratios of labels per instances and property values per instances as deciding factors. These improvements were critical to AML’s effectiveness on the Knowledge Graph track. 1.2.2 AMLC For the complex matching track, we developed algorithms to tackle additional types of EDOAL mappings, namely mappings involving union class constructs. Furthermore, we refined the Attribute Occurrence Restrictions and Attribute Domain Restrictions algorithms developed last year to take into account instance data when available.
These changes allowed AML to match ontologies from the GeoLink dataset, in addition to those from the Conference dataset. 1.3
As was the case last year, the Link Discovery submissions of AML are adapted to these particular tasks and datasets, as their specificities (namely the absence of a Tbox) demand a dedicated submission. The same is also true to some extent of AML’s Complex Matching submission.
As usual, our submission included precomputed dictionaries with translations, to circumvent Microsoftr Translator’s query limit. 1.4
AML is an open source ontology matching system and is available through GitHub: https://github.com/AgreementMakerLight. 2.1
AML’s result was the same as in previous years, with 95% precision, 93.6% recall, 94.3% F-measure, and 83.2% recall++. It remains the best ranking system in this track by both F-measure and recall++.
AML’s result was exactly the same as in recent years, with 74% F-measure according to the full reference alignment 1, 70% F-measure according to the extended reference alignment 2, 78% F-measure according to the discrete uncertain reference alignment, and 77% according to the continuous one. It remains the best ranking system in this track or tied for best by F-measure according to 4 of the 5 sets of reference alignments available. It ranks second by F-measure on the violation free version of reference alignment 2, as enforcing the removal of conservativity violations can produce undesired practical effects that are not aligned with AML’s guiding principles, so our repair algorithm does not take them into account. 2.3
AML’s results were similar to last year, ranking first with 45% F-measure in the different ontologies modality, but second with only 27% F-measure in the same ontologies modality. We are still unsure as to why AML performs worse in the same ontologies modality. 2.4
AMLC was configured only for the Conference and Geolink datasets. It also produced results in the Hydrography dataset, but these were expectedly mediocre. On the conference dataset, AMLC was the only system to participate in the non-populated version (using the simple reference alignment as input). It improved its recall in relation to last year (37% versus 25%) but this came at the expense of precision and so resulted in an identical F-measure of 34%. On the populated version, it had the highest range of coverage (query F-measure) with 46-50%.
On the GeoLink dataset, AMLC obtained a comparably modest F-measure of 32% (the top system had 60%).
AML had an identical performance to last year, as no changes were made to its interactive algorithms. It remains the system with the best F-measure in both the Anatomy and Conference datasets across all error rates (though it also has the best non-interactive F-measure in these datasets). 2.6
AML had an F-measure of 93.3% in FMA-NCI small, 84.1% in FMA-NCI whole, 83.5% in FMA-SNOMED small, 69.7% in FMA-SNOMED whole, 81.8% in SNOMEDNCI small and 76.5% in SNOMED-NCI whole. In comparison with last year, its performance decreased in all large tasks, due to the erroneous addition of an imprecise matching algorithm in the matching pipeline when testing new configurations. Despite this, it remains the best performing system in five of the six tasks. 2.7
AML generated 2029 mappings in the HP-MP task, 330 of which were unique. It ranked third by F-measure according to the 3-vote silver standard, but this does not necessarily reflect its actual performance, as the unique mappings were not evaluated. If half of AML’s unique mappings were proven correct, which is highly likely given the high precision AML obtains in other biomedical tasks, it would rank first in F-measure. In the DOID-ORDO task, it generated by far the most mappings (4781) and the most unique mappings (2342), and as a result had a relatively low F-measure according to the 3-vote silver standard (65.1%). Again, assessing the correctness of the unique mappings would be essential to gauge AML’s true performance. 2.8
AML obtained the highest F-measure in both datasets, with 78.8% in the FLOPO-PTO task and 80.8% in the ENVO-SWEET task. It ranked first in recall and produced both the most mappings and the most unique mappings. 2.9
AML obtained the same results as last year, with an F-measure of 86%, ranking third by F-measure. 2.10
As in previous years, AML produced a perfect result (100% F-measure) in the Linking and all the Spatial tasks. It was among the most efficient systems in the later, and the only system participating in the former. 2.11
AML was able to complete only four of the five tasks due to an unforeseen timeout in the largest task (which it had been able to carry out in testing). It produced an average F-measure of only 70% if the missing task is counted as zero, but of 88% when it is ignored. In fact, it ranked either first or second in F-measure in all the four tasks it completed.
This year, AML was again the system that tackled the most OAEI tracks and datasets, maintaining its status as one of the broadest and best performing matching systems available to the community.
However, unlike AML’s performance in traditional (simple) matching tracks, there is clearly room for improvement for AML in complex matching, as it had modest Fmeasures. We will strive to refine and improve AML’s complex matching pipeline and contribute to the development of this branch of ontology matching. 3.2
We once again laud the efforts of the organizers of both returning and especially new tracks, as the effort involved in organizing them cannot be overstated. Nevertheless, we must again comment on the unsatisfactory evaluation in the Disease and Phenotype track by means of silver standards generated from the alignments produced by the participating systems via voting. We understand the effort required to build a manually curated reference alignment, but we believe that it is paramount to invest in it, in order to enable a proper evaluation of matching systems. 4
Like in recent years, AML was the matching system that participated in the most OAEI tracks and datasets, and it was among the top performing systems in most of them. AML’s performance did not improve in any of the long-standing OAEI tracks, as most of our development effort went into tackling new challenges and extending the range of AML. We improved substantially our results in the knowledge graph track in comparison with last year, thanks to the extensions to AML’s ontology parsing algorithm and its instance matching pipeline. We were also able to extend AML’s complex matching algorithm portfolio, but despite this, AML complex matching performance requires further improvement. We will continue to invest in addressing this aspect of ontology matching in the near future
DF was funded by the EC H2020 grant 676559 ELIXIR-EXCELERATE and the Portuguese FCT Grant 22231 BioData.pt (co-financed by FEDER). CP and FMC were funded by the Portuguese FCT through the LASIGE Research Unit (UID/CEC/00408/2019). FMC was also funded by PTDC/CCI-BIO/28685/2017. CP was also funded by FCT (PTDC/EEI-ESS/4633/2014). The research of IFC and BSB was partially funded by NSF awards CCF-1934915, CNS-1646395, III-1618126, CCF1331800, and III-1213013, and by NIGMS-NIH award R01GM125943.