The AgreementMaker system is unique in that it features a powerful user interface, a flexible and extensible architecture, an integrated evaluation engine that relies on inherent quality measures, and semi-automatic and automatic methods. This paper describes the participation of AgreementMaker in the 2010 OAEI competition in three tracks: benchmarks, anatomy, and conference. After its successful participation in 2009, where it ranked first in the conference track, second in the anatomy track, and obtained good results in the benchmarks track, the goal in this year's participation is to increase the values of precision, recall, and F-measure for each of those tracks.
1.1
The new AgreementMaker system [
In 2009 AgreementMaker was very successful in the OAEI competition. In particular, AgreementMaker ranked (a close) second among ten systems in the anatomy track. AgreementMaker also participated successfully in two other tracks: benchmarks and conference. In the former track, AgreementMaker was ranked first in terms of precision and seventh in terms of recall among thirteen systems and in the latter track AgreementMaker was ranked first with the highest F-measure (57% at a threshold of 75%) among seven competing systems. 1.2
AgreementMaker comprises several matching algorithms or matchers that can be used for matching (or aligning) the source and target ontologies. The matchers are not restricted to any particular domain. The architecture of AgreementMaker relies on a stack of matchers that belong to three different layers (see Figure 1). Specific configurations of the stack have been used for the benchmarks, anatomy, and conference tracks, as discussed in what follows. However, we describe first the different components in the stack: the matchers, the combination and evaluation modules, and the final alignment module.
Matchers can be concept-based (if they consider only one concept) or structural
(if they consider a subgraph of the ontology). The concept-based matchers support the
comparison of strings. They include: the Base Similarity Matcher (BSM) [
Structural matchers include the Descendants’ Similarity Inheritance (DSI)
matcher [
The combination and evaluation modules are used together, as follows. The Linear
Weighted Combination (LWC) [
Benchmarks For the benchmarks track we used the following configuration:
LWC is adopted to combine the results of four string-based matchers, namely ASM, PSMlex, PSMlex, and BSMlex; the last three make use of two lexicons, namely WordNet and a dictionary built from the ontologies; the similarity values computed at this step are then given as input to the IISM structural matcher.
Anatomy For the anatomy track we used the following configuration:
LWC is adopted to combine the results of four string-based matchers, namely PSMlex, VMMlex, and BSMlex; the last three make use of two lexicons, namely WordNet and a dictionary built from the ontologies.
Conference For the conference track we used the following configuration:
GFM( LWC(ASM; PSM) )
LWC is adopted to combine the results of two string-based matchers, namely ASM and PSM; the similarity values computed at this step are then given as input to the GFM structural matcher. 1.3
The AgreementMaker system is available at http://agreementmaker.org/. 1.4
The alignment results obtained by AgreementMaker in the OAEI 2010 are available at http://agreementmaker.org/oaei. 2
In this section, we present the results obtained by AgreementMaker in the OAEI 2010 competition. It participated in three tracks: benchmarks, anatomy, and conference. Tests were carried out on a PC running Ubuntu Linux 10.04 with AMD AthlonTM II X4 635 processor running at 2.9 Ghz and 8 GB RAM. 2.1
In this track, a source ontology is compared to 111 ontologies that describe the same domain. These ontologies can be divided into 3 categories: concept tests cases (1xx cases), systematic tests cases (2xx cases), and real ontology test cases (3xx cases). AgreementMaker employs the algorithm which we described in section 1.2 for aligning two ontologies.
The 2xx benchmarks test cases are subdivided into 3 groups: 1) 201 to 210, 2) 221 to 247 and 3) 248 to 266. The lexical information in the ontologies in group 1 have been altered to change their labels or identifiers. This alteration includes replacing the labels or identifiers with other names that follow a particular naming convention, a random name, a misspelled name or a foreign word. However, the structure of the ontologies is not modified. The test cases in the second group have ontologies that have flattened hierarchies, expanded hierarchies or no hierarchies at all. The test cases in the third group are the most challenging ones to align. This is because the labels have been scrambled such that they comprise a permutation of letters of a particular length. In addition, the structure of the ontology has been flattened, expanded such that it has more depth or removed completely.
The results obtained by AgreementMaker in the benchmarks track are summarized in Table 1. This track consists of two real world ontologies to be matched. The source ontology describes the Adult Mouse Anatomy (with 2744 classes) while the target ontology is the NCI Thesaurus describing the Human Anatomy (with 3304 classes). Matching these ontologies is also challenging in terms of efficiency because these ontologies are relatively large. The anatomy track consists of four subtracks: subtrack 1, which emphasizes Fmeasure, subtrack 2, which emphasizes precision, subtrack 3, which emphasizes recall, and subtrack 4, which tests the capability of extending a partial reference alignment.
The results obtained by AgreementMaker in the anatomy track are summarized in Table 2. We show the precision, recall, and F-measure for subtrack 1, 2 and 3; subtracks 1 and 3 are also evaluated by considering the recall+ measure, which measures how many non trivial correct correspondences, not detectable by string equivalence, can be found in an alignment. Evaluation of subtrack 4 is carried out by analyzing the changes in precision, recall and f-measure when subtrack 1 is compared with subtrack 4. The conference track consists of 15 ontologies from the conference organization domain and each ontology must be matched against every other ontology. Since the AgreementMaker OAEI 2010 matcher stack considers only two ontologies at a time, we compute 120 alignment files, in total containing 2070 individual alignments. The results obtained are summarized in Table 3. Here we show how precision, recall, and F-measure vary depending on the threshold used for the selection of the mappings. Benchmarks In the OAEI 2009 competition, AgreementMaker was first in terms of the precision of discovered mappings. However, in terms of recall, AgreementMaker was outperformed by six other systems (thirteen systems competed). The new matchers used in the OAEI 2010 competition address specifically the issue of the alignment of concepts that are not lexically similar. The results of this effort increased the recall by 18% at a cost of 3% in precision in comparison with last year’s results. An important contribution to this result comes from the IISM matcher, which exploits instances and structural properties of the ontologies and makes the alignment process less sensitive to lexical differences. A detailed comparison between the results achieved in the 2009 and 2010 competitions in terms of the obtained change in precision, recall, and F-measure for each group of test cases, and the overall H-mean is shown in Table 4.
Anatomy In comparison with the results achieved by AgreementMaker in the OAEI 2009 competition, the experimental results obtained this year show that the system significantly improved with respect to precision, recall, F-measure, and recall+. A major contribution to these results comes from the exploitation of lexical resources to improve string-based and vector-based matchers. A comparison between the results achieved in the two competitions in terms of precision, recall, F-measure and recall+ for subtracks 1, 2 and 3 is shown in Table 5 (except for recall+ that is not evaluated on subtrack 2). Remarkably, our algorithms for retrieving non trivial mappings significantly improved, as shown by the gain of 0.15 in recall+. Instead, we do not present the comparison with the results obtained in 2009 on subtrack 4, because this year we did not exploit any specific algorithm for propagating mappings available in the input alignment.
Conference In comparison with the results achieved in OAEI 2009, AgreementMaker significantly improved on precision, recall, and F-measure for thresholds above 0.75 as shown in the graph represented of Figure 2.4, providing more stable results. Remarkably, the new matchers used for the conference track, namely ASM and GFM, can be used on real-world ontologies, since they are based on generic lexical and structural features. Moreover, ASM can be easily adapted to different string-based similarity metrics, and can be extended by adopting a lexicon. In this paper we presented the results of the AgreementMaker system for aligning ontologies in the OAEI 2010 competition in the three tracks in which it participated: benchmarks, anatomy, and conference. It was our goal to improve on the results obtained by AgreementMaker in 2009. To meet this goal, we developed several new matching methods, which could be readily integrated into the AgreementMaker system because of its modularity and extensibility. Our results (which we compare with last year’s results) amply demonstrate that we have met our goal.
Thanks to Catia Pesquita and Francisco M. Couto from Facultade de Ciencias da Universitade de Lisboa, Portugal, for the feedback on the AgreementMaker system, and for useful insights on biomedical ontologies, which significantly increased our understanding of the problem and our results in the anatomy track.