Matcha is an ontology matching system designed to tackle long-standing challenges such as complex and holistic ontology matching. It incorporates all of the key algorithms from AgreementmakerLight over a novel broader core architecture that includes several new algorithms. Matcha performed well overall, achieving the highest F-measure in ten out of 32 distinct OAEI tasks and ranking in the top three in twelve others. In the complex track, Matcha achieved the highest F-measure in seven tasks using the Graph Edit Distance and two using the Tree Edit Distance.
The MELT [
set has a predetermined list of candidate matches, Matcha calculates scores for each candidate. These candidates are then ranked based on the highest score obtained from the various matching algorithms.
Matcha was packaged in a docker container for ease of sharing and running the evaluation, which included, for example, the files necessary for some of the algorithms, such as background knowledge ontologies used in some tracks and the scripts necessary to run the language models. 2. Results Matcha’s results for OAEI are summarized in Table 2, with the exception of the results for the BioML track, which are presented in Table 3, and the results for the complex track, which are presented in Table 4. Matcha achieved the highest F-measure in ten out of 32 distinct OAEI tasks and ranking in the top three in twelve others. In the complex track, Matcha achieved the highest F-measure in seven tasks using the Graph Edit Distance and two using the Tree Edit Distance.
With the same results as last year, Matcha placed first among all systems achieving scores higher than 0.9 in all evaluation metrics (0.951 for precision, 0.931 for recall, 0.941 for F-measure).
There was an increase in participating systems from two last year to four this year (LogMap competes with three variants). Matcha placed first in three out of ten tasks and failing to produce an alignment in one of them (where only one system had any results), similarly to last year. The results are again very heterogeneous, with precision raging from 0.17 to 1.0 (in the idai-pactols_de-de task) and recall from 0.06 to 0.75. Unsurprisingly the best results are mostly achieved in the single-language tasks.
Due to some technical issues, the language model was changed from last year’s MLLM to sentenceBERT, and it is interesting to note that the overall pattern of results were maintained despite this change. One of the improvement points to consider for the next edition is the runtime, as Matcha takes close to 11 minutes while most systems take under a minute to achieve better or comparable results.
This year’s results in this track were lower than the previous year requiring some further exploration, with Matcha placing last behind AgentOM and two LogMap variants. As mentioned by the track’s organizers, Matcha finds a large number of false positives, mostly in situtations where entities share the same name, which greatly impacts precision (0.158). Recall scores far higher with a 0.917, being the second best value among all systems.
Matcha’s complex matcher finds simultaneously simple equivalences, subsumption relationships, and 1:n complex equivalences. For the unpopulated conference task only complex mappings were evaluated and Matcha was unable to find any true positives.
Three new evaluations were debuted this year that evaluate mappings in the alignment based on their content and structure. Matcha performed much better in the metrics that took into account simple mappings (class evaluation and GED evaluation), since it actually outputted few complex mappings. Further exploration is necessary in this track in order to produce more (and correct) complex mappings.
Matcha competed as the only system in the new biomedical dataset achieving results that serve as a good starting point between 0.465 and 0.554 in F-measure.
Compared to last year, Matcha improved on precision but lowered recall resulting in an F-measure that is 0.01 lower (0.63), placing third. Matcha outperforms both baselines.
An additional evaluation was run to assess any diferences in results from sharp, discrete, and continuous settings. From this assessment, it is noted that Matcha performs well in the sharp evaluation in terms of precision (0.85), but in the discrete uncertain setting, while its precision drops, recall improves to 0.77 and then remains at a 0.75 in the continuous evaluation, indicating that it is successful at identifying uncertain matches.
Regarding the evaluation performed based on logical reasoning, Matcha has 90 conservativity principle violations and 115 consistency principle violations in an alignment of 21 mappings. which is an increase from last year and requires further exploration. As the organizers note, conservativity principle violations can be false positives.
Matcha improved over last year’s results placing first in five tasks out of eight tasks. Futhermore, overall it achieves the best average F1-score of 0.64 and the task that failed to produce an alignment in the previous edition was successfully completed this year. The results are somewhat heterogeneous with precision raging from 0.31 to 1.0 (in the arch1_defc-pactols) and recall from 0.23 to 1.0 (in the arch2_idai-pactols). As in the previous year, recall is less variable with more values falling on the top end of the scale (six out of eight are higher than 0.7).
Similarly to the Archaeology Multilingual track, this track uses sentenceBERT as the new language model for the translation module, which still requires further exploration and review.
Matcha places second to last overall, failing to produce an alignment for one of the tasks and producing no property mappings. By not considering empty, erroneous, or not generated alignments, Matcha improves on its results, with precision increasing from 0.54 to 0.68, recall from 0.71 to 0.88, and consequently F-measure from 0.61 to 0.76.
In aligning instances, Matcha finds far more mappings than other systems (29113 mappings when the next system finds 6653) which will significantly decrease precision.
When looking at each of the test cases, a pattern emerges where Matcha has lower precision and higher recall when comparing all systems. However the precision values are low enough that they cannot be compensated by the high recall, leading Matcha to rank very low.
In this track, two main problems arise which need to be assessed and corrected: the lack of property mappings and the excessive amount of instance mappings produced, which directly influence the system’s precision.
Matcha placed second overall by F-measure in this track. Both precision and recall are low and almost equal at 0.26 and 0.25, respectively. Despite other systems having a higher precision, lower recall values decrease the F-measure significantly. The runtime is still a necessary point of improvement as it takes significantly longer than other systems.
This year, Matcha maintained consistent performance without any major system updates, which is a positive outcome given the participation of four new machine learning–based systems. While Matcha’s Bio-ML rankings based on F-score were moderate compared to the other participating models, it demonstrated a stronger relative performance in terms of Mean Reciprocal Rank (MRR), particularly in the unsupervised setting. The middle-range F-scores mainly reflected high precision paired with comparatively low recall, a pattern also observed among most other participating systems, highlighting that the challenge of improving recall without compromising precision remains an open issue. Overall, key Matcha results in the Bio-ML track include: a top-3 MRR ranking in four of the five tasks in the unsupervised setting; first and second places in MRR ranking in the unsupervised and supervised settings of the SNOMED–FMA task, respectively; and a second place in the SNOMED–NCIT (pharm) task. 3. Conclusions Matcha achieved the highest F-measure in ten out of the 32 distinct OAEI tasks and ranked in the top three in twelve others. Overall, results stayed mostly the same compared to the previous edition. It is interesting to note that changes in the language model of the translation module had limited impact in the archaeology-multilingual and digital humanities tracks. The new results in the complex track are interesting to note, as they combine diferent types of mappings and were evaluated following new strategies. Some issues failed to be resolved from last year, such as the the ones linked to the knowledge graph track, meaning Matcha still requires some further review.
Acknowledgements This work was supported by FCT through fellowships https://doi.org/10.54499/2022.11895.BD (Marta Silva), https://doi.org/10.54499/2022.10557.BD (Pedro Cotovio), https://doi.org/10.54499/2025.04034.BD (Lucas Ferraz) and https://doi.org/10.54499/2024.01208.BD (Laura Balbi), and through LASIGE Research Unit, ref. UID/408/2025. It was partially supported by the KATY project which has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 101017453, by the CancerScan project by the EU’s HORIZON Europe research and innovation programme under grant agreement No 101186829, and by project 41, HfPT: Health from Portugal, funded by the Portuguese Plano de Recuperação e Resiliência.
Declaration on Generative AI The authors have not employed any Generative AI tools.
Mouse-Human idai-pactols_de-de idai-pactols_de-en idai-pactols_de-fr idai-pactols_de-it idai-pactols_en-en idai-pactols_en-fr idai-pactols_en-it idai-pactols_fr-fr idai-pactols_fr-it idai-pactols_it-it
OntoFarm (rar2-M3) arch1_defc-pactols arch2_idai-pactols arch3_ironagedanube-pactols arch4_pactols-parthenos cult1_idai-parthenos cult2_oeai-parthenos dhcs1_dha-unesco dhcs2_tadirah-unesco
Coverage —— Complex (GED evaluation) —— 0.526 0.365 0.431 0.529 0.327 0.404 0.587 0.595 0.591 0.500 0.064 0.113 0.645 0.507 0.568 0.372 0.062 0.106 0.421 0.303 0.353 0.048 0.688 0.091 0.287 0.467 0.355 0.245 0.391 0.301 0.465 0.465 0.465 0.529 0.529 0.529 0.554 0.554 0.554
reference
reference Overlap Incorrect cmt-conference cmt-ekaw conference-ekaw cree-swo hydro3-swo hydrOntology_native-swo hydrontology_translated-swo enslaved-wikidata gbo-gmo popgbo-popgmo hp mp wbp