This paper presents the results obtained by TOMATO in the OAEI 2023 evaluation campaign. We describe here the results in the Conference track. We report a general discussion on the results and future improvements of the system. TOMATO (TOolkit for MATching Ontologies) takes inspiration from previous work on ontology matching systems such as POMAP++ [1]. TOMATO is designed as a pairwise matcher, aligning pairs of input ontologies against each other. At its core, TOMATO utilizes machine learning approaches to learn from element similarities. In earlier versions, it focused mainly on stringbased similarity measures of ontology elements [2]. However, in 2023 we started working beyond this initial approach. While string similarities continue to provide baseline features, we have incorporated additional structural and semantic similarity measures. This includes leveraging relationships between ontology entities, as well as leveraging external knowledge sources. The goal is to move beyond solely lexical matching and capture more of the intended meaning during the alignment process. Lacking a robust ground truth beyond the reference alignment, this line of work was shelved temporarily to investigate supplemental evaluation methods. In this iteration of TOMATO, we took a more direct approach to developing the optimal matching strategy. Rather than starting with local strategies as in the previous year[3], our aim was to first identify the best global strategy across all entity types. To achieve this, we focused on empirically determining the most pertinent similarity measures to incorporate. In our evaluations, we considered a broader set of 11 measures: Levenshtein, Jaccard, Jaro-Winkler, cosine, iSub, dice, 3-gram, Monge-Elkan Levenshtein, Monge-Elkan Jaro-Winkler, and overlap coefficient. By systematically evaluating these measures in various combinations through our machine learning models, our goal was to arrive at the global configuration that performed best when
applied uniformly. Once identified, we could then examine how to potentially refine this global strategy by incorporating localized decompositions based on entity characteristics, as we had explored in earlier versions of TOMATO.
This new evaluation-driven approach aimed to build up the optimal strategy incrementally, starting from the best global foundation before incorporating targeted local enhancements. The rationale was that this may yield stronger baseline results compared to our prior method of starting with localized configurations.
The workflow in TOMATO begins by taking as input two OWL ontologies to be aligned, along with their associated reference RDF alignment files. These reference alignments provide examples of entity matches that serve as ground truth for the learning process.
TOMATO then prepares the data by combining all ontology entity pairs (both matched and unmatched based on the references) into a single dataset. This set contains all possible entity couples across the input ontologies, along with their reference match status.
From here, TOMATO can be used in one of two modes: 1. Learning mode: A machine learning model is trained on the full mixed dataset to learn the patterns of matching vs non-matching entities. 2. Matching mode: A pre-trained model is applied to new ontology entity pairs to predict their alignment status.
Unlike previous versions where we explored a variety of similarity measures, this year our focus is on determining which measures are most pertinent to the matching task and which can be eliminated.
By fully combining all entity pairs and reference alignments upfront, TOMATO is able to learn from all available matching examples without viewing the ontologies separately.
The initial step in TOMATO’s workflow is to parse and ingest the input ontologies. We leverage the owlready2 Python library to build an in-memory representation containing all ontology elements and relationships.
Each entity such as classes, data properties and object properties is indexed via a unique identifier, with the preferred identifier being the element’s label if present. As a fallback, the final segment of the entity’s URI is used.
Structural relationships between classes are also extracted and stored. This includes superclasssubclass links as well as relations defined through object properties.
By fully populating an internal graph structure in this way, TOMATO is able to consider both lexical properties of entities as well as their positions and connections within the ontology taxonomy during the matching process. This combined view aims to capture more contextual evidence about intended semantic correspondence compared to considering elements in isolation.
The loaded ontologies can then be queried as needed during the various steps of similarity computation, model training and alignment prediction.
As in prior iterations, we apply standard text preprocessing techniques to clean and normalize entity labels before computing similarities. This includes: • Converting CamelCase to snake_case • Replacing non-alphanumeric symbols with spaces • Performing English stemming • Removing stop words
However, unlike previous versions that employed matching strategies based on element types, in this work we utilize K-means clustering as described below.
All possible entity pairs between the input ontologies are generated without regard to element type (e.g. class vs property). For each pair, we compute similarities across several measures (Levenshtein, Jaro-Winkler, etc.).
New to this year, rather than hardcoding similarity computation strategies, we compute directly over the multi-dimensional similarity space. As we showed last year, this will allow us to focus on the worst possible outcome, which could be further improved in a case-by-case local strategy.
Training a Model We use GridSearchCV from scikit-learn to determine the best combination of similarity measures for predicting matches. The dataset is split 60/40 for training/testing with 10-fold cross validation. The reference alignments are used to label entity pairs as matches (1) or non-matches (0). This labeled similarity matrix is fed to an SVM classifier to train a matching model.
The TOMATO system is based on a machine learning approach that exploits 60% of data for
training and 40% of data for testing. The training set is classicaly used to learn a model that
consists in finding an optimal weighting of different similarity measures. Clearly, we use a subset
of the reference alignments provided by OAEI that is considered as an overfitting. However, it
is important to consider that the obtained model never considered the 40% set aside for testing.
To avoid this overfitting, we plan to use external resources as ground truth during the training
phase[
Computing Alignments A similarity matrix is constructed for the target ontology pair. This matrix is then input to the pre-trained matching model, which outputs the predicted entity alignments.
Unlike previous versions, in this work we utilize a distributed approach to avoid crashes or slowdowns when processing large or dense ontologies. The classification tasks are handled in parallel across computational threads/processes for improved robustness and performance.
In preparation for the OAEI evaluation, we focused on improving our ability to handle large ontologies in a distributed manner. Previously, training classifiers on the full matching dataset could lead to memory issues or lengthy processing times.
Therefore, in this iteration we implemented a multi-threaded resampling scheme during model training. The matching data is partitioned and resampled in parallel across CPU threads, with intermediate results merged to update the shared model.
Additionally, we experimentally found that training classifiers with a class imbalance favoring non-matching examples (e.g. 90% negative, 10% positive) led to more effective models compared to a balanced class distribution. This better reflects the natural skew in real-world ontology alignments.
To validate different modeling configurations, models trained with both global and local strategies were considered. However, due to time constraints of the competition format, we only submitted results using our global strategy models for evaluation.
The enhancements to our data processing and sampling methodology aimed to enable TOMATO to efficiently learn from very large ontologies in real-world settings, while still optimizing for high matching accuracy.
We face a significant class imbalance problem in our ontology matching task, where the number of positive matches between ontologies is much smaller than the number of negative matches. This poses challenges for training effective machine learning models.
During model training, the non-matching or negative examples dominate the training data distribution. As a result, models can become biased towards predicting negatives and fail to properly learn from the minority positive class.
To address class imbalance, we implement a resampling scheme during model training based on a resampling proportion hyperparameter .
The dataset consists of positive matches labeled 1 and negative matches labeled 0. We partition the negative examples into ⌊ 1⌋ subsets of size 1, where 1 is the number of positives.
We then create a balanced training set by combining the positive examples with a randomly selected negative subset. This ensures equal representation of both classes for learning. We vary to study its impact.
We train several models on matching data, varying ∈ [
Table shows optimal performance is achieved with = 41, with macro F1=0.43. Lower leads to poor predictive ability for negatives, while higher may worsen learning of positives.
This table summarizes the key performance metrics for our models across different values of the resampling proportion , showing the optimal value is =41.
The macro F1 score is a weighted average of F1 scores calculated for each class individually, giving equal importance to each class regardless of prevalence.
The micro F1 score aggregates counts across all classes to determine a single F1 score. It places more weight on accuracy for frequent, majority classes compared to macro F1 which balances across classes.
Together these metrics provide a more comprehensive assessment of model performance for both frequent and rare classes in a multiclass classification problem like ontology matching.
The resampling balances overall class distributions during training, addressing biases towards negatives. Higher proportions improve discrimination of matches by emphasizing difficult positive examples. Optimally, sufficient data from both classes is provided for robust learning.
However, residual recall indicates mismatches remain challenging. Future work involves combining resampling with techniques like data augmentation to further boost positive example diversity and learning.
In conclusion, resampling addresses class imbalance effectively, enhancing ontology matching model performance. Careful tuning of resample rates leads to improved classification accuracy.
This section analyzes the evaluation results published on the OAEI 2023 Conference Track results page (https://oaei.ontologymatching.org/2023/results/conference/).
A total of 11 matching systems participated in this track and produced alignments over the Conference domain ontologies: ALIN, AMD, GraphMatcher, LogMap, LogMapLt, LSMatch, Matcha, OLaLa, ProMatch, SORBETMtch, and TOMATO.
The systems were evaluated based on the main reference alignment rar2-M3 using precision, recall, F-measure, and other metrics.
TOMATO achieved a precision of 0.57, recall of 0.47, and F-measure of 0.52 on rar2-M3 as shown in Table 2. This placed it below the StringEquiv baseline but above ProMatch. Compared to 2022 results in Table 3, TOMATO’s precision and F-measure increased slightly while recall decreased.
TOMATO’s performance remains in the mid-range. While recall could improve, gains were seen in precision and F-measure compared to 2022.
In this paper, we presented the results of the TOMATO ontology matching system on the OAEI 2023 Conference track benchmark. TOMATO employs a machine learning approach combined with strategy selection based on domain characteristics.
Our evaluation showed that TOMATO achieved moderate performance. While its precision and F-measure improved slightly over last year, recall decreased. Nonetheless, TOMATO demonstrated the utility of adapting its strategy based on ontology features.
There remain opportunities for enhancing TOMATO’s matching capabilities. In future work, we plan to explore: • Additional machine learning classifiers and similarity measures • Deeper analysis of ontology structures to better leverage structural context • Techniques like data augmentation to improve generalization of learned models • Integration of external sources like knowledge graphs to strengthen matching We are optimistic that such improvements can lift TOMATO’s rankings on this benchmark by further refining its abilities to identify correct mappings. Continued participation in OAEI also allows ongoing evaluation versus state-of-the-art matchers. Overall, the results provide motivation to push the boundaries of adaptive ontology matching.