This paper presents the 2024 results of an enhanced version of the CANARD system, which integrates Large Language Models (LLMs) to address the challenges of complex alignments. By leveraging LLM-based embeddings, the system better captures semantic and contextual relationships, improving both precision and coverage. Four architectural settings - Label Embedding Similarity (LES), Embeddings of SPARQL Query (ESQ), Subgraph Embeddings (SE) and Instance Embeddings (IE) - were explored to improve the alignment quality. Experiments on the Populated Conference dataset from the OAEI Complex Track demonstrate improvements over baseline approaches, with an increase in F-measure up to 45% in some cases. However, challenges such as runtime overhead in IE and noise in SE components were identified, where future work can explore better aggregation techniques or fine-tuned LLMs.
CEUR
ceur-ws.org with the source CQA. In step 4, similar instances in the target KG are retrieved for those retrieved in step 3. In step 5, the subgraph of the target instances is retrieved. In step 6 the labels of the entities in the subgraphs are retrieved and in step 7 their similarity is measured against the labels retrieved in step 2. In step 8 the correspondences with summed similarity higher than a threshold are kept and in step 9 the final alignment is written in EDOAL format. Two main improvements have been implemented, corresponding to the two similarity evaluations in the instance matching step and in the subgraph similarity measure. By using embeddings to compute those similarities, the matcher improves its capacity to retrieve the relevant entities to each case. Also, since LLMs have increased language understanding, using embeddings generated by LLMs improves similarity computation.
Multiple embedding applications are possible in the proposed architecture. Since subgraphs are involved in the similarity computation to the CQA entities (step 7 of Figure 1), some aggregation techniques are applied. For the similarity computation with embeddings three aggregation strategies were considered:
Label Embedding Similarity (LES) Entity labels from the source and target KGs are processed through pre-trained LLMs. Those LLMs have an associated tokenizer that splits the text into multiple tokens. Those tokens are then input to the LLM that generates one embedding for each token. The final embedding for each label is derived by averaging all embedding from the output of the model’s last hidden layer. All embeddings from the CQA side are cross-compared with all tokens in the subgraph labels resulting in n:m comparisons. The similarities lower than a threshold are filtered out and the resulting ones are added to the final similarity.
Embeddings of SPARQL query (ESQ) In this setting the embeddings generated for all entities in the CQA (similar to the LES step) are averaged resulting in a single embedding for the CQA. Then this embedding is compared with the ones from the subgraph entities resulting in 1:m comparisons.
Subgraph embeddings (SE) For this setting, the embedding for the CQA is computed as in the ESQ setting, and the embeddings for the subgraph are aggregated. Two types of aggregation are considered depending on the type of the subgraph. If the CQA is unary (one variable in the SPARQL query) the corresponding subgraph is a triple composed of subject, predicate, object, and also the subject type and object type. For binary CQAs (two variables) the corresponding subgraph is a path that connects the entities retrieved in the variables.
In the instance matching step (step 4 Figure 1), an embedding is generated for all entities in the datasets without the BNodes. Then, for any given entity the most similar ones by computing their embedding similarity. This setup is named Instance Embeddings (IE) and can be applied simultaneously with the CQA embedding settings LES, ESQ, and SE.
Some adaptations were made to evaluate the system in the OAEI complex track. The matcher was evaluated in the Populated Conference dataset as it contains CQAs and instances as required by CANARD to execute and later by the evaluator to compare the results.
The Populated Conference dataset contains 5 populated ontologies and all pairs were evaluated with the proposed approach. For each pair, a range of similarity thresholds was evaluated to determine the optimal value for filtering alignments. Two distinct threshold values are needed in this approach, one for the CQA-related similarity and the other for the instance matching step. For the similarity in step 7 (Figure 1), values ranging between 0.5 and 1.0 incrementing by 0.1 were considered. For the instance matching step (step 4 in Figure 1), thresholds of 0.8, 0.85, and 0.9 were employed to verify the impact of the threshold in this step. Only a few thresholds were considered in the step 4 to reduce the number of combinations to be tested.
The embedding generation (before the main execution), involves processing all KG labels and instances
through LLMs and is computationally intensive. To address this, embeddings were precomputed and
cached for reuse during the matching process. The computation using LLMs was computed using GPU
acceleration and the usage in the evaluation was done only with CPU. The tested models are GritLM-7B
[
The evaluation was automatically performed using the evaluator proposed in [
Considering all parameter variations of input ontologies, embedding models, and thresholds, 1800 combinations of parameters were tested to identify the most promising configurations. This was followed by additional evaluations incorporating embeddings into the linking step for the top-performing settings. These adaptations allowed the system to demonstrate substantial improvements over baseline methods, achieving higher precision and coverage scores on the tested dataset. 1.2.1. Link to the System and Parameters File The baseline approach can be found in https://framagit.org/IRIT_UT2J/ComplexAlignmentGenerator. The embedding variation used in this paper can be found at https://gitlab.irit.fr/melodi/ ontology-matching/complex/canarde.
In this section, the results of the evaluation of the Populated Conference dataset are presented. In the query-oriented evaluation, the GritLM-7B with the ESQ setting was the one with the highest query-oriented f-measure and precision. In the precision-oriented evaluation, the Stella-base model with ESQ setting and the instance embeddings IE setting with 0.85 threshold reaches higher results in all precision-oriented metrics. The results of this evaluation are presented in Table 1.
As shown in Table 1, LES and ESQ achieve the highest performance when LLMs are utilized. These
configurations involve fewer embedding aggregations than SE. Notably, as the number of aggregations
increases, such as in the SE configurations, the performance of all models decreases. This degradation can
be attributed to the loss of semantic information and increased noise caused by combining embeddings
without a weight transformation mechanism or filtering, such as those used in Graph Neural Networks
1https://huggingface.co/infgrad/stella-base-en-v2
model
base (levenshtein)
GritLM-7B (LES)
sfr-mistral (i-LES)
GritLM-7B (ESQ)
sfr-mistral (i-ESQ)
glove (SE)
glove (i-SE)
stella-base (ESQ+IE 0.9)
stella-base (ESQ+IE 0.85)
GritLM-7B (ESQ+IE 0.9)
[
Also, the improved architecture was compared with other matchers in the same dataset. The results of this comparison are presented in Table 2.
Matcher Matcha-DL AMLC CANARD 2018 CANARD 2024 (Stella-base IE 0.85) CANARD 2024 (GritLM-7B ESQ)
The results demonstrate that the integration of Large Language Models (LLMs) enhances the performance of the CANARD framework in the Populated Conference dataset. The usage of embeddings increased the precision and F-measure by up to 45% over the baseline, showing their efectiveness in capturing semantic nuances. The LES and ESQ configurations were the most efective, and applying the instance embeddings also increased the performance of the matcher in the precision-oriented that is instance-based evaluation.
However, some weaknesses were still present for example the Instance Embeddings (IE) setting incurred significant computational overhead due to the exhaustive comparison of embeddings, particularly for large datasets. Also, the aggregation of subgraph embeddings used in the setting SE occasionally introduced noise, leading to reduced performance in comparison to other configurations.
Several improvements can address the identified weaknesses. The IE step can be optimized using approximate similarity measures or clustering techniques to reduce the search space. Techniques like weighted aggregation or Graph Neural Networks (GNNs) could improve the quality of subgraph embeddings by better preserving semantic relationships. Also, using domain-specific fine-tuning of LLMs on ontology-related tasks could further enhance the relevance of embeddings. Is also possible to combine embedding-based methods with symbolic reasoning approaches to enhance the ability to capture complex logical relationships.
This paper presented an enhanced ontology matching system that integrates Large Language Models (LLMs) into the CANARD framework to deal with complex alignment tasks. The use of LLM-based embeddings improved performance in the conducted evaluation of the Populated Conference dataset. While the system excelled in capturing semantic relationships, challenges such as high runtime in certain configurations and reduced performance in subgraph embeddings highlight areas for future optimization. Proposed improvements include eficiency enhancements, fine-tuning of LLMs, and advanced aggregation techniques.