Still few ontology matching approaches focus on generating alignments by a Large Language Model (LLM), especially in the complex matching task. This paper proposes an approach that leverages the capabilities of LLMs to perform complex ontology matching. The method integrates subsets of both source and target ontologies into the prompt and, as a response, the LLM generates alignments in the structured EDOAL format, rather than natural language descriptions. This reduction technique, based on the automatic generation of SPARQL queries, tackles the challenge of large prompt sizes, reduces the search space, and enables eficient processing on consumer-grade hardware. This approach is evaluated on the Conference and Geolink datasets from the OAEI complex track, demonstrating improved scalability and the ability to produce well-formed EDOAL. Key contributions include the development of a SPARQL-based prompt engineering strategy and the application of few-shot learning techniques to complex alignment generation.
Ontology matching techniques have predominantly focused on simple alignment tasks that are insuficient to cover real-world needs. As data complexity and their relationships increase, the challenge of accurately aligning schemes and ontologies and generating expressive correspondences becomes more pronounced. Conversely, advancements in the field of Natural Language Processing (NLP) have led to the development of powerful Large Language Models (LLMs), which have demonstrated remarkable capabilities in understanding and generating natural language. Although such models have been explored for simple alignment tasks, there has been a growing interest in their use for complex matching.
A recent work addressing the problem has been proposed in [
A more general problem, however, is how to automatically modularize an ontology so that each
complex entity is defined inside its module. For example, the entity “Paper” may be related to an entity
“Author” that probably is not part of the complex “Paper” entity, and filtering out those relations is still
a dificult task. In [
To address these limitations, this paper proposes an approach that selects subsets of both source and target ontologies guided by automatically generated SPARQL queries from PageRank graph nodes. Those subsets are inserted into the prompt and the LLM directly outputs the alignment results in a structured format such as EDOAL rather than natural language descriptions.
By incorporating both ontologies subsets into the prompt, an LLM will have more information to work with since it can combine information from diferent parts of the ontology to produce each correspondence and produce better alignment results. In addition, the task of delimiting complex entities, combining them, and finding the complex correspondences between entities is passed to the LLM. This approach simplifies the evaluation and verification of the results and provides a more comprehensive and accurate alignment.
A key aspect of our proposal is a technique to reduce the search space by employing the automatic
generation of SPARQL queries, as explored in [
The main contributions of this paper are as follows: (i) an approach to complex ontology matching by leveraging the capabilities of LLMs to generate complex alignments in structured format via few-shot prompting and; (ii) a technique based on RAG to reduce the prompt size by creating subsets of the input ontologies guided by SPARQL queries.
The rest of the paper is organized as follows. In Section 2 the research motivations are introduced. In Section 3, the proposed architecture is detailed while explaining all its substeps. In Section 4.1 the experiment settings and results are presented. In Section 5 the related work are discussed. Finally, in Section 6 the conclusion of the paper is presented.
Recent LLM advancements have opened new ways to construct matchers for complex ontology matching tasks. Notably, models like ChatGPT-4o2 and Claude 3.53 have demonstrated the ability to perform complex alignments in controlled, toy problem settings by leveraging the input of entire ontologies into the prompt, as exemplified in Figure 1.
After inserting the ontologies in the prompt, the next step consists of providing the instructions and
examples of alignments so the model can learn to write the resulting alignment in the required format
as exemplified in Figure 2. The examples provided and the resulting alignment are in EDOAL [
The output produced by ChatGPT4o is illustrated in Figure 3. The input ontologies were crafted to contain the concept of AcceptedPaper in the source ontology and the concept of Paper that have a property hasAcceptance leading to a range Acceptance in the target ontology. The LLM properly ifnds similar complex entities between source and target ontologies even without examples of complex correspondences in the prompt. Moreover, the LLM properly combines the entities Paper, hasAcceptance, and Acceptance to produce the complex correspondence while following the proposed file structure in the examples.
This case highlights the potential of LLMs to perform complex ontology matching by directly
outputting the alignments in the expected format. However, ChatGPT-4o and Claude 3.5 are private
LLMs that have hardware requirements far beyond those available to most consumers. In this sense,
1In that work, the notion of Competency Question for Alignment (CQA) has been introduced as a way of expressing user
needs in terms of alignments, in the form of SPARQL queries. As it has been also explored in that work, we adopt here an
automatic way of generating such queries, referred to as ‘SPARQL queries’ for short, in this paper.
2https://chatgpt.com/
3https://claude.ai/new
Given the two ontologies below:
enabling open and smaller LLMs to perform this task is a desirable objective. These tests have been
corroborated with Llama-3 4, an open-source LLM available in smaller sizes. It has demonstrated the
ability to find correspondences between entities but produced the output in the wrong format even
if explicitly instructed to output in EDOAL. By giving examples of alignments, a technique named
few-shot learning [
Despite these promising capabilities, significant challenges remain when deploying these models in real-world scenarios. One of the primary obstacles is the substantial size of the prompts due to the whole input of the source and target ontologies in the prompt that can reach the magnitude of millions of tokens. This becomes problematic as consumer-grade GPUs, commonly used in many practical applications, are unable to handle such large inputs eficiently. To deal with these problems, our proposed approach to automatically construct the prompts with examples and in a reduced size is described in the next sections.
To overcome the challenges of large prompt sizes and limited computational resources, an approach for
reducing the search space is proposed. This reduction is based on using generated SPARQL queries to
select the most promising parts of an ontology and writing smaller prompts that the LLM can process.
Here, the use of automatically generated SPARQL is inspired from [
Write a file in EDOAL format containing the complex alignment between the ontology1 and ontology2. You don't need to explain yourself. Just give as response the resulting file without saying anything. Here is one example bellow: <ontology1> # prefixes lib:Book1 a lib:Book ; dcterms:title "The Catcher in the Rye" ; dcterms:creator lib:Author1 ; lib:hasGenre "Fiction" . lib:Author1 a lib:Author ; foaf:name "J.D. Salinger" ; foaf:birthDate "1919-01-01" . </ontology1> <ontology2> # prefixes pub:Book1 a pub:Book ; dcterms:title "To Kill a Mockingbird" ; dcterms:creator pub:Author1 ; pub:publicationYear "1960" . pub:Author1 a pub:Author ; foaf:name "Harper Lee" ; pub:hasNationality "American" .
</ontology2> <alignment> <?xml version="1.0" encoding="utf-8"?> <rdf:RDF #prefixes> <Alignment> <onto1> <Ontology rdf:about="http://example.org/library#"/> </onto1> <onto2> <Ontology rdf:about="http://example.org/publishing#"/> </onto2> <map> <Cell> <entity1 rdf:resource="http://example.org/library#Book"/> <entity2 rdf:resource="http://example.org/publishing#Book"/> <relation>=</relation> <measure>1.0</measure> </Cell> </map> <map> <Cell> <entity1 rdf:resource="http://example.org/library#Author"/> <entity2 rdf:resource="http://example.org/publishing#Author"/> <relation>=</relation> <measure>1.0</measure> </Cell> </map> </Alignment> </rdf:RDF> </alignment> ontologies. However, the user manually writes the CQAs to express their needs, and writing CQAs for each mapping pair is not scalable. In this sense, in this work, the use of automatically generated SPARQL queries is applied to perform the search space reduction.
At the end of the generation step of all prompts, the EDOAL output of the LLM of the same pair of ontologies is later combined. Along with that reduction, a prompt engineering strategy (few-shot) is <?xml version="1.0" encoding="utf-8"?> <rdf:RDF #prefix> <Alignment>
. . . applied to enable the LLM to perform the complex matching task. The proposed method is detailed in the following sections.
The reduction of space is composed of three main steps, which are detailed in the following.
3.1.1. Identifying Pertinent Entities using PageRank
The first step involves identifying the most relevant entities within the ontologies. In [
The PageRank [
The PageRank computes the rank of a node by iterating times all nodes in the ontology and updating their values in each iteration. The rank of a node is the sum of the ranks of its incoming links divided by their outgoing links. The rank of a link is computed with the equation 1. () = * _−1(()) + 1 − (1)
After the final iteration, the rank of all nodes is normalized by dividing the ranks by the maximum
rank value, and the nodes with a rank higher than a threshold will be converted to SPARQL queries. The
conversion template is based on the work [
The template used to produce the documents is given in Figure 5. The template is the same for both
simple and complex entities. Complex entities are retrieved from BNodes and processed to generate a
text representation. For example, a BNode containing owl:unionOf and a chain of rdf:first and rdf:rest
are traversed and the entities are collected in a list and written in a single line of text. For example, the
union of Conference and Event results in the text UnionOf (Conference, Event) that is inserted into the
ifeld Name of the template. The other fields, Parents, Children, Incoming Properties, Outgoing
Properties, and Disjoint with are also lists of entities retrieved from the graph structure of the
ontology. The number of entities in each field is limited to produce reduced-size documents.
3.1.3. Creating Sub-Ontology Prompts
The next step is inspired by Retrieval Augmented Generation (RAG) [
After retrieving the most promising entities, a new ontology is created by inserting the entities subgraphs expanded by traversing the ontology and inserting the same relations triples until a certain depth from the initial entity. This process is applied to source and target ontologies using the same SPARQL query to ensure that entities semantically related to source and target ontologies are present for comparison in the prompt. Finally, a turtle serialization of the subset ontology is produced to be included in the prompt. Serializing the final ontology using turtle format reduces the number of tokens since turtle is less verbose than RDF/XML. The process is illustrated in the Figure 6.
With the sub-ontologies prepared, a prompt is generated by filling in the following template: Given the two ontologies below: <ontology1> <ontology2> Examples of complex alignment between different ontologies: <ontology1> <ontology2> <alignment> Considering that the input ontologies were filtered to include only the entities related to the query: <query> Write a file in EDOAL format containing the complex alignment between the ontology1 and ontology2. You don’t need to explain yourself. Just give as response the resulting alignment file without saying anything else.
For the experiments, three diferent variations of the prompt template were tested. The first does
not include examples, the second includes examples without the SPARQL query used to select the subontology, and the last includes both examples and the query.
After the generation of the prompts, they are input into the LLM to generate the alignment results in EDOAL. The EDOAL format provides a formal representation of the alignments, capturing complex relationships and correspondences between the entities of the ontologies while facilitating the evaluation of the performance of the alignments produced since the automatic evaluation system used in this work processes that format.
In the generation step, the model is prompted without token sampling, only the most probable token is always selected, with a maximum generation length of 2048 tokens. The generated EDOAL is then repaired in the cases where the model doesn’t generate an appropriate file ending (caused by files that are supposed to be larger than 2048 tokens). Then the EDOAL files for all SPARQL queries in the same source and target ontology pairs are combined resulting in a single file. The file contains all the correspondences related to the concepts in all generated SPARQL queries from the same source and target ontology pair. This is the final output of the architecture that will be later evaluated. That process is depicted in Figure 7.
To evaluate the quality of the alignments produced by the LLM, an experiment was conducted by generating several prompts and evaluating their performance using an automated evaluator [15]. The evaluator assesses the performance of the alignments by comparing the number of similar instances retrieved in each correspondence. Metrics such as precision, recall, and F1-score are used to quantify the performance of the alignment process. In addition, no training or fine-tuning was performed in any model during the experiments, the model’s weights are not changed in any way during the execution.
The selected datasets are Conference and Geolink from the OAEI6 complex track. The selected model for retrieval is Salesforce/SFR-Embedding-2_R used in the process of generating the prompts. The LLM used for the response generation is meta-llama/Meta-Llama-3-8B-Instruct both of them retrieved from HuggingFace 7. The code used in the experiments can be found in GitLab 8.
The first step in the execution pipeline is the generation of the queries that will narrow the size of the prompts to generate the response by the LLM. The PageRank was run with 10 iterations, a damping factor of 0.8, and a threshold of 0.4 that selects a maximum of 30 entities. The number of entities selected for each ontology is present in Table 1.
Ontology conference/iasted conference/edas conference/Cocus conference/paperdyne
conference/crs_dr conference/conference conference/confious conference/MyReview conference/sigkdd conference/PCS conference/OpenConf conference/linklings conference/cmt conference/ekaw conference/MICRO conference/confOf geolink/gmo geolink/gbo
After that step, prompt generation was performed as described in Section 3.1.3. It runs with 15 max entities for each field that receives a list of entities in the description document generation. The maximum size of those documents is 4096 characters and the documents are trimmed in case of overflow. The language model used for the embeddings generation is instruction tunned and the instruction passed along with the query was “Given the following SPARQL query, retrieve relevant entities that are related to the query”. The entity inclusion considers the top 2 most similar entities to the SPARQL query and inflates them by traversing depth 2 following outgoing triples and depth 1 following incoming triples and adding them into the resulting ontology. A sample of the number of tokens for all prompts
generated from the pair Cmt-Conference, when tokenized using the Llama3 tokenizer, is present in Table 2. With that strategy, the matching process can run on a 12 GB VRAM GPU.
query whole ontologies
c-person c-preference c-programcommittee
c-bid c-decision c-administrator c-document
c-paper c-conference
For this experiment, three variations were performed in the LLM prompts. The first variation uses a prompt without examples (LLM, no examples), only containing source and target input ontologies and instructions to generate the final alignment file. The next variation is the usage of two diferent examples of alignments between sample ontologies (LLM, no query), one containing only simple correspondences and the other containing examples of complex correspondences. The last variation includes examples and instructions explaining what SPARQL query was used to produce the ontologies subsets (LLM, query+examples).
In total as illustrated in Figure 8, 4730 correspondences were produced in the no examples variation while no complex correspondences were produced. In the no query variation 8567 correspondences were produced while 93 were complex correspondences. And in the query+examples variation 2214 correspondences were produced while 22 were complex. The no query variation is the one that produces the highest number of correspondences and also the highest number of complex correspondences. The variation query+examples generates the lowest number of correspondences however with higher complex entities than the variation no examples. The no examples variation wasn’t able to generate any complex correspondence while a large amount of them are in natural language or in the wrong format. This illustrates the importance of few-shot prompting in the generation of complex correspondences in the right format.
The merged output was evaluated using the automatic evaluator proposed in [15]. It has a CQAoriented and a precision-oriented evaluation. The CQA oriented requires CQAs to operate while the precision-oriented doesn’t require CQAs. The approach was evaluated in the Conference and Geolink datasets. Due to the requirement for CQAs, the Geolink dataset was evaluated only using the precision evaluator and the CQAs for the Conference track are present with the dataset. Moreover, since the evaluator requires instances to work, in the conference dataset, the approach receives as input the Conference dataset and was evaluated in the Populated Conference dataset as it contains instances.
The proposed variations were compared with CANARD [16] matcher and the results of the evaluation are presented in Table 3. The proposed matcher still falls behind the CANARD baseline in most of the metrics. When compared between the variations, the no query have higher results in the evaluation mode while the variation query+examples achieves higher precision. The variation without examples gets zero results in both CQA-based and precision-based evaluations. As can be seen by these results, providing examples is essential to the model’s performance as it learns from the examples how to write the alignments in the EDOAL format.
mode
CANARD LLM (no examples)
LLM (no query) LLM (query+examples)
CANARD LLM (no examples)
LLM (no query) LLM (query+examples)
CANARD LLM (no examples)
LLM (no query)
LLM (query+examples)
One of the reasons for getting worse results than CANARD is that the proposed matcher generates the wrong entity ID prefixes in some correspondences. That can cause worse results as the evaluator uses the correspondences to retrieve the instances and measures the number of the same instances that the correspondences retrieve, however with the wrong IDs, no instances will be retrieved and that correspondence will be considered as a miss. However, while the proposed matcher still has lower performance than CANARD, it can perform the matching even in the absence of instances, contrary to CANARD which requires the ontologies to be populated to match them.
In the Geolink dataset, both CANARD and the proposed matcher fail to generate correct correspondences. In the case of the proposed matcher, the wrong results are caused by not finding any correct correspondences. Some of the found correspondences, for example, the model generates the correspondence between the property http://gmo#hasFix and the image path http://schema.geolink.org/images/gbooverview.svg that is not correct. One possible correspondence found by the proposed matcher is between http://gmo#InformationObject and rdf:about="http://gbo#DigitalObject but if they fail to share common instances they won’t be considered as correct correspondence.
Ontology (including knowledge graph) matching is a task where natural language information is crucial for development. The rise of LLMs has introduced new methods for enhancing the matching process. While LLMs are being adopted in most pipelines for ontology matching, fewer works deal with directly generating the full alignment output by the LLM for complex matching. This section reviews the works in the current state of research on utilizing LLMs to generate correspondences for ontology matching. Most existing methods rely on closed-source LLMs presenting reproducibility issues.
OLaLa [17] is a customizable, open-source framework for ontology matching that uses LLM in the matching pipeline. It extracts matching candidates from input ontologies, which are then analyzed by an LLM using a user-defined prompt. The approach combines the LLMs with other matchers to improve the alignments and maintain the requirement of simple alignments (1:1 correspondences). The correspondence candidates are presented to the LLM for binary or multiple-choice decisions with confidence scores aiding in selecting the final correspondences. Various text extractors provide single text outputs for entities, including preferred labels and verbalized RDF triples. While efective in the simple alignment task, the approach wasn’t explored for the complex matching case. Also, the decision taken by the LLM considers only entity-level information that can be insuficient to decide the correspondence between entities.
In [
More recently, in [18] a pattern-based approach for complex ontology matching has bee proposed. Close to ours, it focuses on the use of SPARQL queries and Large Language Models (LLMs) for detecting and validating complex correspondences between ontologies. Unlike existing methods, this approach avoids lexical matching in the detection phase and relies solely on structural aspects, with LLMs employed only in the final validation step. The approach uses a pattern-based approach to retrieve verbalized candidate correspondences and input into the LLM for verification. However, the output of the LLM is in natural language and the evaluation was performed manually while in this work the LLM output was evaluated automatically.
This paper presents an approach to complex ontology matching by leveraging the capabilities of LLMs and optimizing prompt size using a strategy based on automatically generated SPARQL queries. The technique enhances the scalability and eficiency of ontology matching by reducing the number of entities considered, making it feasible to handle large-scale datasets on consumer-grade hardware. Additionally, the proposed method addresses challenges associated with large search spaces in ontology matching, such as modularization and entity combination, through the use of LLMs. For instance, by directly integrating ontologies into the prompt and outputting results in a structured format like EDOAL, the method enables automatic evaluation of alignment results.
Although a qualitative evaluation is still missing, the initial experiments (fully automated) demonstrated that including examples in the prompts is crucial for performing complex matching tasks, as they significantly improve the accuracy and format of alignments generated by LLMs compared to using the same model without examples. Even with the high amount of data used to train those LLMs and possible training with data from OAEI, the experiments with the LLMs conducted in this work demonstrate that the LLMs are not producing the correct EDOAL format without examples in the prompt, indicating that they haven’t see enough EDOAL data in the training or are not being able to retain it. While the inclusion of examples notably enhances performance, there remains room for optimization, especially in real-world scenarios where the complexity and scale of ontologies introduce additional challenges. Future work will explore advanced prompt engineering techniques, fine-tuning LLMs specifically for complex ontology matching tasks, and developing models capable of handling larger prompts without increasing memory consumption. [13] N. Muennighof, N. Tazi, L. Magne, N. Reimers, MTEB: massive text embedding benchmark, in: A. Vlachos, I. Augenstein (Eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, Association for Computational Linguistics, 2023, pp. 2006–2029. URL: https://doi.org/10.18653/v1/ 2023.eacl-main.148. doi:10.18653/V1/2023.EACL-MAIN.148. [14] J. J. Pan, J. Wang, G. Li, Survey of vector database management systems, CoRR abs/2310.14021 (2023). URL: https://doi.org/10.48550/arXiv.2310.14021. doi:10.48550/ARXIV. 2310.14021. arXiv:2310.14021. [15] É. Thiéblin, O. Haemmerlé, C. Trojahn, Automatic evaluation of complex alignments: An instancebased approach, Semantic Web 12 (2021) 767–787. URL: https://doi.org/10.3233/SW-210437. doi:10. 3233/SW-210437. [16] É. Thiéblin, G. Sousa, O. Haemmerlé, C. Trojahn, CANARD: an approach for generating expressive correspondences based on competency questions for alignment, Semantic Web 15 (2024) 897–929.
URL: https://doi.org/10.3233/SW-233521. doi:10.3233/SW-233521. [17] S. Hertling, H. Paulheim, Olala: Ontology matching with large language models, in: K. B.
Venable, D. Garijo, B. Jalaian (Eds.), Proceedings of the 12th Knowledge Capture Conference 2023, K-CAP 2023, Pensacola, FL, USA, December 5-7, 2023, ACM, 2023, pp. 131–139. URL: https: //doi.org/10.1145/3587259.3627571. doi:10.1145/3587259.3627571. [18] O. Zamazal, Towards pattern-based complex ontology matching using sparql and llm, in: Proceedings of the 20th International Conference on Semantic Systems (SEMANTiCS 2024), SEMANTiCS, Amsterdam, Netherlands, 2024. Poster paper.