This position paper introduces a new way for large language models (LLMs) to interact with Knowledge Graphs (KGs), especially for information extraction. Most relation extraction approaches focus on finding the corresponding textual spans for subject, relation, and object in a text but ignore the actual state of the KG and the schema defined therein. In this work, the function-calling possibility of LLMs is explored to search for already defined entities in the KG and add information in the form of triples to the KG. A huge improvement over previous techniques is that it can also deal with various large ontologies / KGs and that the added information fits to a provided KG on a schema and instance level. Further validation checks can be added to increase the quality of the extracted statements. A small playground is provided to analyze the efectiveness of the approach.
CEUR ceur-ws.org
The rise of the transformer architecture and Large Language Models (LLMs) has led to increased
use of those models in information extraction tasks to enhance extraction quality. For closed
source information extraction (all possible relations are already given) either the model is trained
to extract a fixed set of relations or the possible relations are provided in the prompt. The latter
approach will not scale for large knowledge graphs with like Wikidata [
In this work, we propose an alternative technique to combine LLMs and KGs via function
calling. Some LLMs like GPT-3.5 Turbo (closed source) or Mistral Small [
Given carefully designed functions that interact with the KG, as e.g., searching/creating instances or properties, the information extraction process can be simplified and scaled to larger ontologies and KGs. CEUR Workshop Proceedings
In the NLP community, the closest task to our approach is referred to as relation extraction, and
the survey by Xu et al. [
Given the scenario that some instances, properties, and classes are already defined in the KG, the task is to extract information from natural language text that fits to the given schema. This means that each resource is defined in the KG and no two resources are created for the same real-world entity. The idea is to allow LLMs to interact with the KG via function calls. All possible functions, together with a description of the functions and parameters, are provided in the prompt as well as the text to extract information from. The model can then return a structured query which is executed, and the textual result is provided to the LLM, which can then decide to call another function or stop the chat by providing a final statement without calling a function.
According to the example sentence “Paris is the capital of France“, we will show which functions should exist and which parameters they should expect. All functions are listed in Tables 1, 2, and 3. In the first step, the subject (in this case “Paris“) needs to be searched in the KG because it might already exist. Thus, a function search_instance which expects some text is needed. The result of such a function needs to be a textual representation of possible candidates in the KG which contains enough information to disambiguate the entities. Thus, the identifier (URL), the rdfs:label and rdfs:comment are included in the result. A simple SPARQL query is shown in Table 1. The result of the SPARQL query is a mapping between the selected variables and the possible substitutions, which need to be serialized as text for the LLM. We implemented all serializers available in RDFLib, which are CSV, JSON, TXT, and XML1.
Based on the result of the search_instance function, the LLM needs to decide if the entity described in the text is already represented in the KG or not. In the latter case, the LLM can either call the search_instance function with another text to search for (which increases the possibility of finding entities that are represented with synonyms) or create the entity in the KG. To this end, an additional function create_instance is required (shown in Table 2). It expects a label, a comment, and an instance type. All function parameters can freely be filled with text generated by the LLM.
Due to the fact that the LLM calls an actual function that is implemented in Python, various
1https://rdflib.readthedocs.io/en/stable/plugin_query_results.html
validation checks can be performed. As an example, the instance type parameter needs to be
a class defined in the KG. Most often, LLMs create URIs that are based on public KGs such as
DBpedia [
After the instance for Paris is either found or created in the KG, the same approach is applied to the property (capital of) and object (France). The LLM decides which information to extract from the given text, which entities to search/create, and the order in which they are processed. Finally, the statement containing the subject, predicate, and object can be added to the KG via } if not, return “The instance type of the instance is not a class“. else, create a new URI named instance_uri and execute: INSERT DATA { <{instance_uri}> a <{instance_type}>;
rdfs:label "{label}"; rdfs:comment "{comment}". } return instance_uri check that subject and object are resources defined in the KG and predicate is a property defined in the KG check that subject and object are coherent with domain and range if all validation checks are successful, execute: INSERT DATA {
<{subject}> <{predicate}> <{object}>.
}
SPARQL query / validation code if super_class is given, check that it is a RDFS or OWL class in the KG create a new URI named class_uri and execute: INSERT DATA { <{class_uri}> a owl:Class; rdfs:subClassOf <{super_class}>;
rdfs:label "{label}"; rdfs:comment "{comment}"; } return class_uri if domain and/or range is given, check that it is a class in the KG if super_property_id is given, check that it is a property in the KG if all validation checks are successful, create property_uri and execute: INSERT DATA { <{property_uri}> a rdf:Property; rdfs:subPropertyOf <{super_property_id}>; rdfs:label "{label}"; rdfs:comment "{comment}"; rdfs:domain <{domain}>; rdfs:range <{range}>. } return property_uri the create_statement function.
The validation checks are a huge advancement because it can be ensured that the added instance will have the correct class defined in the KG, which would not be possible otherwise. Thus, it is also a disadvantage to just provide a function execute_sparql (which would also be possible) because of those additional validations.
The whole extraction works in a greedy way because only one text is processed at a time.
Thus, a decision cannot be revoked once taken, e.g., a domain of a property is fixed when the
property is created. Still, there are possibilities to add more functions that change/remove
classes, properties, or instances. One advantage of the greedy approach is that no further
alignment is necessary, which would be the case when each Wikipedia page is processed in
parallel with an empty KG because each resulting KGs needs to be aligned (the amount of
alignments is usually (2), where n is the number of KGs [
We provide an implementation2 of all functions using the LangChain framework3 to easily parse the LLM output and call the corresponding functions implemented in Python using RDFLib. In our first experiments with GPT-3.5 Turbo, it turned out that the best SPARQL result serialization is CSV (probably because it is very structured). But the final goal is to use open-source LLMs to have a reproducible setup and the possibility of running the information extraction model on our hardware without paying for API access.
Due to the fact that most LLMs are not specifically trained for function calls, there is a need for
a dataset showing the use of exactly those functions. Some of the most promising open-source
models are the Mistral models Small [
But all of the models would rely on the definition of the function without any example of
how to apply the functions and what to do with the result of a function call. Thus a dataset
needs to be generated to show how these functions work in actual information extraction
scenarios. A natural choice is Wikipedia because the text is usually checked by multiple persons,
and many knowledge graphs based on Wikipedia exist. For their REBEL approach (Relation
Extraction By End-to-end Language generation) [
In addition, all surface forms8 of entities can be used to showcase that the parameter for
search functions can be changed to actually find the correct entity in the KG. Given the running
example, the search for the entity France could be called multiple times with diferent
textual representations such as search_instance("France") and search_instance("French
Republic"). It shows that the search functions can and should be called multiple times in case
the correct entity is not yet retrieved. In comparison to relation extraction, the dataset also needs
to reflect the diferent states of the KG, e.g., the entity is not contained in the KG or the search
functions return multiple possible candidates. This could be regulated by a hyperparameter
∈ [
In this paper we proposed a new technique to combine Large Language Models and Knowledge Graphs. The basic idea is to let the LLM interact with the KG by function calls that directly reflect the state of the KG. Thus, all extracted triples will be coherent to the schema, which is already defined but can also be changed by the LLM if corresponding functions are provided.
The next step is to create the dataset and fine-tune open-source LLMs such as Mistral or
Llama 3 models [
Up to now, the dataset only provides examples of relations between individuals but not between individuals and literals. The main dificulty is to map the literal value to the text in a Wikipedia article, which can be arbitrarily complex, e.g., dates that are written in completely diferent ways, such as ’2024-01-01’, ’Jan. 1, 2024’, or even not the full information like ’Jan. 1’.
Currently, the search functions only search for entities where the exact search string is
contained in the labels, and the LLM needs to call those functions possibly multiple times. An
improvement would be to implement a semantic search via Sentence-BERT models [
Another line of future work is to extend the number of involved LLMs and to include a human in the loop. Especially when humans create ontologies, the best results are achieved if more people interact together. In the same way, the number of LLMs could be increased. Technically, another function like ask_expert (similarly a function ask_human_expert for human involvement) could be introduced.
7https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Linking#Duplicate_and_repeat_links
8The term “surface form” was mainly introduced by DBpedia Spotlight [
9https://huggingface.co/sentence-transformers/all-mpnet-base-v2