The use of Large Language Models as autonomous agents interacting with tools has shown to improve the performance of several tasks from code generation to API calling and sequencing. This paper proposes a framework for using Large Language Models as autonomous agents for the task of Knowledge Graph construction from unstructured text. Specifically, it focuses on triple extraction, which involves identifying entities and their relationships from text to construct a Knowledge Graph. Our novel framework “Auto-KG agent” incorporates two relation extraction tools, REBEL and KnowGL, in conjunction with Large Language Models. Experimental results on the CONLL04 dataset demonstrate that while multi-tool approaches face challenges like hallucination, LLM-based agents are promising in mitigating biases, major event identification, handling negations and modalities thus enhancing extraction accuracy, particularly for complex linguistic structures. The impetus for this research is to overcome the current limitations of existing systems for Knowledge Graph construction and propose a roadmap for developing a robust framework capable of handling the intricacies of natural language with minimal human interference. The paper also discusses future directions, such as emulating Large Language Model training using reinforcement learning with human feedback, incorporating query decomposition, and integrating a re-ranking module. Through this research, the authors aim to set a new direction for future endeavours in building advanced, reliable systems for knowledge extraction. Overall, this work highlights the potential of LLM-based agents for knowledge graph construction and proposes a framework for harnessing their capabilities.
CEUR
ceur-ws.org modalities and negations, Mitigating biases
The advent of Knowledge Graphs has revolutionized the way we represent and utilize
information in the digital age. By structuring data as triples—consisting of a head entity, a relationship,
and a tail entity (h, r, t) — Knowledge Graphs provide a semantic framework to describe the
varied and countless entities and their interrelations in the objective world. This structured
approach to data organization underpins intelligent applications and has garnered significant
attention in both academic and industrial spheres due to its potent semantic processing capabilities
and open organizational structure [
In the field of Natural Language Processing, extracting relational facts from text is crucial.
Understanding the semantic relationships between entities in unstructured text helps convert
raw data into structured formats. This structured data is extremely valuable for several tasks,
such as building and enhancing Knowledge Bases. These bases are essential for powering
applications that rely on knowledge [
In the realm of information extraction, frameworks like REBEL [
Despite their innovative approaches, REBEL and KnowGL have certain limitations. The performance of these models is heavily influenced by the quality of their pre-training data. Biases or inaccuracies present in the training datasets can propagate through the models, afecting the accuracy of the extracted relations. Furthermore, the ability of these frameworks to generalize across various domains and text types hinges on the extent to which the pre-trained language models are fine-tuned or further pre-trained on domain-specific datasets. While Large Language Models inherently possess a broad understanding of language, their performance in specialized contexts improves significantly with targeted fine-tuning. Additionally, these systems may struggle with complex sentence structures and fail to identify all relevant major events, particularly in sentences laden with modalities or multiple clauses.
To address these challenges, we introduce a novel framework that synergizes the capabilities of REBEL and KnowGL with the nuanced understanding of Large Language Models. Large Language Models demonstrate remarkable eficacy in processing sentences with modalities and complex structures, leading to accurate event identification and triple extraction. This integration not only enhances event detection but also aids in mitigating biases inherent in training data, ensuring a more comprehensive extraction of triples. Apart from introducing a novel framework, we also aim to answer the following research questions: RQ1: How efective are Large Language Models in mitigating biases for extracting triples which are present in the datasets used for training information extraction tools? RQ2: To what extent do Large Language Models accurately handle modalities and negations in natural language, and how does this capability afect the quality of triple extraction? RQ3: Can Large Language Models enhance the identification of events within unstructured text, thereby improving the accuracy and completeness of triple extraction? RQ4: How well do Large Language Models generalize across diferent datasets without the need for extensive training or fine-tuning, particularly in the context of triple extraction for knowledge graph construction? RQ5: What is the impact of using multiple tools versus a single tool on the performance of triple extraction?
The impetus for our research is to overcome the current limitations of existing systems and chart a course for the development of a robust framework capable of handling the intricacies of natural language. Our experiments are designed with this objective in mind and are conducted with the resources available to us. Through this research, we aim to set a new direction for future endeavors in building advanced, reliable systems for knowledge extraction and reasoning.
Using Large Language Models as autonomous agents has become increasingly popular in recent
research. Large Language Models possess advanced reasoning abilities and skills in utilizing
tools, making them well-suited for autonomous operations. They excel in tasks like acquiring
knowledge, understanding instructions, generalizing information, planning, and reasoning,
showcasing their potential for autonomous tasks [
This section has covered the existing studies on LLM-based Agents for Knowledge Graph
Generation from Text. Jiang et. al [
Methodology
Tool Learning [
Description Tasks In-context demonstration and Generation Reg- Tool manipulation, multi-tool ulation usage Learning on high-quality instruction datasets Tool manipulation, multi-tool usage Reasoning and knowledgebased tasks Reasoning algorithm
Dataset
ToolBench [
A knowledgeable selflearning strategy for path planning
HotPotQA [
MAWPS [
[
WebQSP
CWQ
GrailQA [
In the construction of LLM agents, an LLM acts as the primary controller or “brain,” orchestrating
the sequence of operations required to accomplish a task or respond to a user query. These LLM
agents may require additional modules like planning, memory, and tool utilization to enhance
their functionality [
Our research focuses on extracting triples from unstructured text to facilitate Knowledge Graph construction. Triple extraction entails identifying and extracting structured information in the form of triples, which comprise a subject entity, a relation or predicate, and an object entity. This process is crucial in converting text into a structured format suitable for tasks like knowledge representation and information retrieval in natural language processing.
Relation extraction is another vital task we explore, involving the identification and extraction of semantic relationships between entities mentioned in unstructured text. These relationships, such as “is married to” or “works for”, capture meaningful associations and are represented as triples for downstream applications like Knowledge Graph construction.
REBEL [
REBEL uses a special kind of model called autoregressive sequence-to-sequence (seq2seq)
models. These models are good at making text and also understanding natural language. The
main part of REBEL is a seq2seq model based on BART [
KnowGL [
The goal of our framework is to facilitate automatic triple extraction from text inputs. This framework is designed as a multi-tool system utilizing Large Language Models to execute the task of triple extraction.
Figure 1 outlines a framework for training a Large Language Model (LLM) using a method
referred to as RLHF, which stands for Reinforcement Learning from Human Feedback. The
lfowchart is divided into two main sections:
1. The Large Language Model training procedure using RLHF [
The training procedure begins with raw text to pretrain the Large Language Model. This pre-training step is where the model learns from a large corpus of text to understand language patterns and structures.
After pretraining, the model becomes a “Pretrained LLM” which is then subjected to “Supervised Fine-tuning” using Demonstration data. Demonstration data consists of prompts and response pairs. This step involves training the model on specific examples to perform certain tasks and understand particular domains better. This form an instruction-following Chatbot of low quality.
The fine-tuned model, referred to as “SFT LLM” is then used in conjunction with “Human Preference Data” to train a “Reward Model”. The Human Preference data consist of prompts, winner tuples and loser tuples. This reward model evaluates the outputs of the LLM and provides feedback on its performance.
The feedback from the reward model is used in Reinforcement Learning, where the Large
Language Model is further trained using RLHF (Reinforcement Learning with Human Feedback)
training prompts to improve its outputs based on human preferences and feedback. The RLHF
training prompts also consists of prompt, winner tuples and loser tuples. In the end we get a
high-quality instruction-following RLHF LLM [
RLHF LLM Based Autonomous Agents for Triple Extraction for KG Construction The bottom section of the flowchart shows the application of the trained RLHF LLM for a specific task: Autonomous agents for triple extraction to construct Knowledge Graph.
A User inputs a complex query, a command which is then decomposed by the querydecomposition LLM. The decomposed query is processed by the “RLHF LLM” which interacts with a Tool DB (Tool Database). Here the tools in the Tool DB are “REBEL” and “KnowGL”.
The RLHF LLM then performs re-ranking of triples extracted from the complex query which involves selecting the most relevant and accurate triples based on their count of occurrences of triples in the Knowledge Graph. The final output are then re-ranked which would be used in the construction of a Knowledge Graph. The system prompt for RLHF LLM is provided in the Appendix A.
It is a comprehensive framework for training a Large Language Model using human feedback and then applying this model to extract triples for building Knowledge Graph.
We implement the “Auto-KG Agent” framework to facilitate automatic triple extraction from text inputs. This framework is designed as multi-tool system utilizing Large Language Models to extract triples. We utilise REBEL and KnowGL as tools for triple extraction (relation along with entity). More such frameworks can be added as tools in the Tool DB. Large Language Model is asked to return the entities in JSON format. The diagram presented in the second half of the Figure 1 serves as the visual representation for this section.
The current system comprises the second half of the the Figure 1, without the query decomposition Large Language Model and Large Language Model training using RLHF. As of now we only incorporate Tool DB, LLM without RLHF (only system prompt) and re-ranking of triples based on the length of relation extracted. We state in the section 7 for incorporating other modules in the “Auto-KG Agent” framework.
Dataset We evaluate our system’s performance on the CONLL04 dataset [
4
5
1,290 (922)
343 (231)
422 (288) Evaluation Metrics The evaluation process compares the predicted triples extracted from test data with the ground truth triples. Each instance in both datasets is represented as a dictionary, with a unique identifier and a set of triples. Each sentence has an object corresponding to it which stores the triple.
To calculate the true positives (correctly predicted triples), we iterate through each instance in the ground truth data. For each instance, we check if the corresponding instance exists in the predicted data. If it does, we find the intersection of the triples in the ground truth and predicted data, which gives us the number of correct predictions (true positives).
Additionally, we also calculate the number of extra predictions made by the model that is the count of triples not present in the ground truth. However, we don’t calculate scores for them as it would require Human evaluation.
After calculating count of true positives, we compute both micro and macro scores for precision, recall, and F1 score. Micro scores consider the total number of triples in the entire dataset for calculating precision, recall, and F1 score, while macro scores average these metrics across each instance in the dataset.
Overall, this evaluation process enables us to assess the performance of the triple extraction model by quantifying its precision, recall, and F1 score, considering both individual instances and the entire dataset.
We did a strict evaluation where the correctness for triples extracted as a whole is compared with the corresponding head entity, tail entity and relation in the ground truth. Following are the counts of unique relations extracted for diferent frameworks: • REBEL - 68 relations mapped to 5 relations • KnowGL - 58 relations mapped to 5 relations • REBEL + KnowGL - 90 relations mapped to 5 relations Triple Extraction Tools REBEL and KnowGL frameworks are used as triple extraction tools in our experiment. In our evaluation of the REBEL model on the CONLL04 test dataset, we encountered a diverse set of 68 unique relations extracted by the model. To align these with the CONLL04 dataset’s five predefined relations, we undertook a manual mapping process. This process was guided by semantic similarity and contextual relevance, ensuring that each extracted relation was correctly associated with one of the canonical relations such as ‘killed by’, ‘residence’, ‘location’, ‘headquarters location’, and ‘employer’, as originally formatted in the REBEL paper. The necessity for this manual mapping arose from the fact that the REBEL model, trained on multiple datasets, identified relations beyond the scope of the CONLL04 dataset, requiring careful consideration to maintain semantic integrity. The manual mappings for these relations, based on their semantic similarity to the corresponding five relations were carried out. Similarly, KnowGL had 54 unique relations being extracted. We followed the same mapping procedure for it. Figure 2 shows the distribution based on the number of occurrences for 5 types of relations extracted for diferent experiment design settings.
For details, readers can refer to Appendix A. The code for experiments is available in GitHub 1. 1https://github.com/Ananyaiitbhilai/Text2Triple-LLM-Agent RLHF LLM In our experimental setup, we use Large Language Models without any fine-tuning or instruction tuning. However, in future we plan for RLHF LLMs being used to orchestrate the tool execution for triple extraction. Specifically, we used two open-source Large Language Models for benchmarking: “Gemma” and “Mistral”
Gemma [
Mistral-7B-Instruct-v0.2 [
For additional experimental set-up refer Appendix 3.
The Table 3 illustrates the number of relations and instances for diferent experiment design settings.
In Table 4, REBEL refers to all triples extracted by REBEL, REBEL (subset Mistral) refers to removing all the triples which could not be extracted because of the hallucination (not returning triples in expected JSON format rather string format or other format) in the Mistral LLM. So those triples are removed from ground truth and then evaluation is carried out. Similar follows for REBEL (subset Gemma). Here the tools like KnowGL, REBEL are used as a single-tool in conjuction with LLM (Mistral and Gemma), REBEL and KnowGL as a multi-tool in conjuction with LLM (Mistral and Gemma).
P 0.16
From Table 5 it can be observed that Gemma, a significant number of hallucinations were observed, accounting for 99 out of 288 total responses. This higher incidence of hallucination in Gemma was primarily attributed to incorrect JSON format returned by the model. Conversely, Mistral exhibited a lower occurrence of hallucination, with only 26 out of 288 total responses displaying such phenomena. The single-tool here refers to REBEL, and multi-tool has both REBEL and KnowGL. “Hallucination” refers to a phenomenon where the model generates text that is incorrect, nonsensical, or not real.
Mistral (multi-tool) 288 288 288 99 26 42
We investigated the performance of multiple tools versus single tools for relation extraction and observed a notable decline in scores with multi-tool usage, suggesting that single-tool approaches may yield better results as shown in Table 4. We attributed this drop to increased hallucination, particularly more prevalent when employing multiple tools due to hallucination. However, single-tool usage also presented challenges, as occasionally, the returned format did not align with the one specified in the system prompt. Moreover, From Table 4 , it can be also be observed that REBEL with Large Language Model and only REBEL has almost same performance. It is due to the fact that REBEL and KnowGL is being used as a tool to trigger the action of extracting the relations. Both KnowGL and REBEL have the same architecture thus similar biases.
Our findings underscore the need for integrating Large Language Models with extraction tools to harness their full potential. While tools exhibits shortcomings in certain contexts, Large Language Models ofer a complementary approach, particularly in mitigating biases and enhancing extraction accuracy. By empowering Large Language Models to engage in more nuanced planning or decomposing the query, we anticipate significant improvements in relation extraction performance.
Gemma’s higher scores are because of a larger number of responses being generated as strings rather than in the expected JSON format as shown in Table 5. Consequently, these nonformatted responses are removed, resulting in fewer triples available for evaluation as compared to Mistral. This phenomenon contributes to Gemma’s higher scores in triple extraction tasks compared to Mistral.
In order to answer our aforementioned Research Questions, we took certain examples that had multiple events, complex clauses, negation, modalities and then evaluation by human was carried out manually to check for the correctness of triples extracted from the pre-existing tools and our Auto-KG agent.
Event identification and mitigating biases in Triple Extraction Our investigation uncovered instances of flawed relation extraction within the REBEL and KnowGL tools. In a sample sentence, “While Marie Curie and Albert Einstein conducted groundbreaking experiments in their laboratories at the University of Paris, Leonardo da Vinci’s sketches of Renaissance architecture in the bustling streets of Florence sparked inspiration across Italy,” REBEL solely identified the triple “Florence, located in, Italy”. However, this sentence encompasses two distinct events: “The experimentation conducted by Marie Curie and Albert Einstein at the University of Paris”, and “the inspiration sparked by Leonardo da Vinci’s sketches across Italy”. REBEL’s oversight stemmed from its inclination towards location-centric relations, influenced by biases within the training data.
In contrast, our Auto-KG agent showcased promise in mitigating such limitations. They accurately extracted triples from the sentence, capturing nuanced relations such as “Marie Curie, experimented at, University of Paris”, “Albert Einstein, experimented at, University of Paris”, “Leonardo da Vinci’s sketches, located in, Florence”, and “Leonardo da Vinci’s sketches, sparked inspiration in, Italy”. This underscores the Auto-KG agent’s proficiency in comprehending complex linguistic structures and discerning meaningful relations, showcasing their potential for enhancing relation extraction accuracy.
Negation Handling Discrepancies in Triple Extraction In our comparative analysis, we also observed a notable discrepancy in the handling of negation between REBEL and KnowGL tools, as opposed to our Auto-KG agent in triple extraction tasks. Both REBEL and KnowGL demonstrated limitations in efectively managing negation cues within text, resulting in erroneous extraction of triples. Conversely, our tool exhibited robust performance in negation handling, yielding more accurate triple extractions even in the presence of negation cues.
We considered the sentence “Fado does not work at IIT”. REBEL and KnowGL erroneously extract a triple indicating that Fado works at IIT, failing to account for the negation. In contrast, our tool was adept at discerning the negation cue “not” and appropriately adjusting the extracted triple to reflect the absence of the stated relationship, thereby accurately capturing the intended semantics of the sentence.
This discrepancy underscores the nuanced understanding of language exhibited by Large Language Models, enabling them to efectively navigate linguistic complexities such as negation cues in triple extraction tasks. It highlights the potential of leveraging LLM-based approaches to enhance the accuracy and reliability of triple extraction processes in natural language processing applications.
Generalising well on various datasets A significant disparity in the performance of seq2seq based approaches such as REBEL and KnowGL when trained or fine-tuned on specific datasets. While these models exhibit impressive performance within the confines of their training data, they demonstrate limited generalization capabilities beyond the dataset they were trained on. Conversely, Large Language Models showcase remarkable generalization prowess even without explicit training on a particular dataset. This discrepancy underscores the inherent adaptability and robustness of LLMs, enabling them to efectively handle diverse datasets and tasks without the need for extensive training or fine-tuning.
For future, our focus lies on emulating the Large Language Model training methodology using Reinforcement Learning with Human Feedback (RLHF) as detailed in Section 4
Additionally, we intend to incorporate a query-decomposition LLM to partition complex user queries into sub-queries, facilitating more precise event identification and subsequent triple extraction.
Furthermore, our proposed future work entails synergizing LLMs with multiple extraction tools to enhance their generalization capabilities across diverse datasets without requiring explicit training. This approach holds potential to surpass the performance of seq2seq models such as REBEL and KnowGL.
Moreover, we aim to integrate a re-ranking module into our framework. This module will prioritize all extracted triples based on their confidence levels, ensuring a more refined and accurate output.
We also aim to develop a diverse dataset that encompasses a wide range of relations and includes a variety of sentence structures. This dataset is intended to serve as a robust benchmark for evaluating performance in triple extraction tasks.
The paper presents a novel framework that integrates Large Language Models (LLMs) with existing tools like REBEL and KnowGL for the task of triple extraction from unstructured text to construct knowledge graphs. The proposed framework aims to leverage the strengths of LLMs in understanding complex linguistic structures, handling modalities and negations, and mitigating biases inherent in training data. The experimental results on the CONLL04 dataset indicate that while multi-tool approaches face challenges such as hallucination, the integration of LLMs shows promising results in enhancing extraction accuracy.
There are certain limitation of our research work: 1. Limited LLM Models Evaluated: The experiments were confined to using the Gemma (2B parameters) and Mistral-7B LLMs. The performance of other large language models like LaMDA or models with higher parameter counts (e.g., GPT-4) remains unexplored. Future work could extend these experiments to a broader range of LLM architectures and sizes to provide a more comprehensive evaluation.
2. Limited Task Coverage: The current study focused on a specific task: triple extraction for knowledge graph construction. However, knowledge graph construction and reasoning encompass a wide range of tasks, and the performance of LLMs on other tasks, such as entity linking, relation classification, or multi-hop reasoning, remains unexplored. Future research could extend the evaluation to a broader set of tasks to provide a more comprehensive understanding of Large Language Model capabilities in the context of knowledge graph construction and reasoning.
3. Limited Evaluation Dataset: The paper evaluates the proposed framework on the CONLL04 dataset, which comprises sentences extracted from news articles with a limited set of entity types and relation types. This dataset may not fully represent the diversity and complexity of real-world text, potentially limiting the generalizability of the findings to other domains and contexts. In Future, evaluation can be carried out on other datasets
4. Reliance on Manual Mapping: The paper mentions that manual mapping was required to align the relations extracted by REBEL and KnowGL with the canonical relations in the CONLL04 dataset. This manual intervention introduces potential biases and inconsistencies, as the mapping process may not be entirely objective or scalable across larger datasets or domains.
The authors acknowledge these limitations and express their anticipation for future research opportunities that would allow them to further explore these areas and provide a more comprehensive evaluation of LLM capabilities in the context of knowledge graph construction and reasoning. The research sets a new direction for future work in building advanced, reliable systems for knowledge extraction and reasoning. It highlights the potential of LLM-based agents for knowledge graph construction and proposes a comprehensive framework for harnessing their capabilities.
Additional details about System and parameters for the preliminary Experimental set-up • The context size(n_ctx) is the maximum number of tokens that the model can account for when processing a response. this includes the prompt, and the response itself. In our case the context size was set to 2048. • The maximum number of tokens to generate is 2000 in our case. If _ ≤ 0 or
None, the maximum number of tokens to generate is unlimited and depends on n_ctx. • Average inference time per context/sentence for CONLL04 test dataset for extracting triples in conjunction with LLMs was 25 seconds. • Temperature was set to 0 • The gguf files for Mistral and Gemma were run locally on Mac M1.
The system prompt is shown in Figure 3 Relation Mappings Key
Values employer headquarters location killed by location residence derivative work, inception, instance of, owned by, owner of, part of, participant, participant in, performer, twinned administrative body, occupation, field of this occupation, member of political party, work location, language used, participant in, participant, owner of, owned by, member of, notable work, instance of, interested in, ofice held by head of government, chief executive oficer, educated at, subclass of, part of, ofice held by head of state, chairperson, executive body, industry, oficeholder, position held, practiced by, language of work or name, director / manager, employer, field of work, language of work or name, notable work, occupation, member of, member of political party, oficeholder, operator, position held, educated at, founded by, product or material produced, subsidiary, work location, author, ofice held by head of government, used by, uses, candidacy in election, candidate, chairperson, head of government headquarters location, twinned administrative body, applies to jurisdiction, legislative body, military branch, contains administrative territorial entity, parent organization, operating area, legislative body, contains administrative territorial entity, headquarters location, located in the administrative territorial entity, ethnic group, language used, military branch, parent organization, applies to jurisdiction cause of death, perpetrator, convicted of, killed by, place of death, facet of, date of death, main subject, place of death, facet of, significant event location, capital, continent, located in time zone, shares border with, mountain range, located in or next to body of water, candidate, significant place, spouse, place of publication, target, country, located in or next to body of water, location, mouth of the watercourse, point in time, capital, capital of, shares border with, tributary, diplomatic relation, place of publication, spouse place of birth, based on, country of citizenship, date of birth, has part, number of participants, history of topic, place of birth, country of origin, has quality, significant event, occupant, relative, residence Figure 3: System Prompt