Knowledge graphs are widely used in Retrieval Augmented Generation (RAG) and Explainable AI (XAI), since they can illustrate semantic relationships generated by Large Language Models (LLMs). Recent studies focus on generating knowledge graphs from unstructured data to improve RAG performance; however, they do not explain the underlying graph structure. The analysis of synthetic graphs behind modern graph-based RAG systems shows that the structures generated by LLMs fail to capture semantic relations represented in the unstructured texts. The paper addresses these limitations by proposing a novel approach that enhances explainability and reduces hallucinations in knowledge graph generation for RAG systems. The paper introduces AGGILE, a tool for LLM-based graph generation from unstructured data and their visualization for enhanced explainability. Our approach extracts keywords and related concepts from unstructured data and generates predicates that denote semantic relations between the extracted entities. We evaluated our framework using qualitative linguistic criteria on a sample of unstructured Wikipedia data. The results of our qualitative analysis show that the AGGILE tool captures complex semantic relationships from unstructured data, allowing for in-depth LLMs' knowledge exploration. The method is scalable and domain-agnostic; however, a small rate of hallucinations occurs in the generated structures, and there are limitations in the input text size for proper graph modeling. Overall, this work contributes to improving the transparency and controllability of graph RAG. The AGGILE tool is open source: demo versions, source code, and software documentation are available in the project repository at https://github.com/vifirsanova/AGGILE/.
A knowledge graph is a data model that represents objects and relations between them with nodes and
edges. In Artificial Intelligence (AI), knowledge engineering, and Natural Language Processing (NLP),
knowledge graphs are often used to depict semantic and logical relations between concepts to represent
data, cognitive structures, or common knowledge. Graphs are often applied to enhance search engines
and information retrieval systems [
Retrieval Augmented Generation (RAG) [
While there are various open-domain knowledge bases, such as Wikidata [
From our empirical findings and observations made with GraphRAG API [
We suggest that the limitation is caused by the fact the graph structure is generated end-to-end, i.e. the graph is resulted from the input data directly without fragmenting the graph into its structural elements (nodes and edges). The study hypothesis is that knowledge graph generation can become more controllable, providing more advanced explainability options through visualization tools, if the graph synthesis implies sequential generation of a graph components, i.e. nodes and edges. The study proposes a novel approach called Automated Graph Generation for Inference and Language Exploration (AGGILE) that uses prompt-tuning to build knowledge graphs from unstructured data sequentially using the following algorithm: • Extract entities from the unstructured data to form graph nodes • Extract related items for the extracted entities (related nodes) • Generate predicated between the extracted entities and the related items (edges) • Form triplets from two nodes and edges between them • Connect the triplets to a graph • Set weights to edges according to the number of connections with other nodes
The resulting structure is a weighted graph, where the nodes with more relations get higher weights. This model allows for extracting topics, finding clusters and searching through subgraphs, as well as visualizing weighted graphs allowing for fine-grained RAG explainability analysis. The AGGILE tool provides the following functions: • Graph structure generation from unstructured data • Interactive graph visualization • Graph-based RAG function
The approach is evaluated using linguistic qualitative analysis based on criteria involving searching for noise in the generated data, assessing nodes’ and edges’ accuracy and relevance to the data topic, as well as evaluating graph scalability. The methods from linguistics are used to explore the semantic relations built with the AGGILE tool, analyze the word forms and parts-of-speech used by the model to denote nodes and edges, and identify LLM hallucinations and logical errors.
The AGGILE tool is fully compatible with the HuggingFace infrastructure. The tool is open source and available in the project repository at https://github.com/vifirsanova/AGGILE/. The tool generates JSON object with graph structure, HTML file with interactive graph built with Plotly, and answer to user question. The tool supports a wide range of LLMs, while the experiments described in this paper are based on DeepSeek-R1-Distill-Qwen-32B model, a distilled model from DeepSeek-R1 based on Qwen.
The study contributions are the following: • A novel tool for RAG explainability based on automated knowledge graph generation and visualization • A novel method for generating knowledge graphs from unstructured data based on sequential nodes and edges synthesis for RAG and XAI • A novel approach to qualitative knowledge graph assessment for XAI based on linguistic theory Overall, AGGILE provides a toolkit for automated graph generation, explainability analysis and RAG implementation. The study findings show that sequential generation of graph components from unstructured data allows for building scalable RAG models with high explainability potential, enabling transparent and controllable language generation.
The AGGILE tool demonstrate the ability to highlight target domain and topical clusters, as well as iflter irrelevant information. The model can produce weighted relations, allowing to extract subgraphs and learning how LLMs prioritize the information during the processing. Also, the model can extract diverse semantic links, for example, hierarchical and associative relations.
However, the proposed method has several limitations. The model implies several queries to the LLM, increasing the time required to generate the graph. This limits the AGGILE eficiency, however, we plan to solve the issue by using vector representations directly instead of using text-based prompt-tuning. Also, the model shows some logical errors in forming edges between the nodes, and tends to generate relations between identical nodes. The latter problem was fixed by applying a simple rule for filtering triplets with duplicating nodes.
In perspective, we plan to test the AGGILE performance on popular RAG benchmarks, expland the tool interface with tablular representation of the generated nodes and edges, and present an analytical framework for LLMs’ assessment based on the developed linguistic criteria for graph-based XAI.
A knowledge graph (KG) is a structured representation of real-world entities and the relationships
between them [
The semantic linking methodology employed in this study is inspired by WordNet [
A triplet (or triple) is a part of KG (subgraph), consisting of a keyword (subject), a related word (object) and the semantic relation (predicate) between them: (, , ) = (, , ). Some examples of such triplets include: Mathematics () is a branch of () Number Theory (), Geometry () studies () Shapes (), Analysis () is a branch of () Mathematic Logic ().
Retrieval-augmented generation (RAG) is a technique designed to enhance the performance of large
language models (LLMs) by integrating them with an external knowledge base (KB) [
An advanced variant of RAG, known as GraphRAG [
SubgraphRAG [
G-Retriever [
While the aforementioned works primarily focus on KG-based information retrieval, the following studies concentrate on KG generation.
In the first one [
The second study is devoted to explanation generation for learning recommendations [
None of the aforementioned works explicitly addresses the problem of explainability in graph generation itself. They are aimed to assess LLM’s outputs and reasoning without identifying the specific data components contributing most to KG construction. Our work is devoted to exploration of this issue. Moreover, it is designed to handle unstructured user data and introduces a more controllable graph generation method through structured prompt engineering.
The AGGILE is a Python-based tool designed to generate knowledge graphs from unstructured data for the purposes of XAI. The graphs are generated from plain text using LLMs.
The tool generate triplets, which is a set of 2 nodes and 1 edge denoting a relation between them. Each triplet has the following form: subject-predicate-object, where subject and object are nodes, and predicate is an edge.
The tool is based on prompt-tuning methods, prompts are used to guide LLMs in identifying key lexical entities to form the graph. The AGGILE Python class consists of methods for extracting entities, identifying relations between them and forming triplets, as well as constructing and visualizing graph in HTML-format.
• Entity extraction: identifying key concepts, named entities, and keywords from the input text • Relation extraction: determining relationships between extracted entities • Graph construction: visualizing the extracted entities and relationships as a knowledge graph First, the tool extracts N keywords, named entities or produces semantic concepts consisting of one word or a phrase based on the provided text. N is the number of instances defined by user (10 by default). The process is done through LLM prompt-tuning (see Appendix A "Subject extraction"). This list is forms a list subject, and it will contain a list of nodes for the graph.
Next, the tool extracts 5-10 related words for each primary node (subject) and produces a predicate between them. Predicates define the relationship between subjects and objects as a verb or a phrase describing it. The list of secondary nodes denoting related words forms the list of objects, and a list of predicates form the edges of the graph. See Appendix A "Object extraction" and "Predicate generation".
This way of generating graphs is more controllable and predictable due to the ordered prompt structure. LLM inference is implemented through HuggingFace Hub’s InferenceClient, see the project repository for more details.
As a result, triplets are combined into a JSON object used to construct a graph. Visualization is made in HTML-format using Plotly library. Nodes’ colour depends on their degree: the more relations has the node, the higher weight is assigned to it. The weight defines the nodes’ degree.
The AGGILE class also provides a method for implementing RAG based on the generated graphs. However, this study focuses on the explainability potential of the developed tool.
• max_tokens = 1024 for subjects’ extraction, • max_tokens = 512 for objects’ extraction and predicates’ generation, • temperature = 0.5 in all cases, • top_p = 0.1 in all cases.
Either of three lists (subjects, objects, predicates) are generated as JSON objects. The resulting set of prompt templates may be found in Appendix A.
The method was evaluated on several texts accessed through Wikipedia API. The experiments were
conducted using Deepseek-ai/DeepSeek-R1-Distill-Qwen-32B [
The first graph was (see Fig. 2) constructed using the Wikipedia article Mathematics. Two dominants are observed: Mathematics itself and Mathematical Logic. These nodes exhibit the highest number of related nodes, thereby achieving the maximum degree score. All other nodes represent objects related to them, although the source text contains more entities from other disciplines such as engineering, medicine, natural sciences, etc. This fact suggests that the model labeled these concepts as the most significant during the graph generation process.
As for relations, mainly the triplet ’S is a branch of O’ is used, however, there are also some other predicate types including studies, is a type of, includes, is related to, is a specialized field of , is a prerequisite for, is a key component of, is a foundation of. Semantically, most of these relations represent meronymic or holonymic structures (i.e., part–whole relationships), with the exception of studies, which reflects agent-object relationships.
Figure 3 illustrates the graph constructed from the Wikipedia article Pug. This graph demonstrates greater granularity due to the structure of the source text, which contains not only a description of the breed’s physical characteristics but also historical information regarding its origins. Following the provided source, the graph is organized into two thematic clusters, connected by the most significant node, Pug. The nodes above Pug represent the breed’s physical traits. Notably, the term Pekingese appears within this cluster, despite the fact that there is no other direct relationship between pekingeses and pugs apart from the fact that both ’are the breeds of dogs’, as presented in the graph. Most likely the model included Pekingese because it has the similar appearance as Pug, according to Wikipedia page.
The ’historical’ cluster includes the Second Opium War, its events, and information about the pug’s ancestor, the Happa or Hap-pah dog, as well as the role of pugs in the lives of Tibetan monks.
The nodes are denoted with the key entities including the target Pug, its ancestors and Pekingese, the Second Opium War, Chinese emperors, geographical places, monks and monasteries. An inaccuracy in nodes extraction is observed: there is the node Pug as well as the node Pugs. Another cases include Happah dog and its variations: Happa, Happa dog and Happa or Hap-pah dog; Lo-sze, Lo-Sze, Lo-Chiang-Sze; Old Summer Palace, Old Summer Palace (Yuanmingyuan) and Yuanmingyuan. The model does not focus on clustering diferent names of the same objects, probably because it is not queried in the prompt (see Appendix A).
Relations in the graph are represented in a variety of ways: • using a verb: train, exhibits; • using a verb with preposition or particle: is associated with, lived in, unrelated to; • using a verb, a preposition and a noun: has no direct relationship with; • using a noun and a preposition: ancestor of ; • sometimes, the whitespace between words is replaced with an underscore: involved_participants, has_short_coat.
It is not clear at this moment how LLM decides between whitespace and underscore, or how it determines whether to use a particle or preposition: unrelated VS. unrelated to. The grammatical features of predicates depend on the subjects and objects (see ’has’ or ’have’). There are past and present forms of verbs, passive constructions, the diference between singular and plural entities.
The list of inaccuracies in generating predicates includes the following. Firstly, there is an instance of predicate where the node is mentioned. The relation between monasteries and Pugs is expressed as monasteries often have. This forms a triplet monasteries-monasteries often have-Pugs. Secondly, an illogical relation is used — has no direct relationship with. On the one side, this may be considered as a hallucination since the model is asked to extract related concepts. On the other side, the fact that a subject and an object have nothing in common suggests that this is indeed the relationship between them. Lastly, once incorrect information is given: Tibet is a type of royal dogs.
It remains unclear why, despite the comparable length of the Wikipedia articles on Mathematics and Pug, the former resulted in a smaller graph containing fewer concepts. The possible explanation is that the article about Pug contains a broader range of topics, while Mathematics page is devoted only to discipline and its elements. This interpretation is supported by the relations generated by LLM; as was already mentioned, the majority of predicates in the Mathematics graph are expressed ’as a branch of’, reflecting a more hierarchical, discipline-centered structure.
Talking about scalability, the method has demonstrated adaptability to various topics, including animals, history, and science, as concise graphs were generated for all of them. This fact proves that our method is applicable to texts across diferent domains.
Discussing the limitations of the method, the criteria by which the model selects entities for triplets are not clear. In some cases, the subject and object coincide, or the hierarchical relationship between them lacks clarity. Moreover, as previously mentioned, the model does not cluster the names of the same object. Secondly, the actual version of the instrument requires three queries to LLM’s reasoning, which increases computational time. Addressing these limitations will be a focus of our future research.
A graphical user interface was built, using Plotly, Streamlit, and HuggingFace. It is shown on Figures 4 and 5.
User is asked to provide a text into the window and press the Enter button. Then the text is processed and HTML with the visualization is returned. Visualized graph may be zoomed. The color of nodes represents their degree. This functionality was achieved with Plotly.
Additionally, our app supports the function for graphRAG. An answer to user query is provided on the basis of both provided source and generated triplets. However, it is not evaluated yet and is a next step of our work.
The app is hosted on HuggingFace Spaces and built with Streamlit.
The paper presents a novel tool for RAG explainability, AGGILE. The model generates knowledge graphs from unstructured data and visualizes the results using Plotly. The study describes primary qualitative analysis based on linguistic criteria developed by the authors of the paper. The results of qualitative evaluation show that the method is scalable, suggesting that AGGILE is a general-purpose explainability instrument suitable for wide range of domains.
We have proven that shifting from end-to-end graph generated to a more fine-grained procedure, where the process of graph generation is split into separate stages (sequential node extraction and edge generation) lead to improved controllability, enhanced transparency and decreased hallucination ratio.
The proposed approach have notable advantages. The model demonstrate the ability to focus on key concepts, define target domain, and filtering irrelevant entities, ensuring disambiguation (contextual understanding) and the possibility of finding the root concepts from the point of view of the LLMs’ explainability.
The approach can produce fine-grained graph structures, producing subgraphs, clusters, and complex relations. The weighting technique introduced in the paper allows for using the AGGILE tool for assessing concepts that gain more focus by the LLM during the processing.
The model is capable of extracting diverse relations, including hierarchical (e.g. ’a branch of’), associative (’is associated with’), and descriptive (’exhibits’) meanings. The method can be applied to a wide range of topics, suggesting that the proposed tool is a general-purpose explainability instrument.
The method has several limitations. For example, the model execution time is significant; the model generates a graph structure for approximately 3 minutes, since it queries LLM 3 times to generate 2 sets of nodes and 1 set of edges. In perspective, we plan to use vector storage for prompt-tuning to perform all intermediate processes (generation of nodes and edges) directly through vectors, reducing the time required for vectorization to optimize the model processing time.
Also, the model struggles with recognizing word forms, synonyms, and variations of the same lexical entity (e.g., ’Hap-pah dog’ and ’Happa dog’). The model might produce inconsistent redundant edges, e.g. ’is a branch of’ or ’studies’ for the same semantic relation. This indicates that a study towards improving the control over nodes and edges generation is required.
Despite the decreasing amount of hallucinations in our graphs, some generated graphs still have logical errors and hallucinations, especially in forming edges between the nodes. This can misrepresent the source data.
In some cases, the model tends to generate relation between one and the same entity, however, this issue was fixed by a simple heuristics. We added a rule that ensures that 2 nodes in a triplet are always diferent.
The method allows for building representative graphs for with considerably long and contextuallyrich texts. The produced graphs reflect both semantical and grammatical features of the input data, as well as indicate, which parts of input data target the model output, contributing to XAI methodology.
In perspective, we plan to develop methods for enhancing the AGGILE control over LLM outputs that would result in reducing redundancies, filtering hallucination and improving the model consistency. A possible solution is to define a standardized set of relations and semantic hierarchy in a form of rules for prompt-tuning and response formatting through formal grammars and conditional text generation.
Next, we plan to perform the model optimization through switching to intermediate vector representations from text-based prompt-tuning in generating nodes and edges. This step should reduce the model execution time and enhance user experience and the model scalability.
Finally, we plan to develop a multilingual version of the AGGILE tool, and add such features as generating tables with nodes and edges for improved explainability.
During the preparation of this work, the authors used GPT-4 in order to: Grammar and spelling check. After using these tool, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.
""" extract {n} collocations describing key concepts, keywords, named entities from the provided source """
""" extract 5-10 most representative collocations from the provided source that are related to the provided concept """
""" answer the question using graph triplets and provided source """