Knowledge Graphs (KGs) are crucial for Retrieval-Augmented Generation (RAG), but traditional methods have limitations in capturing details and querying academic KGs. The challenges lie in identifying the appropriate KG type for RAG, such as a Metadata KG, and optimizing the integration of Large Language Models (LLMs) with KGs to enhance retrieval and generation. This paper introduces a novel framework combining the Deep Document Model (DDM) concept and a KG-enhanced Query Processing (KGQP) mechanism. DDM provides a comprehensive, hierarchical representation of academic papers using advanced Natural Language Processing (NLP) techniques, while KGQP optimizes complex queries using the KG's structural information and semantic relationships. The framework also integrates KGs with state-of-the-art LLMs to improve knowledge utilization and downstream task performance. Evaluations show that the KG-based approach surpasses vector-based methods in relevance, accuracy, completeness, and readability. This research demonstrates the potential of combining KGs and LLMs for efective academic knowledge management and discovery. § Submission type: Poster §.
CEUR ceur-ws.org
In the current era, where LLMs are extensively applied for complex question-answering tasks,
these models serve as efective tools for understanding. However, relying solely on LLMs is not
suficient to meet the challenges, as they require appropriate methods to process and explore
massive semi-structured data [
In this work, we propose utilizing the Metadata KG of a document set, such as ASKG [
Central to our approach is the DDM concept4, which captures the logical hierarchical structure of documents from section, paragraph to sentence level. By employing NLP techniques, DDM conducts in-depth analysis of textual elements, identifies their hierarchical relationships, and creates a model that reflects the document’s logical flow. Our process integrates PARSE and DDM methods to construct a KG from scientific papers (ASKG). This approach not only transcends the traditional role of knowledge graphs as mere fact providers but also ofers comprehensive metadata and context for collections of research papers. DDM enhances the depth and richness of knowledge representation through its alignment with the Document Object Model Ontology (DOMO). In this way, the DDM’s structure is materialized in an ontology-based metadata KG.
To address the challenge of “AI hallucination” faced by LLMs when dealing with complex
questions, particularly those involving intricate fact verification, we have designed and
implemented the KGQP (KG-enhanced Query Processing) workflow 5. This process harnesses
the potential advantages of LLMs and incorporates an academic KG (ASKG) aligned with a
generic model (DOMO). As shown in Figure 1, this pipeline leverages KG-based context in LLM
question-answering, utilizing structured information from ontology-based KGs like ASKG to
provide accurate context to prompt engineering tasks with LLMs and reduce hallucinations.
The workflow’s main steps as shown in Figure 1 involves:
1. Entity Identification: We utilize GPT-4 for identifying entities in user queries. GPT-4
demonstrates exceptional performance in entity recognition and matching [
2https://w3id.org/kgcp/PARSE
3https://w3id.org/kgcp/
4https://w3id.org/kgcp/DDM
5https://w3id.org/kgcp/KGQP
• String Matching: When exact matching fails to retrieve suficient relevant entities,
we employ more flexible search techniques. These include case-insensitive searches
and searches for any occurrences of the specified substring within the entity label,
allowing for partial matches and retrieving more potentially relevant entities.
3. Entity Matching: We link academic entities with relevant paragraphs using
GISTEmbedding. The semantic similarity between each entity and paragraph pair is calculated
using cosine similarity, with a threshold set at 70%. For each entity, we identify the
paragraph with the highest similarity score. Subsequently, we match the entities identified
in the user’s query with the corresponding entities in our KG.
4. Selection of Most Relevant Paragraphs: We employ Keyword Frequency Matching to
select the most relevant paragraphs.
5. Answer Generation: We utilize LLaMA2 [
This workflow leverages the structured information in our KG to provide more accurate and relevant context for the LLM, thereby improving answer quality and reducing hallucinations.
We compare our KGQP method against a vector-based baseline. In the baseline experiment, a simple chunking approach was employed to preprocess the text data. The dataset used in this experiment consists of 10 scientific papers in the field of computer science. These papers were all published by the same two authors, who are also the experts involved in the subsequent human evaluation. The dataset was divided into chunks with a maximum of 100 tokens per chunk and a 5% overlap ratio between adjacent chunks. This ensures that the content length of each chunk obtained by “Simple Chunking” is comparable to the average length of Paragraphs6 in KGQP process and provides richer contextual information. This chunking process yielded a total of 618 text chunks.
Our evaluation employed both human assessment and quantitative metrics. Three evaluators,
including two human experts and Claude [
We also conducted entity extraction, calculating overlap and Jaccard distance between the
two methods. Semantic distance was measured using GIST-Embedding [
As shown in Table 1, the KG-based method consistently outperformed the vector-based approach across all evaluated dimensions. The most significant improvements were observed in Completeness and Accuracy, demonstrating the KG-based method’s efectiveness in providing more comprehensive and precise answers. These results suggest that our proposed approach efectively enhances question-answering performance in academic contexts.
Our approach shows promise despite challenges posed by varied document structures. Future work will focus on better integrating DDM with the PARSE Pipeline for full automation and extending our framework to support Multimodal Knowledge Graphs and Question Answering for figures and tables.
In conclusion, our KG-based method demonstrates consistent superiority in producing relevant, accurate, and comprehensive answers while maintaining high readability. The observed improvements across all metrics underscore the potential of our approach in enhancing questionanswering performance within academic contexts. As we address current limitations and expand the system’s capabilities, we anticipate further advancements in academic knowledge management and discovery.