This work introduces a novel Graph-based Retrieval-Augmented Generation (GraphRAG) approach, which enhances the accuracy of Large Language Models (LLMs) on the medical Question Answering (QA) task. The proposed approach aims to alleviate limitations of previous ones, such as the overreliance on facts from outdated knowledge bases, the adherence to specialized ontologies, and the lack of utilization of performance-oriented techniques (e.g., model weight quantization). To do so, (i) it automatically extracts factual triplets from medical questions using a state-of-the-art LLM, while allowing fact verification by medical experts; (ii) it connects the entities from these factual triplets into a Knowledge Graph (KG); (iii) it represents graph entities using pre-trained transformer embeddings stored in an in-memory vector framework, and (iv) it leverages these embeddings, firstly by finding similar KG entities to the extracted terms of the user question, and secondly by semantically ranking the top-n most relevant factual triplets to the user question, which are then used as enriched context for the LLM. To verify the validity of our approach, we conduct a series of experiments using a prominent medical QA dataset (MedMCQA) to measure the accuracy of our approach by incorporating various open source LLMs. In addition to these experiments, we investigate the efect of diferent context lengths on the accuracy achieved. To the best of our knowledge, this is the first GraphRAG approach that empirically measures the accuracy of LLMs on QA, while utilizing 4-bit quantization. It is shown that our approach improves model accuracy up to +5.5% over the baseline ones, while it requires significantly less computational resources compared to previous approaches. To enable the reproducibility of our work, we make the extracted triplets and code publicly available.
Recent advancements in the field of Deep Learning have
led to the development of Large Language Models (LLMs),
which have general-purpose language understanding and
reasoning capabilities [
To address these challenges, several strategies have
already been proposed [
A major drawback of common RAG approaches is that
they retrieve independent text passages that are relevant to
the users’ question [
In any case, several GraphRAG approaches demonstrate
a series of limitations: (i) they rely on existing KGs, without
addressing the phase of graph construction from
unstructured data, which requires great manual efort by domain
experts; (ii) they often utilize KGs with outdated domain
knowledge [
Focusing on the medical QA task, the aim of this paper is
to develop and assess an accurate and eficient GraphRAG
approach that addresses the above limitations. The proposed
approach automatically creates KG triplets from
unstructured text (question-answer pairs) and stores them in a
scalable graph database. These triplets can be retrieved from the
KG and used as additional context by an open source LLM
that has small memory requirements due to quantization.
The contributions of this paper are the following:
• we introduce a novel GraphRAG approach, namely
VectorGraphRAG, that does not rely on specialized
ontologies or existing medical KGs, which often
maintain outdated domain knowledge [
The remainder of this paper is organized as follows: stateof-the-art LLMs and medical GraphRAG approaches are described in Section 2, highlighting their benefits and limitations; the proposed approach is described in Section 3; our experimental setup and results are presented in Section 4; ifnally, concluding remarks and future research directions are outlined in Section 5. To facilitate the reproducibility of our work, we make our code and extracted triplets publicly available2.
LLMs can handle various NLP tasks, due to their
autoregressive architecture, extensive training on vast amounts
of data, and alignment with human instructions through
reinforcement learning [
So far, several proprietary and open-source LLMs have
been proposed. Proprietary models, such as OpenAI’s
ChatGPT [
2https://github.com/NC0DER/VectorGraphRAG
but are costly, let alone the fact that their use in sensitive
applications requiring data privacy is not possible in the
EU due to GDPR regulations [
In this study, we consider a series of remarkable
opensource families of models that achieve state-of-the-art
accuracy in several benchmark tests reported in the literature.
These include Llama 3 [
HyKGE [
Labrak et al. [
Bailicai [
MedGraphRAG [
DEEB-RAG [
BioKGQA [
VectorGraphRAG includes two multi-step processes. The ifrst one concerns the extraction and representation of knowledge (see Section 3.2 and Figure 1), while the second one is the retrieval of the appropriate KG triplets to be used for RAG (see Section 3.3 and Figure 2).
Definition 1: Semantic entity and relationship. We define a semantic entity as , where ∈ and a semantic relationship as , where ∈ ; and are the sets of extracted entities and their relationships (predicates), respectively.
Definition 2: Semantic Triplet. We define this triplet as =(,, ), where ̸= and ∈ . and are the head and tail entities, respectively, and is the predicate of these entities. Finally, ={1, ..., } is the set of semantic (factual) triplets, where ⊆ × .
Definition 3: Knowledge Graph (KG). We define an attributed graph=(, , , ), where ={, ..., } is the set of graph vertices and ⊆ × is the set of graph edges. We also define a function that maps semantic entities and predicates to graph vertices and edges, respectively: = {→, → ∀ ∈ , ∈, ∈, ∈} (1) Definition 4: Semantic vector (embedding) computation. We define an encoding function = { → ∀ ∈ }, where ∈ is a semantic vector of dimension , which comprises real value coeficients. This vector semantically represents a textual sequence by leveraging a pre-trained transformers model.
Definition 5: Similarity function. We define a cosine similarity function between two semantic vectors and as: (, ) =
× ||||·|| || (2)
The first step of this process extracts factual triplets from
each question-answer pair, by prompting a state-of-the-art
LLM (i.e., GPT-4o mini). The second step constructs a .csv file
from the set of triplets T ; this format is selected to facilitate
any possible amendments by medical experts. The third
step loads the KG data from the file into a graph database.
We also place uniqueness constraints on the entity names,
to avoid storing duplicates. The fourth step involves the
use of a sentence transformers model [
The first step of this process initializes the graph database
that contains the KG, and loads three important components
into memory: (i) the LLM using 4-bit quantization (GPU);
(ii) the sentence transformers model (RAM); (iii) the entity
embeddings (RAM). We then build a cosine similarity
index (RAM) using FAISS [
The second step concerns the extraction of search terms from the user question. To do this, we remove stop-words, tokenize the sentences, and extract all n-grams from each sentence, where ∈ {1, 2, 3}. Each n-gram is encoded using the embedding model.
The third step finds the top-k most similar KG entities to the extracted search terms from the similarity index, using the function, where S is the set of search terms and the vector embedding for each element of the encoded set. = ((, )|(, )>0.9) ∈() ∧ ∈() (3) The fourth step traverses the graph database to retrieve triplets, whose entities match either the search terms or the top-k most similar ones. We then use the sentence transformer model to encode the question Q and the triplets T to ifnd the top-n most similar triplets using the function: = ((,())|(,())>0.1) (4) ∈( ) Longer text sequences tend to generate lower similarity scores; thus, the cutof in equation 4 is much lower in order to avoid dissimilar triplets (i.e., triplets with similarity scores in the range [-1, 0]). The final step utilizes the top-n triplets as textual context for the LLM by embedding them in the prompt.
In terms of hardware, we utilize a PC with an Intel Core i9-13900K CPU with 64 GB of RAM and an Nvidia RTX 3060 GPU (12 GB of VRAM). With respect to software, we utilize HuggingFace Transformers4 to develop LLM inference code, and we build our KG with Neo4j5. As mentioned earlier, we also use sentence transformers to encode entities into vectors, which are saved into numpy array files. Finally, we extend
Similarly to other works, we validated our approach using the MedMCQA dataset. Specifically, to build our KG, we utilized its training part, which comprises ∼ 183K medical questions. We selected a subset of it comprising ∼ 120K questions, labelled as “single choice”, where each choice contains a single option.
Then, we utilized the data extraction prompt (Table 1) and GPT-4o mini, which was accessed through the OpenAI API6, to create a semantic triplet from each question-answer pair. A post-processing step was applied to remove empty and malformed triplets (e.g., those without a subject or an object). The data extraction step costed 2.49$ dollars’ worth of credits. Alternatively, we could also use an open-source model (e.g., Llama-3.3 70B) to eliminate the credit cost; however, due to memory hardware limitations, this was not possible. The data generated are also available at the repository of this paper7. For our experiments, we utilized the evaluation part of MedMCQA, which consists of 4186 question-answer pairs (not included in the training part). Specifically, we selected the single choice questions (2816 in total).
For the medical QA task, the LLMs were instructed to select the correct answer from a list of possible ones. To extract the answer from the model output, we utilized regular expressions. These answers were then compared with the correct ones to calculate the accuracy score. Table 2 summarizes the results of our experiments concerning the evaluation of several open-source LLMs, while a comparative assessment
7https://github.com/NC0DER/VectorGraphRAG of our approach against previous ones is given in Table 3. To ensure the reproducibility of our results, we use a specific random seed and we set the model temperature to 0.0.
As shown in Table 2, there is an increase in the accuracy of all models considered (up to +5.5%) when using VectorGraphRAG (compared to the case that the models use just their internal knowledge, i.e., Baseline). For the best performance of VectorGraphRAG, we set the number of triplets to 300 and we retrieve the top-10 most similar terms to each question term extracted from the KG.
These parameters are selected based on additional experiments that were performed using Llama-3.1 (see Figures 3, 4). As shown in Figure 3, the model accuracy peaks at 300 factual triplets and then decreases to the baseline accuracy at 600 triplets. Afterwards, it starts increasing, until a local optimum is reached at 1100 triplets; after this point, the accuracy drops significantly. Thus, we infer that we can maximize the LLM accuracy while minimizing the retrieved KG context (measured in the number of triplets). As mentioned above, for each medical term we retrieve the top-k most similar terms. To find the optimal number of k, we experiment with several values, while keeping the number of triplets to 300. As shown in Figure 4, the accuracy fluctuates, with 10 being the best value of k.
We also compared the proposed approach with similar
ones. As shown in Table 3, previous approaches
underperform with the sole exception of MedGraphRAG. An
interesting remark is that our approach achieves near
state-ofthe-art accuracy, while being much more eficient, due to its
performance-oriented design and the use of a single LLM
generation for RAG. In contrast, MedGraphRAG requires
multiple LLM generations to summarize graph communities
that contain medical facts to achieve state-of-the-art
accuracy. Similarly, Bailicai utilizes multiple LLM generations
and performs fine-tuning, both of them being
computationally expensive [
It is also stressed that our approach achieves better performance (even with the quantized LLM), compared to previous ones with non-quantized LLMs (except MedGraphRAG). Thus, by using 4-bit quantized LLMs, we reduce the required memory by 75%, without sacrificing accuracy, which is particularly useful given the high cost of GPUs with large VRAM.
From our experimental results, we also infer that providing only the most relevant information to the LLM and not iflling its entire context window with unnecessary information leads to accuracy and performance benefits. In this study, we utilized several prompts that are listed in Table 1.
VectorGraphRAG overcomes the limitations of previous medical QA approaches. Specifically, it: (i) automates the process of building a KG by using an LLM to extract triplets from QA pairs, thus reducing the labor of domain experts; (ii) utilizes a scalable graph database that loads only a small portion of the KG, thus being able to scale on larger KGs that do not fit entirely in memory; (iii) utilizes a small inmemory vector index to accelerate the entity search based on the top-10 most similar terms; (iv) once the appropriate graph context is retrieved, performs a single LLM generation instead of multiple ones to significantly decrease the total inference time; (v) ranks the triplets and limits them to provide only the most relevant context to the LLM, thus increasing accuracy, and (vi) utilizes LLM 4-bit quantization, which approximately requires a quarter of the amount of VRAM compared to other approaches, without any significant accuracy loss.
Based on the above remarks, we can answer the research
questions listed in the Introduction. Overall, we introduced
a novel approach that uses an LLM for automatic triplet
creation instead of relying on existing medical KGs or
specialized ontologies (RQ1). Our experimental results showed
that using a quantized LLM does not significantly reduce
model accuracy (RQ2). When considering the results from
other approaches with diferent prompting strategies, we
observed that non-GraphRAG approaches underperformed
[
In any case, our approach has a set of limitations. First, it requires a large state-of-the-art LLM (i.e., GPT-4o mini) for creating the triplets. In addition, given the fact that the LLM might hallucinate when producing some of these triplets, a careful validation from domain experts is required. Furthermore, our approach expects questions where there is a single correct answer; for questions where there is no clear answer, we should build an extra processing layer.
As far as future work directions are concerned, we
consider the incorporation of domain-specific ontologies in our
approach, aiming to further improve its accuracy.
Additionally, we plan to support factual triplet extraction from
unstructured text, by building an additional processing layer.
A third work direction is to run experiments with more
evaluation metrics and QA datasets, as well as to employ human
evaluation, aiming to get additional insights about the
proposed approach and generalize our research findings. A
fourth work direction is to evaluate our approach for open
QA [
The authors have not employed any Generative AI tools.