Large Language Models (LLMs) have achieved remarkable success in various Natural Language Processing (NLP) tasks, yet they remain prone to generating factually incorrect content, known as hallucinations. In this context, this work focuses on factuality hallucinations, ofering a comprehensive review of existing detection methods and an empirical evaluation of their efectiveness. In particular, we investigate the role of external knowledge integration by testing hallucination detection approaches that leverage evidence retrieved from a real-world Web search engine. Our experimental analysis compares this knowledge-enhanced strategy with alternative approaches, including uncertainty-based and black-box methods, across multiple benchmark datasets. The results indicate that, while external knowledge generally improves factuality detection, the quality and precision of the retrieval process critically afect performance. Our findings underscore the importance of grounding LLM outputs in verifiable external sources and point to future directions for improving retrieval-augmented hallucination detection systems.
systems, limit their practical applicability, and contribute
to the spread of misinformation [
Processing (NLP) tasks, including text summarization, In this context, we investigate how incorporating
exmachine translation, and conversational AI. Due to their ternal knowledge can improve the efectiveness of
halimpressive ability to understand, interpret, and generate lucination detection in LLMs. Specifically, we explore
human-like language, LLMs have become indispensable the integration of Retrieval-Augmented Generation (RAG)
tools in fields such as education, research, and healthcare. frameworks [
However, despite their capabilities and the significant the aim of enhancing their ability to identify
hallucitechnological advancements they represent, LLMs still nated content by accessing verifiable information.
Thereface some challenges. A particularly critical issue is their fore, in this work, we develop an automated knowledge
tendency to generate the so-called hallucinations, which retrieval system that leverages the Google Search API
are outputs that are plausible but incorrect, under difer- to collect relevant external evidence, which is then
inent perspectives [
Attribution 4.0 International (CC BY 4.0).
lowing categories:
Within the context of LLMs, the term “hallucination”
refers to the generation of content that is either nonsen- • Uncertainty estimation-based: Studies suggest that
outsical or unfaithful to the source content. In the literature, puts produced with high model uncertainty are more
hallucinations are typically categorized into two main prone to hallucinations [
• Zero-resource and black-box: These techniques detect hallucinations by analyzing output consistency and model behavior across multiple generations, without relying on external knowledge or internal model access.
Although these methods are broadly applicable to any
LLM, they may be less efective in scenarios involving
queries with multiple plausible answers or ambiguous
interpretations.
2.2. Faithfulness Hallucinations Belonging to the first category, the work described in
[
Finally, belonging to the third category of methods, The knowledge retrieval system is built entirely upon
in [
In light of the primary limitations identified in the literature for existing hallucination detection approaches, this study proposes a fully automated methodology that completely eliminates the need for human involvement Figure 1: Pipeline of the knowledge retrieval process. in the knowledge retrieval process. Manual retrieval is often labor-intensive and time-consuming; by contrast, the proposed approach leverages an automated pipeline for sourcing and integrating external knowledge, 3.2. Few-Shot Prompting with Knowledge thereby significantly reducing both time and operational Few-shot prompting is a technique in which an LLM costs. Furthermore, the efectiveness of the method is is presented with a limited number of task-specific exvalidated through experiments conducted on three estab- amples to guide its behavior and enhance its ability to perform a given task. However, the model’s responses in this setting are based solely on the knowledge acquired during the pre-training phase. To enhance its performance and expand its informational basis, the framework integrates external knowledge retrieved through the automated retrieval system. This additional context is provided to the model during inference, enabling more accurate and informed task execution. Specifically, the process is structured into the following steps: • The user’s query is encoded using the embedding
model; • The resulting embedding is used to retrieve relevant
information from the vectorized knowledge base; • The retrieved knowledge is incorporated into the prompt, together with a set of examples and the question–answer pair to be assessed; • The LLM evaluates the factuality of the answer by leveraging both its internal knowledge and the external information, classifying the response as either factual (true) or hallucinated (false).
3.3. SelfCheckGPT with Knowledge to conduct the experiments, describes the datasets and the metric used for performance evaluation, and provides The knowledge was also integrated into the SelfCheck- an analysis of the results obtained. GPT framework to improve the quality of the sampled responses. The underlying assumption is that providing the LLM with relevant external information will lead to 4.1. Experimental Setup the generation of more accurate and reliable responses. All experiments were carried out on the Google Colab As a result, when these samples are compared with the platform,1 utilizing a Tesla T4 GPU. The LLM employed target response using one of the SelfCheckGPT variants, for the few-shot prompting approach, response samit becomes easier to assess whether the target response pling, and the LLM-prompt variant of SelfCheckGPT was is hallucinated. The process is structured according to the following steps: 1https://colab.research.google.com/ Llama-3.2-3B-Instruct, accessed using the Transformers For the implementation of SelfCheckGPT, the variants library of Hugging Face.2 For both approaches, the model employed for evaluation purposes are BERTScore, NLI, selected for generating semantic embeddings and as a and LLM prompt (see Section 2.3). In accordance with retriever was jina-embeddings-v3.3 The retrieved knowl- the original SelfCheckGPT configuration, 5 responses edge was segmented into chunks of 256 characters with per query were sampled using a temperature setting of an overlap of 25 characters to preserve semantic coher- 1.0 and a maximum output length of 128 tokens. Figence across segments. The retriever was configured to ure 5 illustrates the prompt provided to the LLM for the return the top 5 most relevant documents according to generation of these sampled responses. similarity to the input query.
The few-shot prompting approach was evaluated by Prompt for Generating Sampled Responses with providing the model with 1, 5, and 10 examples. To gener- Knowledge ate the response, the LLM was set to a temperature value equal to 0.001. Figure 4 presents the prompt structure provided to the LLM to classify a given text as either factual or hallucinated.
4.2. Datasets and Evaluation Metric
For the experimental evaluation, three benchmark
datasets for hallucination detection were selected. Each
dataset includes a user query, the corresponding
LLMgenerated response, and a binary label indicating whether
the response is factually accurate. The datasets employed
are FactAlign [
by an LLM, your objective is to determine if the response is an hallucination or not.
In the context of NLP, an "hallucination" refers to a phenomenon where the LLM generates text that is incorrect, nonsensical, or not real. Based on your knowledge, on the knowledge provided, and on the definition of hallucination provided, analyze the user query and the response of the LLM, and answer the following question: is the response factual or not?
empty or not useful, in which case you have to respond based only on your knowledge.
... ### EXAMPLE N
### LLM TURN
2https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct 3https://huggingface.co/jinaai/jina-embeddings-v3 to facilitate more efective knowledge retrieval through generated by an LLM, and a binary factuality label. For the Google Search API and to simplify both the factu- evaluation purposes, only the entries corresponding to ality classification task performed by the LLM and the user queries in the form of questions were retained. Due generation of sampled responses within the SelfCheck- to computational constraints, a subset of 100 observations GPT framework. Following this filtering step, a random was selected. To mitigate the efects of class imbalance, sample of 100 questions was selected. This limitation an equal number of true and false instances (50 each) was imposed by constraints on computational resources were randomly sampled. A fixed random seed was apand time, which required a balance between the number plied to ensure reproducibility and consistency across all of examples and processing eficiency. Furthermore, to experimental configurations. ensure comparability and consistency across the methods and each variant, a fixed random seed was used to guarantee the reproducibility of the 100 instances across all experiments.
designed for the evaluation of hallucination detection in
LLMs, encompassing five distinct domains, each posing
specific challenges for the models under analysis [
The domains are defined as follows:
FactBench. This dataset was specifically developed
to evaluate FactCheck-GPT, a multi-step framework
designed for the detection and correction of factual errors
in responses generated by LLMs [
This dataset was constructed by selecting 50 prompts from FactPrompts and fact-checking datasets such as TruthfulQA [14]. For each prompt, responses were generated using ChatGPT and subsequently annotated by human evaluators with binary labels indicating factual correctness; • FELM-WK : Subset of the FELM dataset that will be
detailed in the next paragraph; • HaluEval: This benchmark dataset for hallucination detection was constructed by initially considering 52 000 prompts, followed by a filtering procedure aimed at selecting those most likely to elicit hallucinated responses from a LLM. Specifically, each prompt was submitted to ChatGPT three times, and the average semantic similarity among the generated responses was calculated. The 5 000 prompts with the lowest semantic similarity scores were retained to ensure the dataset included only the most challenging queries.
The selected prompts were then resubmitted to ChatGPT to obtain a second set of responses, which were manually annotated as either true or false based on their factual accuracy [15].
FactBench was made publicly available by the authors on GitHub and comprises a total of 4 835 examples, of which 3 838 are labeled as true and 995 as false.5 Each instance includes a user query, the corresponding response 5https://github.com/yuxiaw/Factcheck-GPT/blob/main/Factbench. jsonl • World knowledge: Includes questions related to general
cultural and factual knowledge; • Science and technology: Comprises statements related to scientific facts or citations across disciplines such as physics and biology; • Reasoning: Contains prompts that require multi-step
logical reasoning to produce a correct response; • Recommendation and writing: Involves open questions requiring the model to provide suggestions or generate creative or structured written content; • Math: Encompasses problems that necessitate both logical reasoning and mathematical skills to arrive at correct answers.
FELM was constructed by aggregating prompts from diverse sources, which were then submitted to ChatGPT operating in a zero-shot configuration. The resulting responses were segmented into sentences, each of which was subsequently evaluated by a team of experts. The factual accuracy of each sentence was assessed based on comparison with reliable sources, and sentences were annotated as either true or false accordingly. A response was labeled as true only if all its sentences were assessed as accurate; otherwise, it was classified as false. The FELM dataset was obtained from Hugging Face and comprises a total of 847 instances.6 Each instance includes a user prompt, the corresponding response generated by the LLM, and a factuality label. Of these examples, 566 are labeled as factual, while 281 are labeled as non-factual.
For evaluation, only the World knowledge and Science and technology domains were considered, as the remaining presented substantial limitations for the knowledge retrieval approach (e.g., mathematical prompts such as “What is the value of the expression 1! + 2! + 3! + ... + 6https://huggingface.co/datasets/hkust-nlp/felm 10!” ). As in the previous datasets, only prompts formu- across all three benchmark datasets. With regard to fewlated as questions were retained. To mitigate class im- shot prompting, the ten-shot configuration achieves the balance and accommodate computational constraints, a best performance, followed by the five-shot and one-shot balanced subset of 100 samples—comprising 50 factual variants, respectively. This trend is consistent with the and 50 non-factual instances—was randomly selected. A hypothesis that providing a greater number of examples ifxed random seed was applied to ensure consistency enables the LLM to better internalize the task structure, across experiments. thereby improving generalization and overall accuracy. In this regard, the strategy for selecting examples in Evaluation metric. Since all the datasets employed the few-shot prompting approach could be improved. In in the evaluation are balanced, Accuracy was adopted as the current evaluation, examples were randomly sampled the primary performance metric. It is defined as follows: from the datasets, which may result in class imbalance among the examples shown to the LLM, potentially afAccuracy = + fecting performance. Ensuring a balanced representation + + + of classes in the selected examples would therefore be where TP denotes factual responses correctly classified crucial for enhancing the robustness of the analysis in as factual, TN represents hallucinated responses correctly the few-shot prompting setting. identified as hallucinations, FP corresponds to halluci- Regarding the impact of knowledge integration, on nated responses incorrectly classified as factual, and FN the FactAlign dataset, the only method that underperrefers to factual responses mistakenly classified as hallu- forms when incorporating external knowledge is fewcinations. shot prompting with five examples; all other tested methods either match or surpass the performance of their 4.3. Results and Discussion counterparts without knowledge. A similar trend is observed on FactBench, where all approaches that leverage To evaluate the impact of knowledge integration, the retrieved knowledge perform at least as well as, and often performance of both SelfCheckGPT and the few-shot better than, those without knowledge integration. Finally, prompting approach was evaluated in two configurations: in the FELM dataset, incorporating external knowledge with and without the inclusion of external knowledge. generally leads to performance improvements across A summary of the comparative results is presented in methods, with the sole exception of SelfCheckGPT using Table 1. The notation W/O and W denotes whether the the LLM Prompt, where performance declines by one perevaluated variant operates without or with integrated centage point after knowledge integration. Overall, these knowledge, respectively. For each variant and dataset, analyses suggest that integrating external knowledge the version (with or without knowledge) that achieves generally enhances the performance of the evaluated apthe highest performance is underlined; if both versions proaches across all datasets, with only a few exceptions perform equally, no underlining is applied. where a slight decrease in performance was observed. These performance declines may be attributed to limiModels Variant FactAlign FactBench FELM tations in the knowledge retrieval process. Specifically,
W/O W W/O W W/O W only the first retrieved URL is considered—typically the
BERTScore 59.0 61.0 61.0 60.0 56.0 59.0 most popular, but not necessarily the most informative. SelfCheckGPT LLMNPLrIompt 6672..00 6657..00 6547..00 6693..00 6697..00 6781..00 Additionally, the retrieval system occasionally fails to PFreowm-pShtiontg OFTienvnee---ssshhhooottt 555705...000 555549...000 655293...000 666425...000 655269...000 656392...000 aahcnincteid-sebsrorttehmleeveaacchnqatuncioissinmttioesnnotrodfCuvAeaPtlouTaWCbHleebAerxpetsretorrtniecactlitoikonnnso,s,wsuwlcehhdigcahes. Table 1 Nevertheless, on average, approaches augmented with exComparison between methods with and without integrated ternal knowledge outperform their non-augmented counknowledge, to evaluate its impact on their performance. terparts. This suggests that further improvements in the retrieval process could improve the overall efectiveness of these methods and lead to even greater performance gains.
As shown in Table 1, the SelfCheckGPT framework
consistently outperforms the few-shot prompting
approach across all evaluated conditions. This result aligns
with expectations, given that SelfCheckGPT is specifi- 5. Conclusions and Perspectives
cally designed for hallucination detection, whereas
fewshot prompting is a more general-purpose methodol- In this study, we introduced a fully automated knowledge
ogy. Among the SelfCheckGPT variants, the NLI-based retrieval framework that leverages a custom search
enmethod demonstrates the highest overall efectiveness gine interfacing with the Web via the Google Search API
and eficiency, surpassing the LLM prompting variant
to extract relevant external information. The retrieved
knowledge was subsequently integrated into two distinct
methodologies: (i) few-shot prompting, which consists of
providing a set of examples to guide task execution, and
(ii) SelfCheckGPT, a hallucination detection framework
that generates and compares multiple responses from an
LLM to identify factual inconsistencies. The enhanced
versions of both approaches, incorporating retrieved
knowledge, were evaluated on three benchmark datasets
for hallucination detection—FactAlign, FactBench, and
FELM—spanning a diverse range of domains. The
experimental results indicate that SelfCheckGPT
consistently outperforms the few-shot prompting approach,
demonstrating strong performance across all three
benchmark datasets. Among its variants, the NLI configuration
emerges as the most efective and computationally
eficient. Moreover, the integration of external knowledge
generally enhances the performance of the evaluated
approaches compared to their counterparts without such
integration. Nonetheless, the observed improvements
could be further amplified by refining the knowledge
retrieval process in future work. Specifically, challenges
such as CAPTCHA mechanisms or site access restrictions
that limit automated retrieval should be addressed.
Additionally, the quality of the queries submitted to the search
engine could be improved by leveraging LLMs to
generate more precise and contextually rich queries, thereby
yielding more informative results. Moreover, expanding
the number of retrieved Web sources may lead to more
comprehensive and accurate knowledge; for instance,
retrieving the top five results could increase the relevance
and diversity of the retrieved information. Finally,
future researches may also focus on further refining the
knowledge integration process by leveraging more
advanced and sophisticated RAG techniques [
Knowledge-based, explainable User empowerment in Releasing private data and Assessing Misinformation in online environments);7 ATEQC – Progetti di Ricerca di Ateneo – Quota Competitiva (University Research Projects – Competitive Funding Scheme) PriQuaDeS: Next-generation Privacy- and Quality-preserving Decentralized Social Web
Eccellenza 2023-2027” of the Department of Informatics, Systems and Communication (DISCo), University of Milano-Bicocca, Italy. We further acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer [16], owned by the EuroHPC Joint Undertaking, hosted by CINECA (Italy). Declaration on Generative AI