Retrieval Augmented Generation (RAG) systems often struggle with irrelevant passages that mislead LLMs during answer generation. This work introduces a comprehensive framework for quantifying and understanding the distracting nature of such passages. We propose a novel metric to measure passage-level distraction efects, demonstrating its robustness across diferent models. Our methodology combines retrieval-based approaches with controlled synthetic generation techniques that create distracting content spanning multiple categories. Through experimental validation on standard question-answering benchmarks, we show that passages with higher distraction scores consistently degrade model efectiveness, even when relevant content is present. Leveraging this framework, we construct an enhanced training dataset featuring systematically curated distracting passages. When fine-tuned on this dataset, LLMs demonstrate substantial improvements, achieving up to 7.5% accuracy gains over baselines trained on standard RAG data. Our contributions provide both theoretical insights into distraction mechanisms in RAG and practical solutions for developing more robust retrieval-augmented language models.
The integration of retrieval mechanisms with Large Language Models (LLMs) has become a cornerstone
approach for addressing knowledge-intensive tasks [
Our work addresses these challenges by developing a systematic approach to identify distracting passages and quantify their distracting efect . We demonstrate that despite the apparent model-dependency of distraction susceptibility, the relative distracting efects of passages show remarkable consistency across diferent LLMs, as evidenced by high correlation scores between models. Furthermore, we validate our measure by showing that passages with higher distracting efect scores cause more significant degradation in answer accuracy, even when relevant information is also available to the model.
Our investigation encompasses two complementary methodologies for fetching distracting content.
First, we analyze passages obtained through various retrieval strategies, including a novel
answer⋆This is an extended abstract of [
CEUR
ceur-ws.org
skewed approach designed to surface topically related but answer-irrelevant content. Second, we
generate distracting passages across predefined categories, drawing inspiration from established
taxonomies of problematic content [
Finally, we demonstrate that LLMs fine-tuned using training sets enriched with our identified distracting passages achieve substantially improved accuracy compared to models trained on conventional RAG datasets derived through standard retrieval methods.
We introduce a quantitative framework for measuring the distracting efect of irrelevant passages. Our approach evaluates a passage’s ability to mislead an LLM when the passage does not contain the answer to a given query. We test this by tasking the model to respond with “NO-RESPONSE” when the passage content is insuficient to answer the question. For a given query and an irrelevant passage , we define the distracting efect DE () as:
DE () = 1 − PLLM(NO-RESPONSE|, ) This formulation captures the probability that an LLM will attempt to answer a query based on an irrelevant passage rather than appropriately abstaining. The metric ranges from 0 (no distraction) to 1 (maximum distraction), providing an interpretable measure of the passage’s distraction.
Our analysis demonstrates remarkable consistency in distracting efect measurements across diferent language model architectures. Despite varying model sizes and training procedures, we observe strong correlations (Spearman coeficients ranging from 0.47 to 0.76) in distraction assessments across models from diferent families (Llama, Falcon, and Qwen) ranging from 3B to 70B parameters. This suggests that the distracting efect represents an intrinsic property of passages and that diferent LLMs share the same weaknesses.
This section presents our empirical investigation into diferent methods for obtaining distracting
passages and analyzes their efectiveness in RAG systems. In this paper, we show results on the Natural
Questions (NQ) dataset [
To comprehensively analyze distracting passages, we examine both standard retrieval approaches and novel techniques for obtaining highly distracting content, addressing cases where standard retrieval does not return distracting passages.
Regarding retrieval, we introduce an answer-skewed retriever that seeks passages topically related to queries while avoiding answer-relevant content. This approach modifies the standard dense retrieval embedding by subtracting the answer representation from the query embedding: sub(, ) = () − () , where controls the aggressiveness of answer exclusion1.
Additionally, we generate irrelevant passages using Claude 3.5 Sonnet V2.0. Following established
categorizations [
Figure 1a demonstrates a critical finding: higher-ranked passages consistently show greater distracting efects compared to lower positions across all retrieval strategies. This pattern holds for both standard dense retrieval ( st) and our answer-skewed approach ( sk). Notably, the integration of reranking modules ( +st and +sk) amplifies this phenomenon. While reranking improves overall retrieval quality, it paradoxically elevates the most distracting passages to top positions. This suggests that contemporary retrieval systems tend to surface passages that are semantically related to queries but factually misleading.
Figure 1b reveals the complementary nature of our diferent approaches to identifying distracting passages. For the retrieval-based methods, the analysis focuses on the first non-relevant passage returned by each method, which is also the most distracting passage accessible through that approach, as shown in Figure 1a. The vertical bars show the percentage of queries where each method contributes the most distracting passage. The blue parts highlight the times when no other method reaches the same distracting efect as the others. The substantial contribution from both retrieval-based and generationbased methods indicates that a hybrid approach yields superior coverage of distracting passage types. This diversity is particularly valuable for creating robust training datasets, as we will show in Section 4.
To validate our distracting efect measure, we conducted controlled experiments examining how passages with varying distraction levels influence answer accuracy when combined with relevant content. Our experiments account for positional bias [16, 17] by testing both ordering configurations (gold-first and distracting-first) and reporting averaged results. We establish three experimental conditions to demonstrate the progressive impact of distraction. First, when prompts include only the relevant passage, baseline accuracy reaches 82.6 and 80.6 for Llama-3.2-3B and Llama-3.1-8B, respectively. Second, adding a weak distractor (distracting efect smaller than 0.2) alongside the relevant passage causes modest performance degradation, with accuracy declining to 79.4 and 80.1. Third, incorporating a hard distractor (distracting efect greater than 0.8) produces substantially greater impact, with accuracy dropping more dramatically to 71.5 and 73.9 for the same models.
These results validate that our proposed metric efectively identifies truly distracting passages. The substantial accuracy gap between weak and hard distractors, with hard distractors causing accuracy decreases of 6 to 11 percentage points, demonstrates the metric’s ability to distinguish between diferent levels of distracting content. NQ TriviaQA
Answer accuracy averaged over all 4 test sets. None is the non-fine-tuned baseline, Retrieve, Rerank and Hard are fine-tuning strategies. Metrics: (1) , accuracy on ungrounded instances, (2) , accuracy on grounded instances, and (3) , overall accuracy. Bold values indicate the highest per model and dataset.
Leveraging insights from our distracting efect analysis, we propose a training strategy to fine-tune LLMs for improved robustness in RAG applications. Our approach constructs a training dataset using 800 queries from NQ, where each query is paired with 5 passages selected according to three distinct strategies: (1) Retrieve uses the top 5 passages from standard dense retrieval without reranking; (2) Rerank same as Retrieve, but in this case we enable reranking; and (3) Hard applies a mixed composition where 50% of instances contain the first relevant passage from the reranker plus the 4 most distracting passages, and 50% contain the 5 most distracting passages using methods from Section 3. Lastly, all passages are randomly shufled to eliminate undesired positional biases during training. 7 percentage points gains. The benefits prove particularly pronounced for ungrounded examples, where no relevant passage appears in the prompt and correct answers must be retrieved from the model’s parametric memory. This pattern suggests that training with distracting passages enhances models’ ability to resist misleading contextual information while appropriately relying on their internal knowledge when external context appears unreliable.
This work introduces a formal framework for quantifying the distracting efect of irrelevant passages in RAG systems and demonstrates its efectiveness across multiple LLMs. We reveal that stronger retrieval systems paradoxically surface more highly distracting passages, reducing accuracy by up to 11 percentage points when included in LLM contexts. Leveraging this insight, we develop a comprehensive approach combining retrieved and generated passages to create a challenging dataset with distracting content. Fine-tuning on this dataset enhances model robustness, achieving up to 7.5% accuracy improvements on question-answering benchmarks compared to conventional training approaches. We believe that our framework for quantifying distracting efects will enable new approaches to robust information retrieval in LLM-based systems.
During the preparation of this work, the authors used Claude Sonnet 4 to check grammar and spelling.
This research was conducted while Florin Cuconasu was enrolled in the Italian National Doctorate on Artificial Intelligence at Sapienza University of Rome. The project received support from PNRR MUR project PE0000013-FAIR. 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, pp. 3905–3920. [16] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, Lost in the middle: How language models use long contexts, Transactions of the Association for Computational Linguistics 12 (2024) 157–173. URL: https://aclanthology.org/2024.tacl-1.9/. doi:10.1162/tacl_a_00638. [17] F. Cuconasu, S. Filice, G. Horowitz, Y. Maarek, F. Silvestri, Do rag systems sufer from positional bias?, 2025. URL: https://arxiv.org/abs/2505.15561. arXiv:2505.15561. [18] M. Joshi, E. Choi, D. S. Weld, L. Zettlemoyer, TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 1601–1611.