Large Language Models (LLM) hold immense promise for real-world applications, but their generic knowledge often falls short of domain-specific needs. Fine-tuning, a common approach, can sufer from catastrophic forgetting and hinder generalizability. In-Context Learning (ICL) ofers an alternative, which can leverage Retrieval-Augmented Generation (RAG) to provide LLMs with relevant demonstrations for few-shot learning tasks. This paper explores the desired qualities of a demonstration retrieval system for ICL. We argue that ICL retrieval in this context resembles item-cold-start recommender systems, prioritizing discovery and maximizing information gain over strict relevance. We propose a novel evaluation method that measures the LLM's subsequent performance on NLP tasks, eliminating the need for subjective diversity scores. Our findings demonstrate the critical role of diversity and quality bias in retrieved demonstrations for efective ICL, and highlight the potential of recommender system techniques in this domain.
Large Language Models (LLMs) have emerged as a powerful
tool for natural language processing, demonstrating
remarkable abilities in areas like text completion, summarization,
and question answering [
However, a crucial distinction exists between generic
public knowledge and the specific private knowledge required
for most real-world use cases. While LLMs excel at generic
text completion or chat-like interactions, practical
applications often demand solving specific and repeatable
downstream tasks within a particular domain [
Fine-tuning, a technique where LLMs are trained on large
datasets tailored to the target task, ofers a path towards
adapting LLMs to these domain-specific needs. Yet,
finetuning presents significant challenges. When trained on
tasks-specific data, LLMs tend to forget knowledge and skills
gained in the initial training, a phenomenon referred to as
Catastrophic Forgetting [
As an alternative to fine-tuning, In-Context Learning
(ICL) ofers a promising approach for leveraging LLMs in
Information Retrieval’s Role in RAG Systems (IR-RAG) - 2024
∗Corresponding author.
†Both authors contributed equally to this research.
[
This paper focuses on few-shot learning and the retrieval
of relevant demonstrations for this process, where a
demonstration is some text which is included in the LLM’s context
to demonstrate how the LLM should formulate correct
answers. Few-shot learning presents a well-structured
problem, allowing us to evaluate the quality of the retrieval
algorithm using established classification metrics. Crucially,
we show that enriching a language model with a few-shot
example retriever ofers a powerful method to achieve
finetuning-like behavior, steering the output of the LLM towards
the desired outcome even with limited data. Interestingly,
increasing the context size in prompts beyond a certain
point yields diminishing returns. The most impactful
information resides within a relatively small set of well-chosen
demonstrations, rather than overloading the prompt with
vast amounts of data [
Building upon this concept, this paper identifies
desirable properties for a RAG system under the framework of
few-shot learning. We demonstrate that state-of-the-art
retrieval systems in this context resemble item-cold-start
recommender systems. Unlike exact search algorithms that
prioritize precision and recall, our focus is on discovery, by
maximizing the set of collective information gain from the
retrieved demonstrations. This necessitates solving various
trade-ofs between query relevance, quality scoring, as well
as diversity algorithms to ensure a variety of informative
examples are surfaced. Furthermore, we propose a method
for evaluating RAG system performance through the
subsequent performance of the enriched LLM on established
NLP tasks like question answering or text generation. This
methodology ofers a valuable approach to directly
assessing diversity and quality-based retrieval systems, which
removes the need to define a subjective diversity score, a
historically challenging aspect of evaluating such systems
in academic settings [
CEUR
ceur-ws.org
To summarize, in this paper we study the impact of diversity and quality bias in retrieving demonstrations for ICL. We start by reviewing the use of diversity and other biases in both ICL and Information Retrieval works. We then propose a method for evaluating the performance of diferent retrieval algorithms. Then we present experiments and results demonstrating the impact of diversity and quality bias on an LLM’s ability to generate correct answers. Finally we discuss the applicability of state-of-the-art ICL retrieval algorithms in real-world setting, and show that recommendation engines ofer a better solution than semantic search engines.
This paper sits at the intersection of two distinct, but increasingly intertwined, research areas: In-Context Learning for Large Language Models and Information Retrieval. While ICL focuses on enabling LLMs to learn from carefully selected contextual information, IR deals with retrieving relevant information from document collections. Our work leverages concepts from both fields to address the challenge of few-shot learning with LLMs.
Within the context of RAG, few-shot learning can be defined as a specific scenario where the ”documents” retrieved are actually ”examples” used to guide the LLM. These examples can also be referred to interchangeably as ”In-Context Examples” (ICE) or ”demonstrations”. The importance of ICL in achieving state-of-the-art LLM performance is undeniable, with its ubiquitous presence in top benchmarks across various domains. Consequently, ICL research is a rapidly evolving field with numerous proposed algorithms.
Several noteworthy ICL approaches have emerged that
address the challenge of retrieving informative examples for
few-shot learning. Some methods only promote the
diversity of the demonstrations, like in [
Other approaches focuses on identifying examples where
the LLM exhibits low token-level uncertainty. In [
Finally, neural ranking, one of the most commonly used
ICL approach, typically yielding superior results [
While general purpose pre-trained embeddings like BERT
[
While both relevance and diversity are crucial for
efective few-shot learning, methods yielding the best ICL
results combine these two paradigms rather than prioritizing
one over the other. This is achieved by maximizing a
carefully balanced trade-of between semantic similarity to the
prompt and diversity of the retrieved examples.
Unsupervised techniques can be adapted to prioritize the selection
of examples that are both relevant to the prompt and
dissimilar to each other. In [
The most popular unsupervised approach for achieving
this balance between relevance and diversity is Maximal
Marginal Relevance (MMR). MMR retrieves a set of
examples by iteratively selecting the example that maximizes a
linear combination of the scores of relevance to the prompt
and dissimilarity to the previously retrieved examples. It
was analyzed in [
The concept of diversity in information retrieval has been a long-running topic of research. In this section we propose a short review of the use of diversity in the IR literature and related domains.
The MMR algorithm was analyzed in [
Within IR, the recommender system literature brings an
additional point-of-view on studying diversity in retrieval,
by focusing on the benefit of diverse results for a user,
instead of evaluating methods against a potentially arbitrary
relevance/diversity trade-of. The dificulty of evaluating
the impact of diversity, and the necessity for large scale
real-world recommendation studies has been explored in
[
Learning a trade-of between relevancy and diversity also
naturally occurs in related online frameworks such as active
learning, multi-armed bandits and Bayesian optimization.
In [
Within batch-mode Bayesian optimization, in [
Complementing the discussion on diversity in information retrieval, quality bias also plays a crucial role in efective retrieval. Quality bias refers to the prioritization of documents or examples considered to be more reliable or informative within the retrieved set. Incorporating quality considerations into retrieval algorithms can significantly improve standard unbiased IR metrics.
Several approaches have been explored to address quality bias in pure IR tasks. These can be broadly categorized into content-based and graph-based methods.
Content-based methods leverage existing signals inside the
documents themselves to identify potentially lower-quality
content. Examples include spam detection scores developed
in works like [
Instead of relying on the content itself, graph-based
algorithms inherently capture implicit quality signals within
their ranking model. PageRank, a seminal algorithm for
web search ranking introduced in [
Interestingly, the concept of inherent quality bias in graphbased IR approaches resembles collaborative filtering techniques employed in recommender systems. In an analogous manner to learning-to-rank on a (item, item) graph, collaborative filtering addresses learning-to-rank on a bipartite (user, item) graph. In this way, collaborative filtering also implicitly learns a trade-of between item similarity and popularity, favoring items that are both similar to the user’s past preferences and also generally well-received by other users.
We propose to frame the ICL problem as an item-cold-start recommendation problem, where the query is an unseen item, and the objective is to retrieve from the pool of candidate few-shot demonstrations a set of items maximizing the cumulative reward to the user (the LLM). In this case, the reward is a measure of how much the retrieved items increase the probability that the LLM generates a correct answer. A solution to this optimization problem requires not only relevance, but also diversity and quality in the retrieved items, such that the amount of useful information presented to the LLM in the context is maximized.
Further, we propose to measure the impact of diversity on the retrieved items by directly calculating the probability of the LLM generating a correct answer given the context items. This is in contrast to a typical retrieval context where the retriever is evaluated by calculating some metric relating to the accuracy and recall of documents most similar to the query. In such a setting, it is typical to add a term to the metric which measures the diversity of the retrieved documents to promote more diverse retrievers, knowing that diversity improves the reward to the user but without having an explicit model connecting diversity to the user’s reward. In the case of retrieving demonstrations for inclusion in an LLM’s context, we can directly measure the impact of diversity on the LLM’s reward by calculating the probability of the LLM generating a correct answer.
Consider an answer that should be generated by an LLM in response to a query . The query can be a simple question such as ”Who was the first man to walk on the moon?”, or more general message such as ”I’d like to find red shoes”. The answer could take on many forms, such a factual response ”Neil Armstrong”, a clarifying question ”What size and style of shoe are you looking for?”, a JSON payload to send to an API ”{"search_terms":["red","shoe"]}” etc.
Consider a set of demonstrations , where each demonstration is a pair (, ) answer , or a triple (, , )̄ containing a query and correct
which additionally contains an incorrect answer .̄ Datasets under this later triplet form are commonly used in Contrastive Learning approaches. We call , a subset of demonstrations retrieved from , the context.
⊂ =
{( , , ̄ ), … , ( , , ̄ )}
Given an auto-regressive LLM , the query , and a retrieved context , we define ( ∣ , )
generates the answer . In practice, the tokens of the examples from the context are appended to the tokens of the query , using prompt formatting techniques that may the probability that be optimized to a specific LLM. ( ∣ , ()) is maximized.
Putting it all together, for an unseen query and unseen correct answer , a few-shot retriever retrieve a subset of demonstrations () ∈ such that must eficiently
Consider the probability of generating the correct answer given an empty context ( ∣ ) . We are interested in evaluating how much the context increases the probability of the correct answer . That is, we want a metric which is related to diference of ( ∣ , ) and ( ∣ ) .
In a pure retrieval setting, we would be interested in finding the context which contains the demonstrations that are most similar to the . And we could argue that if there exists a smooth function ∶ → which maps a query to its correct answer, then by retrieving the demonstrations whose queries are nearest to , we should also be retrieving the answers which are closest to , and this should help the language model
generate the correct answer .
However, it is doubtful that the space in which is compared to the demonstrations is one in which the function ∶ →
is smooth, so it is not necessarily true that the retrieved answers are closest to . Nor is it necessarily true that ( ∣ , )
is maximized when contains those answers closest to . Consider that the answer might depend on some information which isn’t contained in or any nearby answer.
Therefore, we prefer to measure ( ∣ , ) directly.
In practice, given that
is an auto-regressive language model, this is done by taking the product of the probability of each token generated by . The model generates text sequentially by predicting one token at a time based on the previously generated tokens. Let = ( 1, 2, … , ) represent a sequence of tokens produced by the model. The probability of the model generating the sequence can be expressed as the joint probability of generating each token in the sequence, conditioned on the tokens that precede it. This can be mathematically represented as: () = ( 1) ⋅ ( 2| 1) ⋅ ( 3
| 1, 2) ⋯ ( | 1, 2, … , −1 ) Thus, ( ∣ , )
is the product of the conditional probabilities of each token, and these probabilities are output by the LLM at inference time and are readily available in APIs serving LLMs such as the OpenAI API.
In binary classification, accuracy is typically used as an evaluation metric, and can be defined as: 1 | | (,,)∈̄ ∑ 1(( ∣ ) > ( ∣̄ ) )
Where: | | is the number of examples in the dataset ; 1(⋅)is the indicator function that returns 1 if the condition is true and 0 otherwise; and ( )̄ is the correct (incorrect) label for example .
Given a retriever and a demonstration (, , )̄ ∈ we introduce the simplified leave-one-out notation () = () . We define the metric MC1 which is related ∖{(,,)}̄ to accuracy:
MC1 = 1 | |
∑ (,,)∈̄ 1( ( ∣ , () to the case that multiple correct answers a and multiple incorrect answers ā are provided for each query. This metric is the average number of correct answers which have greater probability than all incorrect answers.
MC2 = 1 | |
∑ 1 ∑ ∏ 1( ( ∣ , () (, a,ā)∈ |a| ∈ a ∈̄ ā
Finally, we define the related metric MC3. This metric is the ratio of probability of correct answers to the probability of incorrect answers.
These metrics and their names follow those defined in
[
We postulate that an ideal metric should obey the following properties: it should be positive when the retrieved context increases the probability of a correct answer; it should be equal in magnitude when the probability of a correct answer halves or doubles; it should relate to the probability of getting all correct answers such that if any one correct answer is impossible, the metric is minimized. Moreover, in the case that incorrect answers are provided, it should be positive when the context increases the probability of correct answers more than that of incorrect answers.
We define the DPO metric as the negative of the Direct
Preference Optimization loss [
DPO = log ( log ( ∣ , () ( ∣ ) ) − log ( ∣̄ , () ( ∣̄ ) ) )
In the case that incorrect answers are not available, the term containing ̄ can be omitted while preserving the aforementioned properties.
Because the metric is proportional to probability ratio ( ∣ , )/ ( ∣ ) rather than the absolute probability ( ∣ , ) , it is invariant to the number of tokens and frequency of rare vocabulary tokens in the answer. If this were not the case, then the score for an example would get a positive (negative) bias if the correct (incorrect) answer is shorter. Similarly, the score across a set of examples would weigh examples with shorter answers more strongly.
Another aspect of the DPO metric that is worth considering is that by using this metric to optimize the retriever, we are efectively fine-tuning a model. Consider the LLM model ( ∣ ) which assigns a probability of a generation given a prompt . Now consider another model ′ ( ∣ ′), where ′ = () . From this perspective, we can consider that ′ is functionally the same as but with added parameters arising from . And so, by finding a retriever which maximizes the DPO metric, we are in efect fine-tuning the model ′ . This section details the core algorithm employed for fewshot demonstration retrieval, which leverages a greedy strategy to maximize a combination of three key scores: query relevance, demonstration diversity, and demonstration quality bias. The full retrieval algorithm is presented in Algorithm 1.
The relevance score between the query and each candidate
demonstration is calculated using the cosine similarity of
their respective BERT embeddings [
To promote diversity in the retrieved demonstrations and avoid redundancy, we incorporate the Maximal Marginal Relevance (MMR) algorithm. MMR iteratively selects the items that maximizes the combined score of relevance to the query and dissimilarity to the previously chosen items. This ensures a balance between retrieving relevant items and ensuring they cover a variety of information. A parameter, , is used to control the trade-of between relevance and diversity. Higher values of prioritize relevance, whereas lower values prioritize diversity.
While the pre-trained BERT embeddings capture semantic relationships, they do not inherently account for the quality of the few-shot demonstrations. To address this, we explicitly introduce a demonstration quality bias term related to the popularity of an item in a training dataset. This score is computed using the log perplexity of the demonstration answer , given the demonstration question .
1
∑ log ( ∣ ) || ∈
This can be interpreted as measuring the probability of
the correct answer given the query , normalized to the
length of the answer. This can also been interpreted as a
proxy for a popularity bias, akin to the number of
connections of an item in graph-based retrieval algorithms like
recommender systems. Like in the article [
Algorithm 1 MMR with quality bias Require: 1 ≤ ≤ ; 0 ≤ ≤ 1; 0 ≤ ≤ 1 Require: ∈ ℝ ▷ query embedding Require: ∀1 ≤ ≤ , ∈ ℝ ▷ example question embedding Require: ∀1 ≤ ≤ , ∈ ℝ ▷ example quality bias ← ‖‖ ; ∀1 ≤ ≤ , ← ‖ ‖ ∀1 ≤ ≤ , ← ⊤ + (1 − ) 1 ← arg max for 2 ≤ ≤ do ∀1 ≤ ≤ , ← max1≤< ⊤ ∀1 ≤ ≤ , ← − (1 − ) ← arg max∉{ 1,…, −1 } end for
We are interested in a publicly available dataset which meets the following criteria: it should have enough statistical power so that we can resolve small diferences in accuracy, ideally it will have hundreds of examples or more; it doesn’t need to have vast amounts of data as this isn’t a typical setting for few-shot learning, and the cost of conducting experiments can become burdensome; it should provide correct and incorrect answers so that we can report classification metrics from Section 3.2.1; it should be big enough to contain similar examples with partially redundant information so the use of diversity can improve collective information presented to the LLM in a context.
We chose to conduct our experiments using the TruthfulQA
dataset [
The dataset is used in three diferent ways in this paper:
• : this is the dataset as described in [
The DPO metric is calculated over this dataset. • : this is the set of every distinct (, ) pairs contained in . It contains 2,846 such pairs. This is the set of demonstrations from which a context is drawn. That is, ⊂ .
When calculating a metric score for an example ( , a , ā ), all demonstrations with the query are left out from the demonstrations available for inclusion in the context. In this manner, the correct answers a are not included in the context when the LLM is presented with query .
We conducted our experiments using four noteworthy LLMs:
the smaller base text-completion model Mistral-7B-v0.1 (7B
parameters), and the larger instruct-fine-tuned mixture of
models Mixtral-8x22B-Instruct-v0.1 (141B parameters) from
Mistral [
All four models are open-weights LLMs, meaning their internal parameters are publicly available for scrutiny and potential fine-tuning. These modern models stand out for achieving impressive performance on various tasks despite their relatively compact size. This eficiency makes it an attractive option for resource-constrained environments where deploying colossal models might not be feasible.
We implemented the Algorithm 1 and metrics from Section 3.2.1 in python. We computed BERT embeddings using the package sentence_transformers 2, and implemented the retrieval algorithms in numpy. We queried all LLM models using the Together API 3.
We did not perform hyper-parameter tuning, and fixed
the two parameters to = 0.75 and = 0.95 in all
experiments. We fixed the amount of retrieved demonstrations to
= 6 , matching the number of few-shot examples from the
ifxed primer example from the TruthfulQA paper [
To measure the impact of the separated components of
Algorithm 1, relevance, diversity, and bias, we implemented
variants of the retrieval algorithm using only one or two of
the three components:
• Fix: fixed primer examples [
We present the experimental metrics for the 6 retrievers for the 4 diferent LLMs: in Table 1 and 2 for the Mistral models, and Table 3 and 4 for the Llama-3 models.
Our evaluation relies on a combination of metrics to assess the efectiveness of diferent retrieval strategies for ICL. The normalized DPO metric provides the most valuable insights for each LLM individually but cannot be directly compared across models. The three additional classification metrics allow for objective performance comparisons across models. However, these metrics are susceptible to bias based on token sequence length. 1https://llama.meta.com/llama3/ 2https://sbert.net/ 3https://docs.together.ai/docs/inference-python
The impact of few-shot learning is best seen by comparing the three MC metrics for Rel+Div for a smaller model against Fix for a larger model: the smaller models (7B and 8B parameters) enriched with ICL RAG are essentially matching or outperforming the bigger models (141B and 70B parameters) without ICL RAG.
The results consistently demonstrate that incorporating both relevance and diversity into the retrieval strategy leads to superior performance across all metrics and for both LLMs. For all models, and for all metrics, Rel+Div largely outperforms Rel. This finding reinforces the importance of not just retrieving relevant demonstrations but also ensuring a diverse set that maximizes the informative value for the LLM.
Interestingly, the impact of the low perplexity bias yields contrasting results. For both Mistral models, adding this bias results in a decline in performance on almost all metrics. Conversely, both Llama-3 models exhibit overall improvement with the low perplexity bias, in particular with the DPO metric. This intriguing observation suggests that LLMdependent hyper-parameter tuning of might be necessary to optimize retrieval strategies for specific models. Alternatively, the low perplexity bias itself may benefit from further refinement. Using an opposite intuition, we may argue that instead of prioritizing demonstrations the LLM already finds likely, introducing demonstrations that surprise the model the most could be beneficial for certain LLMs, potentially maximizing the learning impact of each demonstration. These findings open exciting new avenues for future research in ICL retrieval strategies, creating a parallel with novelty and serendipity concepts in recommender systems.
Calibrating the amount of diversity in the retrieved set is crucial when optimizing ICL retrieval. We highlight the dificulty of achieving this without our proposed methodology by demonstrating the non-monotonous relationship between the amount of diversity in the retrieved demonstrations and the resulting benefit to the LLM performance. To quantify diversity, we calculate the average cosine similarity between the BERT embeddings of each demonstration pair within the retrieved set. The LLM’s benefit is measured using the DPO metric. We then systematically vary while keeping the LLM fixed ( Llama-3-8B-chat), the quality bias ifxed ( = 0.95), and the number of retrieved demonstrations constant ( = 6 ) to observe the empirical correlation between diversity and DPO. The results are visualized in Figure 1. This experiment underscores the importance of a metric measuring the impact of the retrieved context on the LLM, like DPO. Without such a metric, it would be challenging to efectively calibrate and achieve the optimal balance between relevance and diversity in the retrieved demonstrations.
While the importance of diversity in ICL retrieval is paramount, we note that readily available RAG systems rarely implement it directly within the core retrieval algorithm. There are several practical considerations to keep in mind for successful deployment.
Retrieval latency is crucial at scale. Exhaustive, brute-force
nearest neighbor search is computationally expensive and
impractical. Instead, real-world systems leverage eficient
indexing techniques and approximated kNN algorithms, as
described in [
The computational cost of ICL may be evaluated against the cost of fine-tuning. For instance, consider a large LLM like gpt-3.5 with a current price 6x larger per input tokens between fine-tuned or default model 4. While ICL requires additional input tokens, it is guaranteed to ofer cost savings compared to fine-tuning when < 6 with this model.
An interesting contrast between ICL and fine-tuning is
highlighted in the paper [
Furthermore, building a curated database of few-shot demonstrations ofers significant advantages to practitioners. These demonstrations are not specific to a single LLM but can be readily utilized with any LLM architecture. This eliminates vendor lock-in and lets practitioners leverage the best LLM for the task at hand without concerns about compatibility. Perhaps even more importantly, a well-maintained database of few-shot examples automatically benefits from the continuous advancements in LLM technology. As newer, more powerful pre-trained LLMs become rapidly available, existing demonstrations can be used to enrich them quickly. This ensures applications leverage the latest capabilities without the need to completely re-engineer workflows. This reusability and adaptability position our few-shot learning engine as a powerful tool for harnessing the ever-evolving potential of LLMs to solve real business challenges. 4In May 2024, the price of gpt-turbo-0125 is $0.5/M input tokens, $1.5/M output tokens; and fine-tuned price of $3/M input tokens, $6/M output tokens, $8M fine-tuning tokens Traditional full-text search algorithms like BM25 lead to empirically lower ICL quality. Vector stores ofer a more suitable solution for eficient retrieval based on semantic similarity. Numerous vendors provide vector store solutions, and they can be broadly categorized as follows:
In-Memory vector indexes, such as FAISS and nmslib, ofer exceptional speed with minimal setup complexity, but limited scalability for larger datasets. They may not implement in-place addition or deletion of the indexed vectors. Self-Hosted vector databases, such as Elasticsearch and Postgres, provide a balance between scalability and performance, at a much larger setup complexity. They typically implement eficient addition and deletion of the indexed vectors. SaaS vector stores, such as Pinecone and VertexAI, ofer a convenient option with pre-configured infrastructure and almost no setup complexity. We invite the reader to consult the lists of integrated vector stores of LangChain 5 and LlamaIndex 6 for near-exhaustive lists of available tools.
Due to the complexities of incorporating such rules
directly within retrieval indexing algorithm [
While vector search ofers a powerful foundation for practical ICL retrieval, it often lacks native support for essential considerations like diversity or quality bias. These aspects are crucial for ensuring informative and efective retrieval of few-shot learning examples. Existing tools for recommendation engines, on the other hand, often excel in these areas. Recommendation engines natively incorporate rules that promote diversity by recommending a variety of items, or quality bias by prioritizing most popular products. Future research directions as well as practical systems for ICL retrieval could explore adapting or integrating these well-established techniques from recommender systems to further enhance the efectiveness and sophistication of fewshot learning through information retrieval. State-of-the-art ICL for real-world applications can be achieved by combining the strengths of vector search with the established ”diversity-aware” retrieval approaches from recommender systems.
This paper explored the critical role of information retrieval in ICL for few-shot learning with Large Language Models. Our work identified key desirable properties for ICL 5https://python.langchain.com/docs/integrations/vectorstores/ 6https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_ stores/ retrieval systems. We demonstrated that state-of-the-art retrieval in this domain resembles recommender systems under the item cold-start problems. Unlike traditional information retrieval prioritizing for exact recall, our approach emphasizes discovery by maximizing the collective information gain from retrieved demonstrations. This necessitates balancing query relevance, quality scoring, and diversity to ensure a variety of informative examples are surfaced. Furthermore, we propose a novel evaluation method for ICL retrieval based on the subsequent performance of the enriched LLM on NLP tasks. This approach eliminates the need for subjective diversity scores, a challenge in information retrieval evaluation. Our findings demonstrate the significant impact of diversity and quality bias in retrieving demonstrations for ICL. By incorporating these well-established techniques from recommender systems, we can unlock the full potential of ICL for few-shot learning and empower LLMs to tackle real-world tasks with limited data.
To Ching-Wei Chen, for finding the name RAGSys, and the entire Crossing Minds team for their support during this research.