RAGSys: Item-Cold-Start Recommender as RAG System Emile Contal1,∗,† , Garrin McGoldrick1,† 1 Crossing Minds Inc, San Francisco, USA Abstract Large Language Models (LLM) hold immense promise for real-world applications, but their generic knowledge often falls short of domain-specific needs. Fine-tuning, a common approach, can suffer from catastrophic forgetting and hinder generalizability. In-Context Learning (ICL) offers an alternative, which can leverage Retrieval-Augmented Generation (RAG) to provide LLMs with relevant demonstrations for few-shot learning tasks. This paper explores the desired qualities of a demonstration retrieval system for ICL. We argue that ICL retrieval in this context resembles item-cold-start recommender systems, prioritizing discovery and maximizing information gain over strict relevance. We propose a novel evaluation method that measures the LLM’s subsequent performance on NLP tasks, eliminating the need for subjective diversity scores. Our findings demonstrate the critical role of diversity and quality bias in retrieved demonstrations for effective ICL, and highlight the potential of recommender system techniques in this domain. Keywords Recommender systems, Information Retrieval, Large Language Models, Few-Shot Learning, In-Context-Learning 1. Introduction scenarios with limited data. This approach exploits the demonstrated ability of LLMs for ”meta-learning” – essen- Large Language Models (LLMs) have emerged as a powerful tially, learning how to learn. In [6], the authors prove the tool for natural language processing, demonstrating remark- capacity of LLMs to effectively ingest in-context training able abilities in areas like text completion, summarization, data points and solve statistical optimization problems such and question answering [1]. One of their most intriguing as gradient descent. ICL enables practitioners to leverage Re- capabilities is their potential to learn ”common sense” – trieval Augmented Generation (RAG), that is enriching the general knowledge about the world that allows them to input prompt by information that is retrieved in real-time reason and make inferences beyond the literal meaning of [7]. We refer to [8] for a recent survey on ICL. text. This has fueled excitement about the possibility of This paper focuses on few-shot learning and the retrieval achieving zero-shot learning, where LLMs can solve unseen of relevant demonstrations for this process, where a demon- problems without any prior training on specific tasks [2]. stration is some text which is included in the LLM’s context However, a crucial distinction exists between generic pub- to demonstrate how the LLM should formulate correct an- lic knowledge and the specific private knowledge required swers. Few-shot learning presents a well-structured prob- for most real-world use cases. While LLMs excel at generic lem, allowing us to evaluate the quality of the retrieval text completion or chat-like interactions, practical applica- algorithm using established classification metrics. Crucially, tions often demand solving specific and repeatable down- we show that enriching a language model with a few-shot stream tasks within a particular domain [3]. This typically example retriever offers a powerful method to achieve fine- necessitates knowledge specific to a business or organiza- tuning-like behavior, steering the output of the LLM towards tion, such as understanding internal processes, up-to-date the desired outcome even with limited data. Interestingly, product details, or customer behavior. increasing the context size in prompts beyond a certain Fine-tuning, a technique where LLMs are trained on large point yields diminishing returns. The most impactful infor- datasets tailored to the target task, offers a path towards mation resides within a relatively small set of well-chosen adapting LLMs to these domain-specific needs. Yet, fine- demonstrations, rather than overloading the prompt with tuning presents significant challenges. When trained on vast amounts of data [9]. This highlights the importance of tasks-specific data, LLMs tend to forget knowledge and skills effective retrieval strategies, transforming 𝑘-shot learning gained in the initial training, a phenomenon referred to as into a top-𝑘 information retrieval problem at its core. Catastrophic Forgetting [4]. Consequently, a fine-tuned Building upon this concept, this paper identifies desir- LLM loses some of its ability to generalize to novel exam- able properties for a RAG system under the framework of ples that aren’t well represented in its fine-tuning training few-shot learning. We demonstrate that state-of-the-art data. Moreover, while fine-tuning allows an LLM to memo- retrieval systems in this context resemble item-cold-start rize task-specific information, it doesn’t necessarily allow recommender systems. Unlike exact search algorithms that the LLM to reason about that information [5]. As a final prioritize precision and recall, our focus is on discovery, by consideration, keeping LLMs constantly up-to-date using maximizing the set of collective information gain from the fine-tuning can be infeasible, especially for domains with retrieved demonstrations. This necessitates solving various frequently changing information like e-commerce product trade-offs between query relevance, quality scoring, as well inventory, whereas it is easy to update a database in real- as diversity algorithms to ensure a variety of informative time from which information is retrieved. examples are surfaced. Furthermore, we propose a method As an alternative to fine-tuning, In-Context Learning for evaluating RAG system performance through the sub- (ICL) offers a promising approach for leveraging LLMs in sequent performance of the enriched LLM on established NLP tasks like question answering or text generation. This Information Retrieval’s Role in RAG Systems (IR-RAG) - 2024 methodology offers a valuable approach to directly assess- ∗ Corresponding author. ing diversity and quality-based retrieval systems, which † Both authors contributed equally to this research. removes the need to define a subjective diversity score, a Envelope-Open emile@crossingminds.com (E. Contal); historically challenging aspect of evaluating such systems garrin.mcgoldrick@crossingminds.com (G. McGoldrick) in academic settings [10]. © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings To summarize, in this paper we study the impact of di- 2.1.3. Pure Relevance versity and quality bias in retrieving demonstrations for Notably, a connection can be drawn between traditional full- ICL. We start by reviewing the use of diversity and other text search algorithms and pure relevance approaches. In biases in both ICL and Information Retrieval works. We [13] the authors use BM25 [14], a well-established retrieval then propose a method for evaluating the performance of ranking function commonly used in information retrieval different retrieval algorithms. Then we present experiments tasks. This approach essentially leverages the strengths of and results demonstrating the impact of diversity and qual- BM25 in identifying examples with terms highly similar ity bias on an LLM’s ability to generate correct answers. to the query, to select the most relevant examples for the Finally we discuss the applicability of state-of-the-art ICL specific task at hand. This strategy ensures the retrieved retrieval algorithms in real-world setting, and show that rec- examples are topically relevant to the task while potentially ommendation engines offer a better solution than semantic introducing some variation in the specific phrasing or word- search engines. ing used. Finally, neural ranking, one of the most commonly used 2. Related Work ICL approach, typically yielding superior results [15, 16, 17, 18, 19], is maximizing similarity in a dense embeddings This paper sits at the intersection of two distinct, but in- space. These methods, like KATE [20], utilize 𝑘-Nearest creasingly intertwined, research areas: In-Context Learn- Neighbors (kNN ) search using the cosine distance of sentence ing for Large Language Models and Information Retrieval. embeddings to retrieve the examples most semantically sim- While ICL focuses on enabling LLMs to learn from carefully ilar to the prompt. Scaling this method leverages vector selected contextual information, IR deals with retrieving search algorithms, commonly used in large-scale informa- relevant information from document collections. Our work tion retrieval tasks, where efficient retrieval of semantically leverages concepts from both fields to address the challenge similar documents is crucial. of few-shot learning with LLMs. While general purpose pre-trained embeddings like BERT [21] form a strong baseline, learning specific embeddings 2.1. Few-Shot In-Context Learning with for retrieval, and in particular ICL, is a very active area of research. In [22] the authors build upon BERT and introduce Retrieval Augmented Generation ColBERT that improves the retrieval performances by an Within the context of RAG, few-shot learning can be defined order of magnitude. Other embeddings model have been as a specific scenario where the ”documents” retrieved are proposed for ICL retrieval in [16] and [17]. Some authors actually ”examples” used to guide the LLM. These examples also explored training full language models [15], as well can also be referred to interchangeably as ”In-Context Ex- as LLM [19], showing further improvements compared to amples” (ICE) or ”demonstrations”. The importance of ICL traditional embedding-based approaches. While leading to in achieving state-of-the-art LLM performance is undeni- superior results, these supervised neural ranking models able, with its ubiquitous presence in top benchmarks across for learning-to-rank necessitate orders of magnitude more various domains. Consequently, ICL research is a rapidly training examples data, that is typically not available to evolving field with numerous proposed algorithms. practitioners. In addition, without any explicit metric space such as dense embeddings, efficient retrieval indexing such 2.1.1. Pure Diversity as [23] cannot be used. Several noteworthy ICL approaches have emerged that ad- 2.1.4. Diversity/Relevance Trade-off dress the challenge of retrieving informative examples for few-shot learning. Some methods only promote the diver- While both relevance and diversity are crucial for effec- sity of the demonstrations, like in [11] where the authors tive few-shot learning, methods yielding the best ICL re- utilize 𝑘-means clustering in a dense embeddings space to sults combine these two paradigms rather than prioritizing achieve diversity. By applying 𝑘-means to the sentence em- one over the other. This is achieved by maximizing a care- beddings of the demonstrations, this approach ensures that fully balanced trade-off between semantic similarity to the the retrieved examples cover a variety of semantic spaces, in- prompt and diversity of the retrieved examples. Unsuper- herently increasing the mutual information of the retrieved vised techniques can be adapted to prioritize the selection set, but without taking into account the relevancy to the of examples that are both relevant to the prompt and dis- query. similar to each other. In [24] the authors introduce a greedy method to select relevant demonstrations while ensuring 2.1.2. Pure Quality enough coverage. They define specific coverage strategies adapted to the problem of program generation. In [25] the Other approaches focuses on identifying examples where authors employs an active learning setting, where a voting the LLM exhibits low token-level uncertainty. In [12] the algorithm selects a set of examples penalizing the top-𝑘 clos- authors analyze token probabilities within candidate 0-shot est from already selected ones, using cosine distance in an prompts. By prioritizing prompts where the LLM has the embedding space. highest generation likelihood (low perplexity), this approach The most popular unsupervised approach for achieving aims to select examples that hold the potential for significant this balance between relevance and diversity is Maximal learning gains for the LLM. The intuition that the authors Marginal Relevance (MMR). MMR retrieves a set of exam- give is that a prompt that is more expected by the LLM is ples by iteratively selecting the example that maximizes a more likely to help it extracting the relevant information. linear combination of the scores of relevance to the prompt Accessing the per-token probabilities for all examples incurs and dissimilarity to the previously retrieved examples. It a significant compute cost, but can be pre-computed as they was analyzed in [26] for ICL and was shown to outperform do not depend on the query. simpler methods. Alternatively to MMR, Determinantal Within batch-mode Bayesian optimization, in [42] and Point Processes (DPP) has been used in [18] to optimize [43] the authors analyze two greedy exploration/exploita- the joint information of the selected 𝑘 examples. However, tion algorithms to select the next batch of items maximizing exactly solving the DPP optimization being NP -hard, hence the cumulative reward. Like with recommender systems, the authors also employs greedy maximization. these online settings exemplify the theoretical and empirical importance of diversifying the selected set of items despite 2.2. Diversity in Information Retrieval and the true objective function only including pure relevance, the cumulative reward. Online Learning The concept of diversity in information retrieval has been a 2.3. Quality Biases in Information Retrieval long-running topic of research. In this section we propose a short review of the use of diversity in the IR literature and Complementing the discussion on diversity in information related domains. retrieval, quality bias also plays a crucial role in effective re- trieval. Quality bias refers to the prioritization of documents 2.2.1. In Information Retrieval or examples considered to be more reliable or informative within the retrieved set. Incorporating quality consider- The MMR algorithm was analyzed in [27], and compared ations into retrieval algorithms can significantly improve against other approached like KL -divergence optimization in standard unbiased IR metrics. [28]. Pure retrieval algorithms typically optimize for recall, Several approaches have been explored to address quality not information maximization. Agreeing on a diversity ob- bias in pure IR tasks. These can be broadly categorized into jective function remains a challenge. Diversity is sometimes content-based and graph-based methods. introduced as a heuristic to cover possible interpretations of the same query, instead of minimizing information overlap 2.3.1. Content-Based Quality Biases from near-duplicated results. In [10] the authors leverage a concept of information nuggets with documents to estimate Content-based methods leverage existing signals inside the the redundancy of the set of retrieved documents. Topic documents themselves to identify potentially lower-quality modeling is also employed, such as [29] that uses a taxon- content. Examples include spam detection scores developed omy of categories labelling the documents and the queries. in works like [44] and [45]. By incorporating such scores The desired properties of diverse retrieval are furthermore during retrieval, the system can prioritize higher quality doc- characterized in [30]. A various set of similarity methods uments. More sophisticated content-based approaches don’t and diversification algorithms are analyzed in [31] on sparse limit at spam classification, but extract more generic quality features vectors. Among diversity evaluation methods based features the content of documents. Works like [46] explore on topic modelling, three notable criteria used in the TREC features such as stop-word statistics or entropy of the docu- Diversity track [32], ERR-IA [33], 𝛼-nDCG @𝑘 [10], and NRBP ments to generate quality scores. The authors demonstrate [34], are compared in [35]. that biasing standard retrieval using these features leads to improved retrieval effectiveness even using unbiased IR 2.2.2. In Recommender Systems metrics like nDCG . Within IR, the recommender system literature brings an 2.3.2. Graph-Based Quality Biases additional point-of-view on studying diversity in retrieval, by focusing on the benefit of diverse results for a user, in- Instead of relying on the content itself, graph-based algo- stead of evaluating methods against a potentially arbitrary rithms inherently capture implicit quality signals within relevance/diversity trade-off. The difficulty of evaluating their ranking model. PageRank , a seminal algorithm for the impact of diversity, and the necessity for large scale web search ranking introduced in [47], exemplifies this ap- real-world recommendation studies has been explored in proach. PageRank leverages the links structure between [36]. In [37] and [38] the authors model the user behavior web articles to assign higher importance to web pages that conditioned on the set of retrieved items. In [39] the authors are linked to by other high-quality pages. This process improve the diversity versus relevance trade-off in recom- implicitly prioritizes documents with a higher perceived mender systems by directly learning a ranking model that quality based on the quality of their in-links. favor diversity, instead of only applying diversity re-ranking methods. 2.3.3. Connections to Recommender Systems Interestingly, the concept of inherent quality bias in graph- 2.2.3. In Online Learning based IR approaches resembles collaborative filtering tech- Learning a trade-off between relevancy and diversity also niques employed in recommender systems. In an analogous naturally occurs in related online frameworks such as active manner to learning-to-rank on a (item, item) graph, collab- learning, multi-armed bandits and Bayesian optimization. orative filtering addresses learning-to-rank on a bipartite In [40] the authors modify a learning-to-rank algorithm (user, item) graph. In this way, collaborative filtering also from users feedback, to inherently learn diverse rankings implicitly learns a trade-off between item similarity and and demonstrate a positive impact on the original relevance popularity, favoring items that are both similar to the user’s metric. Other approaches such as [41] also introduce di- past preferences and also generally well-received by other versity in learning-to-rank algorithms while preserving the users. offline settings, but then are limited to evaluate using direct diversity measures. 3. Methodology of the correct answer 𝑎. That is, we want a metric which is related to difference of 𝑝𝑀 (𝑎 ∣ 𝑞, 𝐶) and 𝑝𝑀 (𝑎 ∣ 𝑞). We propose to frame the ICL problem as an item-cold-start In a pure retrieval setting, we would be interested in find- recommendation problem, where the query is an unseen ing the context 𝐶 which contains the 𝑘 demonstrations that item, and the objective is to retrieve from the pool of can- are most similar to the 𝑞. And we could argue that if there didate few-shot demonstrations a set of items maximizing exists a smooth function 𝑓 ∶ 𝑞 → 𝑎 which maps a query to the cumulative reward to the user (the LLM). In this case, its correct answer, then by retrieving the demonstrations the reward is a measure of how much the retrieved items whose queries are nearest to 𝑞, we should also be retrieving increase the probability that the LLM generates a correct the answers which are closest to 𝑎, and this should help the answer. A solution to this optimization problem requires language model 𝑀 generate the correct answer 𝑎. not only relevance, but also diversity and quality in the re- However, it is doubtful that the space in which 𝑞 is com- trieved items, such that the amount of useful information pared to the demonstrations is one in which the function presented to the LLM in the context is maximized. 𝑓 ∶ 𝑞 → 𝑎 is smooth, so it is not necessarily true that Further, we propose to measure the impact of diversity the retrieved answers are closest to 𝑎. Nor is it necessarily on the retrieved items by directly calculating the probability true that 𝑝𝑀 (𝑎 ∣ 𝑞, 𝐶) is maximized when 𝐶 contains those of the LLM generating a correct answer given the context answers closest to 𝑎. Consider that the answer 𝑎 might de- items. This is in contrast to a typical retrieval context where pend on some information which isn’t contained in 𝑎 or any the retriever is evaluated by calculating some metric relat- nearby answer. ing to the accuracy and recall of documents most similar Therefore, we prefer to measure 𝑝𝑀 (𝑎 ∣ 𝑞, 𝐶) directly. to the query. In such a setting, it is typical to add a term In practice, given that 𝑀 is an auto-regressive language to the metric which measures the diversity of the retrieved model, this is done by taking the product of the probabil- documents to promote more diverse retrievers, knowing ity of each token generated by 𝑀. The model generates that diversity improves the reward to the user but with- text sequentially by predicting one token at a time based out having an explicit model connecting diversity to the on the previously generated tokens. Let 𝑎 = (𝑎1 , 𝑎2 , … , 𝑎𝑛 ) user’s reward. In the case of retrieving demonstrations for represent a sequence of tokens produced by the model. The inclusion in an LLM’s context, we can directly measure the probability of the model generating the sequence 𝑎 can be impact of diversity on the LLM’s reward by calculating the expressed as the joint probability of generating each token probability of the LLM generating a correct answer. in the sequence, conditioned on the tokens that precede it. This can be mathematically represented as: 3.1. Problem Statement 𝑝(𝑎) = 𝑝(𝑎1 ) ⋅ 𝑝(𝑎2 |𝑎1 ) ⋅ 𝑝(𝑎3 |𝑎1 , 𝑎2 ) ⋯ 𝑝(𝑎𝑛 |𝑎1 , 𝑎2 , … , 𝑎𝑛−1 ) Consider an answer 𝑎 that should be generated by an LLM in response to a query 𝑞. The query can be a simple question Thus, 𝑝𝑀 (𝑎 ∣ 𝑞, 𝐶) is the product of the conditional proba- such as ”Who was the first man to walk on the moon?”, or bilities of each token, and these probabilities are output by more general message such as ”I’d like to find red shoes”. the LLM at inference time and are readily available in APIs The answer could take on many forms, such a factual re- serving LLMs such as the OpenAI API. sponse ”Neil Armstrong”, a clarifying question ”What size and style of shoe are you looking for?”, a JSON payload to 3.2.1. Classification Metrics send to an API ”{"search_terms":["red","shoe"]} ” etc. In binary classification, accuracy is typically used as an Consider a set of demonstrations 𝒟, where each demon- evaluation metric, and can be defined as: stration is a pair (𝑞, 𝑎) containing a query 𝑞 and correct answer 𝑎, or a triple (𝑞, 𝑎, 𝑎)̄ which additionally contains an 1 ∑ 1(𝑝(𝑦 ∣ 𝑥) > 𝑝(𝑦 ̄ ∣ 𝑥)) incorrect answer 𝑎.̄ Datasets under this later triplet form |𝒟 | (𝑥,𝑦,𝑦)∈𝒟 ̄ are commonly used in Contrastive Learning approaches. We call 𝐶, a subset of demonstrations retrieved from 𝒟, the Where: |𝒟 | is the number of examples in the dataset 𝒟; context. 1(⋅) is the indicator function that returns 1 if the condition is true and 0 otherwise; and 𝑦 (𝑦)̄ is the correct (incorrect) 𝐶 ⊂ 𝒟 = {(𝑞𝑖 , 𝑎𝑖 , 𝑎𝑖̄ ), … , (𝑞𝑛 , 𝑎𝑛 , 𝑎𝑛̄ )} label for example 𝑥. Given a retriever 𝑅𝒟 and a demonstration (𝑞, 𝑎, 𝑎)̄ ∈ 𝒟, Given an auto-regressive LLM 𝑀, the query 𝑞, and a re- we introduce the simplified leave-one-out notation 𝑅(𝑞) = trieved context 𝐶, we define 𝑝𝑀 (𝑎 ∣ 𝑞, 𝐶) the probability that 𝑅𝒟 ∖{(𝑞,𝑎,𝑎)} ̄ (𝑞). We define the metric MC1 which is related 𝑀 generates the answer 𝑎. In practice, the tokens of the to accuracy: examples from the context 𝐶 are appended to the tokens of the query 𝑞, using prompt formatting techniques that may 1 be optimized to a specific LLM. MC1 = ∑ 1(𝑝𝑀 (𝑎 ∣ 𝑞, 𝑅(𝑞)) > 𝑝𝑀 (𝑎 ̄ ∣ 𝑞, 𝑅(𝑞))) |𝒟 | (𝑞,𝑎,𝑎)∈𝒟 ̄ Putting it all together, for an unseen query 𝑞 and unseen correct answer 𝑎, a few-shot retriever 𝑅𝒟 must efficiently In the case that many incorrect answers are provided for retrieve a subset of 𝑘 demonstrations 𝑅𝒟 (𝑞) ∈ 𝒟 𝑘 such that each query a,̄ we can extend this in the same manner as 𝑝𝑀 (𝑎 ∣ 𝑞, 𝑅𝒟 (𝑞)) is maximized. multi-class classification by requiring that the correct an- swer have greater probability than all the incorrect answers: 3.2. Evaluation 1 ∑ ∏ 1(𝑝𝑀 (𝑎 ∣ 𝑞, 𝑅(𝑞)) > 𝑝𝑀 (𝑎 ̄ ∣ 𝑞, 𝑅(𝑞))) Consider the probability of generating the correct answer |𝒟 | (𝑞,𝑎,a)∈𝒟 ̄ 𝑎∈̄ ā 𝑎 given an empty context 𝑝𝑀 (𝑎 ∣ 𝑞). We are interested in evaluating how much the context 𝐶 increases the probability We also define a metric MC2 which extends this further to the case that multiple correct answers a and multiple incorrect answers ā are provided for each query. This metric 3.3. Retrieval Algorithm is the average number of correct answers which have greater This section details the core algorithm employed for few- probability than all incorrect answers. shot demonstration retrieval, which leverages a greedy strat- MC2 = egy to maximize a combination of three key scores: query relevance, demonstration diversity, and demonstration qual- 1 1 ity bias. The full retrieval algorithm is presented in Algo- ∑ ∑ ∏ 1(𝑝𝑀 (𝑎 ∣ 𝑞, 𝑅(𝑞)) > 𝑝𝑀 (𝑎 ̄ ∣ 𝑞, 𝑅(𝑞))) |𝒟 | (𝑞,a,a)∈𝒟 ̄ |a| 𝑎∈a 𝑎∈̄ ā rithm 1. Finally, we define the related metric MC3. This metric is the ratio of probability of correct answers to the probability 3.3.1. Query Relevance of incorrect answers. The relevance score between the query and each candidate ∑𝑎∈a 𝑝𝑀 (𝑎 ∣ 𝑞, 𝑅(𝑞)) demonstration is calculated using the cosine similarity of 1 MC3 = ∑ their respective BERT embeddings [21]. By computing the |𝒟 | (𝑞,a,a)∈𝒟 ̄ ∑𝑎∈̄ ā 𝑝𝑀 (𝑎 ̄ ∣ 𝑞, 𝑅(𝑞)) cosine similarity between the query embedding and the embedding of each demonstration’s query, we obtain a score These metrics and their names follow those defined in that reflects the topical similarity and semantic alignment [48]. While they are easy to interpret, these metrics are not between the query and the candidate demonstration. well normalized: they don’t take into account all possible correct and incorrect answers. As a result, if the sample of 3.3.2. Retrieved Demonstrations Diversity correct and incorrect answers have varying lengths and use of rare vocabulary tokens, these will impact the metrics. To promote diversity in the retrieved demonstrations and avoid redundancy, we incorporate the Maximal Marginal 3.2.2. Direct Preference Optimization Metric Relevance (MMR) algorithm. MMR iteratively selects the items that maximizes the combined score of relevance to the We postulate that an ideal metric should obey the following query and dissimilarity to the previously chosen items. This properties: it should be positive when the retrieved context ensures a balance between retrieving relevant items and increases the probability of a correct answer; it should be ensuring they cover a variety of information. A parameter, equal in magnitude when the probability of a correct an- 𝜆𝑑 , is used to control the trade-off between relevance and swer halves or doubles; it should relate to the probability diversity. Higher values of 𝜆𝑑 prioritize relevance, whereas of getting all correct answers such that if any one correct lower values prioritize diversity. answer is impossible, the metric is minimized. Moreover, in the case that incorrect answers are provided, it should 3.3.3. Demonstration Quality Bias be positive when the context 𝐶 increases the probability of correct answers more than that of incorrect answers. While the pre-trained BERT embeddings capture semantic We define the DPO metric as the negative of the Direct relationships, they do not inherently account for the quality Preference Optimization loss [49], which satisfies these prop- of the few-shot demonstrations. To address this, we explic- erties: itly introduce a demonstration quality bias term related to the popularity of an item in a training dataset. This score is computed using the log perplexity of the demonstration 𝑝𝑀 (𝑎 ∣ 𝑞, 𝑅(𝑞)) 𝑝𝑀 (𝑎 ̄ ∣ 𝑞, 𝑅(𝑞)) answer 𝑎, given the demonstration question 𝑞. DPO = log 𝜎( log − log ) 𝑝𝑀 (𝑎 ∣ 𝑞) 𝑝𝑀 (𝑎 ̄ ∣ 𝑞) 1 ∑ log 𝑝𝑀 (𝑎𝑖 ∣ 𝑞) In the case that incorrect answers are not available, the |𝑎| 𝑎 ∈𝑎 𝑖 term containing 𝑎 ̄ can be omitted while preserving the afore- mentioned properties. This can be interpreted as measuring the probability of Because the metric is proportional to probability ratio the correct answer 𝑎 given the query 𝑞, normalized to the 𝑝𝑀 (𝑎 ∣ 𝑞, 𝐶)/𝑝𝑀 (𝑎 ∣ 𝑞) rather than the absolute probability length of the answer. This can also been interpreted as a 𝑝𝑀 (𝑎 ∣ 𝑞, 𝐶), it is invariant to the number of tokens and proxy for a popularity bias, akin to the number of connec- frequency of rare vocabulary tokens in the answer. If this tions of an item in graph-based retrieval algorithms like were not the case, then the score for an example would get recommender systems. Like in the article [12], the intuition a positive (negative) bias if the correct (incorrect) answer is is that the more frequently a related sequence of tokens oc- shorter. Similarly, the score across a set of examples would curs in the pre-training dataset of the LLM, the more likely weigh examples with shorter answers more strongly. the model will be able to extract its relevant information. Another aspect of the DPO metric that is worth consider- Rather than directly analyzing the massive amount of text ing is that by using this metric to optimize the retriever, we data (often trillions of tokens) used to pre-train the LLM, are effectively fine-tuning a model. Consider the LLM model we focus on the perplexity of the sequence. Perplexity acts 𝑝𝑀 (𝑎 ∣ 𝑞) which assigns a probability of a generation 𝑎 given as a proxy, indicating how surprised the LLM is by the se- a prompt 𝑞. Now consider another model 𝑝𝑀 ′ (𝑎 ∣ 𝑞 ′ ), where quence, essentially, how well it aligns with what the LLM 𝑞 ′ = 𝑅(𝑞). From this perspective, we can consider that 𝑝𝑀 ′ expects to see. A parameter 𝜆𝑏 controls the trade-off be- is functionally the same as 𝑝𝑀 but with added parameters tween relevance/diversity and quality bias. Lower values of arising from 𝑅. And so, by finding a retriever 𝑅 which maxi- 𝜆𝑏 emphasize high-quality demonstrations. mizes the DPO metric, we are in effect fine-tuning the model 𝑝𝑀′ . Algorithm 1 MMR with quality bias When calculating a metric score for an example (𝑞𝑖 , a𝑖 , a𝑖̄ ), Require: 1 ≤ 𝑘 ≤ 𝑛; 0 ≤ 𝜆𝑑 ≤ 1; 0 ≤ 𝜆𝑏 ≤ 1 all demonstrations with the query 𝑞𝑖 are left out from the Require: 𝑄 ∈ ℝ𝑑 ▷ query embedding demonstrations available for inclusion in the context. In Require: ∀1 ≤ 𝑖 ≤ 𝑛, 𝐸𝑖 ∈ ℝ𝑑 ▷ example question this manner, the correct answers a𝑖 are not included in the embedding context when the LLM is presented with query 𝑞𝑖 . Require: ∀1 ≤ 𝑖 ≤ 𝑛, 𝑏𝑖 ∈ ℝ ▷ example quality bias 𝑄 𝐸 𝑄 ← ‖𝑄‖ ; ∀1 ≤ 𝑖 ≤ 𝑛, 𝐸𝑖 ← ‖𝐸𝑖 ‖ 4.1.4. The Language Models 𝑖 ∀1 ≤ 𝑖 ≤ 𝑛, 𝑣𝑖 ← 𝜆𝑏 𝑄𝐸𝑖⊤ + (1 − 𝜆𝑏 )𝑏𝑖 We conducted our experiments using four noteworthy LLMs: 𝐶1 ← arg max𝑖 𝑣𝑖 the smaller base text-completion model Mistral-7B-v0.1 (7B for 2 ≤ 𝑠 ≤ 𝑘 do parameters), and the larger instruct-fine-tuned mixture of ∀1 ≤ 𝑖 ≤ 𝑛, 𝑚𝑖 ← max1≤𝑗<𝑠 𝐸𝑖 𝐸𝐶⊤𝑗 models Mixtral-8x22B-Instruct-v0.1 (141B parameters) from ∀1 ≤ 𝑖 ≤ 𝑛, 𝑤𝑖 ← 𝜆𝑑 𝑣𝑖 − (1 − 𝜆𝑑 )𝑚𝑖 Mistral [50]; as well as a smaller chat-tuned model Llama- 𝐶𝑠 ← arg max𝑖∉{𝐶1 ,…,𝐶𝑠−1 } 𝑤𝑖 3-8B-chat (8B parameters) and a larger chat-tuned model end for Llama-3-70B-chat (70B parameters) from Llama 1 . All four models are open-weights LLMs, meaning their internal parameters are publicly available for scrutiny and 4. Experiments potential fine-tuning. These modern models stand out for achieving impressive performance on various tasks despite 4.1. Experimental Setup their relatively compact size. This efficiency makes it an attractive option for resource-constrained environments 4.1.1. The Dataset Choice where deploying colossal models might not be feasible. We are interested in a publicly available dataset which meets the following criteria: it should have enough statistical 4.2. Implementation Details and Ablation power so that we can resolve small differences in accuracy, Study ideally it will have hundreds of examples or more; it doesn’t need to have vast amounts of data as this isn’t a typical We implemented the Algorithm 1 and metrics from Section setting for few-shot learning, and the cost of conducting 3.2.1 in python. We computed BERT embeddings using the experiments can become burdensome; it should provide package sentence_transformers 2 , and implemented the re- correct and incorrect answers so that we can report classifi- trieval algorithms in numpy. We queried all LLM models cation metrics from Section 3.2.1; it should be big enough using the Together API 3 . to contain similar examples with partially redundant in- We did not perform hyper-parameter tuning, and fixed formation so the use of diversity can improve collective the two parameters to 𝜆𝑑 = 0.75 and 𝜆𝑏 = 0.95 in all experi- information presented to the LLM in a context. ments. We fixed the amount of retrieved demonstrations to 𝑘 = 6, matching the number of few-shot examples from the 4.1.2. The TruthfulQA Dataset fixed primer example from the TruthfulQA paper [48]. To measure the impact of the separated components of We chose to conduct our experiments using the TruthfulQA Algorithm 1, relevance, diversity, and bias, we implemented dataset [48] which meets these requirements. The dataset variants of the retrieval algorithm using only one or two of contains 817 distinct examples, which yields a standard error the three components: in the range of 1% to 2% for accuracy measures in the range of 90% to 50%. Each example contains a single query, and • Fix: fixed primer examples [48] a variable number of correct and incorrect answers. And • Bias: Pure quality bias [12] by considering each distinct (𝑞, 𝑎) pair as a demonstration • Rel: Pure semantic similarity [20] (KATE ) for the purpose of building a context, the retriever is faced • Rel+Bias: Semantic similarity plus quality bias with similar demonstrations as multiple (𝑞, 𝑎) pairs share the same query (on average, the dataset contains 3.5 correct • Rel+Div: Semantic similarity plus diversity [26] answers for each query). • Rel+Bias+Div: Algorithm 1 4.1.3. Generating Demonstrations Pairs and Triplets 4.3. Main Results The dataset is used in three different ways in this paper: We present the experimental metrics for the 6 retrievers for the 4 different LLMs: in Table 1 and 2 for the Mistral models, • 𝒟𝑀𝐶 : this is the dataset as described in [48]. It con- and Table 3 and 4 for the Llama-3 models. tains 817 examples, each of which contains a vari- Our evaluation relies on a combination of metrics to as- able number of correct and incorrect answers. The sess the effectiveness of different retrieval strategies for ICL. metrics MC1, MC2 and MC3, which can accept an The normalized DPO metric provides the most valuable arbitrary number of correct and incorrect answers insights for each LLM individually but cannot be directly as inputs, are calculated over this dataset. compared across models. The three additional classification • 𝒟𝐷𝑃𝑂 : this is the set of every distinct (𝑞, 𝑎, 𝑎)̄ triple metrics allow for objective performance comparisons across contained in 𝒟𝑀𝐶 . It contains 12,485 such triplets. models. However, these metrics are susceptible to bias based The DPO metric is calculated over this dataset. on token sequence length. • 𝒟𝐼 𝐶𝐿 : this is the set of every distinct (𝑞, 𝑎) pairs contained in 𝒟𝑀𝐶 . It contains 2,846 such pairs. This 1 https://llama.meta.com/llama3/ is the set of demonstrations from which a context is 2 https://sbert.net/ drawn. That is, 𝐶 ⊂ 𝒟𝐼 𝐶𝐿 . 3 https://docs.together.ai/docs/inference-python Table 1 Table 2 Evaluation Metrics with Mistral-7B-v0.1 Evaluation Metrics with Mixtral-8x22B-Instruct-v0.1 Method DPO MC1 MC2 MC3 Method DPO MC1 MC2 MC3 Fix -20.40 0.2815 0.2086 0.4285 Fix -19.06 0.5202 0.3896 0.6799 Bias -33.56 0.2411 0.1652 0.3596 Bias -27.30 0.4382 0.3096 0.5948 Rel -12.71 0.4455 0.3664 0.5925 Rel -15.17 0.6193 0.5004 0.7616 Rel+Bias -13.63 0.4602 0.3663 0.5969 Rel+Bias -14.77 0.6389 0.5080 0.7657 Rel+Div -12.37 0.5177 0.3930 0.6616 Rel+Div -12.67 0.6879 0.5181 0.8092 Rel+Div+Bias -14.54 0.4676 0.3592 0.6255 Rel+Div+Bias -13.29 0.6573 0.5071 0.7924 Table 3 Table 4 Evaluation Metrics with Llama-3-8B-chat Evaluation Metrics with Llama-3-70B-chat Method DPO MC1 MC2 MC3 Method DPO MC1 MC2 MC3 Fix -22.12 0.3623 0.2709 0.5195 Fix -23.05 0.4382 0.3375 0.6184 Bias -17.55 0.3831 0.2876 0.5729 Bias -20.41 0.4455 0.3303 0.6424 Rel -17.20 0.4920 0.4046 0.6518 Rel -19.02 0.5483 0.4482 0.6958 Rel+Bias -17.14 0.5043 0.4083 0.6570 Rel+Bias -19.09 0.5532 0.4495 0.7054 Rel+Div -16.14 0.5520 0.4173 0.7009 Rel+Div -13.93 0.6389 0.4834 0.7758 Rel+Div+Bias -15.80 0.5177 0.4007 0.6841 Rel+Div+Bias -13.72 0.6022 0.4621 0.7583 The impact of few-shot learning is best seen by comparing larity between the BERT embeddings of each demonstration the three MC metrics for Rel+Div for a smaller model against pair within the retrieved set. The LLM’s benefit is measured Fix for a larger model: the smaller models (7B and 8B param- using the DPO metric. We then systematically vary 𝜆𝑑 while eters) enriched with ICL RAG are essentially matching or keeping the LLM fixed (Llama-3-8B-chat), the quality bias outperforming the bigger models (141B and 70B parameters) fixed (𝜆𝑏 = 0.95), and the number of retrieved demonstra- without ICL RAG. tions constant (𝑘 = 6) to observe the empirical correlation The results consistently demonstrate that incorporating between diversity and DPO. The results are visualized in both relevance and diversity into the retrieval strategy leads Figure 1. This experiment underscores the importance of to superior performance across all metrics and for both a metric measuring the impact of the retrieved context on LLMs. For all models, and for all metrics, Rel+Div largely the LLM, like DPO. Without such a metric, it would be chal- outperforms Rel. This finding reinforces the importance of lenging to effectively calibrate 𝜆𝑑 and achieve the optimal not just retrieving relevant demonstrations but also ensuring balance between relevance and diversity in the retrieved a diverse set that maximizes the informative value for the demonstrations. LLM. Interestingly, the impact of the low perplexity bias yields contrasting results. For both Mistral models, adding this bias results in a decline in performance on almost all metrics. Conversely, both Llama-3 models exhibit overall improve- ment with the low perplexity bias, in particular with the DPO metric. This intriguing observation suggests that LLM- dependent hyper-parameter tuning of 𝜆𝑏 might be necessary to optimize retrieval strategies for specific models. Alter- natively, the low perplexity bias itself may benefit from further refinement. Using an opposite intuition, we may argue that instead of prioritizing demonstrations the LLM already finds likely, introducing demonstrations that sur- prise the model the most could be beneficial for certain LLMs, potentially maximizing the learning impact of each demonstration. These findings open exciting new avenues for future research in ICL retrieval strategies, creating a par- allel with novelty and serendipity concepts in recommender Figure 1: Diversity Metric and DPO. Non-monotonous relation- systems. ship between a diversity metric, the average cosine similarity between embedding pairs, and the quality metric DPO. Obtained by varying 𝜆𝑑 with Llama-3-8B-chat and 𝑘 = 6. 4.4. Calibrating Diversity using DPO Calibrating the amount of diversity in the retrieved set is crucial when optimizing ICL retrieval. We highlight the difficulty of achieving this without our proposed method- 5. Discussion: Real-World RAG ology by demonstrating the non-monotonous relationship between the amount of diversity in the retrieved demon- Systems strations and the resulting benefit to the LLM performance. While the importance of diversity in ICL retrieval is To quantify diversity, we calculate the average cosine simi- paramount, we note that readily available RAG systems rarely implement it directly within the core retrieval algo- 5.3. Achieving State-of-the-Art Retrieval rithm. There are several practical considerations to keep in with Available Tools mind for successful deployment. Traditional full-text search algorithms like BM25 lead to empirically lower ICL quality. Vector stores offer a more 5.1. Balancing Performance and Efficiency suitable solution for efficient retrieval based on semantic Retrieval latency is crucial at scale. Exhaustive, brute-force similarity. Numerous vendors provide vector store solutions, nearest neighbor search is computationally expensive and and they can be broadly categorized as follows: impractical. Instead, real-world systems leverage efficient In-Memory vector indexes, such as FAISS and nmslib , indexing techniques and approximated kNN algorithms, as offer exceptional speed with minimal setup complexity, but described in [23], to ensure fast retrieval times. This ap- limited scalability for larger datasets. They may not imple- proach is essential for handling large datasets while main- ment in-place addition or deletion of the indexed vectors. taining responsiveness. To seamlessly integrate with exist- Self-Hosted vector databases, such as Elasticsearch and ing retrieval engines and leverage their optimized search Postgres , provide a balance between scalability and perfor- capabilities, a retrieval algorithm for RAG must ensure its mance, at a much larger setup complexity. They typically data is stored in a format compatible with these engines. implement efficient addition and deletion of the indexed vec- Commonly indexed data structures include text itself or tors. SaaS vector stores, such as Pinecone and VertexAI , low-dimensional dense vector embeddings. By adhering offer a convenient option with pre-configured infrastructure to these indexing practices, RAG systems can effectively and almost no setup complexity. We invite the reader to leverage the power of existing retrieval engines and achieve consult the lists of integrated vector stores of LangChain 5 fast, scalable retrieval of informative examples. and LlamaIndex 6 for near-exhaustive lists of available tools. Due to the complexities of incorporating such rules di- rectly within retrieval indexing algorithm [52], none of the 5.2. ICL RAG versus Fine-Tuning solutions known to the authors from any of the above cate- The computational cost of ICL may be evaluated against gory implements diversity or quality biasing of the result. the cost of fine-tuning. For instance, consider a large LLM A common heuristic to mitigate this problem is to retrieve like gpt-3.5 with a current price 6x larger per input tokens a larger set of candidate examples (e.g., double the desired between fine-tuned or default model 4 . While ICL requires number) and then apply diversity techniques like MMR on additional input tokens, it is guaranteed to offer cost savings the retrieved candidates as a post-processing step. Quality compared to fine-tuning when 𝑘 < 6 with this model. biasing can be indirectly achieved by modifying the indexed An interesting contrast between ICL and fine-tuning is embeddings themselves. For instance, reducing the norm of highlighted in the paper [51]. The paper argues that fine- embeddings associated with low-quality content can nudge tuning can be more efficient than few-shot ICL in terms of the retrieval algorithm towards higher-quality examples. cost and latency due to the super-linear increase in LLM An exact implementation in the context of cosine-similarity latency with growing prompt sizes. However, this latency or dot-product relevance is to add an additional column concern is less relevant with inference-throughput opti- storing the quality bias, and set the corresponding value to mized LLM systems built with large GPU clusters, such as 1 in the embedding of the query. commonly used APIs. In these systems, the observed latency While vector search offers a powerful foundation for prac- remains independent of the prompt size. From a latency per- tical ICL retrieval, it often lacks native support for essential spective, adding ICL demonstrations can be considered free. considerations like diversity or quality bias. These aspects Additionally, the paper suggests that ICL requires scanning are crucial for ensuring informative and effective retrieval through 100% of the demonstrations at query time. However of few-shot learning examples. Existing tools for recom- this does not hold when employing real retrieval engines mendation engines, on the other hand, often excel in these with indexing and approximate kNN , which significantly areas. Recommendation engines natively incorporate rules reduce the number of examples scanned during retrieval. that promote diversity by recommending a variety of items, Furthermore, building a curated database of few-shot or quality bias by prioritizing most popular products. Fu- demonstrations offers significant advantages to practition- ture research directions as well as practical systems for ers. These demonstrations are not specific to a single ICL retrieval could explore adapting or integrating these LLM but can be readily utilized with any LLM architec- well-established techniques from recommender systems to ture. This eliminates vendor lock-in and lets practitioners further enhance the effectiveness and sophistication of few- leverage the best LLM for the task at hand without con- shot learning through information retrieval. State-of-the-art cerns about compatibility. Perhaps even more importantly, ICL for real-world applications can be achieved by com- a well-maintained database of few-shot examples automat- bining the strengths of vector search with the established ically benefits from the continuous advancements in LLM ”diversity-aware” retrieval approaches from recommender technology. As newer, more powerful pre-trained LLMs systems. become rapidly available, existing demonstrations can be used to enrich them quickly. This ensures applications lever- age the latest capabilities without the need to completely 6. Conclusion re-engineer workflows. This reusability and adaptability This paper explored the critical role of information retrieval position our few-shot learning engine as a powerful tool in ICL for few-shot learning with Large Language Mod- for harnessing the ever-evolving potential of LLMs to solve els. Our work identified key desirable properties for ICL real business challenges. 4 5 In May 2024, the price of gpt-turbo-0125 is $0.5/M input tokens, $1.5/M https://python.langchain.com/docs/integrations/vectorstores/ 6 output tokens; and fine-tuned price of $3/M input tokens, $6/M output https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_ tokens, $8M fine-tuning tokens stores/ retrieval systems. We demonstrated that state-of-the-art b2e63e36c57e153b9015fece2352a9f9-Paper-Conference. retrieval in this domain resembles recommender systems pdf. under the item cold-start problems. Unlike traditional infor- [7] O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, mation retrieval prioritizing for exact recall, our approach A. Shashua, K. Leyton-Brown, Y. Shoham, In-Context emphasizes discovery by maximizing the collective informa- Retrieval-Augmented Language Models, Transactions tion gain from retrieved demonstrations. This necessitates of the Association for Computational Linguistics 11 balancing query relevance, quality scoring, and diversity (2023) 1316–1331. doi:10.1162/tacl_a_00605 . to ensure a variety of informative examples are surfaced. [8] Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, Furthermore, we propose a novel evaluation method for ICL X. Sun, J. Xu, L. Li, Z. Sui, A survey for in-context retrieval based on the subsequent performance of the en- learning, CoRR abs/2301.00234 (2023). riched LLM on NLP tasks. This approach eliminates the need [9] D. Machlab, R. Battle, LLM In-Context Recall is Prompt for subjective diversity scores, a challenge in information re- Dependent, CoRR (2024) arXiv:2404.08865. doi:10. trieval evaluation. Our findings demonstrate the significant 48550/arXiv.2404.08865 . arXiv:2404.08865 . impact of diversity and quality bias in retrieving demon- [10] C. L. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, strations for ICL. By incorporating these well-established A. Ashkan, S. Büttcher, I. MacKinnon, Novelty and techniques from recommender systems, we can unlock the diversity in information retrieval evaluation, in: Pro- full potential of ICL for few-shot learning and empower ceedings of the 31st Annual International ACM SIGIR LLMs to tackle real-world tasks with limited data. Conference on Research and Development in Infor- mation Retrieval, SIGIR ’08, Association for Comput- ing Machinery, New York, NY, USA, 2008, p. 659–666. 7. Acknowledgments doi:10.1145/1390334.1390446 . [11] W. Yu, D. Iter, S. Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, To Ching-Wei Chen, for finding the name RAGSys, and the M. Zeng, M. Jiang, Generate rather than retrieve: entire Crossing Minds team for their support during this Large language models are strong context generators, research. in: International Conference for Learning Representa- tion (ICLR), 2023. References [12] H. Gonen, S. Iyer, T. Blevins, N. Smith, L. Zettlemoyer, Demystifying prompts in language models via per- [1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plexity estimation, in: H. Bouamor, J. Pino, K. Bali plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- (Eds.), Findings of the Association for Computational try, A. Askell, et al., Language models are few-shot Linguistics: EMNLP 2023, Association for Computa- learners, Advances in neural information processing tional Linguistics, Singapore, 2023, pp. 10136–10148. systems 33 (2020) 1877–1901. doi:10.18653/v1/2023.findings- emnlp.679 . [2] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, [13] S. Agrawal, C. Zhou, M. Lewis, L. Zettlemoyer, Large language models are zero-shot reasoners, in: M. Ghazvininejad, In-context examples selection S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, for machine translation, in: A. Rogers, J. Boyd- K. Cho, A. Oh (Eds.), Advances in Neural Infor- Graber, N. Okazaki (Eds.), Findings of the Asso- mation Processing Systems, volume 35, Curran ciation for Computational Linguistics: ACL 2023, Associates, Inc., 2022, pp. 22199–22213. URL: https: Association for Computational Linguistics, Toronto, //proceedings.neurips.cc/paper_files/paper/2022/file/ Canada, 2023, pp. 8857–8873. doi:10.18653/v1/2023. 8bb0d291acd4acf06ef112099c16f326-Paper-Conference. findings- acl.564 . pdf. [14] S. Robertson, S. Walker, S. Jones, M. M. Hancock- [3] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, Beaulieu, M. Gatford, Okapi at trec-3, in: S. Wang, L. Wang, W. Chen, LoRA: Low-rank adap- Overview of the Third Text REtrieval Confer- tation of large language models, in: International ence (TREC-3), Gaithersburg, MD: NIST, 1995, Conference on Learning Representations, 2022. URL: pp. 109–126. URL: https://www.microsoft.com/en-us/ https://openreview.net/forum?id=nZeVKeeFYf9. research/publication/okapi-at-trec-3/. [4] Y. Luo, Z. Yang, F. Meng, Y. Li, J. Zhou, Y. Zhang, [15] O. Rubin, J. Herzig, J. Berant, Learning to retrieve An empirical study of catastrophic forgetting prompts for in-context learning, in: M. Carpuat, M.-C. in large language models during continual de Marneffe, I. V. Meza Ruiz (Eds.), Proceedings of the fine-tuning, ArXiv abs/2308.08747 (2023). URL: 2022 Conference of the North American Chapter of https://api.semanticscholar.org/CorpusID:261031244. the Association for Computational Linguistics: Hu- [5] L. Berglund, M. Tong, M. Kaufmann, M. Balesni, man Language Technologies, Association for Com- A. C. Stickland, T. Korbak, O. Evans, The rever- putational Linguistics, Seattle, United States, 2022, sal curse: Llms trained on ”a is b” fail to learn ”b pp. 2655–2671. doi:10.18653/v1/2022.naacl- main. is a”, ArXiv abs/2309.12288 (2023). URL: https://api. 191 . semanticscholar.org/CorpusID:262083829. [16] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, [6] Y. Bai, F. Chen, H. Wang, C. Xiong, S. Mei, Trans- R. Majumder, F. Wei, Text Embeddings by Weakly- formers as statisticians: Provable in-context learning Supervised Contrastive Pre-training, CoRR (2022) with in-context algorithm selection, in: A. Oh, arXiv:2212.03533. doi:10.48550/arXiv.2212.03533 . T. Neumann, A. Globerson, K. Saenko, M. Hardt, arXiv:2212.03533 . S. Levine (Eds.), Advances in Neural Information [17] X. Li, K. Lv, H. Yan, T. Lin, W. Zhu, Y. Ni, G. Xie, Processing Systems, volume 36, Curran Asso- X. Wang, X. Qiu, Unified demonstration retriever for ciates, Inc., 2023, pp. 57125–57211. URL: https: in-context learning, in: A. Rogers, J. Boyd-Graber, //proceedings.neurips.cc/paper_files/paper/2023/file/ N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lin- [26] X. Ye, S. Iyer, A. Celikyilmaz, V. Stoyanov, G. Dur- guistics (Volume 1: Long Papers), Association for rett, R. Pasunuru, Complementary explanations Computational Linguistics, Toronto, Canada, 2023, pp. for effective in-context learning, in: A. Rogers, 4644–4668. doi:10.18653/v1/2023.acl- long.256 . J. Boyd-Graber, N. Okazaki (Eds.), Findings of the As- [18] J. Ye, Z. Wu, J. Feng, T. Yu, L. Kong, Compositional ex- sociation for Computational Linguistics: ACL 2023, emplars for in-context learning, in: A. Krause, E. Brun- Association for Computational Linguistics, Toronto, skill, K. Cho, B. Engelhardt, S. Sabato, J. Scarlett (Eds.), Canada, 2023, pp. 4469–4484. doi:10.18653/v1/2023. Proceedings of the 40th International Conference on findings- acl.273 . Machine Learning, volume 202 of Proceedings of Ma- [27] J. Carbonell, J. Goldstein, The use of mmr, diversity- chine Learning Research, PMLR, 2023, pp. 39818–39833. based reranking for reordering documents and pro- URL: https://proceedings.mlr.press/v202/ye23c.html. ducing summaries, in: Proceedings of the 21st Annual [19] L. Wang, N. Yang, F. Wei, Learning to retrieve in- International ACM SIGIR Conference on Research and context examples for large language models, in: Y. Gra- Development in Information Retrieval, SIGIR ’98, As- ham, M. Purver (Eds.), Proceedings of the 18th Confer- sociation for Computing Machinery, New York, NY, ence of the European Chapter of the Association for USA, 1998, p. 335–336. doi:10.1145/290941.291025 . Computational Linguistics (Volume 1: Long Papers), [28] C. X. Zhai, W. W. Cohen, J. Lafferty, Beyond indepen- Association for Computational Linguistics, St. Julian’s, dent relevance: methods and evaluation metrics for Malta, 2024, pp. 1752–1767. subtopic retrieval, in: Proceedings of the 26th Annual [20] J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, W. Chen, International ACM SIGIR Conference on Research and What makes good in-context examples for GPT-3?, Development in Informaion Retrieval, SIGIR ’03, As- in: E. Agirre, M. Apidianaki, I. Vulić (Eds.), Proceed- sociation for Computing Machinery, New York, NY, ings of Deep Learning Inside Out (DeeLIO 2022): The USA, 2003, p. 10–17. doi:10.1145/860435.860440 . 3rd Workshop on Knowledge Extraction and Integra- [29] R. Agrawal, S. Gollapudi, A. Halverson, S. Ieong, Di- tion for Deep Learning Architectures, Association for versifying search results, in: Proceedings of the Sec- Computational Linguistics, Dublin, Ireland and Online, ond ACM International Conference on Web Search 2022, pp. 100–114. doi:10.18653/v1/2022.deelio- 1. and Data Mining, WSDM ’09, Association for Com- 10 . puting Machinery, New York, NY, USA, 2009, p. 5–14. [21] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: URL: https://doi.org/10.1145/1498759.1498766. doi:10. Pre-training of deep bidirectional transformers for 1145/1498759.1498766 . language understanding, in: J. Burstein, C. Doran, [30] S. Gollapudi, A. Sharma, An axiomatic approach for T. Solorio (Eds.), Proceedings of the 2019 Conference result diversification, in: Proceedings of the 18th In- of the North American Chapter of the Association for ternational Conference on World Wide Web, WWW Computational Linguistics: Human Language Tech- ’09, Association for Computing Machinery, New York, nologies, Volume 1 (Long and Short Papers), Asso- NY, USA, 2009, p. 381–390. doi:10.1145/1526709. ciation for Computational Linguistics, Minneapolis, 1526761 . Minnesota, 2019, pp. 4171–4186. doi:10.18653/v1/ [31] M. R. Vieira, H. L. Razente, M. C. N. Barioni, M. Had- N19- 1423 . jieleftheriou, D. Srivastava, C. Traina, V. J. Tsotras, [22] O. Khattab, M. Zaharia, Colbert: Efficient and effective On query result diversification, in: Proceedings of the passage search via contextualized late interaction over 2011 IEEE 27th International Conference on Data Engi- bert, in: Proceedings of the 43rd International ACM neering, ICDE ’11, IEEE Computer Society, USA, 2011, SIGIR Conference on Research and Development in p. 1163–1174. doi:10.1109/ICDE.2011.5767846 . Information Retrieval, SIGIR ’20, Association for Com- [32] C. L. Clarke, N. Craswell, I. Soboroff, Overview of puting Machinery, New York, NY, USA, 2020, p. 39–48. the trec 2009 web track., in: Trec, volume 9, 2009, pp. doi:10.1145/3397271.3401075 . 20–29. [23] A. Beygelzimer, S. Kakade, J. Langford, Cover [33] O. Chapelle, D. Metlzer, Y. Zhang, P. Grinspan, Ex- trees for nearest neighbor, in: Proceedings of the pected reciprocal rank for graded relevance, in: Pro- 23rd International Conference on Machine Learning, ceedings of the 18th ACM Conference on Information ICML ’06, Association for Computing Machinery, New and Knowledge Management, CIKM ’09, Association York, NY, USA, 2006, p. 97–104. doi:10.1145/1143844. for Computing Machinery, New York, NY, USA, 2009, 1143857 . p. 621–630. doi:10.1145/1645953.1646033 . [24] I. Levy, B. Bogin, J. Berant, Diverse demonstrations [34] C. L. Clarke, M. Kolla, O. Vechtomova, An effec- improve in-context compositional generalization, in: tiveness measure for ambiguous and underspecified A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Proceed- queries, in: Proceedings of the 2nd International ings of the 61st Annual Meeting of the Association for Conference on Theory of Information Retrieval: Ad- Computational Linguistics (Volume 1: Long Papers), vances in Information Retrieval Theory, ICTIR ’09, Association for Computational Linguistics, Toronto, Springer-Verlag, Berlin, Heidelberg, 2009, p. 188–199. Canada, 2023, pp. 1401–1422. doi:10.18653/v1/2023. doi:10.1007/978- 3- 642- 04417- 5_17 . acl- long.78 . [35] C. L. Clarke, N. Craswell, I. Soboroff, A. Ashkan, A [25] H. Su, J. Kasai, C. H. Wu, W. Shi, T. Wang, J. Xin, R. Z. comparative analysis of cascade measures for nov- 0037, M. Ostendorf, L. Zettlemoyer, N. A. Smith, T. Y. elty and diversity, in: Proceedings of the Fourth 0009, Selective annotation makes language models bet- ACM International Conference on Web Search and ter few-shot learners, in: The Eleventh International Data Mining, WSDM ’11, Association for Comput- Conference on Learning Representations, ICLR 2023, ing Machinery, New York, NY, USA, 2011, p. 75–84. Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023. doi:10.1145/1935826.1935847 . URL: https://openreview.net/pdf?id=qY1hlv7gwg. [36] C.-N. Ziegler, S. M. McNee, J. A. Konstan, G. Lausen, Improving recommendation lists through topic diver- scale hypertextual web search engine, Computer sification, in: Proceedings of the 14th International Networks and ISDN Systems 30 (1998) 107–117. Conference on World Wide Web, WWW ’05, Associa- URL: https://www.sciencedirect.com/science/article/ tion for Computing Machinery, New York, NY, USA, pii/S016975529800110X, proceedings of the Seventh 2005, p. 22–32. doi:10.1145/1060745.1060754 . International World Wide Web Conference. [37] S. Vargas, P. Castells, Rank and relevance in novelty [48] S. C. Lin, J. Hilton, O. Evans, Truthfulqa: Measuring and diversity metrics for recommender systems, in: how models mimic human falsehoods, in: Annual Proceedings of the Fifth ACM Conference on Recom- Meeting of the Association for Computational Lin- mender Systems, RecSys ’11, Association for Comput- guistics, 2021. URL: https://api.semanticscholar.org/ ing Machinery, New York, NY, USA, 2011, p. 109–116. CorpusID:237532606. doi:10.1145/2043932.2043955 . [49] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, [38] S. Vargas, Novelty and diversity enhancement and S. Ermon, C. Finn, Direct preference optimization: evaluation in recommender systems and information Your language model is secretly a reward model, retrieval, in: Proceedings of the 37th International in: A. Oh, T. Neumann, A. Globerson, K. Saenko, ACM SIGIR Conference on Research & Development M. Hardt, S. Levine (Eds.), Advances in Neural in Information Retrieval, SIGIR ’14, Association for Information Processing Systems, volume 36, Curran Computing Machinery, New York, NY, USA, 2014, p. Associates, Inc., 2023, pp. 53728–53741. URL: https: 1281. doi:10.1145/2600428.2610382 . //proceedings.neurips.cc/paper_files/paper/2023/file/ [39] Y. Zheng, C. Gao, L. Chen, D. Jin, Y. Li, Dgcn: Diver- a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference. sified recommendation with graph convolutional net- pdf. works, in: Proceedings of the Web Conference 2021, [50] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bam- WWW ’21, Association for Computing Machinery, ford, D. Singh Chaplot, D. de las Casas, F. Bressand, New York, NY, USA, 2021, p. 401–412. doi:10.1145/ G. Lengyel, G. Lample, L. Saulnier, L. Renard Lavaud, 3442381.3449835 . M.-A. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, [40] F. Radlinski, R. Kleinberg, T. Joachims, Learning di- T. Lacroix, W. El Sayed, Mistral 7B, arXiv e-prints verse rankings with multi-armed bandits, in: Pro- (2023) arXiv:2310.06825. doi:10.48550/arXiv.2310. ceedings of the 25th International Conference on Ma- 06825 . arXiv:2310.06825 . chine Learning, ICML ’08, Association for Comput- [51] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, ing Machinery, New York, NY, USA, 2008, p. 784–791. M. Bansal, C. A. Raffel, Few-shot parameter-efficient doi:10.1145/1390156.1390255 . fine-tuning is better and cheaper than in-context [41] Y. Zhu, Y. Lan, J. Guo, X. Cheng, S. Niu, Learning learning, in: S. Koyejo, S. Mohamed, A. Agarwal, for search result diversification, in: Proceedings of D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural the 37th International ACM SIGIR Conference on Re- Information Processing Systems, volume 35, Curran search & Development in Information Retrieval, SIGIR Associates, Inc., 2022, pp. 1950–1965. URL: https: ’14, Association for Computing Machinery, New York, //proceedings.neurips.cc/paper_files/paper/2022/file/ NY, USA, 2014, p. 293–302. doi:10.1145/2600428. 0cde695b83bd186c1fd456302888454c-Paper-Conference. 2609634 . pdf. [42] T. Desautels, A. Krause, J. W. Burdick, Parallelizing [52] L. Li, C.-Y. Chan, Efficient indexing for diverse query exploration-exploitation tradeoffs in gaussian process results, Proc. VLDB Endow. 6 (2013) 745–756. doi:10. bandit optimization, Journal of Machine Learning 14778/2536360.2536373 . Research 15 (2014) 4053–4103. URL: http://jmlr.org/ papers/v15/desautels14a.html. [43] E. Contal, D. Buffoni, A. Robicquet, N. Vayatis, Parallel gaussian process optimization with upper confidence bound and pure exploration, in: H. Blockeel, K. Kerst- ing, S. Nijssen, F. Železný (Eds.), Machine Learning and Knowledge Discovery in Databases, Springer Berlin Heidelberg, Berlin, Heidelberg, 2013, pp. 225–240. [44] A. Ntoulas, M. Najork, M. Manasse, D. Fetterly, De- tecting spam web pages through content analysis, in: Proceedings of the 15th International Conference on World Wide Web, WWW ’06, Association for Com- puting Machinery, New York, NY, USA, 2006, p. 83–92. doi:10.1145/1135777.1135794 . [45] G. V. Cormack, M. D. Smucker, C. L. A. Clarke, Ef- ficient and effective spam filtering and re-ranking for large web datasets, Inf. Retr. 14 (2011) 441–465. doi:10.1007/s10791- 011- 9162- z . [46] M. Bendersky, W. B. Croft, Y. Diao, Quality-biased ranking of web documents, in: Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM ’11, Association for Comput- ing Machinery, New York, NY, USA, 2011, p. 95–104. doi:10.1145/1935826.1935849 . [47] S. Brin, L. Page, The anatomy of a large-