=Paper=
{{Paper
|id=Vol-3752/paper6
|storemode=property
|title=Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework
|pdfUrl=https://ceur-ws.org/Vol-3752/paper6.pdf
|volume=Vol-3752
|authors=Zackary Rackauckas,Arthur Câmara,Jakub Zavrel
|dblpUrl=https://dblp.org/rec/conf/llm4eval/RackauckasCZ24
}}
==Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework==
https://ceur-ws.org/Vol-3752/paper6.pdf
Evaluating RAG-Fusion with RAGElo : an Automated
Elo-based Framework
Zackary Rackauckas1 , Arthur Câmara2 and Jakub Zavrel2
1
Columbia University, New York, NY, USA
2
Zeta Alpha, Amsterdam, The Netherlands
Abstract
Challenges in the automated evaluation of Retrieval-Augmented Generation (RAG) Question-
answering (QA) systems include hallucination problems in domain-specific knowledge and the
lack of gold standard benchmarks for company-internal tasks. This results in difficulties in
evaluating RAG variations, like RAG -Fusion (RAGF ) in the context of a product QA task at Infineon
Technologies. To solve these problems, we propose a comprehensive evaluation framework,
which leverages Large Language Models (LLMs) to generate large datasets of synthetic queries
based on real user queries and in-domain documents, uses LLM-as-a-judge to rate retrieved
documents and answers, evaluates the quality of answers, and ranks different variants of
Retrieval-Augmented Generation (RAG ) agents with RAGElo ’s automated Elo-based competition.
LLM-as-a-judge rating of a random sample of synthetic queries shows a moderate, positive
correlation with domain expert scoring in relevance, accuracy, completeness, and precision.
While RAGF outperformed RAG in Elo score, a significance analysis against expert annotations
also shows that RAGF significantly outperforms RAG in completeness, but underperforms in
precision. In addition, Infineon’s RAGF assistant demonstrated slightly higher performance in
document relevance based on MRR@5 scores. We find that RAGElo positively aligns with the
preferences of human annotators, though due caution is still required. Finally, RAGF ’s approach
leads to more complete answers based on expert annotations and better answers overall based
on RAGElo ’s evaluation criteria.
Keywords
Retrieval-augmented generation, Elo-based evaluation, LLM-as-a-judge, RAG-Fusion
1. Introduction
The text-generating capabilities of LLMs, together with their text understanding abilities,
have allowed conversational Question-Answering (QA) systems to experience a consider-
able leap in performance, with near-human text quality and reasoning capabilities [1].
However, these systems can be prone to hallucinations [2, 3], as they sometimes produce
seemingly plausible but factually incorrect answers.
The general inability of such models to identify unanswerable questions [4, 5] can
exacerbate hallucinations, especially in enterprise settings. In such scenarios, user
questions may require specific domain knowledge to be answered properly. This knowledge
LLM4Eval 2024:The First Workshop on Large Language Models for Evaluation in Information Retrieval,
18 July 2024, Washington, DC
Envelope-Open zcr2105@columbia.edu (Z. Rackauckas); camara@zeta-alpha.com (A. Câmara); zavrel@zeta-alpha.com
(J. Zavrel)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
is usually out-of-domain for most LLMs, but is present in private and confidential internal
documents from the company.
One such company is Infineon, a leading manufacturer of semiconductors. Given its
wide range of equipment, information about its products is spread across multiple, highly
technical documents, including datasheets and selection guides of hundreds of pages.
Therefore, an internal retrieval augmented conversational QA system was developed by
Infineon for internal users such as account managers, field application engineers, and sales
operations specialists. This system allows professionals to ask questions about products
from the whole catalog while in the field.
One of the features of Infineon’s conversational agent is the usage of RAG -Fusion (RAGF ),
a technique for increasing the quality of the generated answers by generating variations
of the user question and combining the rankings produced by these variations using
rank-fusion methods (i.e., recriprocal rank fusion (RRF) [6]) into a ranking that has both
more diverse and higher quality answers.
However, evaluating these systems bring complications common to retrieval augmented
agents, especially in enterprise settings, stemming from the lack of comprehensive test
datasets. Ideally, such a test set would comprise a large set of real user questions from
a query log, paired with “golden answers” provided by experts. The lack of such a
test set leads to two main issues. First, evaluation of answers generated by LLMs by
traditional n-gram evaluation metrics such as ROUGE [7], BLEU [8], and METEOR [9]
is not possible, given the lack of ground truth answers. Second, and as a consequence,
evaluating the quality of the answers generated by the LLM systems would require
in-domain experts (potentially from within the company) in a process that is both slow
and costly [10].
One approach for tackling the lack of an extensive test set is to use synthetic queries
generated by LLMs as a proxy of user queries [11]. However, the lack of in-domain
knowledge of LLMs makes queries naively generated by these models unreliable and
prone to hallucinations, especially when generating queries about specific products and
their specifications (c.f., Table 1 for examples of real user’s questions submitted to the
system).
To solve this, we propose to use a process similar to InPars [12] to create a set of
synthetic evaluation queries. We ask LLMs to generate queries based on portions of
existing documentation injected into the prompt. To increase similarity to real user
queries, we include existing user questions as few-shot examples to the prompt. With
this process, we are able to generate a large set of high-quality synthetic queries for
evaluating our systems. Figure 2 describes the process of generating synthetic queries
and the output of a search agent. Table 2 shows a sample of these queries.
To tackle the second issue, a lack of ground truth “golden answers,” we leverage an
LLM-as-a-judge process, where a strong LLM is used to evaluate the quality of the
answers generated by the RAG agent’s LLM [13]. We then follow the practice of judging
generated answers in a pairwise fashion [14], prompting the judge LLM to select the
better answer between two candidates generated by different RAG pipelines. (c.f. Section
6 with details of our pipelines).
Finally, to mitigate the lack of in-domain knowledge of the judging LLM, we also
annotate the relevance of the documents retrieved by the pipelines being evaluated and
inject the relevant documents in the context used by the judging LLM. This allows the
judging LLM to better assess for hallucinations and completeness and better align the
quality of the evaluations to those conducted by experts.
This process is mediated by RAGElo 1 , a toolkit for evaluating RAG systems inspired
by the Elo ranking system. RAGElo provides an easy-to-use CLI and Python library
for using LLMs to evaluate retrieval results and answers produced by RAG pipelines.
By combining a retrieval evaluator, a pairwise answer annotator, and an Elo-inspired
tournament, RAGElo leverages powerful LLMs to agnostically annotate and rank different
RAG pipelines. We notice that, although noisy, the LLM annotations generated by
RAGElo are generally well aligned with experts’ judgments of relative system quality,
allowing for fast experimentation and comparisons between different RAG implementations
without the frequent intervention of experts as annotators.
This paper evaluates multiple implementations of Infineon’s retrieval augmented con-
versational agent using RAGElo : a traditional Retrieval-Augmented Generation and a
RAG -Fusion implementation. RAG -Fusion generates multiple variations of the user question
and combines the rankings produced by these queries into a more diverse set of documents.
The documents are then fed into the LLM. We also analyze these same agents under
a keyword-based retrieval regimen (i.e., the retriever uses BM25 to retrieve and rank
documents), a dense retriever, and a hybrid retriever that combines the ranking generated
by the BM25 and the dense retrievers using RRF. Our goal is to answer the following
questions:
• Does the evaluation framework proposed by RAGElo align with the preferences of
human annotators for answers generated by RAG -based conversational agents?
• Does the RAGF approach of submitting multiple variations of the user question and
combining their rankings lead to better answers?
Table 1
Sample of questions submitted by users to the Infineon RAG-Fusion system
User-submitted queries
What is the country of origin of IM72D128, and how does geopolitical exposure affect the market and
my SAM for the microphone?
What is the IP rating of mounted IM72D128?
Tell me microphones that have been released since January 2023 based on the datasheet revision
history.
We need to confirm whether the IFX waterproof MIC has a sleeping mode and wake-up functions.
Table 2
Sample of synthetic queries for evaluating Infineon’s RAG assistant. GPT4 refers to OpenAI’s gpt-
4-turbo-2024-04-09 model. Opus, Sonnet and Haiku refer to Anthropic’s Claude 3 models opus-
20240229 , sonnet-20240229 and haiku-20240307 , respectively.
model Query
GPT4 What are some typical consumer applications for TLV496x-xTA/B sensors?
GPT4 What specific ISO 26262 readiness is available for the KP253 sensor?
Opus How small of a form factor can I achieve for a battery-powered air quality device using
Infineon’s PAS CO2 sensor?
Sonnet Can Infineon’s sensors support bus configurations or daisy-chaining for simplified wiring and
reduced complexity in IoT systems?
Haiku Which TLE4971 current sensor models are available in the TISON-8-6 package?
To cross-sell a MEMS microphone and a
How to cross-sell a MEMS microphone and a XENSIV XENSIV sensor to customers, you can follow
sensor to customers? these steps:
1. Identify customer's requirements (…)
Original query
Dense KNN search Top-k retrieved
Agent generated answer
documents
Traditional RAG Agent
What are the key features of Infineon's MEMS
microphones and XENSIV sensors (…)
Here are the key features, benefits, and
How can Infineon’s MEMS microphones and XENSIV applications of Infineon's MEMS microphone
sensors be integrated for enhanced audio (…) and XENSIV sensor products, along with
successful cross-selling strategies: (…)
What are the most suitable applications and
industries for Infineon's MEMS microphones (…) Dense KNN search Fused top-k
Agent generated answer
retrieved documents
Query variations generated by agent (RRF)
Per query top-k
retrieved documents
RAG Fusion Agent
Figure 1: A traditional Retrieval-Augmented Generation pipeline compared to a RAG -Fusion pipeline.
While a traditional RAG agent submits only the original query to the search system, a RAGF agent first
generates variations of the user query and combines the rankings induced by these queries into a final
ranking using RRF. The resulting top-k passages are fed into the LLM for generating the answer to the
user’s query.
2. Related Work
Several evaluation systems for RAG have been proposed to address flaws in current
evaluation methods. For instance, Facts as a Function (FaaF) [15] is an end-to-end
factual evaluation algorithm specially created for RAG pipelines. By creating functions
from ground truth facts, FaaF focuses on the quality of generation and retrieval by
calling LLMs. FaaF has substantially increased efficiency and cost-effectiveness, achieving
reduced error rates compared to traditional evaluation methods. The reliance on a set of
ground truths does not meet our goal of applying an automated evaluation toolkit to
our pipelines. Recently, researchers have moved to eliminate the need for ground truths.
This is especially important when automatically evaluating agents that retrieve highly
technical documents from a large database, such as the Infineon RAGF conversational
agent. RAGElo eliminates this reliance by using an LLM-as-a-judge, a method studied in
1
https://github.com/zetaalphavector/ragelo
numerous recent works.
SelfCheckGPT demonstrates the ability to leverage LLMs to detect and rank factuality
with zero resources [16]. In addition, it has been demonstrated that GPT3.5 Turbo
outperforms ground truth baselines in fact-checking with a "1/2-shot" method [17]. A
model built to classify statements as true or false based on the activations of an LLM’s
hidden layers had up to 83% classification accuracy [18]. This evidence supports RAGElo ’s
usage of LLM-as-a-judge.
Automated evaluation metrics can also be applied to RAG-based agents. BARTScore,
an automated metric based on the BART architecture, has also outperformed most
metrics on categories including factuality [19, 20]. Besides automated evaluation metrics,
several automated evaluation frameworks have been created with a similar goal to RAGElo .
Focusing on faithfulness, answer relevance, and content relevance, RAGAS leverages
LLM prompting to focus on situations where ground truths and human annotations are
not present in a dataset [21]. Prediction-powered inference aims to decrease the number
of human annotations needed for machine learning prediction on a dataset of images
of galaxies with approximately 300,000 annotations [22]. The ARES toolkit leverages
prediction-powered inference to evaluate RAG systems with fewer human annotations.
Like RAGElo , ARES automatically evaluates RAG systems using synthetically generated
data [23].
ARAGOG highlights Hypothetical Document Embedding (HyDE) and LLM reranking
as effective methods for enhancing retrieval precision while also exploring the effectiveness
of Sentence Window Retrieval and the potential of the Document Summary Index in
improving RAG systems [24].
While the aforementioned frameworks evaluate answers on relevance, faithfulness, and
correctness metrics, RAG can also be evaluated on noise and counterfactual robustness,
negative rejection, and information integration [25].
In addition to answers, frameworks have also been created to evaluate documents.
Corrective Retrieval Augmented Generation (CRAG) builds on RAG by employing a
retrieval evaluator to ensure that only the optimal documents are fed into the LLM
prompt prior to the answer generation phase [26].
Due to its Elo-based ranking system for answers, its use of LLM-as-a-judge, and its
relevance evaluation of the intermediate retrieval steps in a RAG pipeline, RAGElo is a
unique evaluation toolkit. In this study, we use it to compare a simple RAG versus a more
sophisticated RAGF system on a knowledge-intensive industry-specific domain.
3. Retrieval Augmented QA with rank fusion
While answers generated by traditional retrieval augmented systems are based on a
number of documents retrieved from a single query, RAGF introduces additional variation
into the retrieval process. Upon receiving a query from the user, a RAGF agent leverages
a large language model to generate a set of queries based on the original [27]. Table 3
shows examples of queries generated by the agent based on the query, “How to cross-sell
a MEMS microphone and a XENSIV sensor to customers?”.
Table 3
Queries Generated from “How to cross-sell a MEMS microphone and a XENSIV sensor to customers?”
LLM-Generated Query
What are the key features of Infineon’s MEMS microphones and XENSIV sensors that can be
highlighted while cross-selling?
How can Infineon’s MEMS microphones and XENSIV sensors be integrated for enhanced audio and
motion sensing capabilities in various applications?
What are the most suitable applications and industries for Infineon’s MEMS microphones and XENSIV
sensors to maximize cross-selling potential?
After generating the variations for the user query, the RAGF agent submits the original
and the generated queries to a retrieval system [28] that returns the top-𝑘 relevant
documents 𝑑, 𝑑1 , 𝑑2 , … 𝑑𝑘 from the set of all documents 𝐷 for each query. The rankings
induced by these queries are then combined using recriprocal rank fusion (RRF) [6] into
a final, higher-quality set of passages. The intuition behind RAGF is that submitting
variations of the same query and combining the final rankings increases the likelihood of
relevant passages being injected into the LLM prompt. In contrast, non-relevant passages
retrieved by a single query are discarded. Figure 1 describes how RAG and RAGF differ.
1
𝑅𝑅𝐹 𝑆𝑐𝑜𝑟𝑒(𝑑 ∈ 𝐷) = ∑ . (1)
𝑟∈𝑅 𝑟(𝑑) +𝑘
4. Development of a synthetic test set
claude-3-opus-20240229
claude-3-haiku-20240307
Documents
(passages) claude-3-sonnet-20240229
gpt-4-turbo-2024-04-09
Query generators
Expert questions
(few-shot)
What security features does the OPTIGA
Trust M provide for IoT devices (…)
Which TLE4971 current sensor models are
available in the TISON-8-6 package?
For applications with fast switching
technologies like SiC, which (…)
Are there any self-diagnosis features
available in the KP23x analog sensor?
Pooled queries
Figure 2: Process for creating synthetic queries. We prompt multiple LLMs to generate queries based
on existing documents. We include some existing user queries in the prompt as few-shot examples.
As previously discussed, one of the main issues when evaluating the quality of a QA
system in an enterprise setting is that, frequently, companies do not have a large enough
existing collection of queries to evaluate such systems’ quality. Therefore, in this work,
we propose to adopt a strategy previously used by methods for generating synthetic
queries for training retrieval systems, such as InPars [12] and Promptagator [29].
Similar to these approaches, we randomly sample passages from documents within
our collection and prompt an LLM to generate questions that users may ask about
these portions. However, one difference in our approach to generating training queries
is the size of these passages. When generating queries for training a retrieval system,
we ideally want to keep the passages short to fit in the dense encoder’s relatively short
context windows. However, when generating queries for evaluating QA systems (including
retrieval augmented), we are not bound to the limit of the embedding model used for
retrieval. Rather, a longer passage may yield questions that require multiple shorter
passages to be answered. Therefore, we submit relatively long passages to the LLMs.
Specifically, each passage is extracted from up to ten pages of PDF documents (about
2000 tokens 2 )
To keep the questions generated as diverse as possible, we prompt four different LLMs
to generate up to ten questions based on the same documents. Our test set collection
contains a mix of queries generated by OpenAI’s GPT-4 turbo [30] and Anthropic’s
Claude-3 [31] Opus, Sonnet, and Haiku models 3 . From a set of 𝑁 = 840 queries, we
sampled 200 queries across all four models. Half of the queries are selected from GPT-4
generated queries, and the other half from Claude 3 queries. Among the Claude 3 queries,
to ensure the quality of the queries and their diversity, we again sample according to
each model size. Ultimately, our test set contains 100 queries from GPT-4-turbo, 50 from
Claude 3 Opus, 30 from Sonnet, and 20 from Haiku.
Finally, to increase the quality of the generated queries, We asked for an account
manager, a sales operations specialist, a marketing representative, and a business devel-
opment manager to create queries that they would submit to the conversational agent
from the perspective of their role. They were instructed to produce queries regarding
products from the XENSIV sensor product line, consisting of MEM microphones, radar,
current, magnetic, pressure, and environmental sensors. We compiled a list of 23 of these
queries to use as a base for experimentation and used them as few-shot examples in
the query generation prompt. Figure 2 illustrates our method for generating synthetic
queries based on existing user queries and document passages.
5. LLM-as-a-Judge for RAG pipelines
Even with a suitable set of synthetic questions for evaluating our RAG conversational
agent, assessing whether a given answer properly answers a question is not trivially done.
If a ground-truth “golden answer” is available, one can use traditional syntactic-based
2
all LLMs used in our experiments had long context windows of 128k or 200k tokens.
3
We did not use GPT-3.5 or open source models due to their shorter context window at the time of
writing.
Agent A answer
Agent B answer
Agent A: Yes, the KP23x sensor has Both assistants correctly answer
some self-diagnosis features.
the question (…)
according to document , the
Agent B:The KP23x analog sensor series sensors have features such as (…)
includes self-diagnosis capabilities, Based on these observations,
Are there any self-diagnosis such as built-in monitoring
features available in the Assistant B provides a more
functionality and ISO 26262 (…)
KP23x analog sensor? accurate, relevant, and focused
response to the user's question
regarding the KP23x series sensors.
Final Verdict: [[B]]
RAGElo Retrieval Evaluator RAGElo Retrieval Evaluator
Retrieved
documents and Pointwise retrieved Pairwise answer
Synthetic test query Search agents generated answers document evaluation judgement FInal evaluation
Figure 3: The RAGElo evaluation pipeline. First, documents retrieved by the agents are evaluated
pointwise according to their relevance to the user’s question. Then, the agents’ answers are evaluated
pairwise, using the retrieved relevant documents from both agents as reference.
metrics such as BLEU, METEOR or ROUGE [8, 9, 7]. Without such reference answers,
one would require human annotators with a considerable understanding of the question’s
topic to manually assess the quality of the answers produced by each system. However,
this is a costly process.
Alternatively, several LLM-as-a-Judge methods have been proposed, where another
LLM is asked to evaluate the quality of answers generated by other LLMs. Nevertheless,
in an enterprise setting, the answers usually require the LLM to access knowledge not
present in their training datasets but rather contained in documents internal to the
company. This is usually accomplished using a RAG pipeline like the one described above.
Therefore, the judging LLM also needs access to similar knowledge to accurately evaluate
the agent’s answers’ quality.
Therefore, in this work, we rely on RAGElo , an open-source RAG evaluation toolkit that
evaluates the answers generated by each agent and the documents retrieved by them. By
injecting the annotation of retrieved documents, pooled by the agents being evaluated,
on the answer evaluation step, this method allows for the judging LLM to evaluate if
the generated answer was able to use all the information available about the question
properly and to check for any hallucinations. As the documents used for generating the
answers are included in the answer evaluating prompt, an agent that incorrectly cites
information from a source or refers to information not present in these documents is
likely hallucinating and should have its evaluation adjusted accordingly. As we explore
in Section 8, this two-step process results in a high correlation between human expert
annotators and the judging LLM, enabling higher reliability and trust when evaluating
different RAG pipelines. This process is also illustrated in Figure 3.
5.1. Evaluation aspects
While our main evaluation focuses on the pairwise comparison between the two agents,
RAGElo also allows us to evaluate answers pointwise. In this setting, similar to other
works [32], we prompt the judging LLM to evaluate the answers according to multiple
criteria:
• Relevance: Does the answer address the user’s question?
• Accuracy: Is the answer factually correct, based on the documents provided?
• Completeness: Does the answer provide all the information needed to answer the
user’s question?
• Precision: If the user’s question is about a specific product, does the answer provide
the answer for that specific product?
6. Retrieval pipelines
We not only experiment with different search agents (i.e., RAG and RAGF . We are also
interested in how different retrieval methods may impact the quality of the final answers
generated by these agents.
6.1. Retrieval methods
Our corpus consists of passages extracted from the Infineon XENSIV Product Selection
Guide, a 117-page document with detailed information on every product in the XENSIV
family. This document included technical information about all Infineon XENSIV sensors,
consumer and automotive sensor applications, guidance in selecting the correct sensor,
and other comprehensive and detailed information about the product line.
The passages are embedded using multilingual-e5-base [33] 4 and indexed using
OpenSearch, allowing us to perform both KNN-based vector search, keyword-based
search with BM25 [34], and RRF based hybrids thereof.
6.2. QA Systems Implementation
We mainly evaluate two agents: a naive RAG pipeline, where the agent first retrieves top-𝑘
passages that are then templated into a prompt, and the Infineon RAG -Fusion (RAGF )
agent. Upon receiving a query, a naive RAG agent takes the following actions:
1. Retrieve the top k most relevant passages from the search system.
2. Perform a Chat Completions API call, prompting the LLM with instructions for
generating an answer based on the five relevant passages.
3. Process and output the Chat Completions response.
Meanwhile, the Infineon RAGF conversational assistant uses a similar framework and
performs the following steps upon receiving a query:
1. Perform a Chat Completions API call to generate four new queries based on the
original query using a prompt tailored to the agent’s original goal.
2. Retrieve the top k most relevant passages for each query.
3. Using RRF, combines the top-𝑘 passages induced by all queries into a final ranking.
4. Perform a Chat Completions API call prompting the LLM with carefully worded
instructions for generating an answer based on the top-𝑘 fused passages
5. Process and output the Chat Completions response.
4
https://huggingface.co/intfloat/multilingual-e5-base
7. Experiments
7.1. Comparing LLM-as-a-judge to expert annotators
While LLM-as-a-judge is a theoretically viable algorithm for rating RAG and RAGF answers,
we must establish whether the results agree with the annotations of domain experts.
Figure 4 provides a Bland-Altman plot to visually represent the LLM and human
judgments’ agreement.
Figure 4: Bland-Altman plot to visualize the comparison between LLM-as-a-judge and expert answers.
The bias of approximately 0.12 indicates that, on average, LLM scores were slightly
higher than human scores. The limits of agreement ranged from approximately -1.17 to
1.41. demonstrating substantial variability in the difference between LLM and human
evaluators.
Next, we compared LLM-as-a-judge to expert annotators with Kendall’s 𝜏. Kendalls
𝜏 is a nonparametric measure that quantifies the degree of association between two
monotonic continuous or ordinal variables by calculating the proportion of concordance
and discordance among pairwise ranks, offering valuable insight into their rank correlation
[35, 36]. We used the SciPy Stats Kendalltau function to calculate a tau-b score and a
p-value for the combined ratings of all columns, flattened into a 1-D array with RAG and
RAGF ratings combined [37]. The tau-b value, a nonparametric measure of association, is
calculated using the following formula [38]:
(𝑃 − 𝑄)
𝜏𝑏 = (2)
√(𝑃 + 𝑄 + 𝑇 ) ⋅ (𝑃 + 𝑄 + 𝑈 )
P represents the number of concordant pairs, Q represents the number of discordant
pairs, T represents the number of ties exclusive to x, and U represents the number of ties
exclusive to y.
This test returned 𝜏 ≈ 0.56, indicating a moderate, positive correlation [39] with a
p-value against a null hypothesis of no association of 𝑝 < 0.01 (99.99% confidence level).
For comparison, in similar experiments judging human versus LLM judgments, Faggioli
et al. found 𝜏 values of 𝜏 = 0.76 and 𝜏 = 0.86[40].
Following the same methodology, we also calculated Spearman’s 𝜌, a similar nonpara-
metric correlation measure. This resulted in 𝜌 ≈ 0.59 with 𝑝 < 0.01, demonstrating a
statistically significant, moderate positive correlation [36].
7.2. RAG vs RAGF
7.2.1. Quality of retrieved documents
We assessed document retrieval quality using Mean Reciprocal Rank@5 (MRR@5), which
averages the inverse ranks of the first relevant result within the top five positions across
all queries. The formula is given by
|𝑄|
1 1
𝑀𝑅𝑅@5 = ∑ , (3)
|𝑄| 𝑖=1 rank𝑖
where |𝑄| is the total number of queries and rank𝑖 is considered only if it’s within the top
five, otherwise it counts as zero [41].
MRR@5 scores were calculated for each agent and each retrieval method considering
two categories:
1. MRR@5 score for documents deemed “somewhat relevant” or “very relevant.”
2. MRR@5 score for documents deemed “very relevant.”
The results can be seen below in Table 4.
Table 4
Mean MRR@5 scores for RAG vs RAG-F. The retrieval method columns indicate if the retrieval component
used was vector search only (KNN), keywords only (BM25) or hybrid (KNN and BM25, combined with
RRF).
Agent Retrieval Method Very Relevant Somewhat Relevant
RAG KNN 0.407 0.828
RAG BM25 0.821 0.955
RAG Hybrid 0.746 0.949
RAG-F KNN 0.396 0.810
RAG-F BM25 0.855 0.970
RAG-F Hybrid 0.758 0.961
7.2.2. Pairwise evaluation of answers
We then ran RAGElo games to evaluate end-to-end answer quality of RAG vs RAGF with
different base retriever configurations a task that cannot rely on standard Information
Retrieval metrics. These RAGElo results show more victories for RAGF than RAG ; For
example, when using BM25 as a base retriever, RAG F won 49% of the games, RAG won
14.5%, and RAG and RAGF are tied in 36.5% of the times. The resulting Elo scores for all six
variants are shown in table 6, which give a robust ranking of the systems, without reliance
on a gold standard. It is interesting to see, that for both RAGF as well as RAG , BM25 is a
strong baseline that is not surpassed by generic embeddings in these experiments.
Next, we compared the RAGElo outcome to the preference of our Infineon human
annotator. We performed two-tailed paired t-tests to compare RAG against RAGF on
each category from the Infineon representatives’ human evaluations with 𝛼 = .05. As
expected, due to its larger variety of retrieved results, RAGF significantly outperforms RAG
in completeness at the 95% confidence level with 𝑝 ≈ 0.01 However, on the precision of
answers, RAG significantly outperformed RAGF at the 95% confidence level with 𝑝 ≈ 0.04.
Table 5
RAG vs RAGF Win percentage between pairwise comparison of the agent’s answers using GPT-4o as a
judge with RAGElo .
BM25 KNN Hybrid
Agent AVG
RAG RAGF RAG RAGF RAG RAGF
RAG — 14.5% 49.5% 52.5% 29.0% 28.5% 34.8%
BM25
RAGF 49.0% — 58.5% 51.5% 53.5% 30.5% 48.6%
RAG 33.0% 27.0% — 20.0% 26.0% 31.0% 27.4%
KNN
RAGF 34.5% 30.0% 37.0% — 30.5% 32.0% 32.8%
RAG 41.5% 21.0% 51.5% 48.0% — 20.5% 36.0%
Hybrid
RAGF 46.0% 35.0% 49.0% 45.5% 43.5% — 44.3%
Table 6
Elo Ranking for all agents averaged over 500 tournaments.
Agent Retrieval Elo score
RAGF BM25 571.0
RAGF Hybrid 550.0
RAG Hybrid 497.0
RAG BM25 487.0
RAGF KNN 470.0
RAG KNN 436.0
8. Discussion
As observed above, we found statistically significant, moderate positive correlations
between LLM ratings and human annotations. This indicates a consistent association
between the ratings from LLM-as-a-judge and those by Infineon experts. We find that
on average, LLM scores are slightly higher than those of human annotators. This means
that while relevance judgements on individual queries should not be fully reliable, and
IR metrics derived from LLM-as-a-judge should not be equated with regular relevance
scores without further calibration, we can still make good use of this approach to rank-
order systems. These findings collectively support the validity of our LLM evaluation
method, which assesses conversational system outputs based on a combination of relevance,
accuracy, completeness, and recall.
The style of evaluation and the different dimensions it takes into account are specified
in the prompts given to the LLM in the RAGElo evaluation, which are provided in
Appendix A. Specifically, while the initial LLM-as-a-judge is given specific criteria to
focus on only four categories, we instructed RAGElo ’s impartial judge LLM to value more
than the initial four categories:
Your evaluation should consider factors such as comprehensiveness, correct-
ness, helpfulness, completeness, accuracy, depth, and level of detail of their
responses.
Since RAGF significantly outperformed RAG in the completeness category, the RAGElo
judge LLM likely weighed completeness higher than precision. In addition, based
on manual observation of a small random sample of answers, RAGF produced more
comprehensive answers and featured higher depth and level of detail due to the multiple
query generation. However, games where RAG won were most likely influenced by a
significantly more precise answer than that of RAGF . While RAGF values comprehensive
answers that offer multiple perspectives to the user, RAG produces shorter answers
that answer the original query only. Since completeness is defined as the extent to
which a user’s question was answered, it can be presumed that RAGF ’s longer and more
comprehensive answers may tend to be more complete. And since precision relates to the
agent mentioning the correct product or product family, it can be presumed that RAGF ’s
longer answers have more room to consider other products or product families, leading
to reduced answer precision. While the human annotation was done by Infineon experts,
different humans may rate answers differently, even if following the same set of criteria.
A larger number of documents or a database of non-technical documents may have led
to a different outcome. RAGF can be applied to not only Infineon documents but also any
documents database to retrieve. This includes not only enterprise uses but also uses in
education, such as mathematics and language learning. The algorithm can be tuned to
different use cases by tweaking the internal LLM prompt. For example, the Infineon RAGF
bot was prompted to "think like an engineer." However, an educator RAGF bot could be
prompted to "think like a teacher." Future work includes exploring other applications of
RAGF , especially in education. In addition, we will experiment with different prompts for
both LLM-as-a-judge and RAGElo while using different quantities and types of documents
with the same retrieval algorithms.
Based on the calculated MMR@5 scores, we found that the RAGF agent mostly outper-
forms the RAG agent in ranking both highly relevant and somewhat relevant documents
retrieved. This evidence search on multiple query variants produced, on average, slightly
more higher-ranked relevant documents than using only the original user query. We also
see that using vector search with embeddings is not a silver bullet, as for our test queries,
BM25 seriously outperforms it. Since retrieval quality is highly dependent on the quality
of the embeddings and their fit to the domain, this outcome will likely be changed by
fine-tuning the embeddings, and adding additional intelligent re-rankers, which we leave
here for future work, as the evaluation framework would remain the same.
9. Conclusion
Overall, we found that the evaluation framework proposed by RAGElo positively aligns
with the preferences of human annotators for RAG and RAGF with due caution due to a
moderate correlation and variability of scoring. We found that the RAGF approach leads to
better answers most of the time, according to the RAGElo evaluation. According to expert
scoring, the RAGF approach significantly outperforms in completeness compared to RAG
but significantly underperforms in precision compared to RAG. Based on these results,
we cannot confidently assert that RAGF ’s approach leads to better answers generally.
However, the results do support that RAGF ’s approach leads to more complete answers
and a higher proportion of better answers under evaluation by RAGElo .
Since RAGElo is generally applicable to all retrieval-augmented algorithms, in future
work, we also intend to test different agents other than RAG and RAGF , including those
with different reranking algorithms, different embedding models, and different LLMs. In
addition, due to RAGF ’s underperformance in document relevance, we may also leverage
CRAG to reduce this gap. We will also investigate the reflection of human sensitivity in
expert ratings, especially whether the LLMs should or can reflect human sensitivities.
Acknowledgments
We thank Brooks Felton from Infineon for his support during this work. We also thank
the Infineon sales team for providing valuable feedback.
References
[1] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Nee-
lakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger,
T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse,
M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish,
A. Radford, I. Sutskever, D. Amodei, Language Models are Few-Shot Learners, 2020.
doi:10.48550/arXiv.2005.14165 . arXiv:2005.14165 .
[2] Y. Xiao, W. Y. Wang, On Hallucination and Predictive Uncertainty in Conditional
Language Generation, in: Proceedings of the 16th Conference of the European
Chapter of the Association for Computational Linguistics: Main Volume, Association
for Computational Linguistics, Online, 2021, pp. 2734–2744. doi:10.18653/v1/2021.
eacl- main.236 .
[3] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto,
P. Fung, Survey of Hallucination in Natural Language Generation, ACM Computing
Surveys 55 (2023) 248:1–248:38. doi:10.1145/3571730 .
[4] Z. Yin, Q. Sun, Q. Guo, J. Wu, X. Qiu, X.-J. Huang, Do large language models
know what they dont know?, in: Findings of the Association for Computational
Linguistics: ACL 2023, 2023, pp. 8653–8665.
[5] A. Amayuelas, L. Pan, W. Chen, W. Wang, Knowledge of knowledge: Exploring
known-unknowns uncertainty with large language models, 2023. URL: https://arxiv.
org/abs/2305.13712.
[6] G. V. Cormack, C. L. A. Clarke, S. Buettcher, Reciprocal rank fusion outperforms
condorcet and individual rank learning methods, in: SIGIR 2019, SIGIR ’09,
Association for Computing Machinery, New York, NY, USA, 2009, p. 758759. URL:
https://doi.org/10.1145/1571941.1572114. doi:10.1145/1571941.1572114 .
[7] C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text
Summarization Branches Out, Association for Computational Linguistics, Barcelona,
Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013.
[8] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic
evaluation of machine translation, in: ACL 2002, ACL ’02, Association for Computa-
tional Linguistics, USA, 2002, p. 311318. URL: https://doi.org/10.3115/1073083.1073135.
doi:10.3115/1073083.1073135 .
[9] A. Lavie, A. Agarwal, Meteor: an automatic metric for mt evaluation with high levels
of correlation with human judgments, in: StatMT 2007, StatMT ’07, Association
for Computational Linguistics, USA, 2007, p. 228231.
[10] Z. Yang, A. Moffat, A. Turpin, Pairwise crowd judgments: Preference, absolute, and
ratio, in: Proceedings of the 23rd Australasian Document Computing Symposium,
ADCS ’18, Association for Computing Machinery, New York, NY, USA, 2018. URL:
https://doi.org/10.1145/3291992.3291995. doi:10.1145/3291992.3291995 .
[11] N. Arabzadeh, C. L. A. Clarke, A comparison of methods for evaluating generative
ir, 2024. URL: https://arxiv.org/abs/2404.04044.
[12] V. Jeronymo, L. Bonifacio, H. Abonizio, M. Fadaee, R. Lotufo, J. Zavrel, R. Nogueira,
InPars-v2: Large Language Models as Efficient Dataset Generators for Information
Retrieval, 2023. doi:10.48550/arXiv.2301.01820 . arXiv:2301.01820 .
[13] H. Huang, Y. Qu, J. Liu, M. Yang, T. Zhao, An empirical study of llm-as-a-judge
for llm evaluation: Fine-tuned judge models are task-specific classifiers, 2024. URL:
https://arxiv.org/abs/2403.02839.
[14] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li,
D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, I. Stoica, Judging llm-as-a-judge with
mt-bench and chatbot arena, 2023. URL: https://arxiv.org/abs/2306.05685.
[15] V. Katranidis, G. Barany, Faaf: Facts as a function for the evaluation of generated
text, 2024. arXiv:2403.03888 .
[16] P. Manakul, A. Liusie, M. J. F. Gales, Selfcheckgpt: Zero-resource black-box hallu-
cination detection for generative large language models, 2023. arXiv:2303.08896 .
[17] T. Zhang, H. Luo, Y.-S. Chuang, W. Fang, L. Gaitskell, T. Hartvigsen, X. Wu, D. Fox,
H. Meng, J. Glass, Interpretable unified language checking, 2023. arXiv:2304.03728 .
[18] A. Azaria, T. Mitchell, The internal state of an llm knows when it’s lying, 2023.
arXiv:2304.13734 .
[19] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoy-
anov, L. Zettlemoyer, Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension, 2019. arXiv:1910.13461 .
[20] W. Yuan, G. Neubig, P. Liu, Bartscore: Evaluating generated text as text generation,
in: M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, J. W. Vaughan (Eds.),
Advances in Neural Information Processing Systems, volume 34, Curran Associates,
Inc., 2021, pp. 27263–27277. URL: https://proceedings.neurips.cc/paper_files/paper/
2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf.
[21] S. Es, J. James, L. Espinosa-Anke, S. Schockaert, Ragas: Automated evaluation of
retrieval augmented generation, 2023. arXiv:2309.15217 .
[22] A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, T. Zrnic, Prediction-
powered inference, 2023. arXiv:2301.09633 .
[23] J. Saad-Falcon, O. Khattab, C. Potts, M. Zaharia, Ares: An automated evaluation
framework for retrieval-augmented generation systems, 2024. arXiv:2311.09476 .
[24] M. Eibich, S. Nagpal, A. Fred-Ojala, Aragog: Advanced rag output grading, 2024.
arXiv:2404.01037 .
[25] J. Chen, H. Lin, X. Han, L. Sun, Benchmarking large language models in retrieval-
augmented generation, 2023. arXiv:2309.01431 .
[26] S.-Q. Yan, J.-C. Gu, Y. Zhu, Z.-H. Ling, Corrective retrieval augmented generation,
2024. arXiv:2401.15884 .
[27] G. Fazlija, Toward Optimising a Retrieval Augmented Generation Pipeline using
Large Language Model, Master’s thesis, 2024.
[28] Z. Rackauckas, Rag-fusion: A new take on retrieval augmented generation,
International Journal on Natural Language Computing 13 (2024) 3747. URL:
http://dx.doi.org/10.5121/ijnlc.2024.13103. doi:10.5121/ijnlc.2024.13103 .
[29] Z. Dai, V. Y. Zhao, J. Ma, Y. Luan, J. Ni, J. Lu, A. Bakalov, K. Guu, K. B.
Hall, M.-W. Chang, Promptagator: Few-shot dense retrieval from 8 examples, 2022.
arXiv:2209.11755 .
[30] OpenAI, Gpt-4 turbo and gpt-4, 2024. arXiv:gpt-4-turbo-and-gpt-4 .
[31] Anthropic, The claude 3 model family: Opus, sonnet, haiku, 2024. arXiv:Model
Card Claude 3.pdf .
[32] P. Thomas, S. Spielman, N. Craswell, B. Mitra, Large language models can accurately
predict searcher preferences, 2023. arXiv:2309.10621 .
[33] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, F. Wei, Multilingual e5 text
embeddings: A technical report, 2024. arXiv:2402.05672 .
[34] S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, M. Gatford, Okapi
at TREC-3, in: D. K. Harman (Ed.), Proceedings of The Third Text REtrieval
Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994, volume
500-225 of NIST Special Publication, National Institute of Standards and Technology
(NIST), 1994, pp. 109–126. URL: http://trec.nist.gov/pubs/trec3/papers/city.ps.gz.
[35] N. D. Edwards, E. de Jong, S. T. Ferguson, Graphing methods for kendall’s 𝜏, 2023.
arXiv:2308.08466 .
[36] S. Perreault, Efficient inference for kendall’s tau, 2022. arXiv:2206.04019 .
[37] scipy.stats.kendalltau, ???? URL: https://docs.scipy.org/doc/scipy/reference/generated/
scipy.stats.kendalltau.html#r4cd1899fa369-2.
[38] M. G. KENDALL, The treatment of ties in ranking problems, Biometrika 33 (1945)
239–251. doi:https://doi.org/10.1093/biomet/33.3.239 .
[39] P. Schober, C. Boer, L. A. Schwarte, Correlation coefficients: Appropriate use and
interpretation, Anesthesia & Analgesia 126 (2018) 1763–1768. doi:https://doi.org/
10.1213/ane.0000000000002864 .
[40] G. Faggioli, L. Dietz, C. L. A. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando,
E. Kanoulas, M. Potthast, B. Stein, H. Wachsmuth, Perspectives on large language
models for relevance judgment, in: Proceedings of the 2023 ACM SIGIR International
Conference on Theory of Information Retrieval, ICTIR 23, ACM, 2023. URL:
http://dx.doi.org/10.1145/3578337.3605136. doi:10.1145/3578337.3605136 .
[41] A. Jadon, A. Patil, A comprehensive survey of evaluation techniques for recommen-
dation systems, 2024. arXiv:2312.16015 .
A. RAGElo ’s prompts and configurations
A.1. Retrieval Evaluator
We used the default RAGElo ’s ReasonerEvaluator , which has the following system prompt:
You a r e an e x p e r t document a n n o t a t o r . Your j o b i s t o e v a l u a t e
whether a document c o n t a i n s r e l e v a n t i n f o r m a t i o n t o answer a
user ’ s question .
P l e a s e a c t a s an i m p a r t i a l r e l e v a n c e a n n o t a t o r f o r a s e a r c h
e n g i n e . Your g o a l i s t o e v a l u a t e t h e r e l e v a n c y o f t h e
documents g i v e n a u s e r q u e s t i o n .
You s h o u l d w r i t e one s e n t e n c e e x p l a i n i n g why t h e document i s
r e l e v a n t o r not f o r t h e u s e r q u e s t i o n . A document can be :
− Not r e l e v a n t : The document i s not on t o p i c .
− Somewhat r e l e v a n t : The document i s on t o p i c but d o e s not
f u l l y answer t h e u s e r q u e s t i o n .
− Very r e l e v a n t : The document i s on t o p i c and an s we r s t h e u s e r ’
s question .
[ user question ]
{ query }
[ document c o n t e n t ]
{ document }
A.2. Answer evaluators
For the pointwise evaluator used in Section 5.1, we used the following prompt with
RAGElo ’s CustomPromptAnswerEvaluator :
You a r e an i m p a r t i a l j u d g e f o r e v a l u a t i n g t h e q u a l i t y o f t h e
r e s p o n s e s p r o v i d e d by an AI a s s i s t a n t t a s k e d t o answer u s e r s
’ q u e s t i o n s about t h e c a t a l o g u e o f IoT s e n s o r s produced by
Infineon .
You w i l l be g i v e n t h e u s e r ’ s q u e s t i o n and t h e answer produced
by t h e a s s i s t a n t .
The agent ’ s answer was g e n e r a t e d based on a s e t o f documents
r e t r i e v e d by a s e a r c h e n g i n e .
You w i l l be p r o v i d e d with t h e r e l e v a n t documents r e t r i e v e d by
the search engine .
Your t a s k i s t o e v a l u a t e t h e answer ’ s q u a l i t y based on t h e
r e s p o n s e ’ s r e l e v a n c e , acc uracy , and c o m p l e t e n e s s .
## Rules f o r e v a l u a t i n g an answer :
− ∗∗ R e l e v a n c e ∗ ∗ : Does t h e answer a d d r e s s t h e u s e r ’ s q u e s t i o n ?
− ∗∗ Accuracy ∗ ∗ : I s t h e answer f a c t u a l l y c o r r e c t , based on t h e
documents p r o v i d e d ?
− ∗∗ Completeness ∗ ∗ : Does t h e answer p r o v i d e a l l t h e i n f o r m a t i o n
needed t o answer t h e u s e r ’ s q u e s t i o n ?
− ∗∗ P r e c i s i o n ∗ ∗ : I f t h e u s e r ’ s q u e s t i o n i s about a s p e c i f i c
product , d o e s t h e answer p r o v i d e t h e answer f o r t h a t
s p e c i f i c product ?
## S t e p s t o e v a l u a t e an answer :
1 . ∗∗ Understand t h e u s e r ’ s i n t e n t ∗ ∗ : E x p l a i n i n your own words
what t h e u s e r ’ s i n t e n t i s , g i v e n t h e q u e s t i o n .
2 . ∗∗ Check i f t h e answer i s c o r r e c t ∗ ∗ : Think s t e p −by−s t e p
whether t h e answer c o r r e c t l y a ns w er s t h e u s e r ’ s q u e s t i o n .
3 . ∗∗ E v a l u a t e t h e q u a l i t y o f t h e answer ∗ ∗ : E v a l u a t e t h e q u a l i t y
o f t h e answer based on i t s r e l e v a n c e , acc uracy , and
completeness .
4 . ∗∗ A s s i g n a s c o r e ∗ ∗ : Produce a s i n g l e l i n e JSON o b j e c t with
t h e f o l l o w i n g keys , each with a s i n g l e s c o r e between 0 and
2 , where 2 i s t h e h i g h e s t s c o r e on t h a t a s p e c t :
− " relevance "
− 0 : The answer i s not r e l e v a n t t o t h e u s e r ’ s q u e s t i o n .
− 1 : The answer i s p a r t i a l l y r e l e v a n t t o t h e u s e r ’ s
question .
− 2 : The answer i s f u l l y r e l e v a n t t o t h e u s e r ’ s q u e s t i o n .
− " accuracy "
− 0 : The answer i s f a c t u a l l y i n c o r r e c t .
− 1 : The answer i s p a r t i a l l y c o r r e c t .
− 2 : The answer i s f u l l y c o r r e c t .
− " completeness "
− 0 : The answer d o e s not p r o v i d e enough i n f o r m a t i o n t o
answer t h e u s e r ’ s q u e s t i o n .
− 1 : The answer o n l y a n sw e rs some a s p e c t s o f t h e u s e r ’ s
question .
− 2 : The answer f u l l y a ns w er s t h e u s e r ’ s q u e s t i o n .
− " precision "
− 0 : The answer d o e s not mention t h e same p r o d u c t o r
product l i n e as the user ’ s question .
− 1 : The answer mentions a s i m i l a r p r o d u c t o r p r o d u c t l i n e ,
but not t h e same a s t h e u s e r ’ s q u e s t i o n .
− 2 : The answer mentions t h e e x a c t same p r o d u c t o r p r o d u c t
l i n e as the user ’ s question .
The l a s t l i n e o f your answer must be a SINGLE LINE JSON o b j e c t
with t h e k e y s " r e l e v a n c e " , " a c c u r a c y " , " c o m p l e t e n e s s " , and "
p r e c i s i o n " , each with a s i n g l e s c o r e between 0 and 2 .
[DOCUMENTS RETRIEVED]
{ documents }
[ User Query ]
{ query }
[ Agent answer ]
{ answer }
For the pairwise evaluation between agents used for the results in Tables 5 and 6, we
used RAGElo ’s PairwiseAnswerEvaluator with the following parameters:
pairwise_evaluator_config = PairwiseEvaluatorConfig (
n_games_per_query =15 ,
h a s _ c i t a t i o n s=F a l s e ,
include_raw_documents=True ,
i n c l u d e _ a n n o t a t i o n s=True ,
document_relevance_thresh old =2,
f a c t o r s=" t h e c o m p r e h e n s i v e n e s s , c o r r e c t n e s s , h e l p f u l n e s s ,
c o m p l e t e n e s s , ac curacy , depth , and l e v e l o f d e t a i l o f
t h e i r r e s p o n s e s . Answers a r e c o m p r e h e n s i v e i f they show
t h e u s e r m u l t i p l e p e r s p e c t i v e s i n a d d i t i o n t o but s t i l l
r e l e v a n t to the i n t e n t of the o r i g i n a l question . " ,
)
This generates 15 random games between two agents per query (i.e., all possible unique
games for 6 agents) and tells the evaluator that:
• The answers do not include specific citations to any passage (has_citations=False )
• Include the full text of the retrieved passages in the evaluation prompt
(include_raw_documents=True )
• Inject the output of the retrieval evaluator into the prompt
(include_annotations=True )
• Ignore any passage with a relevance score below 2
(document_relevance_threshold=2 )
• Consider these factors when selecting the best answer
factors=… )
These parameters produce the following final prompt used for evaluating the answers:
P l e a s e a c t a s an i m p a r t i a l j u d g e and e v a l u a t e t h e q u a l i t y o f
t h e r e s p o n s e s p r o v i d e d by two AI a s s i s t a n t s t a s k e d t o answer
t h e q u e s t i o n below based on a s e t o f documents r e t r i e v e d by
a search engine .
You s h o u l d c h o o s e t h e a s s i s t a n t t h a t b e s t an s we r s t h e u s e r
q u e s t i o n based on a s e t o f r e f e r e n c e documents t h a t may o r
may not be r e l e v a n t .
For each r e f e r e n c e document , you w i l l be p r o v i d e d with t h e t e x t
o f t h e document a s w e l l a s r e a s o n s why t h e document i s o r
i s not r e l e v a n t .
Your e v a l u a t i o n s h o u l d c o n s i d e r f a c t o r s such a s
comprehensiveness , correctness , h e l p f u l n e s s , completeness ,
accurac y , depth , and l e v e l o f d e t a i l o f t h e i r r e s p o n s e s .
Answers a r e c o m p r e h e n s i v e i f they show t h e u s e r m u l t i p l e
p e r s p e c t i v e s i n a d d i t i o n t o but s t i l l r e l e v a n t t o t h e i n t e n t
of the o r i g i n a l question .
D e t a i l s a r e o n l y u s e f u l i f they answer t h e u s e r ’ s q u e s t i o n . I f
an answer c o n t a i n s non−r e l e v a n t d e t a i l s , i t s h o u l d not be
p r e f e r r e d o v e r one t h a t o n l y u s e s r e l e v a n t i n f o r m a t i o n .
Begin your e v a l u a t i o n by e x p l a i n i n g why each answer c o r r e c t l y
ans we rs t h e u s e r ’ s q u e s t i o n . Then , you s h o u l d compare t h e
two r e s p o n s e s and p r o v i d e a s h o r t e x p l a n a t i o n o f t h e i r
d i f f e r e n c e s . Avoid any p o s i t i o n b i a s e s and e n s u r e t h a t t h e
o r d e r i n which t h e r e s p o n s e s were p r e s e n t e d d o e s not
i n f l u e n c e your d e c i s i o n . Do not a l l o w t h e l e n g t h o f t h e
r e s p o n s e s t o i n f l u e n c e your e v a l u a t i o n . Be a s o b j e c t i v e a s
possible .
A f t e r p r o v i d i n g your e x p l a n a t i o n , output your f i n a l v e r d i c t by
s t r i c t l y f o l l o w i n g t h i s format : " [ [ A ] ] " i f a s s i s t a n t A i s
b e t t e r , " [ [ B ] ] " i f a s s i s t a n t B i s b e t t e r , and " [ [ C ] ] " f o r a
tie .
[ User Q u e s t i o n ]
{ query }
[ R e f e r e n c e Documents ]
{ documents }
[ The S t a r t o f A s s i s t a n t A’ s Answer ]
{ answer_a }
[ The End o f A s s i s t a n t A’ s Answer ]
[ The S t a r t o f A s s i s t a n t B’ s Answer ]
{answer_b }
[ The End o f A s s i s t a n t B’ s Answer ]