Challenges in the automated evaluation of Retrieval-Augmented Generation (RAG) Questionanswering (QA) systems include hallucination problems in domain-specific knowledge and the lack of gold standard benchmarks for company-internal tasks. This results in dificulties in evaluating RAG variations, like RAG-Fusion (RAGF) in the context of a product QA task at Infineon Technologies. To solve these problems, we propose a comprehensive evaluation framework, which leverages Large Language Models (LLMs) to generate large datasets of synthetic queries based on real user queries and in-domain documents, uses LLM-as-a-judge to rate retrieved documents and answers, evaluates the quality of answers, and ranks diferent variants of Retrieval-Augmented Generation (RAG) agents with RAGElo's automated Elo-based competition. LLM-as-a-judge rating of a random sample of synthetic queries shows a moderate, positive correlation with domain expert scoring in relevance, accuracy, completeness, and precision. While RAGF outperformed RAG in Elo score, a significance analysis against expert annotations also shows that RAGF significantly outperforms precision. In addition, Infineon's RAGF assistant demonstrated slightly higher performance in preferences of human annotators, though due caution is still required. Finally, RAGF's approach leads to more complete answers based on expert annotations and better answers overall based on RAGElo's evaluation criteria.
CEUR ceur-ws.org
The text-generating capabilities of LLMs, together with their text understanding abilities,
have allowed conversational Question-Answering (QA) systems to experience a
considerable leap in performance, with near-human text quality and reasoning capabilities [
The general inability of such models to identify unanswerable questions [
2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). is usually out-of-domain for most LLMs, but is present in private and confidential internal documents from the company.
One such company is Infineon, a leading manufacturer of semiconductors. Given its wide range of equipment, information about its products is spread across multiple, highly technical documents, including datasheets and selection guides of hundreds of pages. Therefore, an internal retrieval augmented conversational QA system was developed by Infineon for internal users such as account managers, field application engineers, and sales operations specialists. This system allows professionals to ask questions about products from the whole catalog while in the field.
One of the features of Infineon’s conversational agent is the usage of RAG -Fusion (RAGF ),
a technique for increasing the quality of the generated answers by generating variations
of the user question and combining the rankings produced by these variations using
rank-fusion methods (i.e., recriprocal rank fusion (RRF) [
However, evaluating these systems bring complications common to retrieval augmented
agents, especially in enterprise settings, stemming from the lack of comprehensive test
datasets. Ideally, such a test set would comprise a large set of real user questions from
a query log, paired with “golden answers” provided by experts. The lack of such a
test set leads to two main issues. First, evaluation of answers generated by LLMs by
traditional n-gram evaluation metrics such as ROUGE [
One approach for tackling the lack of an extensive test set is to use synthetic queries
generated by LLMs as a proxy of user queries [
To solve this, we propose to use a process similar to InPars [
To tackle the second issue, a lack of ground truth “golden answers,” we leverage an
LLM-as-a-judge process, where a strong LLM is used to evaluate the quality of the
answers generated by the RAG agent’s LLM [
Finally, to mitigate the lack of in-domain knowledge of the judging LLM, we also annotate the relevance of the documents retrieved by the pipelines being evaluated and inject the relevant documents in the context used by the judging LLM. This allows the judging LLM to better assess for hallucinations and completeness and better align the quality of the evaluations to those conducted by experts.
This process is mediated by RAGElo 1, a toolkit for evaluating RAG systems inspired by the Elo ranking system. RAGElo provides an easy-to-use CLI and Python library for using LLMs to evaluate retrieval results and answers produced by RAG pipelines. By combining a retrieval evaluator, a pairwise answer annotator, and an Elo-inspired tournament, RAGElo leverages powerful LLMs to agnostically annotate and rank diferent RAG pipelines. We notice that, although noisy, the LLM annotations generated by RAGElo are generally well aligned with experts’ judgments of relative system quality, allowing for fast experimentation and comparisons between diferent RAG implementations without the frequent intervention of experts as annotators.
This paper evaluates multiple implementations of Infineon’s retrieval augmented conversational agent using RAGElo: a traditional Retrieval-Augmented Generation and a RAG-Fusion implementation. RAG-Fusion generates multiple variations of the user question and combines the rankings produced by these queries into a more diverse set of documents. The documents are then fed into the LLM. We also analyze these same agents under a keyword-based retrieval regimen (i.e., the retriever uses BM25 to retrieve and rank documents), a dense retriever, and a hybrid retriever that combines the ranking generated by the BM25 and the dense retrievers using RRF. Our goal is to answer the following questions: • Does the evaluation framework proposed by RAGElo align with the preferences of human annotators for answers generated by RAG-based conversational agents? • Does the RAGF approach of submitting multiple variations of the user question and combining their rankings lead to better answers?
What is the country of origin of IM72D128, and how does geopolitical exposure afect the market and my SAM for the microphone? What is the IP rating of mounted IM72D128? Tell me microphones that have been released since January 2023 based on the datasheet revision history.
We need to confirm whether the IFX waterproof MIC has a sleeping mode and wake-up functions.
Several evaluation systems for RAG have been proposed to address flaws in current
evaluation methods. For instance, Facts as a Function (FaaF) [
numerous recent works.
SelfCheckGPT demonstrates the ability to leverage LLMs to detect and rank factuality
with zero resources [
Automated evaluation metrics can also be applied to RAG-based agents. BARTScore,
an automated metric based on the BART architecture, has also outperformed most
metrics on categories including factuality [
ARAGOG highlights Hypothetical Document Embedding (HyDE) and LLM reranking
as efective methods for enhancing retrieval precision while also exploring the efectiveness
of Sentence Window Retrieval and the potential of the Document Summary Index in
improving RAG systems [
While the aforementioned frameworks evaluate answers on relevance, faithfulness, and
correctness metrics, RAG can also be evaluated on noise and counterfactual robustness,
negative rejection, and information integration [
In addition to answers, frameworks have also been created to evaluate documents.
Corrective Retrieval Augmented Generation (CRAG) builds on RAG by employing a
retrieval evaluator to ensure that only the optimal documents are fed into the LLM
prompt prior to the answer generation phase [
Due to its Elo-based ranking system for answers, its use of LLM-as-a-judge, and its relevance evaluation of the intermediate retrieval steps in a RAG pipeline, RAGElo is a unique evaluation toolkit. In this study, we use it to compare a simple RAG versus a more sophisticated RAGF system on a knowledge-intensive industry-specific domain.
While answers generated by traditional retrieval augmented systems are based on a
number of documents retrieved from a single query, RAGF introduces additional variation
into the retrieval process. Upon receiving a query from the user, a RAGF agent leverages
a large language model to generate a set of queries based on the original [
After generating the variations for the user query, the RAGF agent submits the original
and the generated queries to a retrieval system [
Documents (passages) Expert questions (few-shot) claude-3-opus-20240229 claude-3-haiku-20240307 claude-3-sonnet-20240229 gpt-4-turbo-2024-04-09
Query generators What security features does the OPTIGA Trust M provide for IoT devices (…) WahviacihlaTbLlEe49i7n1tchuerTrIeSnOtNs-8en-s6opramcokadegles? are FtoerchanpoplloigciaetsiolniskewiStihC,fawshticshwi(t…c)hing Are there any self-diagnosis features available in the KP23x analog sensor?
Pooled queries
As previously discussed, one of the main issues when evaluating the quality of a QA
system in an enterprise setting is that, frequently, companies do not have a large enough
existing collection of queries to evaluate such systems’ quality. Therefore, in this work,
we propose to adopt a strategy previously used by methods for generating synthetic
queries for training retrieval systems, such as InPars [
Similar to these approaches, we randomly sample passages from documents within our collection and prompt an LLM to generate questions that users may ask about these portions. However, one diference in our approach to generating training queries is the size of these passages. When generating queries for training a retrieval system, we ideally want to keep the passages short to fit in the dense encoder’s relatively short context windows. However, when generating queries for evaluating QA systems (including retrieval augmented), we are not bound to the limit of the embedding model used for retrieval. Rather, a longer passage may yield questions that require multiple shorter passages to be answered. Therefore, we submit relatively long passages to the LLMs. Specifically, each passage is extracted from up to ten pages of PDF documents (about 2000 tokens 2)
To keep the questions generated as diverse as possible, we prompt four diferent LLMs
to generate up to ten questions based on the same documents. Our test set collection
contains a mix of queries generated by OpenAI’s GPT-4 turbo [
Finally, to increase the quality of the generated queries, We asked for an account manager, a sales operations specialist, a marketing representative, and a business development manager to create queries that they would submit to the conversational agent from the perspective of their role. They were instructed to produce queries regarding products from the XENSIV sensor product line, consisting of MEM microphones, radar, current, magnetic, pressure, and environmental sensors. We compiled a list of 23 of these queries to use as a base for experimentation and used them as few-shot examples in the query generation prompt. Figure 2 illustrates our method for generating synthetic queries based on existing user queries and document passages.
Even with a suitable set of synthetic questions for evaluating our RAG conversational agent, assessing whether a given answer properly answers a question is not trivially done. If a ground-truth “golden answer” is available, one can use traditional syntactic-based 2all LLMs used in our experiments had long context windows of 128k or 200k tokens. 3We did not use GPT-3.5 or open source models due to their shorter context window at the time of writing.
Agent A answer
Agent B answer Are there any self-diagnosis features available in the KP23x analog sensor?
Synthetic test query
Search agents
RAGElo RetrievalEvaluator gednoecRruaemtteredienvatesndsawnders dPoociunmtweinsteerveatrliueavteiodn
AsgoemnetsAe:lfY-edsi,agtnhoesiKsP2f3exatsuernesso.r has Btohtehqausessitsitoann(t…s)correctly answer AifsgnuuecncnlchtutadiBseo:snbTauhslieeilltKtfyP--2ida3ninxadmgaoInnnSoaiOsltio2osg6r2ics6nae2gpnas(bo…ir)listeireise,s asAaBcescacnscsosiuerosrddrtaoisatnnnehgt,tahtvBreoeepslrdfeeooevovcaabiutnsdmuteeer,rsnevtsaaan<stdmduioocforchone_csaiu,sds>e(,d…)the rreegsaprodnisnegttohtehKePu2s3exrs'esrqiueesssteinosnors.
Final Verdict: [[B]] RAGElo RetrievalEvaluator
Pairwise answer judgement
FInal evaluation
metrics such as BLEU, METEOR or ROUGE [
Alternatively, several LLM-as-a-Judge methods have been proposed, where another LLM is asked to evaluate the quality of answers generated by other LLMs. Nevertheless, in an enterprise setting, the answers usually require the LLM to access knowledge not present in their training datasets but rather contained in documents internal to the company. This is usually accomplished using a RAG pipeline like the one described above. Therefore, the judging LLM also needs access to similar knowledge to accurately evaluate the agent’s answers’ quality.
Therefore, in this work, we rely on RAGElo, an open-source RAG evaluation toolkit that
evaluates the answers generated by each agent and the documents retrieved by them. By
injecting the annotation of retrieved documents, pooled by the agents being evaluated,
on the answer evaluation step, this method allows for the judging LLM to evaluate if
the generated answer was able to use all the information available about the question
properly and to check for any hallucinations. As the documents used for generating the
answers are included in the answer evaluating prompt, an agent that incorrectly cites
information from a source or refers to information not present in these documents is
likely hallucinating and should have its evaluation adjusted accordingly. As we explore
in Section 8, this two-step process results in a high correlation between human expert
annotators and the judging LLM, enabling higher reliability and trust when evaluating
diferent RAG pipelines. This process is also illustrated in Figure 3.
5.1. Evaluation aspects
While our main evaluation focuses on the pairwise comparison between the two agents,
RAGElo also allows us to evaluate answers pointwise. In this setting, similar to other
works [
We not only experiment with diferent search agents (i.e., RAG and RAGF. We are also interested in how diferent retrieval methods may impact the quality of the final answers generated by these agents. 6.1. Retrieval methods Our corpus consists of passages extracted from the Infineon XENSIV Product Selection Guide, a 117-page document with detailed information on every product in the XENSIV family. This document included technical information about all Infineon XENSIV sensors, consumer and automotive sensor applications, guidance in selecting the correct sensor, and other comprehensive and detailed information about the product line.
The passages are embedded using multilingual-e5-base [
Meanwhile, the Infineon RAGF conversational assistant uses a similar framework and performs the following steps upon receiving a query: 1. Perform a Chat Completions API call to generate four new queries based on the original query using a prompt tailored to the agent’s original goal. 2. Retrieve the top k most relevant passages for each query. 3. Using RRF, combines the top- passages induced by all queries into a final ranking. 4. Perform a Chat Completions API call prompting the LLM with carefully worded instructions for generating an answer based on the top- fused passages 5. Process and output the Chat Completions response.
7.1. Comparing LLM-as-a-judge to expert annotators While LLM-as-a-judge is a theoretically viable algorithm for rating RAG and RAGF answers, we must establish whether the results agree with the annotations of domain experts.
Figure 4 provides a Bland-Altman plot to visually represent the LLM and human judgments’ agreement.
The bias of approximately 0.12 indicates that, on average, LLM scores were slightly higher than human scores. The limits of agreement ranged from approximately -1.17 to 1.41. demonstrating substantial variability in the diference between LLM and human evaluators.
Next, we compared LLM-as-a-judge to expert annotators with Kendall’s . Kendalls
is a nonparametric measure that quantifies the degree of association between two
monotonic continuous or ordinal variables by calculating the proportion of concordance
and discordance among pairwise ranks, ofering valuable insight into their rank correlation
[
( − ) √( + + ) ⋅ ( + + ) (2) P represents the number of concordant pairs, Q represents the number of discordant pairs, T represents the number of ties exclusive to x, and U represents the number of ties exclusive to y.
This test returned ≈ 0.56, indicating a moderate, positive correlation [
Following the same methodology, we also calculated Spearman’s , a similar
nonparametric correlation measure. This resulted in ≈ 0.59 with < 0.01, demonstrating a
statistically significant, moderate positive correlation [
We assessed document retrieval quality using Mean Reciprocal Rank@5 (MRR@5), which averages the inverse ranks of the first relevant result within the top five positions across all queries. The formula is given by
||
@5 = 1 ∑
|| =1 rank
1 ,
(3)
where || is the total number of queries and rank is considered only if it’s within the top
ifve, otherwise it counts as zero [
MRR@5 scores were calculated for each agent and each retrieval method considering two categories: 1. MRR@5 score for documents deemed “somewhat relevant” or “very relevant.” 2. MRR@5 score for documents deemed “very relevant.”
The results can be seen below in Table 4.
We then ran RAGElo games to evaluate end-to-end answer quality of RAG vs RAGF with diferent base retriever configurations a task that cannot rely on standard Information Retrieval metrics. These RAGElo results show more victories for RAGF than RAG; For example, when using BM25 as a base retriever, RAGF won 49% of the games, RAG won 14.5%, and RAG and RAGF are tied in 36.5% of the times. The resulting Elo scores for all six variants are shown in table 6, which give a robust ranking of the systems, without reliance
BM25 KNN
RAGF RAG RAGF RAG RAGF
RAG 49.5% 58.5% — 37.0% 51.5% 49.0%
on a gold standard. It is interesting to see, that for both RAGF as well as RAG, BM25 is a strong baseline that is not surpassed by generic embeddings in these experiments.
Next, we compared the RAGElo outcome to the preference of our Infineon human annotator. We performed two-tailed paired t-tests to compare RAG against RAGF on each category from the Infineon representatives’ human evaluations with = .05 . As expected, due to its larger variety of retrieved results, RAGF significantly outperforms RAG in completeness at the 95% confidence level with ≈ 0.01 However, on the precision of answers, RAG significantly outperformed RAGF at the 95% confidence level with ≈ 0.04. As observed above, we found statistically significant, moderate positive correlations between LLM ratings and human annotations. This indicates a consistent association between the ratings from LLM-as-a-judge and those by Infineon experts. We find that on average, LLM scores are slightly higher than those of human annotators. This means that while relevance judgements on individual queries should not be fully reliable, and IR metrics derived from LLM-as-a-judge should not be equated with regular relevance scores without further calibration, we can still make good use of this approach to rankorder systems. These findings collectively support the validity of our LLM evaluation method, which assesses conversational system outputs based on a combination of relevance, accuracy, completeness, and recall.
The style of evaluation and the diferent dimensions it takes into account are specified in the prompts given to the LLM in the RAGElo evaluation, which are provided in Appendix A. Specifically, while the initial LLM-as-a-judge is given specific criteria to focus on only four categories, we instructed RAGElo’s impartial judge LLM to value more than the initial four categories:
Your evaluation should consider factors such as comprehensiveness, correctness, helpfulness, completeness, accuracy, depth, and level of detail of their responses.
Since RAGF significantly outperformed RAG in the completeness category, the RAGElo judge LLM likely weighed completeness higher than precision. In addition, based on manual observation of a small random sample of answers, RAGF produced more comprehensive answers and featured higher depth and level of detail due to the multiple query generation. However, games where RAG won were most likely influenced by a significantly more precise answer than that of RAGF. While RAGF values comprehensive answers that ofer multiple perspectives to the user, RAG produces shorter answers that answer the original query only. Since completeness is defined as the extent to which a user’s question was answered, it can be presumed that RAGF’s longer and more comprehensive answers may tend to be more complete. And since precision relates to the agent mentioning the correct product or product family, it can be presumed that RAGF’s longer answers have more room to consider other products or product families, leading to reduced answer precision. While the human annotation was done by Infineon experts, diferent humans may rate answers diferently, even if following the same set of criteria.
A larger number of documents or a database of non-technical documents may have led to a diferent outcome. RAGF can be applied to not only Infineon documents but also any documents database to retrieve. This includes not only enterprise uses but also uses in education, such as mathematics and language learning. The algorithm can be tuned to diferent use cases by tweaking the internal LLM prompt. For example, the Infineon RAGF bot was prompted to "think like an engineer." However, an educator RAGF bot could be prompted to "think like a teacher." Future work includes exploring other applications of RAGF, especially in education. In addition, we will experiment with diferent prompts for both LLM-as-a-judge and RAGElo while using diferent quantities and types of documents with the same retrieval algorithms.
Based on the calculated MMR@5 scores, we found that the RAGF agent mostly outperforms the RAG agent in ranking both highly relevant and somewhat relevant documents retrieved. This evidence search on multiple query variants produced, on average, slightly more higher-ranked relevant documents than using only the original user query. We also see that using vector search with embeddings is not a silver bullet, as for our test queries, BM25 seriously outperforms it. Since retrieval quality is highly dependent on the quality of the embeddings and their fit to the domain, this outcome will likely be changed by ifne-tuning the embeddings, and adding additional intelligent re-rankers, which we leave here for future work, as the evaluation framework would remain the same.
Overall, we found that the evaluation framework proposed by RAGElo positively aligns with the preferences of human annotators for RAG and RAGF with due caution due to a moderate correlation and variability of scoring. We found that the RAGF approach leads to better answers most of the time, according to the RAGElo evaluation. According to expert scoring, the RAGF approach significantly outperforms in completeness compared to RAG but significantly underperforms in precision compared to RAG. Based on these results, we cannot confidently assert that RAGF’s approach leads to better answers generally. However, the results do support that RAGF’s approach leads to more complete answers and a higher proportion of better answers under evaluation by RAGElo.
Since RAGElo is generally applicable to all retrieval-augmented algorithms, in future work, we also intend to test diferent agents other than RAG and RAGF, including those with diferent reranking algorithms, diferent embedding models, and diferent LLMs. In addition, due to RAGF’s underperformance in document relevance, we may also leverage CRAG to reduce this gap. We will also investigate the reflection of human sensitivity in expert ratings, especially whether the LLMs should or can reflect human sensitivities.
We thank Brooks Felton from Infineon for his support during this work. We also thank the Infineon sales team for providing valuable feedback.
A.1. Retrieval Evaluator We used the default RAGElo’s ReasonerEvaluator, which has the following system prompt: You a r e an e x p e r t document a n n o t a t o r . Your j o b i s t o e v a l u a t e whether a document c o n t a i n s r e l e v a n t i n f o r m a t i o n t o answer a u s e r ’ s q u e s t i o n .
P l e a s e a c t a s an i m p a r t i a l r e l e v a n c e a n n o t a t o r f o r a s e a r c h e n g i n e . Your g o a l i s t o e v a l u a t e t h e r e l e v a n c y o f t h e documents g i v e n a u s e r q u e s t i o n .
You s h o u l d w r i t e one s e n t e n c e e x p l a i n i n g why t h e document i s r e l e v a n t o r not f o r t h e u s e r q u e s t i o n . A document can be : − Not r e l e v a n t : The document i s not on t o p i c . − Somewhat r e l e v a n t : The document i s on t o p i c but d o e s not f u l l y answer t h e u s e r q u e s t i o n . − Very r e l e v a n t : The document i s on t o p i c and a n s w e r s t h e u s e r ’ s q u e s t i o n . [ u s e r q u e s t i o n ]
{ query } [ document c o n t e n t ]
{ document } A.2. Answer evaluators For the pointwise evaluator used in Section 5.1, we used the following prompt with RAGElo’s CustomPromptAnswerEvaluator: You a r e an i m p a r t i a l judge f o r e v a l u a t i n g the q u a l i t y o f the r e s p o n s e s provided by an AI a s s i s t a n t tasked to answer u s e r s ’ q u e s t i o n s about the c a t a l o g u e o f IoT s e n s o r s produced by I n f i n e o n .
You w i l l be g i v e n the user ’ s q u e s t i o n and the answer produced by the a s s i s t a n t .
The agent ’ s answer was g e n e r a t e d based on a s e t o f documents r e t r i e v e d by a s e a r c h e n g i n e .
You w i l l be provided with the r e l e v a n t documents r e t r i e v e d by the s e a r c h e n g i n e .
Your t a s k i s to e v a l u a t e the answer ’ s q u a l i t y based on the response ’ s r e l e v a n c e , accuracy , and c o m p l e t e n e s s . ## Rules f o r e v a l u a t i n g an answer : − ∗∗ Relevance ∗ ∗ : Does the answer a d d r e s s the user ’ s q u e s t i o n ? − ∗∗ Accuracy ∗ ∗ : I s the answer f a c t u a l l y c o r r e c t , based on the documents provided ? − ∗∗ Completeness ∗ ∗ : Does the answer p r o v i d e a l l the i n f o r m a t i o n needed to answer the user ’ s q u e s t i o n ? − ∗∗ P r e c i s i o n ∗ ∗ : I f the user ’ s q u e s t i o n i s about a s p e c i f i c product , does the answer p r o v i d e the answer f o r that s p e c i f i c product ? ## Steps to e v a l u a t e an answer : 1 . ∗∗ Understand the user ’ s i n t e n t ∗ ∗ : Explain i n your own words what the user ’ s i n t e n t i s , g i v e n the q u e s t i o n . 2 . ∗∗ Check i f the answer i s c o r r e c t ∗ ∗ : Think step −by−s t e p whether the answer c o r r e c t l y answers the user ’ s q u e s t i o n . 3 . ∗∗ Evaluate the q u a l i t y o f the answer ∗ ∗ : Evaluate the q u a l i t y o f the answer based on i t s r e l e v a n c e , accuracy , and c o m p l e t e n e s s . 4 . ∗∗ Assign a s c o r e ∗ ∗ : Produce a s i n g l e l i n e JSON o b j e c t with the f o l l o w i n g keys , each with a s i n g l e s c o r e between 0 and 2 , where 2 i s the h i g h e s t s c o r e on that a s p e c t : − " r e l e v a n c e " − 0 : The answer i s not r e l e v a n t to the user ’ s q u e s t i o n . − 1 : The answer i s p a r t i a l l y r e l e v a n t to the user ’ s
q u e s t i o n .
− 2 : The answer i s f u l l y r e l e v a n t to the user ’ s q u e s t i o n . − " accuracy " − 0 : The answer i s f a c t u a l l y i n c o r r e c t . − 1 : The answer i s p a r t i a l l y c o r r e c t .
− 2 : The answer i s f u l l y c o r r e c t . − " c o m p l e t e n e s s " − 0 : The answer does not p r o v i d e enough i n f o r m a t i o n to answer the user ’ s q u e s t i o n . − 1 : The answer only answers some a s p e c t s o f the user ’ s q u e s t i o n .
− 2 : The answer f u l l y answers the user ’ s q u e s t i o n . − " p r e c i s i o n " − 0 : The answer does not mention the same product or product l i n e as the user ’ s q u e s t i o n . − 1 : The answer mentions a s i m i l a r product or product l i n e , but not the same as the user ’ s q u e s t i o n . − 2 : The answer mentions the e x a c t same product or product l i n e as the user ’ s q u e s t i o n .
The l a s t l i n e o f your answer must be a SINGLE LINE JSON o b j e c t with the keys " r e l e v a n c e " , " accuracy " , " c o m p l e t e n e s s " , and " p r e c i s i o n " , each with a s i n g l e s c o r e between 0 and 2 . [DOCUMENTS RETRIEVED] { documents } [ User Query ] { query } [ Agent answer ] { answer }
For the pairwise evaluation between agents used for the results in Tables 5 and 6, we used RAGElo’s PairwiseAnswerEvaluator with the following parameters: p a i r w i s e _ e v a l u a t o r _ c o n f i g = P a i r w i s e E v a l u a t o r C o n f i g ( n_games_per_query=15 , h a s _ c i t a t i o n s=False , include_raw_documents=True , i n c l u d e _ a n n o t a t i o n s=True , document_relevance_threshold =2, f a c t o r s=" the comprehensiveness , c o r r e c t n e s s , h e l p f u l n e s s , completeness , accuracy , depth , and l e v e l o f d e t a i l o f t h e i r r e s p o n s e s . Answers a r e comprehensive i f they show the u s e r m u l t i p l e p e r s p e c t i v e s i n a d d i t i o n to but s t i l l r e l e v a n t to the i n t e n t o f the o r i g i n a l q u e s t i o n . " , ) This generates 15 random games between two agents per query (i.e., all possible unique games for 6 agents) and tells the evaluator that: • The answers do not include specific citations to any passage ( has_citations=False) • Include the full text of the retrieved passages in the evaluation prompt (include_raw_documents=True) • Inject the output of the retrieval evaluator into the prompt (include_annotations=True) • Ignore any passage with a relevance score below 2
(document_relevance_threshold=2) • Consider these factors when selecting the best answer
factors=…)
These parameters produce the following final prompt used for evaluating the answers: P l e a s e a c t as an i m p a r t i a l judge and e v a l u a t e the q u a l i t y o f the r e s p o n s e s provided by two AI a s s i s t a n t s tasked to answer the q u e s t i o n below based on a s e t o f documents r e t r i e v e d by a s e a r c h e n g i n e .
You should choose the a s s i s t a n t that b e s t answers the u s e r q u e s t i o n based on a s e t o f r e f e r e n c e documents that may or may not be r e l e v a n t .
For each r e f e r e n c e document , you w i l l be provided with the t e x t o f the document as w e l l as r e a s o n s why the document i s or i s not r e l e v a n t .
Your e v a l u a t i o n should c o n s i d e r f a c t o r s such as comprehensiveness , c o r r e c t n e s s , h e l p f u l n e s s , completeness , accuracy , depth , and l e v e l o f d e t a i l o f t h e i r r e s p o n s e s . Answers a r e comprehensive i f they show the u s e r m u l t i p l e p e r s p e c t i v e s i n a d d i t i o n to but s t i l l r e l e v a n t to the i n t e n t o f the o r i g i n a l q u e s t i o n .
D e t a i l s a r e only u s e f u l i f they answer the user ’ s q u e s t i o n . I f an answer c o n t a i n s non−r e l e v a n t d e t a i l s , i t should not be p r e f e r r e d over one that only u s e s r e l e v a n t i n f o r m a t i o n . Begin your e v a l u a t i o n by e x p l a i n i n g why each answer c o r r e c t l y answers the user ’ s q u e s t i o n . Then , you should compare the two r e s p o n s e s and p r o v i d e a s h o r t e x p l a n a t i o n o f t h e i r d i f f e r e n c e s . Avoid any p o s i t i o n b i a s e s and e n s u r e that the o r d e r i n which the r e s p o n s e s were p r e s e n t e d does not i n f l u e n c e your d e c i s i o n . Do not a l l o w the l e n g t h o f the r e s p o n s e s to i n f l u e n c e your e v a l u a t i o n . Be as o b j e c t i v e as p o s s i b l e .
A f t e r p r o v i d i n g your e x p l a n a t i o n , output your f i n a l v e r d i c t by s t r i c t l y f o l l o w i n g t h i s format : " [ [A] ] " i f a s s i s t a n t A i s b e t t e r , " [ [ B ] ] " i f a s s i s t a n t B i s b e t t e r , and " [ [ C ] ] " f o r a t i e . [ User Question ] { query } [ R e f e r e n c e Documents ] { documents } [ The S t a r t o f A s s i s t a n t A’ s Answer ] {answer_a} [ The End o f A s s i s t a n t A’ s Answer ] [ The S t a r t o f A s s i s t a n t B’ s Answer ] {answer_b} [ The End o f A s s i s t a n t B’ s Answer ]