The text-generating capabilities of LLMs, together with their text understanding abilities, have allowed conversational Question-Answering (QA) systems to experience a considerable leap in performance, with near-human text quality and reasoning capabilities [1]. However, these systems can be prone to hallucinations [2,3], as they sometimes produce seemingly plausible but factually incorrect answers.
The general inability of such models to identify unanswerable questions [4,5] can exacerbate hallucinations, especially in enterprise settings. In such scenarios, user questions may require specific domain knowledge to be answered properly. This knowledge LLM4Eval 2024:The First Workshop on Large Language Models for Evaluation in Information Retrieval, 18 July 2024, Washington, DC Envelope zcr2105@columbia.edu (Z. Rackauckas); camara@zeta-alpha.com (A. CΓ’mara); zavrel@zeta-alpha.com (J. Zavrel) is usually out-of-domain for most LLMs, but is present in private and confidential internal documents from the company.
One such company is Infineon, a leading manufacturer of semiconductors. Given its wide range of equipment, information about its products is spread across multiple, highly technical documents, including datasheets and selection guides of hundreds of pages. Therefore, an internal retrieval augmented conversational QA system was developed by Infineon for internal users such as account managers, field application engineers, and sales operations specialists. This system allows professionals to ask questions about products from the whole catalog while in the field.
One of the features of Infineon's conversational agent is the usage of RAG-Fusion (RAGF), a technique for increasing the quality of the generated answers by generating variations of the user question and combining the rankings produced by these variations using rank-fusion methods (i.e., recriprocal rank fusion (RRF) [6]) into a ranking that has both more diverse and higher quality answers.
However, evaluating these systems bring complications common to retrieval augmented agents, especially in enterprise settings, stemming from the lack of comprehensive test datasets. Ideally, such a test set would comprise a large set of real user questions from a query log, paired with "golden answers" provided by experts. The lack of such a test set leads to two main issues. First, evaluation of answers generated by LLMs by traditional n-gram evaluation metrics such as ROUGE [7], BLEU [8], and METEOR [9] is not possible, given the lack of ground truth answers. Second, and as a consequence, evaluating the quality of the answers generated by the LLM systems would require in-domain experts (potentially from within the company) in a process that is both slow and costly [10].
One approach for tackling the lack of an extensive test set is to use synthetic queries generated by LLMs as a proxy of user queries [11]. However, the lack of in-domain knowledge of LLMs makes queries naively generated by these models unreliable and prone to hallucinations, especially when generating queries about specific products and their specifications (c.f., Table 1 for examples of real user's questions submitted to the system).
To solve this, we propose to use a process similar to InPars [12] to create a set of synthetic evaluation queries. We ask LLMs to generate queries based on portions of existing documentation injected into the prompt. To increase similarity to real user queries, we include existing user questions as few-shot examples to the prompt. With this process, we are able to generate a large set of high-quality synthetic queries for evaluating our systems. Figure 2 describes the process of generating synthetic queries and the output of a search agent. Table 2 shows a sample of these queries.
To tackle the second issue, a lack of ground truth "golden answers," we leverage an LLM-as-a-judge process, where a strong LLM is used to evaluate the quality of the answers generated by the RAG agent's LLM [13]. We then follow the practice of judging generated answers in a pairwise fashion [14], prompting the judge LLM to select the better answer between two candidates generated by different RAG pipelines. (c.f. Section 6 with details of our pipelines).
Finally, to mitigate the lack of in-domain knowledge of the judging LLM, we also annotate the relevance of the documents retrieved by the pipelines being evaluated and inject the relevant documents in the context used by the judging LLM. This allows the judging LLM to better assess for hallucinations and completeness and better align the quality of the evaluations to those conducted by experts.
This process is mediated by RAGElo 1 , a toolkit for evaluating RAG systems inspired by the Elo ranking system. RAGElo provides an easy-to-use CLI and Python library for using LLMs to evaluate retrieval results and answers produced by RAG pipelines. By combining a retrieval evaluator, a pairwise answer annotator, and an Elo-inspired tournament, RAGElo leverages powerful LLMs to agnostically annotate and rank different RAG pipelines. We notice that, although noisy, the LLM annotations generated by RAGElo are generally well aligned with experts' judgments of relative system quality, allowing for fast experimentation and comparisons between different RAG implementations without the frequent intervention of experts as annotators.
This paper evaluates multiple implementations of Infineon's retrieval augmented conversational agent using RAGElo: a traditional Retrieval-Augmented Generation and a RAG-Fusion implementation. RAG-Fusion generates multiple variations of the user question and combines the rankings produced by these queries into a more diverse set of documents. The documents are then fed into the LLM. We also analyze these same agents under a keyword-based retrieval regimen (i.e., the retriever uses BM25 to retrieve and rank documents), a dense retriever, and a hybrid retriever that combines the ranking generated by the BM25 and the dense retrievers using RRF. Our goal is to answer the following questions:
β’ Does the evaluation framework proposed by RAGElo align with the preferences of human annotators for answers generated by RAG-based conversational agents? β’ Does the RAGF approach of submitting multiple variations of the user question and combining their rankings lead to better answers?
Table 1 Sample of questions submitted by users to the Infineon RAG-Fusion system User-submitted queries What is the country of origin of IM72D128, and how does geopolitical exposure affect the market and my SAM for the microphone? What is the IP rating of mounted IM72D128? Tell me microphones that have been released since January 2023 based on the datasheet revision history. We need to confirm whether the IFX waterproof MIC has a sleeping mode and wake-up functions. While a traditional RAG agent submits only the original query to the search system, a RAGF agent first generates variations of the user query and combines the rankings induced by these queries into a final ranking using RRF. The resulting top-k passages are fed into the LLM for generating the answer to the user's query.
Several evaluation systems for RAG have been proposed to address flaws in current evaluation methods. For instance, Facts as a Function (FaaF) [15] is an end-to-end factual evaluation algorithm specially created for RAG pipelines. By creating functions from ground truth facts, FaaF focuses on the quality of generation and retrieval by calling LLMs. FaaF has substantially increased efficiency and cost-effectiveness, achieving reduced error rates compared to traditional evaluation methods. The reliance on a set of ground truths does not meet our goal of applying an automated evaluation toolkit to our pipelines. Recently, researchers have moved to eliminate the need for ground truths. This is especially important when automatically evaluating agents that retrieve highly technical documents from a large database, such as the Infineon RAGF conversational agent. RAGElo eliminates this reliance by using an LLM-as-a-judge, a method studied in numerous recent works.
SelfCheckGPT demonstrates the ability to leverage LLMs to detect and rank factuality with zero resources [16]. In addition, it has been demonstrated that GPT3.5 Turbo outperforms ground truth baselines in fact-checking with a "1/2-shot" method [17]. A model built to classify statements as true or false based on the activations of an LLM's hidden layers had up to 83% classification accuracy [18]. This evidence supports RAGElo's usage of LLM-as-a-judge.
Automated evaluation metrics can also be applied to RAG-based agents. BARTScore, an automated metric based on the BART architecture, has also outperformed most metrics on categories including factuality [19,20]. Besides automated evaluation metrics, several automated evaluation frameworks have been created with a similar goal to RAGElo. Focusing on faithfulness, answer relevance, and content relevance, RAGAS leverages LLM prompting to focus on situations where ground truths and human annotations are not present in a dataset [21]. Prediction-powered inference aims to decrease the number of human annotations needed for machine learning prediction on a dataset of images of galaxies with approximately 300,000 annotations [22]. The ARES toolkit leverages prediction-powered inference to evaluate RAG systems with fewer human annotations. Like RAGElo, ARES automatically evaluates RAG systems using synthetically generated data [23].
ARAGOG highlights Hypothetical Document Embedding (HyDE) and LLM reranking as effective methods for enhancing retrieval precision while also exploring the effectiveness of Sentence Window Retrieval and the potential of the Document Summary Index in improving RAG systems [24].
While the aforementioned frameworks evaluate answers on relevance, faithfulness, and correctness metrics, RAG can also be evaluated on noise and counterfactual robustness, negative rejection, and information integration [25].
In addition to answers, frameworks have also been created to evaluate documents. Corrective Retrieval Augmented Generation (CRAG) builds on RAG by employing a retrieval evaluator to ensure that only the optimal documents are fed into the LLM prompt prior to the answer generation phase [26].
Due to its Elo-based ranking system for answers, its use of LLM-as-a-judge, and its relevance evaluation of the intermediate retrieval steps in a RAG pipeline, RAGElo is a unique evaluation toolkit. In this study, we use it to compare a simple RAG versus a more sophisticated RAGF system on a knowledge-intensive industry-specific domain.
While answers generated by traditional retrieval augmented systems are based on a number of documents retrieved from a single query, RAGF introduces additional variation into the retrieval process. Upon receiving a query from the user, a RAGF agent leverages a large language model to generate a set of queries based on the original [27]. Table 3 shows examples of queries generated by the agent based on the query, "How to cross-sell a MEMS microphone and a XENSIV sensor to customers?".
Queries Generated from "How to cross-sell a MEMS microphone and a XENSIV sensor to customers?"
What are the key features of Infineon's MEMS microphones and XENSIV sensors that can be highlighted while cross-selling? How can Infineon's MEMS microphones and XENSIV sensors be integrated for enhanced audio and motion sensing capabilities in various applications? What are the most suitable applications and industries for Infineon's MEMS microphones and XENSIV sensors to maximize cross-selling potential?
After generating the variations for the user query, the RAGF agent submits the original and the generated queries to a retrieval system [28] that returns the top-π relevant documents π, π 1 , π 2 , β¦ π π from the set of all documents π· for each query. The rankings induced by these queries are then combined using recriprocal rank fusion (RRF) [6] into a final, higher-quality set of passages. The intuition behind RAGF is that submitting variations of the same query and combining the final rankings increases the likelihood of relevant passages being injected into the LLM prompt. In contrast, non-relevant passages retrieved by a single query are discarded. Figure 1 describes how RAG and RAGF differ. As previously discussed, one of the main issues when evaluating the quality of a QA system in an enterprise setting is that, frequently, companies do not have a large enough existing collection of queries to evaluate such systems' quality. Therefore, in this work, we propose to adopt a strategy previously used by methods for generating synthetic queries for training retrieval systems, such as InPars [12] and Promptagator [29].
Similar to these approaches, we randomly sample passages from documents within our collection and prompt an LLM to generate questions that users may ask about these portions. However, one difference in our approach to generating training queries is the size of these passages. When generating queries for training a retrieval system, we ideally want to keep the passages short to fit in the dense encoder's relatively short context windows. However, when generating queries for evaluating QA systems (including retrieval augmented), we are not bound to the limit of the embedding model used for retrieval. Rather, a longer passage may yield questions that require multiple shorter passages to be answered. Therefore, we submit relatively long passages to the LLMs. Specifically, each passage is extracted from up to ten pages of PDF documents (about 2000 tokens 2 )
To keep the questions generated as diverse as possible, we prompt four different LLMs to generate up to ten questions based on the same documents. Our test set collection contains a mix of queries generated by OpenAI's GPT-4 turbo [30] and Anthropic's Claude-3 [31] Opus, Sonnet, and Haiku models 3 . From a set of π = 840 queries, we sampled 200 queries across all four models. Half of the queries are selected from GPT-4 generated queries, and the other half from Claude 3 queries. Among the Claude 3 queries, to ensure the quality of the queries and their diversity, we again sample according to each model size. Ultimately, our test set contains 100 queries from GPT-4-turbo, 50 from Claude 3 Opus, 30 from Sonnet, and 20 from Haiku.
Finally, to increase the quality of the generated queries, We asked for an account manager, a sales operations specialist, a marketing representative, and a business development manager to create queries that they would submit to the conversational agent from the perspective of their role. They were instructed to produce queries regarding products from the XENSIV sensor product line, consisting of MEM microphones, radar, current, magnetic, pressure, and environmental sensors. We compiled a list of 23 of these queries to use as a base for experimentation and used them as few-shot examples in the query generation prompt. Figure 2 illustrates our method for generating synthetic queries based on existing user queries and document passages.
Even with a suitable set of synthetic questions for evaluating our RAG conversational agent, assessing whether a given answer properly answers a question is not trivially done. If a ground-truth "golden answer" is available, one can use traditional syntactic-based 2 all LLMs used in our experiments had long context windows of 128k or 200k tokens.
We did not use GPT-3.5 or open source models due to their shorter context window at the time of writing.
Are there any self-diagnosis features available in the KP23x analog sensor?
Agent B:The KP23x analog sensor series includes self-diagnosis capabilities, such as built-in monitoring functionality and ISO 26262 (β¦)
Agent A: Yes, the KP23x sensor has some self-diagnosis features.
Both assistants correctly answer the question (β¦) according to document <doc_id>, the sensors have features such as (β¦) Based on these observations, Assistant B provides a more accurate, relevant, and focused response to the user's question regarding the KP23x series sensors. The RAGElo evaluation pipeline. First, documents retrieved by the agents are evaluated pointwise according to their relevance to the user's question. Then, the agents' answers are evaluated pairwise, using the retrieved relevant documents from both agents as reference.
metrics such as BLEU, METEOR or ROUGE [8,9,7]. Without such reference answers, one would require human annotators with a considerable understanding of the question's topic to manually assess the quality of the answers produced by each system. However, this is a costly process.
Alternatively, several LLM-as-a-Judge methods have been proposed, where another LLM is asked to evaluate the quality of answers generated by other LLMs. Nevertheless, in an enterprise setting, the answers usually require the LLM to access knowledge not present in their training datasets but rather contained in documents internal to the company. This is usually accomplished using a RAG pipeline like the one described above. Therefore, the judging LLM also needs access to similar knowledge to accurately evaluate the agent's answers' quality.
Therefore, in this work, we rely on RAGElo, an open-source RAG evaluation toolkit that evaluates the answers generated by each agent and the documents retrieved by them. By injecting the annotation of retrieved documents, pooled by the agents being evaluated, on the answer evaluation step, this method allows for the judging LLM to evaluate if the generated answer was able to use all the information available about the question properly and to check for any hallucinations. As the documents used for generating the answers are included in the answer evaluating prompt, an agent that incorrectly cites information from a source or refers to information not present in these documents is likely hallucinating and should have its evaluation adjusted accordingly. As we explore in Section 8, this two-step process results in a high correlation between human expert annotators and the judging LLM, enabling higher reliability and trust when evaluating different RAG pipelines. This process is also illustrated in Figure 3.
While our main evaluation focuses on the pairwise comparison between the two agents, RAGElo also allows us to evaluate answers pointwise. In this setting, similar to other works [32], we prompt the judging LLM to evaluate the answers according to multiple criteria:
β’ Relevance: Does the answer address the user's question?
β’ Accuracy: Is the answer factually correct, based on the documents provided? β’ Completeness: Does the answer provide all the information needed to answer the user's question? β’ Precision: If the user's question is about a specific product, does the answer provide the answer for that specific product?
We not only experiment with different search agents (i.e., RAG and RAGF. We are also interested in how different retrieval methods may impact the quality of the final answers generated by these agents.
Our corpus consists of passages extracted from the Infineon XENSIV Product Selection Guide, a 117-page document with detailed information on every product in the XENSIV family. This document included technical information about all Infineon XENSIV sensors, consumer and automotive sensor applications, guidance in selecting the correct sensor, and other comprehensive and detailed information about the product line.
The passages are embedded using multilingual-e5-base [33] 4 and indexed using OpenSearch, allowing us to perform both KNN-based vector search, keyword-based search with BM25 [34], and RRF based hybrids thereof.
We mainly evaluate two agents: a naive RAG pipeline, where the agent first retrieves top-π passages that are then templated into a prompt, and the Infineon RAG-Fusion (RAGF) agent. Upon receiving a query, a naive RAG agent takes the following actions:
1. Retrieve the top k most relevant passages from the search system. 2. Perform a Chat Completions API call, prompting the LLM with instructions for generating an answer based on the five relevant passages.
Meanwhile, the Infineon RAGF conversational assistant uses a similar framework and performs the following steps upon receiving a query:
1. Perform a Chat Completions API call to generate four new queries based on the original query using a prompt tailored to the agent's original goal. 2. Retrieve the top k most relevant passages for each query.
While LLM-as-a-judge is a theoretically viable algorithm for rating RAG and RAGF answers, we must establish whether the results agree with the annotations of domain experts.
Figure 4 provides a Bland-Altman plot to visually represent the LLM and human judgments' agreement. The bias of approximately 0.12 indicates that, on average, LLM scores were slightly higher than human scores. The limits of agreement ranged from approximately -1.17 to 1.41. demonstrating substantial variability in the difference between LLM and human evaluators.
Next, we compared LLM-as-a-judge to expert annotators with Kendall's π. Kendalls π is a nonparametric measure that quantifies the degree of association between two monotonic continuous or ordinal variables by calculating the proportion of concordance and discordance among pairwise ranks, offering valuable insight into their rank correlation [35,36]. We used the SciPy Stats Kendalltau function to calculate a tau-b score and a p-value for the combined ratings of all columns, flattened into a 1-D array with RAG and RAGF ratings combined [37]. The tau-b value, a nonparametric measure of association, is calculated using the following formula [38]:
P represents the number of concordant pairs, Q represents the number of discordant pairs, T represents the number of ties exclusive to x, and U represents the number of ties exclusive to y. This test returned π β 0.56, indicating a moderate, positive correlation [39] with a p-value against a null hypothesis of no association of π < 0.01 (99.99% confidence level). For comparison, in similar experiments judging human versus LLM judgments, Faggioli et al. found π values of π = 0.76 and π = 0.86 [40].
Following the same methodology, we also calculated Spearman's π, a similar nonparametric correlation measure. This resulted in π β 0.59 with π < 0.01, demonstrating a statistically significant, moderate positive correlation [36].
We assessed document retrieval quality using Mean Reciprocal Rank@5 (MRR@5), which averages the inverse ranks of the first relevant result within the top five positions across all queries. The formula is given by
where |π| is the total number of queries and rank π is considered only if it's within the top five, otherwise it counts as zero [41]. MRR@5 scores were calculated for each agent and each retrieval method considering two categories:
1. MRR@5 score for documents deemed "somewhat relevant" or "very relevant." 2. MRR@5 score for documents deemed "very relevant."
The results can be seen below in Table 4.
Mean MRR@5 scores for RAG vs RAG-F. The retrieval method columns indicate if the retrieval component used was vector search only (KNN), keywords only (BM25) or hybrid (KNN and BM25, combined with RRF).
We then ran RAGElo games to evaluate end-to-end answer quality of RAG vs RAGF with different base retriever configurations a task that cannot rely on standard Information Retrieval metrics. These RAGElo results show more victories for RAGF than RAG; For example, when using BM25 as a base retriever, RAGF won 49% of the games, RAG won 14.5%, and RAG and RAGF are tied in 36.5% of the times. The resulting Elo scores for all six variants are shown in table 6, which give a robust ranking of the systems, without reliance on a gold standard. It is interesting to see, that for both RAGF as well as RAG, BM25 is a strong baseline that is not surpassed by generic embeddings in these experiments. Next, we compared the RAGElo outcome to the preference of our Infineon human annotator. We performed two-tailed paired t-tests to compare RAG against RAGF on each category from the Infineon representatives' human evaluations with πΌ = .05. As expected, due to its larger variety of retrieved results, RAGF significantly outperforms RAG in completeness at the 95% confidence level with π β 0.01 However, on the precision of answers, RAG significantly outperformed RAGF at the 95% confidence level with π β 0.04.
As observed above, we found statistically significant, moderate positive correlations between LLM ratings and human annotations. This indicates a consistent association between the ratings from LLM-as-a-judge and those by Infineon experts. We find that on average, LLM scores are slightly higher than those of human annotators. This means that while relevance judgements on individual queries should not be fully reliable, and IR metrics derived from LLM-as-a-judge should not be equated with regular relevance scores without further calibration, we can still make good use of this approach to rankorder systems. These findings collectively support the validity of our LLM evaluation method, which assesses conversational system outputs based on a combination of relevance, accuracy, completeness, and recall.
The style of evaluation and the different dimensions it takes into account are specified in the prompts given to the LLM in the RAGElo evaluation, which are provided in Appendix A. Specifically, while the initial LLM-as-a-judge is given specific criteria to focus on only four categories, we instructed RAGElo's impartial judge LLM to value more than the initial four categories: Your evaluation should consider factors such as comprehensiveness, correctness, helpfulness, completeness, accuracy, depth, and level of detail of their responses.
Since RAGF significantly outperformed RAG in the completeness category, the RAGElo judge LLM likely weighed completeness higher than precision. In addition, based on manual observation of a small random sample of answers, RAGF produced more comprehensive answers and featured higher depth and level of detail due to the multiple query generation. However, games where RAG won were most likely influenced by a significantly more precise answer than that of RAGF. While RAGF values comprehensive answers that offer multiple perspectives to the user, RAG produces shorter answers that answer the original query only. Since completeness is defined as the extent to which a user's question was answered, it can be presumed that RAGF's longer and more comprehensive answers may tend to be more complete. And since precision relates to the agent mentioning the correct product or product family, it can be presumed that RAGF's longer answers have more room to consider other products or product families, leading to reduced answer precision. While the human annotation was done by Infineon experts, different humans may rate answers differently, even if following the same set of criteria.
A larger number of documents or a database of non-technical documents may have led to a different outcome. RAGF can be applied to not only Infineon documents but also any documents database to retrieve. This includes not only enterprise uses but also uses in education, such as mathematics and language learning. The algorithm can be tuned to different use cases by tweaking the internal LLM prompt. For example, the Infineon RAGF bot was prompted to "think like an engineer." However, an educator RAGF bot could be prompted to "think like a teacher." Future work includes exploring other applications of RAGF, especially in education. In addition, we will experiment with different prompts for both LLM-as-a-judge and RAGElo while using different quantities and types of documents with the same retrieval algorithms.
Based on the calculated MMR@5 scores, we found that the RAGF agent mostly outperforms the RAG agent in ranking both highly relevant and somewhat relevant documents retrieved. This evidence search on multiple query variants produced, on average, slightly more higher-ranked relevant documents than using only the original user query. We also see that using vector search with embeddings is not a silver bullet, as for our test queries, BM25 seriously outperforms it. Since retrieval quality is highly dependent on the quality of the embeddings and their fit to the domain, this outcome will likely be changed by fine-tuning the embeddings, and adding additional intelligent re-rankers, which we leave here for future work, as the evaluation framework would remain the same.
Overall, we found that the evaluation framework proposed by RAGElo positively aligns with the preferences of human annotators for RAG and RAGF with due caution due to a moderate correlation and variability of scoring. We found that the RAGF approach leads to better answers most of the time, according to the RAGElo evaluation. According to expert scoring, the RAGF approach significantly outperforms in completeness compared to RAG but significantly underperforms in precision compared to RAG. Based on these results, we cannot confidently assert that RAGF's approach leads to better answers generally. However, the results do support that RAGF's approach leads to more complete answers and a higher proportion of better answers under evaluation by RAGElo.
Since RAGElo is generally applicable to all retrieval-augmented algorithms, in future work, we also intend to test different agents other than RAG and RAGF, including those with different reranking algorithms, different embedding models, and different LLMs. In addition, due to RAGF's underperformance in document relevance, we may also leverage CRAG to reduce this gap. We will also investigate the reflection of human sensitivity in expert ratings, especially whether the LLMs should or can reflect human sensitivities.
We used the default RAGElo's ReasonerEvaluator, which has the following system prompt: You a r e an e x p e r t document a n n o t a t o r . Your j o b i s t o e v a l u a t e whether a document c o n t a i n s r e l e v a n t i n f o r m a t i o n t o answer a u s e r ' s q u e s t i o n .
P l e a s e a c t a s an i m p a r t i a l r e l e v a n c e a n n o t a t o r f o r a s e a r c h e n g i n e . Your g o a l i s t o e v a l u a t e t h e r e l e v a n c y o f t h e documents g i v e n a u s e r q u e s t i o n .
You s h o u l d w r i t e one s e n t e n c e e x p l a i n i n g why t h e document i s r e l e v a n t o r not f o r t h e u s e r q u e s t i o n . A document can be : β Not r e l e v a n t : The document i s not on t o p i c . β Somewhat r e l e v a n t : The document i s on t o p i c but d o e s not f u l l y answer t h e u s e r q u e s t i o n . β Very r e l e v a n t : The document i s on t o p i c and an s we r s t h e u s e r ' s q u e s t i o n .
[ u s e r q u e s t i o n ] { query } [ document c o n t e n t ] { document }
For the pointwise evaluator used in Section 5.1, we used the following prompt with
RAGElo's CustomPromptAnswerEvaluator:
You a r e an i m p a r t i a l j u d g e f o r e v a l u a t i n g t h e q u a l i t y o f t h e r e s p o n s e s p r o v i d e d by an AI a s s i s t a n t t a s k e d t o answer u s e r s ' q u e s t i o n s about t h e c a t a l o g u e o f IoT s e n s o r s produced by I n f i n e o n .
You w i l l be g i v e n t h e u s e r ' s q u e s t i o n and t h e answer produced by t h e a s s i s t a n t . The agent ' s answer was g e n e r a t e d based on a s e t o f documents r e t r i e v e d by a s e a r c h e n g i n e . You w i l l be p r o v i d e d with t h e r e l e v a n t documents r e t r i e v e d by t h e s e a r c h e n g i n e . Your t a s k i s t o e v a l u a t e t h e answer ' s q u a l i t y based on t h e ## S t e p s t o e v a l u a t e an answer : 1 . * * Understand t h e u s e r ' s i n t e n t * * : E x p l a i n i n your own words what t h e u s e r ' s i n t e n t i s , g i v e n t h e q u e s t i o n . 2 . * * Check i f t h e answer i s c o r r e c t * * : Think s t e p βbyβs t e p whether t h e answer c o r r e c t l y a ns w er s t h e u s e r ' s q u e s t i o n . 3 . * * E v a l u a t e t h e q u a l i t y o f t h e answer * * : E v a l u a t e t h e q u a l i t y o f t h e answer based on i t s r e l e v a n c e , acc uracy , and c o m p l e t e n e s s . 4 . * * A s s i g n a s c o r e * * : Produce a s i n g l e l i n e JSON o b j e c t with t h e f o l l o w i n g keys , each with a s i n g l e s c o r e between 0 and 2 , where 2 i s t h e h i g h e s t s c o r e on t h a t a s p e c t : β " r e l e v a n c e " β 0 : The answer i s not r e l e v a n t t o t h e u s e r ' s q u e s t i o n . β 1 : The answer i s p a r t i a l l y r e l e v a n t t o t h e u s e r ' s q u e s t i o n . β 2 : The answer i s f u l l y r e l e v a n t t o t h e u s e r ' s q u e s t i o n . β " a c c u r a c y " β 0 : The answer i s f a c t u a l l y i n c o r r e c t . β 1 : The answer i s p a r t i a l l y c o r r e c t . t h e u s e r m u l t i p l e p e r s p e c t i v e s i n a d d i t i o n t o but s t i l l r e l e v a n t t o t h e i n t e n t o f t h e o r i g i n a l q u e s t i o n . " , )
This generates 15 random games between two agents per query (i.e., all possible unique games for 6 agents) and tells the evaluator that:
β’ The answers do not include specific citations to any passage (has_citations=False)
β’ Include the full text of the retrieved passages in the evaluation prompt (include_raw_documents=True) β’ Inject the output of the retrieval evaluator into the prompt (include_annotations=True) β’ Ignore any passage with a relevance score below 2 (document_relevance_threshold=2) β’ Consider these factors when selecting the best answer factors=β¦)
These parameters produce the following final prompt used for evaluating the answers:
P l e a s e a c t a s an i m p a r t i a l j u d g e and e v a l u a t e t h e q u a l i t y o f t h e r e s p o n s e s p r o v i d e d by two AI a s s i s t a n t s t a s k e d t o answer t h e q u e s t i o n below based on a s e t o f documents r e t r i e v e d by a s e a r c h e n g i n e .
You s h o u l d c h o o s e t h e a s s i s t a n t t h a t b e s t an s we r s t h e u s e r q u e s t i o n based on a s e t o f r e f e r e n c e documents t h a t may o r may not be r e l e v a n t .
For each r e f e r e n c e document , you w i l l be p r o v i d e d with t h e t e x t o f t h e document a s w e l l a s r e a s o n s why t h e document i s o r i s not r e l e v a n t . Answers a r e c o m p r e h e n s i v e i f they show t h e u s e r m u l t i p l e p e r s p e c t i v e s i n a d d i t i o n t o but s t i l l r e l e v a n t t o t h e i n t e n t o f t h e o r i g i n a l q u e s t i o n . D e t a i l s a r e o n l y u s e f u l i f they answer t h e u s e r ' s q u e s t i o n . I f an answer c o n t a i n s nonβr e l e v a n t d e t a i l s , i t s h o u l d not be p r e f e r r e d o v e r one t h a t o n l y u s e s r e l e v a n t i n f o r m a t i o n .
Begin your e v a l u a t i o n by e x p l a i n i n g why each answer c o r r e c t l y ans we rs t h e u s e r ' s q u e s t A f t e r p r o v i d i n g your e x p l a n a t i o n , output your f i n a l v e r d i c t by s t r i c t l y f o l l o w i n g t h i s format : "
We thank Brooks Felton from Infineon for his support during this work. We also thank the Infineon sales team for providing valuable feedback.