The LLMJudge challenge1 is organized as part of the LLM4Eval2 workshop [1] at SIGIR 2024. Test collections are essential for evaluating information retrieval (IR) systems. The evaluation and tuning of a search system is largely based on relevance labels, which indicate whether a document is useful for a specific search and user. However, collecting relevance judgments on a large scale is costly and resource-intensive. Consequently, typical experiments rely on third-party labelers who may not always produce accurate annotations. The LLMJudge challenge aims to explore an alternative approach by using LLMs to generate relevance judgments. Recent studies have shown that LLMs can generate reliable relevance judgments for search systems. However, it remains unclear which LLMs can match the accuracy of human labelers, which prompts are most efective, how fine-tuned open-source LLMs compare to closed-source LLMs like GPT-4, whether there are biases in synthetically generated data, and if data leakage afects the quality of generated labels. This challenge will investigate these questions, and the collected data will be released as a package to support automatic relevance judgment research in information retrieval and search.
Automatic relevance judgment has recently received significant attention in the Information Retrieval
(IR) community. In earlier studies, Faggioli et al. [
The challenge will be, given the query and document as input, how they are relevant. Here, we use
four-point scale judgments to evaluate the relevance of the query to document as follows:
• [
• [0] Irrelevant: The passage has nothing to do with the query.
The task is, by providing the datasets that include queries, documents, and query-document files to
participants, to ask LLMs to generate a score [
The LLMJudge challenge dataset is built upon the passage retrieval task dataset of the TREC 2023 Deep
Learning track3 (TREC-DL 2023) [
Participants’ results will then be evaluated in two methods after submission: • automated evaluation metrics on human labels in the test set hidden from the participants; • system ordering evaluation of multiple search systems on human judgments and LLM-based judgments
In order to evaluate the quality of the generated labels, we used Cohen’s to see the labeler’s agreement with LLMJudge test data at query-document level and the Kendall’s to check the labeler’s agreement with LLMJudge test data on system ordering, i.e., the runs that submitted to TREC DL 2023. In total, we had 39 submissions (i.e., the 39 labelers) from 7 groups from National Institute of Standards and 3https://microsoft.github.io/msmarco/TREC-Deep-Learning.html
1 0.98 0.96 0.94 0.92 tau0.9 0.88 0.86 0.84 0.82 0.8 0
NISTRetrieval-instruct0 prophet
RMITIR-llama70B 0.05 0.1 0.15 0.2 0.3 0.35 0.4 0.45 0.5 Technology (NIST), RMIT University, The University of Melbourne, University of New Hampshire, University of Waterloo, Included Health, and University of Amsterdam.
Figure 1 shows the performance of submitted labelers on the LLMJudge test set. The x-axis represents Cohen’s , and the y-axis shows the labelers’ agreement on system ordering. Labelers exhibit low variability in Kendall’s but greater variability in Cohen’s . Most labelers cluster within a narrow range of values, indicating consistent system rankings but more variation in inter-rater reliability, as measured by Cohen’s . This suggests that while labelers generally agree on rankings, their exact labels are less consistent, leading to the observed variability in .
The challenge is organized as a joint efort by the University College London, Microsoft, the University of Amsterdam, the University of Waterloo, and the University of Padua. The views expressed in the content are solely those of the authors and do not necessarily reflect the views or endorsements of their employers and/or sponsors. This work is supported by the Engineering and Physical Sciences Research Council [EP/S021566/1], the EPSRC Fellowship titled “Task Based Information Retrieval” [EP/P024289/1], CAMEO, PRIN 2022 n. 2022ZLL7MW.