LLM4Eval 2024
Large Language Models for Evaluation in Information Retrieval 2024

Table of Contents

• Preface
  Summary: There were 26 papers submitted for peer-review to this workshop. Out of these, 7 papers were accepted for this volume as original research papers. Additionally, other 16 papers, 11 original contributions and 5 already published works, were accepted and presented during the workshop.

LLMJudge Challenge Overivew

• LLMJudge: LLMs for Relevance Judgments 1-3
  Hossein A. Rahmani , Emine Yilmaz , Nick Craswell , Bhaskar Mitra , Paul Thomas , Charles L. A. Clarke , Mohammad Aliannejadi , Clemencia Siro , Guglielmo Faggioli

Research Papers

• The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches 4-18
  Bhashithe Abeysinghe , Ruhan Circi
• Exploring Large Language Models for Relevance Judgments in Tetun 19-30
  Gabriel de Jesus , Sérgio Nunes
• EXAM++: LLM-based Answerability Metrics for IR Evaluation 31-50
  Naghmeh Farzi , Laura Dietz
• A Novel Evaluation Framework for Image2Text Generation 51-65
  Jia-Hong Huang , Hongyi Zhu , Yixian Shen , Stevan Rudinac , Alessio M. Pacces , Evangelos Kanoulas
• Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction 66-91
  Hyunwoo Kim , Yoonseo Choi , Taehyun Yang , Honggu Lee , Chaneon Park , Yongju Lee , Jin Young Kim , Juho Kim
• Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework 92-112
  Zackary Rackauckas , Arthur Câmara , Jakub Zavrel
• Toward Automatic Relevance Judgment using Vision-Language Models for Image-Text Retrieval Evaluation 113-123
  Jheng-Hong Yang , Jimmy Lin