Preface: The First Workshop on Evaluation Methodologies, Testbeds and Community for Information Access Research (EMTCIR 2024) Makoto P. Kato1 , Noriko Kando2 , Charles L. A. Clarke3 and Yiqun Liu4 1 University of Tsukuba, Japan 2 National Institute of Informatics, Japan 3 University of Waterloo, Canada 4 Tsinghua University, China Evaluation campaigns, where researchers share important tasks, collaboratively develop test collec- tions, and have discussion to advance technologies, are still important events to strategically address core challenges in information access research. The goal of this workshop is to discuss information access tasks that are worth addressing as a community, share new resources and evaluation method- ologies, and encourage researchers to ultimately propose new evaluation campaigns in NTCIR, TREC, CLEF, FIRE, etc. To accept a wide range of contributions, EMTCIR 2024 called for four types of papers, namely, emerging task, ongoing task, resource, and evaluation papers. As a result, we received five submissions in total: an emerging task paper, two ongoing task papers, a resource paper, and an evaluation paper. Each paper was reviewed by three program committee members. All the papers were accepted at the workshop and will be discussed in-person on December 12, 2024. The first half of the workshop mainly focus on the presentation of accepted contributions, while the latter half focuses on round-table discussion for exploring new tasks. After the round-table discussion, each table is expected to have a short presentation on a new task. We greatly thank the program committee members for providing constructive reviews, and are grateful to the workshop co-chairs and reviewers of SIGIR-AP 2024 for their useful feedback and support. We hope that new evaluation campaigns emerge from this workshop. Program Committee Members • Qingyao Ai, Tsinghua University • Marwah Alaofi, RMIT University • Chung-Chi Chen, National Institute of Advanced Industrial Science and Technology • Charles L. A. Clarke, University of Waterloo • Makoto P. Kato, University of Tsukuba • Yiqun Liu, Tsinghua University • Alistair Moffat, The University of Melbourne • Jian-Yun Nie, University de Montreal • Tetsuya Sakai, Waseda University • Shoko Wakamiya, Nara Institute of Science and Technology EMTCIR ’24: The First Workshop on Evaluation Methodologies, Testbeds and Community for Information Access Research, December 12, 2024, Tokyo, Japan Envelope-Open mpkato@acm.org (M. P. Kato); kando@nii.ac.jp (N. Kando); claclark@uwaterloo.ca (C. L. A. Clarke); yiqunliu@tsinghua.edu.cn (Y. Liu) Orcid 0000-0002-9351-0901 (M. P. Kato); 0000-0002-2133-0215 (N. Kando); 0000-0001-8178-9194 (C. L. A. Clarke); 0000-0002-0140-4512 (Y. Liu) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Preface: The First Workshop on User Modelling in Conversational Information Retrieval (UM-CIR) Praveen Acharya1 , Gareth J. F. Jones1 , Xiao Fu2 , Aldo Lipani2 , Fabio Crestani3 and Noriko Kando4 1 Dublin City University, Dublin, Ireland 2 University College London, London, U.K. 3 Universitá della Svizzera Italiana, Lugano, Switzerland 4 National Institute of Informatics, Tokyo, Japan Conversational Information Retrieval (CIR) in which a user engages in a multi-step dialogue with a search system has attracted growing interest in recent years. This attention has expanded further since the emergence of conversational agents that leverage generative AI methods. Within the information retrieval community, a substantial body of research has emerged, particularly centred around initiatives such as the TREC Conversational Assistance Track (CAsT)1 and Interactive Knowledge Assistance Track (iKAT)2 . These tracks have been instrumental in providing datasets that facilitate research in CIR and enable a comparative analysis of various approaches to conversational search. Most of the existing efforts within these tracks have concentrated on the interactive dialogue between the searcher and the CIR system. The tasks generally overlook the potential contribution of User modelling for effective CIR. Recognizing the importance of this dimension the goal of the workshop was to create a collaborative framework for investigating user modelling and its evaluation in the context of CIR. Participants were invited to share their insights and proposals regarding User modelling in CIR, particularly in relation to algorithm design, system personalization, and the methods through which these models can be simulated and assessed. By fostering dialogue and collaboration among researchers and practitioners, we aim edto deepen the community’s understanding of how effective User modelling might enhance conversational search experiences and lead to more refined and user-centred retrieval systems. The workshop included a keynote presentation entitles “Immersive Personalisation for Conversational Information-seeking: The Future of Interactive Information Retrieval” by Johanne R. Trippas, RMIT University, Australia, three reviewed paper presentations, and structured discussion between the workshop participants. Acknowledgments This work was conducted with the financial support of the Science Foundation Ireland Centre for Research Training in Artificial Intelligence under Grant No. 18/CRT/6223 and partially as part of the SFI ADAPT Centre at DCU (Grant No. 13/RC/2106_P2) (www.adaptcentre.ie), Swiss Federal Institute of Technology (ETH) Leading House Asia Project on “Evaluation of Empathic Conversational Search for Depression Assessment and Support”, Swiss National Science Foundation (SNSF) Grant “Personality and Conversational Information Access”, and National Institute of Informatics (NII) Internship programme and NII MoU Grant. UM-CIR 2024: The 1st Workshop on User Modelling in Conversational Information Retrieval, December 12, 2024, Tokyo, Japan Envelope-Open praveen.acharya2@mail.dcu.ie (P. Acharya); Gareth.Jones@dcu.ie (G. J. F. Jones); xiao.fu.20@ucl.ac.uk (X. Fu); aldo.lipani@ucl.ac.uk (A. Lipani); fabio.crestani@usi.ch (F. Crestani); kando@nii.ac.jp (N. Kando) Orcid 0000-0001-5181-9831 (P. Acharya); 0000-0003-2923-8365 (G. J. F. Jones); 0000-0003-4676-8608 (X. Fu); 0000-0002-3643-6493 (A. Lipani); 0000-0001-8672-0700 (F. Crestani); 000-0002-2133-0215 (N. Kando) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 https://www.treccast.ai/ 2 https://www.trecikat.com/