Evaluation campaigns, where researchers share important tasks, collaboratively develop test collections, and have discussion to advance technologies, are still important events to strategically address core challenges in information access research. The goal of this workshop is to discuss information access tasks that are worth addressing as a community, share new resources and evaluation methodologies, and encourage researchers to ultimately propose new evaluation campaigns in NTCIR, TREC, To accept a wide range of contributions, EMTCIR 2024 called for four types of papers, namely, emerging task, ongoing task, resource, and evaluation papers. As a result, we received five submissions in total: an emerging task paper, two ongoing task papers, a resource paper, and an evaluation paper. Each paper was reviewed by three program committee members. All the papers were accepted at the workshop and will be discussed in-person on December 12, 2024. The first half of the workshop mainly focus on the presentation of accepted contributions, while the latter half focuses on round-table discussion for exploring new tasks. After the round-table discussion, each table is expected to have a short presentation on a new task.
CEUR ceur-ws.org
Praveen Acharya1, Gareth J. F. Jones1, Xiao Fu2, Aldo Lipani2, Fabio Crestani3 and Noriko Kando4
Conversational Information Retrieval (CIR) in which a user engages in a multi-step dialogue with a search system has attracted growing interest in recent years. This attention has expanded further since the emergence of conversational agents that leverage generative AI methods. Within the information retrieval community, a substantial body of research has emerged, particularly centred around initiatives such as the TREC Conversational Assistance Track (CAsT)1 and Interactive Knowledge Assistance Track (iKAT)2. These tracks have been instrumental in providing datasets that facilitate research in CIR and enable a comparative analysis of various approaches to conversational search. Most of the existing eforts within these tracks have concentrated on the interactive dialogue between the searcher and the CIR system. The tasks generally overlook the potential contribution of User modelling for efective CIR.
Recognizing the importance of this dimension the goal of the workshop was to create a collaborative framework for investigating user modelling and its evaluation in the context of CIR. Participants were invited to share their insights and proposals regarding User modelling in CIR, particularly in relation to algorithm design, system personalization, and the methods through which these models can be simulated and assessed. By fostering dialogue and collaboration among researchers and practitioners, we aim edto deepen the community’s understanding of how efective User modelling might enhance conversational search experiences and lead to more refined and user-centred retrieval systems.
The workshop included a keynote presentation entitles “Immersive Personalisation for Conversational Information-seeking: The Future of Interactive Information Retrieval” by Johanne R. Trippas, RMIT University, Australia, three reviewed paper presentations, and structured discussion between the workshop participants.
Acknowledgments This work was conducted with the financial support of the Science Foundation Ireland Centre for Research Training in Artificial Intelligence under Grant No. 18/CRT/6223 and partially as part of the SFI ADAPT Centre at DCU (Grant No. 13/RC/2106_P2) (www.adaptcentre.ie), Swiss Federal Institute of Technology (ETH) Leading House Asia Project on “Evaluation of Empathic Conversational Search for Depression Assessment and Support”, Swiss National Science Foundation (SNSF) Grant “Personality and Conversational Information Access”, and National Institute of Informatics (NII) Internship programme and NII MoU Grant. UM-CIR 2024: The 1st Workshop on User Modelling in Conversational Information Retrieval, December 12, 2024, Tokyo, Japan