Preface

QPP++ 2023: Query Performance Prediction and Its Evaluation in New Tasks is the first edition of a workshop that aims to foster a discussion within the community on how Query Performance Prediction (QPP) can be applied to new techniques in Information Retrieval (IR) and how such techniques can be exploited to define new QPP models. This first edition was hosted by the European Conference on Information Retrieval (ECIR) 2023 in Dublin (Ireland).

QPP++ 2023 received nine scientific submissions, of which seven papers (four long and three short) were accepted. Two to three program committee members reviewed each submission, and the program chairs oversaw the reviewing. The accepted papers included authors from 8 countries and 14 institutions, as some publications resulted from international collaborations. Researchers addressed the following challenges: QPP for conversational search, known-item search and passage retrieval, QPP in the learning-to-rank and neural information retrieval domains, issues with using correlation metrics to evaluate QPP, QPP evaluation using pointwise approaches, continuous evaluation, and using information theory for QPP.

Motivation

The advent of large language models and the rise of new tasks, such as conversational search, semantic search and question answering, enabled by the availability of new powerful technological tools, have led to a previously unseen rapid growth in the variety and quality of Information Retrieval (IR) systems. Several ancillary research fields have also flourished due to the scientific uptake of new Natural Language Processing (NLP) methodologies, facilitating advancement in new IR tasks. The Query Performance Prediction and Its Evaluation in New Tasks (QPP++ 2023) workshop [3] aimed to further fuel such growth in the renowned and important area of Query Performance Prediction (QPP).

The QPP task is defined as estimating search effectiveness in the absence of human relevance judgments [1]. Since its introduction at the beginning of the 21st century, QPP has established itself as an essential tool in numerous tasks, including model selection [1,14], query suggestion [1,14], and rank fusion [11]. The QPP++ 2023 workshop was a collaborative effort of researchers to master the new tools made available by the NLP community and learn how to effectively use them for the QPP task. The workshop focused on applying QPP in traditional scenarios, such as ad-hoc retrieval, and in new domains, including conversational and semantic search, passage retrieval, and question answering. QPP++ 2023 also allowed the community to reexamine past weaknesses and challenges linked to the QPP task, such as its evaluation, while establishing a roadmap to organize and guide the community's future efforts to advance the QPP research field.

QPP and Novel Search Paradigms

Given the recent developments in IR, the prediction quality of existing QPP approaches may be significantly affected in new domains and scenarios for the following three reasons. First, some of the traditional predictors exploit statistics derived from the collection [6], while new IR models often use indexes of embeddings or apply machine learning to re-rank documents [8]. Second, the vast majority of the recently developed retrieval models in IR utilize semantic information that, with a few notable exceptions [9,13], is rarely exploited by QPP models. This, in turn, impairs the performance of traditional QPP models applied on IR systems based on new paradigms [4]. Finally, QPP can be used for new processes such as selective query processing [2].

The QPP++ 2023 workshop aimed to provide a platform for the community to jointly discuss ways to address these challenges and create a better alignment between the latest technologies, retrieval models, and QPP approaches. Along with the challenges mentioned above, the recent advances in NLP present great opportunities for enhancing the state of the art in QPP. The workshop also sought to encourage collaboration between researchers to exploit these opportunities.

QPP and its Evaluation on New Tasks

The quality of QPP methods is typically evaluated by computing the correlation between the scores assigned to queries by a QPP method and the true performance values, e.g., Average Precision (AP), attained for these queries using relevance judgements. Previous research demonstrated the unreliability of this approach when multiple experimental factors (i.e., IR models, corpora, and predictors) are considered [7,12,5]. In addition, researchers demonstrated that high correlation does not necessarily translate to improved retrieval effectiveness [10,7]. These issues are further exacerbated in new domains, such as question answering or conversational search, where the evaluation of the retrieval models is often more challenging. The QPP++ 2023 workshop aimed at fostering discussion in the community regarding these challenges.

The workshop provided a forum for researchers and practitioners to discuss the following key research challenges emerging following the recent advances in IR:

-Can existing QPP techniques be exploited, or which new QPP theories and models need to be devised, for new tasks, such as image retrieval, passageretrieval, question answering, and conversational search? -How can new technologies, such as contextualized embeddings, large language models, and neural networks be exploited to improve QPP? -How should QPP techniques be evaluated, including best practices, datasets, and resources? -Should QPP be evaluated in the same manner for different IR tasks? -What changes should we make to the QPP evaluation paradigm to accommodate new domains and IR techniques?

The workshop is expected to have two main outcomes:

-We intend to compile the workshop proceedings from the submitted papers. The proceedings will be published in the CEUR-WS.org proceedings series. -We intend to draft a position paper describing the roadmap identified during the discussions and submit it to the SIGIR forum.

Organization