QPP++ 2023: Query Performance Prediction and Its Evaluation in New Tasks is the first edition of a workshop that aims to foster a discussion within the community on how Query Performance Prediction (QPP) can be applied to new techniques in Information Retrieval (IR) and how such techniques can be exploited to define new QPP models. This first edition was hosted by the European Conference on Information Retrieval (ECIR) 2023 in Dublin (Ireland). QPP++ 2023 received nine scientific submissions, of which seven papers (four long and three short) were accepted. Two to three program committee members reviewed each submission, and the program chairs oversaw the reviewing. The accepted papers included authors from 8 countries and 14 institutions, as some publications resulted from international collaborations. Researchers addressed the following challenges: QPP for conversational search, known-item search and passage retrieval, QPP in the learning-to-rank and neural information retrieval domains, issues with using correlation metrics to evaluate QPP, QPP evaluation using pointwise approaches, continuous evaluation, and using information theory for QPP.
The advent of large language models and the rise of new tasks, such as
conversational search, semantic search and question answering, enabled by the
availability of new powerful technological tools, have led to a previously unseen rapid
growth in the variety and quality of Information Retrieval (IR) systems. Several
ancillary research fields have also flourished due to the scientific uptake of new
Natural Language Processing (NLP) methodologies, facilitating advancement in
new IR tasks. The Query Performance Prediction and Its Evaluation in New
Tasks (QPP++ 2023) workshop [
The QPP task is defined as estimating search efectiveness in the absence of
human relevance judgments [
The QPP++ 2023 workshop aimed to provide a platform for the community to jointly discuss ways to address these challenges and create a better alignment between the latest technologies, retrieval models, and QPP approaches. Along with the challenges mentioned above, the recent advances in NLP present great opportunities for enhancing the state of the art in QPP. The workshop also sought to encourage collaboration between researchers to exploit these opportunities.
QPP and its Evaluation on New Tasks The quality of QPP methods is typically
evaluated by computing the correlation between the scores assigned to queries by
a QPP method and the true performance values, e.g., Average Precision (AP),
attained for these queries using relevance judgements. Previous research
demonstrated the unreliability of this approach when multiple experimental factors
(i.e., IR models, corpora, and predictors) are considered [
The workshop provided a forum for researchers and practitioners to discuss the following key research challenges emerging following the recent advances in IR: – Can existing QPP techniques be exploited, or which new QPP theories and models need to be devised, for new tasks, such as image retrieval, passageretrieval, question answering, and conversational search? – How can new technologies, such as contextualized embeddings, large language models, and neural networks be exploited to improve QPP? – How should QPP techniques be evaluated, including best practices, datasets, and resources? – Should QPP be evaluated in the same manner for diferent IR tasks? – What changes should we make to the QPP evaluation paradigm to accommodate new domains and IR techniques?
– We intend to compile the workshop proceedings from the submitted papers.
The proceedings will be published in the CEUR-WS.org proceedings series. – We intend to draft a position paper describing the roadmap identified during the discussions and submit it to the SIGIR forum.
The QPP++ workshop would not have been possible without the tremendous efort of several people including the program committee members and the participants who contributed their time to making the workshop a success.