Several recent studies have explored the interaction efects between topics, systems, corpora, and components when measuring retrieval efectiveness. However, all of these previous studies assume that a topic or information need is represented by a single query. In reality, users routinely reformulate queries to satisfy an information need. Recently there has been renewed interest in the notion of “query variations” which are essentially multiple user formulations for an information need. Like many retrieval models, some queries are highly efective while others are not. In this work 1, we explore the fundamental problem of studying the interaction components of an IR experimental collection. Our findings show that query formulations have a comparable efect size to the topic factor itself, which is known to be the factor with the greatest efect size in prior ANOVA studies. This suggests that topic dificulty is an artifact of the collection considered and highlights the importance of further research in understanding link between the complexity of a topic and the query rewriting in IR related tasks.
The interplay between simple keyword queries and large document collections has challenged
researchers in Information Retrieval (IR) for more than half a century. Some queries are highly
efective, while others perform poorly, and changing the ranking models to compensate for
dificult queries can have negative efects on the performance of queries that were performing
well previously. This notion of query dificulty has received a great deal of attention over the
years. For example, NIST ran the Robust Track in 2004 and 2005 to reexamine sets of queries
which had performed poorly across all systems evaluated in the Ad hoc track [
Data and Methods. We used the following collections: TREC Robust 2004 Ad Hoc, TREC
Common CORE 2017, and TREC Common CORE 2018 for our experiments. The Robust Ad
Hoc track contains approximately 528K documents; the TREC 2017 Common CORE contains
over 1.8 million articles; finally, the TREC Common CORE 2018 track roughly containing 600K
news articles. A large seed set of 3,402 human curated query formulations originally developed
using the TREC Robust 2004 Ad Hoc search collection were used in our experiments [
To determine the impact of the topics, their formulations and the systems on the overall
performance, we run an ANOVA [
RQ2 and RQ3: Efect Size of Query Formulation across Corpora . To study the efect of the topics and query formulations across corpora, we slightly change the ANOVA model, by adding the additional corpus factor. Table 2 shows the results for the ANOVA across-corpora. The introduction of the new factor allows to compute the efect of some interactions between factors that could not be computed previously. All the factors are statistically significant. We can observe that the topic factor has a large-size efect even across corpora and that the system factor becomes a moderately large-size efect, being bigger than in the single corpus case (see Table 1); the corpus factor has a medium-size efect. We also note that the query formulation factor has a remarkably large-size efect, even across corpora, observed here for the first time, suggesting it is a key contributor to topic dificulty. Both the query formulation*corpus interaction and the topic*system*corpus interaction, observed here for the first time, are clearly important large-size efects. Overall, these findings provide further evidence supporting the possibility that dificult topics do not actually exist in any absolute sense.
Topic Dificulty . To determine whether the dificulty is an intrinsic property of the topic we execute the following experiment. We randomly sample 20,000 permutations of the topics. For each of such permutations and for each pair system-corpus, using a greedy approach we select formulation to represent each of the topics to maximize the correlation between the ranking of the topics based on the Average Precision (AP) and the random permutation. We then select, among all the rankings of topics, the one that maximizes the correlation with the random permutation of topics. We finally compute the Kendall’s correlation between the sampled and the constructed rankings of topics. We have a mean Kendall’s of 0.85, indicating that the queries selected to induce the desired topic rankings were consistently close to the arbitrary target ordering. This is a strong evidence that topics can be “arbitrarily” easy or dificult across many diferent formulations, corpora and system combinations. This is empirical evidence that topic dificulty is not an intrinsic property of an information need – meaning that query formulation based on a corpus and retrieval system, can be combined to sort topics arbitrarily based on a performance goal. How can such a finding help researchers develop more efective retrieval systems? Firstly, it is worth noting that current evaluation paradigms usually consider a single formulation for each topic. Such an arrangement prevents us to observe system behavior with small changes to each query. We believe that multiple formulations are a key omission in our current evaluation campaigns, and we are hopeful future campaigns will incorporate them into their methodology. If our goal is to model real performance of systems, collections creators should explore how to best include multiple formulations of each topic. Our isolation of multiple formulations of topics has allowed us to study in detail the concept of “topic dificulty”, which is construct of a specific retrieval configuration – the collection, the system and the query which represents the topic – and not a property intrinsic to the topic alone.
In this work, we have presented a comprehensive ANOVA analysis that compares the efect sizes across multiple corpora and retrieval system configurations. We have also generalized previous model configurations in order to incorporate a new nesting factor which maps an information need (topic) to multiple query formulations. The removal of the constraint of a 1:1 mapping between a query and a topic has led to several interesting observations which have important implications on the notion of topic dificulty. We also propose an analysis methodology, based on a permutation algorithm, to further explore topic dificulty. Based on this new knowledge, we were able to show conclusive evidence that topic dificulty is not an intrinsic property and therefore query formulations should be included in future evaluation campaigns.