The evaluation of information retrieval systems is performed using test collections. The classical Cranfield evaluation paradigm is defined on one fixed corpus of documents and topics. Following this paradigm, several systems can only be compared over the same test collections (documents, topics, assessments). In this work, we explore in a systematic way the impact of similarity of test collections on the comparability of the experiments: characterizing the minimal changes between the collections upon which the performance of IR system evaluated can be compared. To do that, we create pair instances of sub-test collections from one reference collection with controlled overlapping elements, and we compare the Ranking of Systems (RoS) of a defined list of IR systems. We can then compute the probability that the RoS are the same across the sub-test collections. We experiment with our framework proposed on the TREC-COVID collections, and two of our findings show that: a) the ranking of systems, according to the MaP, is very stable even for overlaps smaller than 10% for documents, relevance assessments and positive relevance assessments sub-collections, and b) stability is not ensured for MaP, Rprec, Bpref and ndcg evaluation measures even when considering large overlap for the topics.
Classical evaluation of information retrieval systems follows the Cranfield paradigm, based on
the use of a common test collection to evaluate all the systems in comparison. One evaluation
is then a snapshot of the behaviour of systems on a fixed dataset. In a way to study in the
large the quality of a system, a common approach is to test a system on several test collections.
Testing a system on several test collections assesses then the system’s ability to answer diverse
information needs and to cope with various type of datasets. However, diferences between test
collections, according to the content, the structure, or the way the collection has been compiled
can have a huge impact on the results of a single’s system evaluation [
The question we focus may have an impact to other fields of IR than the evaluation:
• knowing how a test collection evolution afects the stability of the systems evaluation
measures may be of great value for Web search engines, where documents, topics and
relevance assessments are constantly changing ;
• deep learning approaches for IR [
This problem has been partly studied in the state of the art but, to our knowledge, a comprehensive study on the documents, topics and assessments does not exist yet. Our proposal is then a) to create, from one test collection, multiple controlled pairs of sub-collections according to documents, topics and assessments, and b) to study the stability of the ranking of several systems between theses sub-collections. With this, we are then able to explore the impact of these pairs. We show that the topics dimension has a greater impact than the assessments and the documents dimensions.
In the following, we present first the state of the art in section 2, before detailing our proposal in section 3. In section 4, we present the experimental setting. Section 5 details the results, before the discussion in part 6 and the conclusion.
We present here existing works that focus on the impact of variation of test collections on the evaluation of the quality of systems.
Classically, diferent test collections are used to measure and test the reproducibility of
system results [
Few works are focusing specifically on the impact of topics in test collections from a ranking
of systems perspective. Robertson and Kanoulas [
Other works study the impact of corpus, assessments or topics changes on the performance of
the systems. Sanderson et al. [
Recent work of Rashidi et al. [
Compared to the state of the art, we investigate here how do the changes, not limited to
document corpus but also including topics and assessments, may afect the comparability of sets
of systems. Moreover, we investigate to what extent it is possible to compare systems evaluated
in changing test collections. Our research questions are: How to quantify the
diference/similarity between test collections? And what diferences in the test collection do guarantee the
comparability of the systems results? We hypothesize that similar test collections produce
the same Ranking of Systems (RoS), similarly to Voorhees et al. [
Our goal in this paper is to propose a way to estimate to which extent changes in a test collection implies changes in the ranking of systems tests on it. Such problem is important to solve, as it may be used to evaluate the stability of a test collection.
Before going into detail into the framework that we build, we first define formally what are comparable test collections.
Definition 1. Two test collections 1 and 2 are comparable according to an evaluation measure , if for a given set of information retrieval systems, the ranking of the systems in according to is the same in 1 and 2.
The performance of systems evaluated in one test collection depends on the features of this
test collection [
Based on these components, we will study the impact of changes using the chosen elements, i.e.,
components or subsets of them, The idea of using elements that may difer from the components
allow us to study more closely specific parts of the test collections: we are then using an
approach similar to Rashidi et al. [
Definition 2. For one element under consideration from a test collection , let 1 and 2 be sub-collections of that difer only by the element , with |1.| = |2.|, all the other elements || being equal. The overlapping level of 1 and 2 is defined as = |1.| with = 1. ∩ 2..
Such overlap, in [
According to this, we define an experimental protocol that assesses the comparability of test collections according to one element of one test collection.
Definition 3. The protocol that studies the threshold of comparability for one test collection , one evaluation measure , for a given set of overlap values , according to one similarity measure for the ranks of systems Δ applied on sub-collections pairs for a set of systems and a threshold , is defined as follows: • for each overlapping level ∈ , build controlled overlapping pairs (,1,,2) of subsets of according to the element ; • compare the RoS of a given set of systems in evaluated on one side on ,1 and on the other side on ,2, using , is done using the ranked lists ,1 and ,2, in a way to assess the impact of the overlapping over the element . This is done by a function Δ which estimates the similarity between the lists ; • compute the probability ,(Δ(.,1, .,2) >= ) for which the Δ(,1, ,2) is larger than for an overlap on a given element , for ∈ [1, ]. This may be computed using classical maximum likelihood estimate on the n pairs generated, i.e. ,(Δ(.,1, .,2) >= ) = |{|∈[1,],Δ(,1,,2)>= }| .
Following this protocol, we are able to define the minimal overlap for which the probability of having the same RoS is large enough.
As an example, if we consider the element ., the protocol considers the number of
documents overlapping in sub-test collections. For . we consider the number of common
topics in both test collections. For . we extract the proportion of common judged documents
for each topic. In a way to obtain robust results, has to be large enough (typically greater
than 50 [
In this part, we defined a protocol for calculating the impact of the overlap between diferent elements on the retrieval results. To show the feasibility of our proposal, we now show the experimental results using the TREC-COVID collection.
We use the complete TREC-COVID test collection [
For each overlapping level , we create 50 test collection pairs, so = 50. We evaluate 10
classical IR systems with and without Bo1 relevance feedback: S = {BM25, DLH, DirichletLM, PL2,
TF_IDF, BM25_Bo1, DLH_Bo1, DirichletLM_Bo1, PL2_Bo1, TF_IDF_Bo1}, implemented using
PyTerrier [
According to the state of the art, we define the elements for a test collection in a
nonexhaustive way as follows:
• .: the set of documents (similar to Sanderson et al. [
Next, we will show the impact of each of these elements considered independently on the TREC-COVID collection.
(a) Overlap of docs vs. probability of same RoS (b) Overlap of topics vs. probability of same RoS (c) Overlap of assessments vs. probability of same (d) Overlap of positive assessments vs. probability
RoS of same RoS
The Figure 1 presents the probabilities of having the same ranking of the considered systems, respectively on ., ., . and . .
For the evaluation measures presented in Figure 1a regarding the overlap of ., we see, as expected, that the probability of similar RoS increases as the overlaps increase. In this figure, the MaP is very stable, as .,(Δ(.,1, .,2) >= 90%) = 1 for each document overlap o greater than 5%. This underlines the fact that the corpus is very focused on one topic area (COVID-related documents). The Bpref evaluation measure (green triangle) has the lower probability for all overlaps tested: the probability of having the same RoS in larger than 90% only for overlaps greater than 65%. A detailed look on the 50 runs for the overlaps of 5% shows that: a) on average, the average for MAP values is 0.998 and average for the Rprec values is 0.886: both values are very large, and b) overall 50 of the 50 MAP values are larger than 0.9 where only 24 of the 50 Bprec values are above 0.9. Mainly the large diference comes from the threshold : if = 0.8, then the Rprec and MAP behave similarly.
The Figure 1b, focusing on topics, exhibits expected behaviors: the more the overlap of topics, the higher the probability of the same RoS. However, we see that the MaP is not as stable as Rprec: the Rprec has almost most of the time the higher probability of RoS similarity. Here, the Bpref is still the least stable measure.
Figures 1c and 1d, corresponding to the overlaps of the assessments and the overlaps of the positive assessments respectively, are the most flat ones for all the evaluation measures. MaP and ndcg always reach the probability of 1 for each overlap values considered. For . the 1Download link: https://ir.nist.gov/covidSubmit/data.html less stable measure is Bpref and for . it is Rprec.
The low slopes in the graphs mean a stable comparability across overlaps, and the intercept is interpreted as the projected minimum comparability value when there is no overlapping elements. The metric with the lowest slope and highest intercept, for three of the four features, is MaP. The comparability of test collections for the RPrec is higher than for MaP only for the Topics experiments. The metric with highest slope and lowest intercept is Bpref in all the analyzed elements, leading to a larger sensitivity of this measure.
Figure 2 presents under a radar view the lower overlap values for which the probability of having a similarity of RoS larger than 0.9 is equal to 1. The overlap value of 100 means that we did not get any full stability for a partial overlap considered. We see that the only element for which we are not able to get any stability on the TREC-COVID collection is . (i.e., the topics splits).
From the Figures 1a, 1c and 1d and 2, we see that the MaP is able to cope with the diferences
between sub-collections. When considering the documents ., relevance assessments .
and relevance assessments . that are relevant, the MaP is able to cope even with very low
overlaps (5%). So, MAP gives us a good comparison over two completely diferent collections,
for the same set of topics. The only element for which none of the evaluation measure is
reaching a probability of 1 is ., reflecting the fact that there is a low stability of the ranking of
systems across very similar collections according to the topics. This finding is in contradiction
with Carterette et al. [
In Figure 1a, we see that the MaP, Rprec and ndcg are very high for each overlap considered.
This may be explained by the fact that the corpus focuses on one area of the topics related to
Covid, and that there is a large redundancy between the documents. It might be just the case
that MaP is just stable even across diferent collections, but the table of Fang et al. [
The Figure 1b exhibits the large impact of the overlapping on topics: as the topics do not
behave similarly, the non-overlapping topics lead to very diferent ranking of systems. We see
on the left part of Figure 1b that the probability of similar ranking for bref is very low (for
instance 0.16 for an overlap of 8%). This findings is mainly caused by the fact that the topics in
a test collection are classically built manually, and are supposed to be very diferent (as shown
in Figure 2 of Banks et al. [
The two elements that are considering the relevant assessments . and . have similar behaviors in Figures 1c and 1d: for almost all overlaps the probabilities are greater than 0.8. This shows that using only partial overlaps of assessments smaller than 50% leads to similar rankings of systems for MaP and ndcg evaluation measures. The Rprec (precision computed at the number of relevant documents for a topic) is more sensitive to the overlaps of positive assessments, as this measures is based on the number of positive relevant assessments. Rprec is especially sensitive to small overlaps of assessments because the positive assessments form roughly 1/3 of all available assessments.
As presented above, Figure 2 describes graphically the minimal overlapping, for each element and each evaluation measure considered, that leads to a probability of 1 to get a 90% similarity between ranking of systems: the larger the area, the smaller the overlap. From this Figure, we see on one side that the Rpec and Bpref evaluation measures are more sensitive to the overlaps, whatever element we consider, and on the other side that the topics are very sensitive to any overlap ratios (orange line, surface on 100% for all evaluation measures). For this Figure, we conclude that for MaP and ndcg evaluation measures, having a test collection composed of only 5% of the assessments and 5% of the positive assessments lead to the same ranking of systems: such result may relax the need for N-fold validation in the case of evaluation of learning-based IR systems, or may constrain an N-fold validation experiment by using splits that lead to same ranking of systems.
Our findings are quite consistent results with table 1 of Ferro and Sanderson [
We have presented a protocol that supports the study the impact of of a test collection on the ranking of systems. Our proposal formalizes this crucial part which needs be defined to perform such study. We then applied the protocol on the TREC-COVID collection. The outcomes of this study show that the documents and topics are the considered elements that have the most impact on the stability of the ranking of systems. We also showed that each evaluation measure behaves very diferently in our experiments. A future work could extend our proposal to study sub-collections overlaps across several measures.
As a future work, we also would like to extend our proposal to be able to consider jointly
several elements, so that we may detect dependencies between elements, as in [
This work was supported by the ANR Kodicare bi-lateral project, grant ANR-19-CE23-0029 of the French Agence Nationale de la Recherche, and by the Austrian Science Fund (FWF).