Reproducibility has become increasingly important for many research areas, among those IR is not an exception and has started to be concerned with reproducibility and its impact on research results. This paper describes our second attempt to propose a lab on reproducibility named CENTRE, held during CLEF 2019. The aim of CENTRE is to run both a replicability and reproducibility challenge across all the major IR evaluation campaigns and to provide the IR community with a venue where previous research results can be explored and discussed. This paper reports the participant results and preliminary considerations on the second edition of CENTRE@CLEF 2019.
Reproducibility is becoming a primary concern in many areas of science [
Also in Information Retrieval (IR) replicability and reproducibility of the
experimental results are becoming a more and more central discussion items in the
research community [
Nevertheless, it has been repeatedly shown that best TREC systems still outperform o -the-shelf open source systems [4{6, 22, 23]. This is due to many di erent factors, among which lack of tuning on a speci c collection when using default con guration, but it is also caused by the lack of the speci c and advanced components and resources adopted by the best systems.
It has been also shown that additivity is an issue, since adding a component
on top of a weak or strong base does not produce the same level of gain [
Moreover, besides the problems encountered in replicating/reproducing
research, we lack any well established measure to assess and quantify the extent
to which something has been replicated/reproduced. In other terms, even if a
later researcher can manage to replicate or reproduce an experiment, to which
extent can we claim that the experiment is successfully replicated or reproduced?
For the replicability task we can compare the original measure score with the
score of the replicated run, as done in [
Finally, both a Dagstuhl Perspectives Workshop [
Overall, the above considerations stress the need and urgency for a systematic approach to reproducibility and generalizability in IR. Therefore, the goal of CLEF, NTCIR, TREC REproducibility (CENTRE) at CLEF 2019 is to run a joint CLEF/NTCIR/TREC task on challenging participants: { to replicate and reproduce best results of best/most interesting systems in previous editions of CLEF/NTCIR/TREC by using standard open source IR systems; { to contribute back to the community the additional components and resources developed to reproduce the results in order to improve existing open source systems; { to start exploring the generalizability of our ndings and the possibility of predicting IR system performance; { to investigate possible measures for replicability and reproducibility in IR.
The paper is organized as follows: Section 2 introduces the setup of the lab; Section 3 discusses the participation and the experimental outcomes; and, Section 4 draws some conclusions and outlooks possible future works.
Similarly to its previous edition, CENTRE@CLEF 2019 o ered the following two tasks: { Task 1 - Replicability : the task focuses on the replicability of selected methods on the same experimental collections; { Task 2 - Reproducibility : the task focuses on the reproducibility of selected methods on di erent experimental collections;
For Replicability and Reproducibility we refer to the ACM Artifact Review and Badging de nitions6: { Replicability (di erent team, same experimental setup): the measurement can be obtained with stated precision by a di erent team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a di erent location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts. In CENTRE@CLEF 2019 this meant to use the same collections, topics and ground-truth on which the methods and solutions have been developed and evaluated. { Reproducibility (di erent team, di erent experimental setup): The measurement can be obtained with stated precision by a di erent team, a di erent measuring system, in a di erent location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently. In CENTRE@CLEF 2019 this meant to use a di erent experimental collection, but in the same domain, from those used to originally develop and evaluate a solution.
For Task 1 and Task 2, CENTRE@CLEF 2019 teams up with the
OpenSource IR Replicability Challenge (OSIRRC) [
Besides Task 1 and Task 2, CENTRE@CLEF 2019 o ered also a new pilot task: { Task 3 - Generalizability : the task focuses on collection performance prediction and the goal is to rank (sub-)collections on the basis of the expected performance over them.
In details, Task 3 was instatiated as follows: 6 https://www.acm.org/publications/policies/ artifact-review-badging { Training : participants need to run plain BM25 and, if they wish, also their own system on the test collection used for TREC 2004 Robust Track (they are allowed to use the corpus, topics and qrels). Participants need to identify features of the corpus and topics that allow them to predict the system score with respect to Average Precision (AP). { Validation: participants can use the test collection used for TREC 2017 Common Core Track (corpus, topics and qrels) to validate their method and determine which set of features represent the best choice for predicting AP score for each system. Note that the TREC 2017 Common Core Track topics are an updated version of the TREC 2004 Robust track topics. { Test (submission): participants need to use the test collection used for TREC 2018 Common Core Track (only corpus and topics). Note that the TREC 2018 Common Core Track topics are a mix of \old" and \new" topics, where old topics were used in TREC 2017 Common Core track. Participants will submit a run for each system (BM25 and their own system) and an additional le (one for each system) including the AP score predicted for each topic. The score predicted can be a single value or a value with the corresponding con dence interval. 2.2
For the previous edition of CENTRE@CLEF 2018 [
To select the target runs from the TREC 2017 and 2018 Common Core Tracks
we did not consider the impact of the proposed approaches in terms of number
of citations, since both the tracks are recent and the citations received by the
submitted papers are not signi cant. Therefore, we looked at the nal ranking
of runs reported in the tracks overviews [
Below we list the runs selected as targets of replicability and reproducibility among which the participants can choose. For each run, we specify the corresponding collection for replicability and for reproducibility. For more information, the list also provides references to the papers describing those runs as well as the overviews describing the overall task and collections.
{ Runs: WCrobust04 and WCrobust0405 [18]
Task Type: TREC 2017 Common Core Track [
Reproducibility: TREC Washington Post Corpus, with TREC 2018
Common Core Topics
Task Type: TREC 2018 Common Core Track [
Since these runs were not originally thought for being used as targets of a replicability/reproducibility exercise, we contacted the authors of the papers to inform them and ask their consent to use the runs.
The participants in CENTRE@CLEF 2019 were not provided with the corpora necessary to perform the tasks. The following collections were needed to perform the task: { The New York Times Annotated Corpus7 contains over 1:8 million articles written and published by the New York Times between January 1, 1987 and June 19, 2007. The text in this corpus is formatted in News Industry Text Format (NITF), which is an XML speci cation that provides a standardized representation for the content and structure of discrete news articles. The dataset is available upon payment of a fee. { The TREC Washington Post Corpus8 contains 608 180 news articles and blog posts from January 2012 through August 2017. The articles are stored in JSON format, and include title, byline, date of publication, kicker (a section header), article text broken into paragraphs, and links to embedded images and multimedia. The dataset is publicly available and free of charge. { The TREC 2004 Robust Corpus9 corresponds to the set of documents on TREC disks 4 and 5, minus the Congressional Record. This document set contains approximately 528 000 documents. The dataset is available upon payment of a fee.
Finally, Table 1 reports the topics used for the three tasks, with the corresponding number of documents and pool sizes. An example of topic is reported in the Figure 1 for TREC 2018 Common Core Track. 2.3
Task 1 - Replicability: As done in the previous edition of CENTRE [
Note that the de nition of Kendall's in Equation (2) is originally proposed for permutations of the same set of items, therefore it is not applicable whenever two rankings do not contain the same set of documents. However, for real rankings of systems it is highly likely that two lists do not contain the same set of items, thus we performed some pre-processing with the runs before computing Kendall's in Equation (2).
In details, consider a xed topic t, the original ranking rt;orig and the replicated ranking rt;replica. If one of the rankings contains a document that is not retrieved by the other ranking, we de ne the rank position of that document as zero. For example, if for a document d, d 2 rt;orig, but d 62 rt;replica, then the rank position of d in rt;replica is zero. Whenever the two rankings contains the same set of documents, Equation (2) is not a ected by this pre-processing step and the computation of Kendall's tau is performed as usual. Furthermore, if two rankings retrieves di erent documents and place them in the same rank positions, Kendall's tau will still be equal to 1, and the comparison is performed just with respect to the relative order of the documents retrieved by both the rankings.
Task 2 - Reproducibility: Since for the reproducibility runs we do not have an already existing run to compare against, we compare the reproduced run score with respect to a baseline run, to see whether the improvement over the baseline is comparable between the original collection C and the new collection D. In particular we compute the E ect Ratio (ER), which is also exploited in CENTRE@NTCIR 14 [25].
In details, given two runs, we refer to the A-run, as the advanced run, and B-run, as the baseline run, where the A-run has been reported to outperform the B-run on the original test collection C. The intuition behind ER is to evaluate to which extent the improvement on the original collection C is reproduced on a new collection D. For any evaluation measure M , let MiC (A) and MiC (B) denote the score of the A-run and that of the B-run for the i-th topic of collection C (1 i TC ). Similarly, let MiD(A0) and MiD(B0) denote the scores for the reproduced A-run and B-run respectively, on the new collection D. Then, ER is computed as follows:
ER(
1 PTD MoCrig) = TD 1 iP=1TC TC i=1
(3) where MiC;orig = MiC (A) MiC (B) is the per-topic improvement of the original advanced and baseline runs for the i-th topic on C. Similarly MiD;reproduced = MiC (A0) MiC (B0) is the per-topic improvement of the reproduced advanced and baseline runs for the i-th topic on D. Note that the per-topic improvement can be negative, for those topics where the advanced run fails to outperform the baseline run.
If ER 0, that means that the replicated A-run failed to outperform the replicated B-run: the replication is a complete failure. If 0 < ER < 1, the replication is somewhat successful, but the e ect is smaller compared to the original experiment. If ER = 1, the replication is perfect in the sense that the original e ect has been recovered as is. If ER > 1, the replication is successful, and the e ect is actually larger compared to the original experiment.
Finally, ER in Equation (3) is instantiated with respect to AP, nDCG and ERR. Furthermore, as suggested in [25], ER is computed even for the replicability task, by replacing MiD;reproduced with MiC;replica in Equation (3). Task 3 - Generalizability: For the generalizability task we planned to compare the predicted run score with the original run score. This is measured with Mean Absolute Error and RMSE between the predicted and original measures scores, with respect to AP, nDCG and ERR. However, we did not receive any run for the generalizability task, so we did not put in practice this part of the evaluation task. 3
19 groups registered for participating in CENTRE@CLEF2019, but unfortunately only one group succeeded in submitting two replicability runs and two reproducibility runs. No runs were submitted for the generalizability task.
The team from the University of Applied Science TH Koln [
Eventually, the participating team submitted four o cial runs and four uno cial runs described in Table 2. The runs and all the code is publicly available online11. 10 https://catalog.ldc.upenn.edu/LDC2012T21 11 https://bitbucket.org/centre_eval/c2019_irc/src/master/
The paper by Grossman and Cormack [18] exploits the principle of automatic routing runs: rst, a logistic regression model is trained with the relevance judgments from one or more collections for each topic, then the model is used to predict relevance assessments of documents from a di erent collection. Both the training and the prediction phases are done on a topic-wise basis.
The routing process represented a challenge for the participating team, which initially submitted a set of four o cial runs, where some of the topics were missing. For example, the o cial run irc task1 WCrobust0405 001 contains only 33 topics, while the corresponding original run WCrobust0405 contains all the 50 topics. The participating team could not understand how to derive document rankings for those topics such that no training topics were available for the logistic regression model. For example, when they were attempting to replicate WCrobust0405, they exploited as training set the intersection between the topics from TREC 2004 Robust and TREC 2005 Robust. Then, for the prediction phase, only 33 topics from TREC 2017 Common Core were contained in the training set, and no prediction could be performed for the remaining topics. Due to similar issues, the o cial irc task2 WCrobust04 001 and irc task2 WCrobust0405 001 contain 25 and 15 topics respectively.
Afterwards, the participating team contacted the authors of the original paper, Grossman and Cormack [18], to understand how to derive rankings even when there are no training topics available. The authors clari ed that for WCrobust0405 the training set contains both the topics from TREC 2004 Robust and TREC 2005 Robust, and when a topic is not contained in TREC 2005 Robust, they used just the TREC 2004 Robust collection as training set. Therefore, the authors submitted four additional uno cial runs, where both irc task1 WCrobust04 001 and irc task1 WCrobust0405 001 contain all the 50 topics, while the reproduced runs irc task2 WCrobust04 001 and irc task2 WCrobust04 001 contain 25 topics. Note that some of the topics are missing for the reproduced runs, since no training data is available for 25 out of the 50 topics of TREC 2018 Common Core.
In the following we report the evaluation results for the replicability and reproducibility tasks, both for the o cial and uno cial submissions.
Table 3 and Table 4 report AP, nDCG and ERR scores for the o cial replicated runs. As shown by RMSE, the replication task was fairly successful with respect to AP and nDCG, while when ERR is considered, RMSE is greater than 0:2, showing that it is harder to replicate ERR than the other evaluation measures. Indeed, it is well known that ERR is highly sensitive to the position of relevant documents at the very beginning of the ranking, thus even the misplacement of a single relevant documents may cause a signi cant drop in ERR score.
Furthermore, as the cut-o increases, even RMSE for AP and nDCG increases, showing that the replication is less accurate at lower cut-o levels. On the other side, RMSE for ERR is almost constant when the cut-o increases, showing once more that ERR focuses on the top rank positions rather than considering the whole ranking. 0:0506 0:2252 0:3821 0:1442 0:3883 0:6299 0:0473 0:2541 0:4428 0:1490 0:4268 0:6883 nDCG@1000 ERR@100
Similarly, Table 5 reports AP, nDCG and ERR scores for the uno cial replicated run irc task1 WCrobust0405 001. Note that the o cial and uno cial replicated run irc task1 WCrobust04 001 are identical, therefore the evaluation scores for this uno cal run are the same reported in Table 3 and are omitted in the following.
Again, we can observe that the replication task is more successful for RMSE with respect to AP and nDCG than ERR. Furthermore, RMSE increases as the cut-o increases, meaning that the accuracy of the replicated run decreases as the cut-o level increases.
By comparing the o cial and uno cial evaluation results for irc task1WCrobust0405 001, in Table 4 and Table 5 respectively, we can note that RMSE score are quite similar, showing that the uno cial run is fairly accurate even on the additional topics.
Table 6 reports ER for the replication task with the o cial runs. We considered WCrobust0405 as advanced run and WCrobust04 as baseline run, therefore the per-topic improvement is computed as WCrobust0405 scores minus WCrobust04 scores for each topic. For the replicated o cial runs, we needed to select from irc task1 WCrobust04 001 the 33 topics contained in irc task1 WCrobust0405 001, otherwise we could not compute the mean per-topic improvement.
ER shows that the replication task is fairly successful for AP, while it is less successful for nDCG and ERR. Furthermore, ER > 1 highlights that the di erence between the advanced and the baseline run is more pronounced in the replicated runs than in the original runs. Again, it can be noted that as the cut-o increases, the accuracy of the replicability exercise decreases for AP and nDCG, while it is almost constant for ERR. 0:0078 0:0446 0:0556 0:0233 0:0597 0:0578 0:0078 0:0446 0:0556 0:0233 0:0597 0:0578
between the original and replicated runs.
Replicated Run irc task1 WCrobust04 001 o cial irc task1 WCrobust0405 001 o cial
Original Run WCrobust04
WCrobust0405 irc task1 WCrobust0405 001 uno cial
WCrobust0405 0:0222 0:0034 0:0107 0:0073 0:0316 0:0199 0:0021 0:0046
Fig. 3. First 10 rank positions for irc task1 WCrobust04 001 for topic 307 form TREC 2017 Common
Core Track.
Analogously, Table 7 reports ER for the replication task with the uno cial runs. We considered WCrobust0405 as advanced run and WCrobust04 as baseline run, therefore the per-topic improvement is computed as in Table 6 and the rst column is equal. Both the replicated uno cial runs contain the same 50 topics, therefore the per-topic improvement is computed as irc task1 WCrobust0405 001 scores minus irc task1 WCrobust04 001 scores for each topic.
When the whole set of 50 topics is considered, the replication is fairly successful with respect to all the measure, with ER ranging between 0:83 and 1:12. The only exception is represented by AP@10, where the replicated runs fails to replicate the per-topic improvements. Again, the accuracy of the replicated runs decreases as the cut-o increases.
Table 8 reports the Kendall's correlation between the original and replicated runs, both for the o cial and uno cial runs. We computed Kendall's at di erent cut-o levels, where we rst trimmed the runs at the speci ed cut-o and subsequently computed Kendall's between the trimmed runs.
Table 8 shows that the replication was not successful for any of the runs in terms of Kendall's . This means that even if the considered replicated runs were similar to the original runs in terms of placement of relevant and non relevant documents, they actually retrieves di erent documents.
Figure 2 and Figure 3 shows the rst 10 rank positions for WCrobust04 and its replicated version irc task1 WCrobust04 001, for topic 307 from TREC 2017 Common Core Track. We can observe that even if the runs retrieves a similar set of documents, the relative position of each document is di erent. For example, document 309412 is at rank position 1 for the original run, but at nDCG@1000 0:0078 0:0446 0:0556 0:0233 0:0597 0:0578 0:1042 0:1019 0:1019 0:0122 0:0431 0:0579 0:0298 0:0767 0:0898 0:0124 0:0142 0:0135
ER 1:5641 0:9664 1:0414 1:2790 1:2848 1:5536 0:1190 0:1394 0:1325 rank position 2 for the replicated run, similary document 733642 is at rank position 1 for the replicated run and at rank position 5 for the original run. Moreover, document 241240 is at rank position 3 for the replicated run, but it does not apper on the rst 10 positions for the original run.
Table 8, Figure 2 and Figure 3 highlights how hard is to replicate the exact ranking of documents. Therefore, whenever a replicability task is considered, comparing the evaluation scores with RMSE or ER might not be enough, since these approaches consider just the position of relevant and not relevant documents, and overlook the actual ranking of documents.
Finally, Table 9 reports the mean per-topic improvement and ER for the o cial runs from the reproducibility task. As done for the replicability task, we considered WCrobust0405 as advanced run and WCrobust04 as baseline run on the test collection from TREC 2017 Common Core Track. For the reproduced o cial runs, we needed to select from irc task2 WCrobust04 001 the 15 topics contained in irc task2 WCrobust0405 001 from TREC 2018 Common Core Track, otherwise we could not compute the per topic improvement.
Furthermore, when the cut-o increases, the accuracy of the reproducibility exercise increases for AP, while it decreases for nDCG and remains almost constant for ERR. nDCG@1000 0:0078 0:0446 0:0556 0:0233 0:0597 0:0578 0:1042 0:1019 0:1019 0:0065 0:0241 0:0336 0:0155 0:0426 0:0509 0:0004 0:0033
Similarly, Table 10 reports the mean per-topic improvement and ER for the uno cial runs from the reproducibility task. We considered WCrobust0405 as advanced run and WCrobust04 as baseline run on TREC 2017 Common Core, therefore the per-topic improvement is computed as in Table 9 and the rst column is equal. Both the reproduced uno cial runs contain the same 25 topics, therefore the per-topic improvement is computed as irc task2 WCrobust0405 001 scores minus irc task2 WCrobust04 001 scores for each topic.
The best reproducibility results are obtained with respect to AP@10 and nDCG@1000, thus the e ect of the advanced run over the baseline run is better reproduced at the beginning of the ranking for AP, and when the whole ranked list is considered, for nDCG. Again, ERR is the hardest measure to be reproduced, indeed it has the lowest ER score for each cut-o level. 4
This paper reports the results on the second edition of CENTRE@CLEF2019.
A total of 19 participants enrolled in the lab, however just one group managed
to submit two replicability runs and two reproducibility runs. As reported in
Section 3, the participating team could not reproduce the runs from Benham et
Al. [
The experimental results show that the replicated runs are fairly successful with respect to AP and nDCG, while the lowest replicability results are obtained with respect to ERR. As ERR mainly focuses on the beginning of the ranking, misplacing even a single relevant document can deteriorate ERR score and have a great impact on the replicability evaluation scores.
Moreover, whenever replicability is considered, RMSE and ER are not enough to evaluate the replicated runs. Indeed, they only account for the position of relevant and not relevant documents by considering the similarity between the original scores and the replicated scores, and they overlook the actual ranking of documents. When the runs are evaluated with Kendall's to account for the actual position of the documents in the ranking, the experiments show that the replicability is not successful at all, with Kendall's values close to 0. This con rms that, even if it is possible to achieve similar scores in terms of IR evaluation measures, it is challenging to replicate the same documents ranking.
When it comes to reproducibility, there are no well-established evaluation measures to determine to which extent a system can be reproduced. Therefore, we compute ER, rstly exploited in [25], which focuses on the reproduction of the improvement of an advanced run over a baseline run. The experiments show that reproducibility was fairly successful in terms of AP@10 and nDCG@1000, while, similarly to the replicability task, ERR is the hardest measure in terms of reproducibility success.
Finally, as reported in [
For the next edition of the lab we are planning to propose some changes in the lab organization to increase the interest and participation of the research community. First, we will target for more popular systems to be replicated and reproduced, moreover we will consider other tasks than the AdHoc, as for example the medical or other popular domains. 17. Fuhr, N.: Some Common Mistakes In IR Evaluation, And How They Can Be
Avoided. SIGIR Forum 51(3), 32{41 (December 2017)
18. Grossman, M.R., Cormack, G.V.: MRG UWaterloo and WaterlooCormack
Participation in the TREC 2017 Common Core Track. In: Voorhees and Ellis [26]
19. Jungwirth, M., Hanbury, A.: Replicating an Experiment in Cross-lingual
Information Retrieval with Explicit Semantic Analysis. In: Cappellato et al. [