Reproducibility has become increasingly important for many research areas, among those IR is not an exception and has started to be concerned with reproducibility and its impact on research results. This paper describes our rst attempt to propose a lab on reproducibility named CENTRE and held during CLEF 2018. The aim of CENTRE is to run a reproducibility challenge across all the major IR evaluation campaigns and to provide the IR community with a venue where previous research results can be explored and discussed. This paper reports the participant results and preliminary considerations on the rst edition of CENTRE@CLEF 2018, as well as some suggestions for future editions.
Reproducibility is becoming a primary concern in many areas of science [
Even if reproducibility has become part of the review forms at major conferences like SIGIR, this is more a qualitative assessment performed by a reviewer on the basis of what can be understood from a paper rather than an actual \proof" of the reproducibility of the experiments reported in the paper. Since 2015, the ECIR conference started a new track focused on reproducibility of previously published results. This conference track led to a stable enough ow of 3-4 reproducibility papers accepted each year but, unfortunately, this valuable e ort did not produce a systematic approach to reproducibility: submitting authors adopted di erent notions of reproducibility, they adopted very diverse experimental protocols, they investigated the most disparate topics, resulting 4 https://www.acm.org/publications/policies/
artifact-review-badging in a very fragmented picture of what was reproduced and what not, and the outcomes of these reproducibility papers are spread over a series of potentially disappearing repositories and Web sites.
Moreover, if we consider open source IR systems, they are typically used as: { starting point by new-comers in the eld, which take them almost o -theshelf using default con guration to begin experience with IR and/or speci c search tasks; { base system on top of which to add a new component/technique you are interested to develop, keeping all the rest in the default con guration; { baseline for comparison, again using default con guration.
Nevertheless, it has been repeatedly shown that best TREC systems still
outperform o -the-shelf open source systems [2{4, 23, 24]. This is due to many
di erent factors, among which lack of tuning on a speci c collection when using
default con guration, but it is also caused by the lack of the speci c and advanced
components and resources adopted by the best systems. It has been also shown
that additivity is an issue, since adding a component on top of a weak or strong
base does not produce the same level of gain [
Therefore, the goal of CENTRE@CLEF 20185 is to run a joint task across CLEF/NTCIR/TREC on challenging participants: { to reproduce best results of best/most interesting systems in previous editions of CLEF/NTCIR/TREC by using standard open source IR systems; { to contribute back to the community the additional components and resources developed to reproduce the results in order to improve existing open source systems.
The paper is organized as follows: Section 2 introduces the setup of the lab; Section 3 discusses the participation and the experimental outcomes; and, Section 4 draws some conclusions and outlooks possible future works. 2 2.1
The CENTRE@CLEF 2018 lab o ered two pilot tasks: { Task 1 - Replicability: the task focused on the replicability of selected methods on the same experimental collections; { Task 2 - Reproducibility : the task focused on the reproducibility of selected methods on the di erent experimental collections. 5 http://www.centre-eval.org/clef2018/
where we adopted the ACM Artifact Review and Badging de nition of replicability and reproducibility: { Replicability (di erent team, same experimental setup): the measurement can be obtained with stated precision by a di erent team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a di erent location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts.
In CENTRE@CLEF 2018 this meant to use the same collections, topics and ground-truth on which the methods and solutions have been developed and evaluated. { Reproducibility (di erent team, di erent experimental setup): The measurement can be obtained with stated precision by a di erent team, a di erent measuring system, in a di erent location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently. In CENTRE@CLEF 2018 this meant to use a di erent experimental collection, but in the same domain, from those used to originally develop and evaluate a solution. 2.2
For this rst edition of CENTRE we prefer to select the target runs among the Ad Hoc tasks in previous editions of CLEF, TREC, and NTCIR. We decided to focus on the Ad Hoc retrieval since it is a general and well known task. Moreover, the algorithms and the approaches used for Ad Hoc retrieval are widely used as basis for all the other types of tasks.
For each evaluation campaign, CLEF, TREC, and NTCIR, we considered all the Ad Hoc tracks and we examined the submitted papers and the proposed approaches. CLEF Ad Hoc tracks were proposed from 2000 to 2008, for TREC campaign we focused on the Web Track, from 2009 to 20011, while for NTCIR we checked the WWW track of 2017. To select the target papers among all the submitted ones we considered the following criteria: { the popularity of the track, by accounting for the number of participating groups and the number of submitted runs; { the impact of the proposed approach, measured by the number of citations received by the paper; { the year of publication, since we preferred more recent papers; { the tools used by the author, we discarded all those papers that were not using publicly available retrieval systems as Lucene, Terrier, Solr, and Indri.
Below we list the runs selected as targets of replicability and reproducibility among which the participants can choose. For each run, it is speci ed the collection for replicability and the collections for reproducibility; for more information, the list also provides references to the papers describing those runs as well as the overviews describing the overall task and collections.
Since these runs were not originally thought for being used as targets of a replicability/reproducibility exercise, we contacted the authors of the papers to inform them and ask their consent to use the runs.
{ Run: AUTOEN [
Task type: CLEF Ad Hoc Multilingual Task
Replicability: Multi-8 Two Years On with topics of CLEF 2005 [
Task type: CLEF Ad Hoc, Bilingual Task
Replicability: TEL English (BL) with topics of CLEF 2008 [
Task type: TREC Ad Hoc Web Task
Replicability: ClueWeb12 Category A with topics of TREC 2013 [
ClueWeb12 Category A and B with topics of TREC 2014 [
Task type: TREC Ad Hoc Web Task
Replicability: ClueWeb12 Category A with topics of TREC 2014 [
ClueWeb12 Category B with topics of TREC 2014 [
Task type: NTCIR Ad Hoc Web Task
Replicability: ClueWeb12 Category B with topics of NTCIR-13 [
Table 1 reports the number of documents and the languages of the documents contained in the provided corpora.
Finally, Table 2 reports the topics used for the replicability and reproducibility tasks, with the corresponding number of documents, documents' languages and pool sizes. An example of topic for each evaluation campaign is reported in the Figure 1 for CLEF, 2 for TREC, and 3 for NTCIR. 2.3
The quality of the replicability runs has been evaluated from two points of view: Fig. 3. Example of topic for NTCIR-13, We Want Web Track.
v m RMSE = tuu m1 X i=1
APorig;i
{ E ectiveness : how close are the performance scores of the replicated systems
to those of the original ones. This is measured using the Root Mean Square
Error (RMSE) [
Since for the reproducibility runs we do not have an already existing run to compare against, we planned to compare the reproduced run score with respect to a baseline run to see whether the improvement over the baseline is comparable between the original and the new dataset. However, we did not receive any reproducibility runs so we cannot put in practice this part of the evaluation task. 3
17 groups registered for participating in CENTRE@CLEF2018 but, unfortunately, only one group succeeded in submitting one replicability run.
Technical University of Wien (TUW) [
The run AH-TEL-BILI-X2DE-CLEF2008.KARLSRUHE.AIFB ONB EN by Cimiano and Sorg uses a cuto of 1,000 documents and so it has to be compared to esalength1000 top1000, which adopts the same cut-o . However, since TUW submitted runs also for cuto s 10 and 100 documents, we compare them against versions of the run AH-TEL-BILI-X2DE-CLEF2008.KARLSRUHE.AIFB ONB EN capped at 10 and 100 documents per topic.
The paper by Cimiano and Sorg [
TUW encountered the following issues in replicating the original run:
{ the Wikipedia underlying database dump of 2008 was no longer available
and they have to resort to the static HTML dump of Wikipedia in the same
period;
{ the above issue caused a processing of Wikipedia articles sensibly di erent
from the original one in [
All these issues prevented TUW from successfully replicating the original run. Indeed the Mean Average Precision (MAP) of the run by Cimiano and Sorg was 0:0667 while the MAP of the run esalength1000 top1000 by TUW is 0:0030.
The detailed results at di erent cuto s for all the submitted runs are reported in table 4. It clearly emerges that all the above mentioned issues caused the TUW runs to severely underperform with respect to the original run and it is hard to say, in general, the extent to which it is possible to replicate it due to the changes in the language resources available.
The di culties encountered in replicating the run are further stressed by the RMSE between AH-TEL-BILI-X2DE-CLEF2008.KARLSRUHE.AIFB ONB EN and esalength1000 top1000, computed according to eq. (1), which is 0:1132 and the average Kendall's correlation among the ranked lists of retrieved documents, computed according to eq. (2), which is 5:69 10 04.
Table 5 reports the results of the comparison with RMSE and Kendall's between the original and all the replicated runs. It can be noted how the RMSE deteriorates as the cuto size increases as it is intuitive since it should be easier to stay closer to the original one when dealing with few top-k documents.
Finally, Table 6 shows the Kendall's correlation between the submitted and the original runs computed for each single topic. We can observe as the general trend is to have very low correlations, i.e. very di erent document rankings between the topics, but there are a few exceptions. For example, topic 467-AH has a correlation of 0:6644 with Explicit Semantic Analysis (ESA) length 1,000 and 10 documents cuto , which suddenly drops to 0:0253 and 0:0038 for cuto s 100 and 1,000, respectively, further stressing the fact that it should be easier to replicate the very top-k documents. Another interesting example is topic 490-AH for which the correlation at ESA length 1,000 and 10 documents cuto is 0:6170, indicating that the right documents have been retrieved but they have been ranked in almost reversed order. 4
This paper reports the results on the rst edition of CENTRE@CLEF2018. A total of 17 participants enrolled in the lab, however just one group managed to submit a run. As reported in the results section, the group encountered many
between the original and the submitted runs computed for each
substantial issues which prevented them to actually replicate the targeted run,
as described in more detail in their paper [
These results support anecdotal evidence in the eld about how much di
cult is to actually replicate (and even more reproduce) research results, even in
a eld with such a long experimental tradition as IR is. However, the lack of
participation is a signal that the community is somehow overlooking this important
issue. As it also emerged from a recent survey within the SIGIR community [
For the next edition of the lab we are planning to propose some changes in the lab organization to increase the interest and participation of the research community. First, we will target for newer and more popular systems to be reproduced, moreover we will consider other tasks than the AdHoc, as for example the medical or other popular domains.