Overview of the 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) Ryan Clancy,1 Nicola Ferro,2 Claudia Hauff,3 Jimmy Lin,1 Tetsuya Sakai,4 Ze Zhong Wu1 1 University of Waterloo 2 University of Padua 3 TU Delft 4 Waseda University ABSTRACT standard test collections. The solution that we have developed The Open-Source IR Replicability Challenge (OSIRRC 2019), orga- is known as “the jig”. nized as a workshop at SIGIR 2019, aims to improve the replicability (2) Build a curated library of Docker images that work with the jig of ad hoc retrieval experiments in information retrieval by gathering to capture a diversity of systems and retrieval models. a community of researchers to jointly develop a common Docker (3) Explore the possibility of broadening our efforts to include specification and build Docker images that encapsulate a diversity additional tasks, diverse evaluation methodologies, and other of systems and retrieval models. We articulate the goals of this benchmarking initiatives. workshop and describe the “jig” that encodes the Docker specifica- Trivially, by supporting replicability, our proposed solution enables tion. In total, 13 teams from around the world submitted 17 images, repeatability as well (which, as a recent case study has shown [14], most of which were designed to produce retrieval runs for the is not as easy as one might imagine). It is not our goal to directly ad- TREC 2004 Robust Track test collection. This exercise demonstrates dress reproducibility, although we do see our efforts as an important the feasibility of orchestrating large, community-based replication stepping stone. experiments with Docker technology. We envision OSIRRC be- We hope that the fruits of this workshop can fuel empirical coming an ongoing community-wide effort to ensure experimental progress in ad hoc retrieval by providing competitive baselines replicability and sustained progress on standard test collections. that are easily replicable. The “prototypical” research paper of this mold proposes an innovation and demonstrates its value by comparing against one or more baselines. The often-cited meta- 1 INTRODUCTION analysis of Armstrong et al. [2] from a decade ago showed that The importance of repeatability, replicability, and reproducibility is researchers compare against weak baselines, and a recent study broadly recognized in the computational sciences, both in support- by Yang et al. [13] revealed that, a decade later, the situation has ing desirable scientific methodology as well as sustaining empirical not improved much—researchers are still comparing against weak progress. The Open-Source IR Replicability Challenge (OSIRRC baselines. Lin [9] discussed social aspects of why this persists, but 2019), organized as a workshop at SIGIR 2019, aims to improve the there are genuine technical barriers as well. The growing complex- replicability of ad hoc retrieval experiments in information retrieval ity of modern retrieval techniques, especially neural models that by building community consensus around a common technical are sensitive to hyperparameters and other details of the training specification, with reference implementations. This overview paper regime, poses challenges for researchers who wish to demonstrate is an extended version of an abstract that appears in the SIGIR that their proposed innovation improves upon a particular method. proceedings. Solutions that address replicability facilitate in-depth comparisons In order to precisely articulate the goals of this workshop, it is between existing and proposed approaches, potentially leading to first necessary to establish common terminology. We use the above more insightful analyses and accelerating advances. terms in the same manner as recent ACM guidelines pertaining to Overall, we are pleased with progress towards the first two goals artifact review and badging:1 of the workshop. A total of 17 Docker images, involving 13 differ- • Repeatability (same team, same experimental setup): a researcher ent teams from around the world, were submitted for evaluation, can reliably repeat her own computation. comprising the OSIRRC 2019 “image library”. These images col- • Replicability (different team, same experimental setup): an inde- lectively generated 49 replicable runs for the TREC 2004 Robust pendent group can obtain the same result using the authors’ own Track test collection, 12 replicable runs for the TREC 2017 Common artifacts. Core Track test collection, and 19 replicable runs for the TREC 2018 • Reproducibility (different team, different experimental setup): an Common Core Track test collection. With respect to the third goal, independent group can obtain the same result using artifacts this paper offers our future vision—but its broader adoption by the which they develop completely independently. community at large remains to be seen. This workshop tackles the replicability challenge for ad hoc docu- ment retrieval, with three explicit goals: 2 BACKGROUND There has been much discussion about reproducibility in the sci- (1) Develop a common Docker specification to support images that ences, with most scientists agreeing that the situation can be charac- capture systems performing ad hoc retrieval experiments on terized as a crisis [3]. We lack the space to provide a comprehensive review of relevant literature in the medical, natural, and behav- Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons ioral sciences. Within the computational sciences, to which at least License Attribution 4.0 International (CC BY 4.0). OSIRRC 2019 co-located with SIGIR 2019, 25 July 2019, Paris, France. a large portion of information retrieval research belongs, there 1 https://www.acm.org/publications/policies/artifact-review-badging have been many studies and proposed solutions, for example, a 1 recent Dagstuhl seminar [7]. Here, we focus on summarizing the own code, and in turn publish the resulting image to be further immediate predecessor of this workshop. used by others. Our workshop was conceived as the next iteration of the Open- Source IR Reproducibility Challenge (OSIRRC), organized as part of 3.1 General Design the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and As defined by the Merriam-Webster dictionary, a jig is “a device Generalizability of Results (RIGOR) [1]. This event in turn traces used to maintain mechanically the correct positional relationship its roots back to a series of workshops focused on open-source IR between a piece of work and the tool or between parts of work systems, which is widely understood as an important component during assembly”. The central activity of this workshop revolved of reproducibility. The Open-Source IR Reproducibility Challenge2 around the co-design and co-implementation of a jig and Docker brought together developers of open-source search engines to pro- images that work with the jig for ad hoc retrieval. Of course, in our vide replicable baselines of their systems in a common environment context, the relationship is computational instead of mechanical. on Amazon EC2. The product is a repository that contains all code Shortly after the acceptance of the workshop proposal at SIGIR necessary to generate ad hoc retrieval baselines, such that with a 2019, we issued a call for participants who were interested in con- single script, anyone with a copy of the collection can replicate tributing Docker images to our effort; the jig was designed with the the submitted runs. Developers from seven different systems con- input of these participants. In other words, the jig and the images tributed to the evaluation, which was conducted on the GOV2 co-evolved with feedback from members of the community. The collection. The details of their experience are captured in an ECIR code of the jig is open source and available on GitHub.3 2016 paper [10]. Our central idea is that each image would expose a number of In OSIRRC 2019, we aim to address two shortcomings with the “hooks” that correspond to a point in the prototypical lifecycle of previous exercise as a concrete step in moving the field forward. an ad hoc retrieval experiment: for example, indexing a collection, From the technical perspective, the RIGOR 2015 participants de- running a batch of queries, etc. These hooks then tie into code that veloped scripts in a shared VM environment, and while this was captures whatever retrieval model a particular researcher wishes to sufficient to support cross-system comparisons at the time, the package in the image—for example, a search engine implemented in scripts were not sufficiently constrained, and the entire setup suf- Java or C++. The jig is responsible for triggering the hooks in each fered from portability and isolation issues. Thus, it would have image in a particular sequence according to a predefined lifecycle been difficult for others to reuse the infrastructure to replicate the model, e.g., first index the collection, then run a batch of queries, results—in other words, the replicability experiments themselves finally evaluate the results. We have further built tooling that ap- were difficult to replicate. We believe that Docker, which is a popu- plies the jig to multiple images to facilitate large-scale experiments. lar standard for containerization, offers a potential solution to these More details about the jig are provided in the next section, but first technical challenges. we overview a few design decisions. Another limitation of the previous exercise was its focus on Quite deliberately, the current jig does not make any demands “bag of words” baselines, and while some participants did submit about the transparency of a particular image. For example, the systems that exploited richer models (e.g., term dependence models search hook can run an executable whose source code is not publicly and pseudo-relevance feedback), there was insufficient diversity in available. Such an image, while demonstrating replicability, would the retrieval models examined. Primarily due to these issues, the not allow other researchers to inspect the inner workings of a exercise has received less follow-up and uptake than the organizers particular retrieval method. While such images are not forbidden had originally hoped. in our design, they are obviously less desirable than images based on open code. In practice, however, we anticipate that most images 3 DOCKER AND “THE JIG” will be based on open-source code. From a technical perspective, our efforts are built around Docker, a One technical design choice that we have grappled with is how widely-adopted Linux-centric technology for delivering software in to get data “into” and “out of” a container. To be more concrete, lightweight packages called containers. The Docker Engine hosts for ad hoc retrieval the container needs access to the document one or more of these containers on physical machines and manages collection and the topics. The jig also needs to be able to obtain their lifecycle. One key feature of Docker is that all containers run the run files generated by the image for evaluation. Generically, on a single operating system kernel; isolation is handled by Linux there are three options for feeding data to an image: first, the data kernel features such as cgroups and kernel namespaces. This makes can be part of the image itself; second, the data can be fetched containers far more lightweight than virtual machines, and hence from a remote location by the image (e.g., via curl, wget, or some easier to manipulate. Containers are created from images, which other network transfer mechanism); third, the jig could mount an are typically built by importing base images (for example, capturing external data directory that the container has access to. The first a specific software distribution) and then overlaying custom code. two approaches are problematic for our use case: images need to The images themselves can be manipulated, combined, and modified be shareable, or resources need to be placed at a publicly-accessible as first-class citizens in a broad ecosystem. For example, a group location online. This is not permissible for document collections can overlay several existing images from public sources, add in its where researchers are required to sign license agreements before using. Furthermore, both approaches do not allow the possibility of testing on blind held-out data. We ultimately opted for the third 2 Note that the exercise is more accurately characterized as replicability and not repro- ducibility; the event predated ACM’s standardization of terminology. 3 https://github.com/osirrc/jig 2 approach: the jig mounts a (read-only) data directory that makes jig Docker the document collection available at a known location, as part of image User specifies the contract between the jig and the image (and similarly for topics). : Starts image A separate directory that is writable serves as the mechanism for prepare the jig to gather output runs from the image for evaluation. This phase Triggers hook init hook method makes it possible for images to be tested on blind held-out Triggers hook documents and topics, as long as the formats have been agreed to index hook in advance. Creates snapshot Finally, any evaluation exercise needs to define the test collection. We decided to focus on newswire test collections because their smaller sizes support a shorter iteration and debug cycle (compared to, for example, larger web collections). In particular, we asked : Triggers hook with snapshot participants to focus on the TREC 2004 Robust Track test collection, search search hook in part because of its long history: a recent large-scale literature phase run files meta-analysis comprising over one hundred papers [13] provides a rich context to support historical comparisons. trec_eval Participants were also asked to accommodate the following two (more recent) test collections if time allowed: • TREC 2017 Common Core Track, on the New York Times Anno- tated Corpus. Figure 1: Interactions between the jig and the Docker image • TREC 2018 Common Core Track, on the Washington Post Corpus. in the canonical evaluation flow. Finally, a “reach” goal was to support existing web test collections (e.g., GOV2 and ClueWeb). Although a few submitted images do support one or more of these collections, no formal evaluation was them readable by the image for indexing (see discussion in the conducted on them. previous section). The jig triggers two hooks in the image: • First, the init hook is executed. This hook is meant for actions 3.2 Implementation Details such as downloading artifacts, cloning repositories and compiling In this section we provide a more detailed technical description source code, or downloading external resources (e.g., a knowledge of the jig. Note, however, that the jig is continuously evolving as graph). Alternatively, these steps can be encoded directly in the we gather more image contributions and learn about our design image itself, thus making init a no-op. These two mechanisms shortcomings. We invite interested readers to consult our code largely lead to the same end result, and so it is mostly a matter repository for the latest details and design revisions. To be clear, we of preference for the image developer. describe v0.1.1 of the jig, which was deployed for the evaluation. • Next, the index hook is executed. The jig passes in a JSON string The jig is implemented in Python and communicates with the containing information such as the collection name, path, format, Docker Engine via the Docker SDK for Python.4 In the current etc. required for indexing. The image manages its own index, specification, each hook corresponds to a script in the image that which is not directly visible to the jig. has a specific name, resides at a fixed location, and obeys a speci- After indexing has completed, the jig takes a snapshot of the image fied contract dictating its behavior. Each script can invoke its own via a Docker commit. This is useful as indexing generally takes interpreter: common implementations include bash and Python. longer than a retrieval run, and this design allows multiple runs (at Thus, via these scripts, the image has freedom to invoke arbitrary different times) to be performed using the same index. code. In the common case, the hooks invoke features of an existing During the search phase, the user issues a command specifying open-source search engine packaged in the image. an image’s repository (i.e., name) and tag (i.e., version), the collec- From the perspective of a user who is attempting to replicate tion to search, and a number of auxiliary parameters such as the results using an image, two commands are available: one for prepa- topics file, qrels file, and output directory. This hook is meant to ration (the prepare phase) and another for actually performing perform the actual ad hoc retrieval runs, after which the jig evalu- the ad hoc retrieval run (the search phase). The jig handles the ates the output with trec_eval. Just as in the index hook, relevant execution lifecycle, from downloading the image to evaluating run parameters are encoded in JSON. The image places run files in the files using trec_eval. This is shown in Figure 1 as a timeline in the /output directory, which is mapped back to the host; this allows canonical lifecycle, with the jig on the left and the Docker image the jig to retrieve the run files for evaluation. on the right. The two phases are described in detail below. In addition to the two main hooks for ad hoc retrieval experi- During the prepare phase, the user issues a command specifying ments, the jig also supports additional hooks for added functionality an image’s repository (i.e., name) and tag (i.e., version) along with (not shown in Figure 1). The first of these is the interact hook a list of collections to index. As part of the contract between the jig that allows a user to interactively explore an image in the state that and an image, the jig mounts the document collections and makes has been captured via a snapshot, after the execution of the index hook. This allows, for example, the user to “enter” an interactive 4 https://docker-py.readthedocs.io/en/stable/ shell in the image (via standard Docker commands), and allows 3 the user to explore the inner workings of an image. The hook also Following the deadline for submitting images, the organizers ran allows users to interact with services that a container may choose all images “from scratch” with v0.1.1 of the jig and the latest release to expose, such as an interactive search interface, or even Jupyter of each participant’s image. Using our script (see Section 3.2), each notebooks. With the interact hook, the container is kept alive image was executed sequentially on a virtual machine instance in in the foreground, unlike the other hooks which exit immediately the Microsoft Azure cloud. Note that it would have been possible once execution has finished. to speed up the experiments by running the images in parallel, Finally, images may also implement a train hook, enabling an each on its own virtual machine instance, but this was not done. image to train a retrieval model, tune hyper-parameters, etc. after We used the instance type Standard_D64s_v3, which according the index hook has been executed. The train hook allows the user to Azure documentation is either based on the 2.4 GHz Intel Xeon to specify training and test splits for a set of topics, along with E5-2673 v3 (Haswell) processor or 2.3 GHz Intel Xeon E5-2673 v4 a model directory for storing the model. This output directory is (Broadwell) processor. Since we have no direct control over the mapped back to the host and can be passed to the search hook physical hardware, it is only meaningful to compare efficiency for use during retrieval. Currently, training is limited to the CPU, (i.e., performance metrics such as query latency) across different although progress has been made to support GPU-based training. images running on the same virtual machine instance. Nevertheless, In the current design, the jig runs one image at a time, but addi- our evaluations focused solely on retrieval effectiveness. This is a tional tooling around the jig includes a script that further automates shortcoming, since a number of images packaged search engines all interactions with an image so that experiments can be run end to that emphasize query evaluation efficiency. end with minimal human supervision. This script creates a virtual The results of running the jig on the submitted images com- machine in the cloud (currently, Microsoft Azure), installs Docker prise the “official” OSIRRC 2019 image library, and is available engine and associated dependencies, and then runs the image using on GitHub.6 We have captured all log output, run files, as well as the jig. All output is then captured for archival purposes. trec_eval output. These results are summarized below. For the TREC 2004 Robust Track test collection, 13 images gen- 4 SUBMITTED IMAGES AND RESULTS erated a total of 49 runs, the results of which are shown in Ta- Although we envision OSIRRC to be an ongoing effort, the reality of ble 1; the specific version of the image is noted. Effectiveness is a physical SIGIR workshop meant that it was necessary to impose measured using standard rank retrieval metrics: average precision an arbitrary deadline at which to “freeze” image development. This (AP), precision at rank cutoff 30 (P30), and NDCG at rank cutoff 20 occurred at the end of June, 2019. At that point in time, we received (NDCG@20). The table does not include runs from the following 17 images by 13 different teams, listed alphabetically as follows: images: Solrini and Elastirini (which are identical to Anserini runs), • Anserini (University of Waterloo) EntityRetrieval (where relevance judgments are not available since • Anserini-bm25prf (Waseda University) it was designed for a different task), and IRC-CENTRE2019 (which • ATIRE (University of Otago) was not designed to produce results for this test collection). • Birch (University of Waterloo) As the primary goal of this workshop is to build community, • Elastirini (University of Waterloo) infrastructure, and consensus, we deliberately attempt to minimize • EntityRetrieval (Ryerson University) direct comparisons of run effectiveness in the presentation: runs • Galago (University of Massachusetts) are grouped by image, and the image themselves are sorted alpha- • ielab (University of Queensland) betically. Nevertheless, a few important caveats are necessary for • Indri (TU Delft) proper interpretation of the results: Most runs perform no parame- • IRC-CENTRE2019 (Technische Hochschule Köln) ter tuning, although at least one implicitly encodes cross-validation • JASS (University of Otago) results (e.g., Birch). Also, runs might use different parts of the com- • JASSv2 (University of Otago) plete topic: the “title”, “description”, and “narrative” (as well as • NVSM (University of Padua) various combinations). For details, we invite the reader to consult • OldDog (Radboud University) the overview paper by each participating team. • PISA (New York University and RMIT University) We see that the submitted images generate runs that use a diverse • Solrini (University of Waterloo) set of retrieval models, including query expansion and pseudo- • Terrier (TU Delft and University of Glasgow) relevance feedback (Anserini, Anserini-bm25prf, Indri, Terrier), term proximity (Indri and Terrier), conjunctive query processing All except for two images were designed to replicate runs for the (OldDog), and neural ranking models (Birch and NVSM). Several im- TREC 2004 Robust Track test collection, which was the primary ages package open-source search engines that are primarily focused target for the exercise. The EntityRetrieval image was designed on efficiency (ATIRE, JASS, JASSv2, PISA). Although we concede to perform entity retrieval (as opposed to ad hoc retrieval). The that there is an under-representation of neural approaches, rela- IRC-CENTRE2019 image packages a submission to the CENTRE tive to the amount of interest in the community at present, there reproducibility effort,5 which targets a specific set of runs from are undoubtedly replication challenges with neural ranking mod- the TREC 2017 Common Core Track. A number of images also els, particularly with their training regimes. Nevertheless, we are support the Common Core Track test collections from TREC 2017 pleased with the range of systems and retrieval models that are and 2018. Finally, a few images also provide support for the GOV2 represented in these images. and ClueWeb test collections, although these were not evaluated. 5 http://www.centre-eval.org/ 6 https://github.com/osirrc/osirrc2019-library 4 Results from the TREC 2017 Common Core Track test collection well as replicability of those reproducibility efforts. The jig also are shown in Table 2. On this test collection, we have 12 runs from supports mechanisms for evaluations on document collections and 6 images. Results from the TREC 2018 Common Core Track test information needs beyond those that an image was originally de- collection are shown in Table 3: there are 19 runs from 4 images. signed for. This aligns with intuitive notions of what it means for a technique to be generalizable. Overall, we believe that our efforts have moved the field of infor- 5 FUTURE VISION AND ONGOING WORK mation retrieval forward both in terms of supporting “good science” Our efforts complement other concurrent activities in the commu- as well as sustained, cumulative empirical progress. This work nity. SIGIR has established a task force to implement ACM’s policy shows that it is indeed possible to coordinate a large, community- on artifact review and badging [5], and our efforts can be viewed as wide replication exercise in ad hoc retrieval, and that Docker pro- a technical feasibility study. This workshop also complements the vides a workable foundation for a common interface and lifecycle recent CENTRE evaluation tasks jointly run at CLEF, NTCIR, and specification. We invite the broader community to join our efforts! TREC [6, 11]. One of the goals of CENTRE is to define appropriate measures to determine whether and to what extent replicability 6 ACKNOWLEDGEMENTS and reproducibility have been achieved, while our efforts focus on We would like to thank all the participant who contributed Docker how these properties can be demonstrated technically. Thus, the jig images to the workshop. This exercise would not have been possible can provide the means to achieve CENTRE goals. Given fortuitous without their efforts. Additional thanks to Microsoft for providing alignment in schedules, participants of CENTRE@CLEF2019 [4] credits on the Azure cloud. were encouraged to participate in our workshop, and this in fact led to the contribution of the IRC-CENTRE2019 image. REFERENCES From the technical perspective, we see two major shortcomings [1] Jaime Arguello, Matt Crane, Fernando Diaz, Jimmy Lin, and Andrew Trotman. of the current jig implementation. First, the training hook is not as 2015. Report on the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR). SIGIR Forum 49, 2 (2015), 107–116. well-developed as we would have liked. Second, the jig lacks GPU [2] Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. support. Both will be remedied in a future iteration. Improvements That Don’t Add Up: Ad-Hoc Retrieval Results Since 1998. In We have proposed and prototyped a technical solution to the Proceedings of the 18th International Conference on Information and Knowledge Management (CIKM 2009). Hong Kong, China, 601–610. replicability challenge specifically for the SIGIR community, but [3] Monya Baker. 2016. Is There a Reproducibility Crisis? Nature 533 (2016), 452–454. the changes we envision will not occur without a corresponding [4] Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, and Ian Soboroff. 2019. Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility cultural shift. Sustained, cumulative empirical progress will only Realm. In Experimental IR Meets Multilinguality, Multimodality, and Interaction. be made if researchers use our tools in their evaluations, and this Proceedings of the Tenth International Conference of the CLEF Association (CLEF will only be possible if images for the comparison conditions are 2019). Lugano, Switzerland. [5] Nicola Ferro and Diane Kelly. 2018. SIGIR Initiative to Implement ACM Artifact available. This means that the community needs to adopt the norm Review and Badging. SIGIR Forum 52, 1 (2018), 4–10. of associating research papers with source code for replicating [6] Nicola Ferro, Maria Maistro, Tetsuya Sakai, and Ian Soboroff. 2018. Overview of results in those papers. However, as Voorhees et al. [12] reported, CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm. In Proceedings of the Ninth International Conference of the CLEF Association (CLEF having a link to a repository in a paper is far from sufficient. The 2018). Avignon, France, 239–246. jig provides the tools to package ad hoc retrieval experiments in a [7] Juliana Freire, Norbert Fuhr, and Andreas Rauber (Eds.). 2016. Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science. standard way, but these tools are useless without broad adoption. Schloss Dagstuhl–Leibniz-Zentrum für Informatik, Germany. The incentive structures of academic publishing need to adapt to [8] Frank Hopfgartner, Allan Hanbury, Henning Müller, Ivan Eggel, Krisztian Ba- encourage such behavior, but unfortunately this is beyond the scope log, Torben Brodt, Gordon V. Cormack, Jimmy Lin, Jayashree Kalpathy-Cramer, Noriko Kando, Makoto P. Kato, Anastasia Krithara, Tim Gollub, Martin Potthast, of our workshop. Evelyne Viega, and Simon Mercer. 2018. Evaluation-as-a-Service for the Compu- Given appropriate extensions, we believe that the jig can be tational Sciences: Overview and Outlook. ACM Journal of Data and Information augmented to accommodate a range of batch retrieval tasks. One Quality (JDIQ) 10, 4 (November 2018), 15:1–15:32. [9] Jimmy Lin. 2018. The Neural Hype and Comparisons Against Weak Baselines. important future direction is to add support for tasks beyond batch SIGIR Forum 52, 2 (2018), 40–51. retrieval, for example, to support interactive retrieval (with real or [10] Jimmy Lin, Matt Crane, Andrew Trotman, Jamie Callan, Ishan Chattopadhyaya, John Foley, Grant Ingersoll, Craig Macdonald, and Sebastiano Vigna. 2016. To- simulated user input) and evaluations on private and other sensitive ward Reproducible Baselines: The Open-Source IR Reproducibility Challenge. In data. Moreover, our effort represents a first systematic attempt Proceedings of the 38th European Conference on Information Retrieval (ECIR 2016). to embody the Evaluation-as-a-Service paradigm [8] via Docker Padua, Italy, 408–420. [11] Tetsuya Sakai, Nicola Ferro, Ian Soboroff, Zhaohao Zeng, Peng Xiao, and Maria containers. We believe that there are many possible paths forward Maistro. 2019. Overview of the NTCIR-14 CENTRE Task. In Proceedings of the building on the ideas presented here. 14th NTCIR Conference on Evaluation of Information Access Technologies. Tokyo, Finally, we view our efforts as a stepping stone toward repro- Japan. [12] Ellen M. Voorhees, Shahzad Rajput, and Ian Soboroff. 2016. Promoting Repeata- ducibility, and beyond that, generalizability. While these two im- bility Through Open Runs. In Proceedings of the 7th International Workshop on portant desiderata are not explicit goals of our workshop, we note Evaluating Information Access (EVIA 2016). Tokyo, Japan, 17–20. [13] Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the that the jig itself can provide the technical vehicle for delivering “Neural Hype”: Weak Baselines and the Additivity of Effectiveness Gains from reproducibility and generalizability. Some researchers would want Neural Ranking Models. In Proceedings of the 42nd Annual International ACM to package their own results in a Docker image. However, there is SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019). Paris, France. nothing that would prevent researchers from reproducing another [14] Ruifan Yu, Yuhao Xie, and Jimmy Lin. 2018. H2 oloo at TREC 2018: Cross- team’s results, that is then captured in a Docker image conforming Collection Relevance Transfer for the Common Core Track. In Proceedings of the to our specifications. This would demonstrate reproducibility as Twenty-Seventh Text REtrieval Conference (TREC 2018). Gaithersburg, Maryland. 5 Image Version Run AP P30 NDCG@20 Anserini v0.1.1 bm25 0.2531 0.3102 0.4240 Anserini v0.1.1 bm25.rm3 0.2903 0.3365 0.4407 Anserini v0.1.1 bm25.ax 0.2895 0.3333 0.4357 Anserini v0.1.1 ql 0.2467 0.3079 0.4113 Anserini v0.1.1 ql.rm3 0.2747 0.3232 0.4269 Anserini v0.1.1 ql.ax 0.2774 0.3229 0.4223 Anserini-bm25prf v0.2.2 b=0.20_bm25_bm25prf 0.2916 0.3396 0.4419 Anserini-bm25prf v0.2.2 b=0.40_bm25_bm25prf 0.2928 0.3438 0.4418 ATIRE v0.1.1 ANT_r4_100_percent.BM25+.s-stem.RF 0.2184 0.3199 0.4211 Birch v0.1.0 mb_2cv.cv.a 0.3241 0.3756 0.4722 Birch v0.1.0 mb_2cv.cv.ab 0.3240 0.3756 0.4720 Birch v0.1.0 mb_2cv.cv.abc 0.3244 0.3767 0.4738 Birch v0.1.0 mb_5cv.cv.a 0.3266 0.3783 0.4769 Birch v0.1.0 mb_5cv.cv.ab 0.3278 0.3795 0.4817 Birch v0.1.0 mb_5cv.cv.abc 0.3278 0.3790 0.4831 Birch v0.1.0 qa_2cv.cv.a 0.3014 0.3507 0.4469 Birch v0.1.0 qa_2cv.cv.ab 0.3003 0.3494 0.4475 Birch v0.1.0 qa_2cv.cv.abc 0.3003 0.3494 0.4475 Birch v0.1.0 qa_5cv.cv.a 0.3102 0.3574 0.4628 Birch v0.1.0 qa_5cv.cv.ab 0.3090 0.3577 0.4615 Birch v0.1.0 qa_5cv.cv.abc 0.3090 0.3577 0.4614 Galago v0.0.2 output_robust04 0.1948 0.2659 0.3732 ielab v0.0.1 robust04-1000 0.1826 0.2605 0.3477 Indri v0.2.1 bm25.title 0.2338 0.2995 0.4041 Indri v0.2.1 bm25.title.prf 0.2563 0.3041 0.3995 Indri v0.2.1 bm25.title+desc 0.2702 0.3274 0.4517 Indri v0.2.1 bm25.title+desc.prf.sd 0.2971 0.3562 0.4448 Indri v0.2.1 dir1000.title 0.2499 0.3100 0.4201 Indri v0.2.1 dir1000.title.sd 0.2547 0.3146 0.4232 Indri v0.2.1 dir1000.title.prf 0.2812 0.3248 0.4276 Indri v0.2.1 dir1000.title.prf.sd 0.2855 0.3295 0.4298 Indri v0.2.1 dir1000.desc 0.2023 0.2581 0.3635 Indri v0.2.1 jm0.5.title 0.2242 0.2839 0.3689 JASS v0.1.1 JASS_r4_10_percent 0.1984 0.2991 0.4055 JASSv2 v0.1.1 JASSv2_10 0.1984 0.2991 0.4055 NVSM v0.1.0 robust04_test_topics_run 0.1415 0.2197 0.2757 OldDog v1.0.0 bm25.robust04.con 0.1736 0.2526 0.3619 OldDog v1.0.0 bm25.robust04.dis 0.2434 0.2985 0.4002 PISA v0.1.3 robust04-1000 0.2534 0.3120 0.4221 Terrier v0.1.7 bm25 0.2363 0.2977 0.4049 Terrier v0.1.7 bm25_qe 0.2762 0.3281 0.4332 Terrier v0.1.7 bm25_prox 0.2404 0.3033 0.4082 Terrier v0.1.7 bm25_prox_qe 0.2781 0.3288 0.4307 Terrier v0.1.7 dph 0.2479 0.3129 0.4198 Terrier v0.1.7 dph_qe 0.2821 0.3369 0.4425 Terrier v0.1.7 dph_prox 0.2501 0.3166 0.4206 Terrier v0.1.7 dph_prox_qe 0.2869 0.3376 0.4435 Terrier v0.1.7 pl2 0.2241 0.2918 0.3948 Terrier v0.1.7 pl2_qe 0.2538 0.3126 0.4163 Table 1: Results on the TREC 2004 Robust Track test collection. 6 Image Version Run AP P30 NDCG@20 Anserini v0.1.1 bm25 0.2087 0.4293 0.3877 Anserini v0.1.1 bm25.rm3 0.2823 0.5093 0.4467 Anserini v0.1.1 bm25.ax 0.2787 0.4980 0.4450 Anserini v0.1.1 ql 0.2032 0.4467 0.3958 Anserini v0.1.1 ql.rm3 0.2606 0.4827 0.4226 Anserini v0.1.1 ql.ax 0.2613 0.4953 0.4429 ATIRE v0.1.1 ANT_c17_100_percent 0.1436 0.4087 0.3742 IRC-CENTRE2019 v0.1.3 wcrobust04 0.2971 0.5613 0.5143 IRC-CENTRE2019 v0.1.3 wcrobust0405 0.3539 0.6347 0.5821 JASS v0.1.1 JASS_c17_10_percent 0.1415 0.4080 0.3711 JASSv2 v0.1.1 JASSv2_c17_10 0.1415 0.4080 0.3711 PISA v0.1.3 core17-1000 0.2078 0.4260 0.3898 Table 2: Results on the TREC 2017 Common Core Track test collection. Image Version Run AP P30 NDCG@20 Anserini v0.1.1 bm25 0.2495 0.3567 0.4100 Anserini v0.1.1 bm25.ax 0.2920 0.4027 0.4342 Anserini v0.1.1 bm25.rm3 0.3136 0.4200 0.4604 Anserini v0.1.1 ql 0.2526 0.3653 0.4204 Anserini v0.1.1 ql.ax 0.2966 0.4060 0.4303 Anserini v0.1.1 ql.rm3 0.3073 0.4000 0.4366 OldDog v1.0.0 bm25.con 0.1802 0.3167 0.3650 OldDog v1.0.0 bm25.dis 0.2381 0.3313 0.3706 PISA v0.1.3 core18-1000 0.2384 0.3500 0.3927 Terrier v0.1.7 bm25 0.2326 0.3367 0.3800 Terrier v0.1.7 bm25_qe 0.2975 0.4040 0.4290 Terrier v0.1.7 bm25_prox 0.2369 0.3447 0.3954 Terrier v0.1.7 bm25_prox_qe 0.2960 0.4067 0.4318 Terrier v0.1.7 dph 0.2427 0.3633 0.4022 Terrier v0.1.7 dph_qe 0.3055 0.4153 0.4369 Terrier v0.1.7 dph_prox 0.2428 0.3673 0.4140 Terrier v0.1.7 dph_prox_qe 0.3035 0.4167 0.4462 Terrier v0.1.7 pl2 0.2225 0.3227 0.3636 Terrier v0.1.7 pl2_qe 0.2787 0.3933 0.3975 Table 3: Results on the TREC 2018 Common Core Track test collection. 7