The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the “jig” that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide efort to ensure experimental replicability and sustained progress on standard test collections.
The importance of repeatability, replicability, and reproducibility is broadly recognized in the computational sciences, both in supporting desirable scientific methodology as well as sustaining empirical progress. The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by building community consensus around a common technical specification, with reference implementations. This overview paper is an extended version of an abstract that appears in the SIGIR proceedings.
In order to precisely articulate the goals of this workshop, it is ifrst necessary to establish common terminology. We use the above terms in the same manner as recent ACM guidelines pertaining to artifact review and badging:1 • Repeatability (same team, same experimental setup): a researcher can reliably repeat her own computation. • Replicability (diferent team, same experimental setup): an independent group can obtain the same result using the authors’ own artifacts. • Reproducibility (diferent team, diferent experimental setup): an independent group can obtain the same result using artifacts which they develop completely independently.
This workshop tackles the replicability challenge for ad hoc document retrieval, with three explicit goals: (1) Develop a common Docker specification to support images that capture systems performing ad hoc retrieval experiments on standard test collections. The solution that we have developed is known as “the jig”. (2) Build a curated library of Docker images that work with the jig to capture a diversity of systems and retrieval models. (3) Explore the possibility of broadening our eforts to include additional tasks, diverse evaluation methodologies, and other benchmarking initiatives.
Trivially, by supporting replicability, our proposed solution enables
repeatability as well (which, as a recent case study has shown [
We hope that the fruits of this workshop can fuel empirical
progress in ad hoc retrieval by providing competitive baselines
that are easily replicable. The “prototypical” research paper of
this mold proposes an innovation and demonstrates its value by
comparing against one or more baselines. The often-cited
metaanalysis of Armstrong et al. [
Overall, we are pleased with progress towards the first two goals of the workshop. A total of 17 Docker images, involving 13 diferent teams from around the world, were submitted for evaluation, comprising the OSIRRC 2019 “image library”. These images collectively generated 49 replicable runs for the TREC 2004 Robust Track test collection, 12 replicable runs for the TREC 2017 Common Core Track test collection, and 19 replicable runs for the TREC 2018 Common Core Track test collection. With respect to the third goal, this paper ofers our future vision—but its broader adoption by the community at large remains to be seen. 2
There has been much discussion about reproducibility in the
sciences, with most scientists agreeing that the situation can be
characterized as a crisis [
Our workshop was conceived as the next iteration of the
OpenSource IR Reproducibility Challenge (OSIRRC), organized as part of
the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and
Generalizability of Results (RIGOR) [
In OSIRRC 2019, we aim to address two shortcomings with the previous exercise as a concrete step in moving the field forward. From the technical perspective, the RIGOR 2015 participants developed scripts in a shared VM environment, and while this was suficient to support cross-system comparisons at the time, the scripts were not suficiently constrained, and the entire setup suffered from portability and isolation issues. Thus, it would have been dificult for others to reuse the infrastructure to replicate the results—in other words, the replicability experiments themselves were dificult to replicate. We believe that Docker, which is a popular standard for containerization, ofers a potential solution to these technical challenges.
Another limitation of the previous exercise was its focus on “bag of words” baselines, and while some participants did submit systems that exploited richer models (e.g., term dependence models and pseudo-relevance feedback), there was insuficient diversity in the retrieval models examined. Primarily due to these issues, the exercise has received less follow-up and uptake than the organizers had originally hoped. 3
From a technical perspective, our eforts are built around Docker, a widely-adopted Linux-centric technology for delivering software in lightweight packages called containers. The Docker Engine hosts one or more of these containers on physical machines and manages their lifecycle. One key feature of Docker is that all containers run on a single operating system kernel; isolation is handled by Linux kernel features such as cgroups and kernel namespaces. This makes containers far more lightweight than virtual machines, and hence easier to manipulate. Containers are created from images, which are typically built by importing base images (for example, capturing a specific software distribution) and then overlaying custom code. The images themselves can be manipulated, combined, and modified as first-class citizens in a broad ecosystem. For example, a group can overlay several existing images from public sources, add in its own code, and in turn publish the resulting image to be further used by others. 3.1
As defined by the Merriam-Webster dictionary, a jig is “a device used to maintain mechanically the correct positional relationship between a piece of work and the tool or between parts of work during assembly”. The central activity of this workshop revolved around the co-design and co-implementation of a jig and Docker images that work with the jig for ad hoc retrieval. Of course, in our context, the relationship is computational instead of mechanical.
Shortly after the acceptance of the workshop proposal at SIGIR 2019, we issued a call for participants who were interested in contributing Docker images to our efort; the jig was designed with the input of these participants. In other words, the jig and the images co-evolved with feedback from members of the community. The code of the jig is open source and available on GitHub.3
Our central idea is that each image would expose a number of “hooks” that correspond to a point in the prototypical lifecycle of an ad hoc retrieval experiment: for example, indexing a collection, running a batch of queries, etc. These hooks then tie into code that captures whatever retrieval model a particular researcher wishes to package in the image—for example, a search engine implemented in Java or C++. The jig is responsible for triggering the hooks in each image in a particular sequence according to a predefined lifecycle model, e.g., first index the collection, then run a batch of queries, ifnally evaluate the results. We have further built tooling that applies the jig to multiple images to facilitate large-scale experiments. More details about the jig are provided in the next section, but first we overview a few design decisions.
Quite deliberately, the current jig does not make any demands about the transparency of a particular image. For example, the search hook can run an executable whose source code is not publicly available. Such an image, while demonstrating replicability, would not allow other researchers to inspect the inner workings of a particular retrieval method. While such images are not forbidden in our design, they are obviously less desirable than images based on open code. In practice, however, we anticipate that most images will be based on open-source code.
One technical design choice that we have grappled with is how to get data “into” and “out of” a container. To be more concrete, for ad hoc retrieval the container needs access to the document collection and the topics. The jig also needs to be able to obtain the run files generated by the image for evaluation. Generically, there are three options for feeding data to an image: first, the data can be part of the image itself; second, the data can be fetched from a remote location by the image (e.g., via curl, wget, or some other network transfer mechanism); third, the jig could mount an external data directory that the container has access to. The first two approaches are problematic for our use case: images need to be shareable, or resources need to be placed at a publicly-accessible location online. This is not permissible for document collections where researchers are required to sign license agreements before using. Furthermore, both approaches do not allow the possibility of testing on blind held-out data. We ultimately opted for the third 2Note that the exercise is more accurately characterized as replicability and not reproducibility; the event predated ACM’s standardization of terminology.
approach: the jig mounts a (read-only) data directory that makes the document collection available at a known location, as part of the contract between the jig and the image (and similarly for topics). A separate directory that is writable serves as the mechanism for the jig to gather output runs from the image for evaluation. This method makes it possible for images to be tested on blind held-out documents and topics, as long as the formats have been agreed to in advance.
Finally, any evaluation exercise needs to define the test collection.
We decided to focus on newswire test collections because their
smaller sizes support a shorter iteration and debug cycle (compared
to, for example, larger web collections). In particular, we asked
participants to focus on the TREC 2004 Robust Track test collection,
in part because of its long history: a recent large-scale literature
meta-analysis comprising over one hundred papers [
Participants were also asked to accommodate the following two (more recent) test collections if time allowed: • TREC 2017 Common Core Track, on the New York Times Annotated Corpus. • TREC 2018 Common Core Track, on the Washington Post Corpus. Finally, a “reach” goal was to support existing web test collections (e.g., GOV2 and ClueWeb). Although a few submitted images do support one or more of these collections, no formal evaluation was conducted on them. 3.2
In this section we provide a more detailed technical description of the jig. Note, however, that the jig is continuously evolving as we gather more image contributions and learn about our design shortcomings. We invite interested readers to consult our code repository for the latest details and design revisions. To be clear, we describe v0.1.1 of the jig, which was deployed for the evaluation.
The jig is implemented in Python and communicates with the Docker Engine via the Docker SDK for Python.4 In the current specification, each hook corresponds to a script in the image that has a specific name, resides at a fixed location, and obeys a speciifed contract dictating its behavior. Each script can invoke its own interpreter: common implementations include bash and Python. Thus, via these scripts, the image has freedom to invoke arbitrary code. In the common case, the hooks invoke features of an existing open-source search engine packaged in the image.
From the perspective of a user who is attempting to replicate results using an image, two commands are available: one for preparation (the prepare phase) and another for actually performing the ad hoc retrieval run (the search phase). The jig handles the execution lifecycle, from downloading the image to evaluating run ifles using trec_eval. This is shown in Figure 1 as a timeline in the canonical lifecycle, with the jig on the left and the Docker image on the right. The two phases are described in detail below.
During the prepare phase, the user issues a command specifying an image’s repository (i.e., name) and tag (i.e., version) along with a list of collections to index. As part of the contract between the jig and an image, the jig mounts the document collections and makes
User specifies <image>:<tag> prepare
phase <image>:<tag> search phase jig
Starts image Triggers hook Triggers hook Creates snapshot
<snapshot> Triggers hook with snapshot
run files trec_eval
Docker image init hook index hook search hook them readable by the image for indexing (see discussion in the previous section). The jig triggers two hooks in the image: • First, the init hook is executed. This hook is meant for actions such as downloading artifacts, cloning repositories and compiling source code, or downloading external resources (e.g., a knowledge graph). Alternatively, these steps can be encoded directly in the image itself, thus making init a no-op. These two mechanisms largely lead to the same end result, and so it is mostly a matter of preference for the image developer. • Next, the index hook is executed. The jig passes in a JSON string containing information such as the collection name, path, format, etc. required for indexing. The image manages its own index, which is not directly visible to the jig.
After indexing has completed, the jig takes a snapshot of the image via a Docker commit. This is useful as indexing generally takes longer than a retrieval run, and this design allows multiple runs (at diferent times) to be performed using the same index.
During the search phase, the user issues a command specifying an image’s repository (i.e., name) and tag (i.e., version), the collection to search, and a number of auxiliary parameters such as the topics file, qrels file, and output directory. This hook is meant to perform the actual ad hoc retrieval runs, after which the jig evaluates the output with trec_eval. Just as in the index hook, relevant parameters are encoded in JSON. The image places run files in the /output directory, which is mapped back to the host; this allows the jig to retrieve the run files for evaluation.
In addition to the two main hooks for ad hoc retrieval experiments, the jig also supports additional hooks for added functionality (not shown in Figure 1). The first of these is the interact hook that allows a user to interactively explore an image in the state that has been captured via a snapshot, after the execution of the index hook. This allows, for example, the user to “enter” an interactive shell in the image (via standard Docker commands), and allows the user to explore the inner workings of an image. The hook also allows users to interact with services that a container may choose to expose, such as an interactive search interface, or even Jupyter notebooks. With the interact hook, the container is kept alive in the foreground, unlike the other hooks which exit immediately once execution has finished.
Finally, images may also implement a train hook, enabling an image to train a retrieval model, tune hyper-parameters, etc. after the index hook has been executed. The train hook allows the user to specify training and test splits for a set of topics, along with a model directory for storing the model. This output directory is mapped back to the host and can be passed to the search hook for use during retrieval. Currently, training is limited to the CPU, although progress has been made to support GPU-based training.
In the current design, the jig runs one image at a time, but additional tooling around the jig includes a script that further automates all interactions with an image so that experiments can be run end to end with minimal human supervision. This script creates a virtual machine in the cloud (currently, Microsoft Azure), installs Docker engine and associated dependencies, and then runs the image using the jig. All output is then captured for archival purposes. 4
Although we envision OSIRRC to be an ongoing efort, the reality of a physical SIGIR workshop meant that it was necessary to impose an arbitrary deadline at which to “freeze” image development. This occurred at the end of June, 2019. At that point in time, we received 17 images by 13 diferent teams, listed alphabetically as follows: • Anserini (University of Waterloo) • Anserini-bm25prf (Waseda University) • ATIRE (University of Otago) • Birch (University of Waterloo) • Elastirini (University of Waterloo) • EntityRetrieval (Ryerson University) • Galago (University of Massachusetts) • ielab (University of Queensland) • Indri (TU Delft) • IRC-CENTRE2019 (Technische Hochschule Köln) • JASS (University of Otago) • JASSv2 (University of Otago) • NVSM (University of Padua) • OldDog (Radboud University) • PISA (New York University and RMIT University) • Solrini (University of Waterloo) • Terrier (TU Delft and University of Glasgow) All except for two images were designed to replicate runs for the TREC 2004 Robust Track test collection, which was the primary target for the exercise. The EntityRetrieval image was designed to perform entity retrieval (as opposed to ad hoc retrieval). The IRC-CENTRE2019 image packages a submission to the CENTRE reproducibility efort, 5 which targets a specific set of runs from the TREC 2017 Common Core Track. A number of images also support the Common Core Track test collections from TREC 2017 and 2018. Finally, a few images also provide support for the GOV2 and ClueWeb test collections, although these were not evaluated.
Following the deadline for submitting images, the organizers ran all images “from scratch” with v0.1.1 of the jig and the latest release of each participant’s image. Using our script (see Section 3.2), each image was executed sequentially on a virtual machine instance in the Microsoft Azure cloud. Note that it would have been possible to speed up the experiments by running the images in parallel, each on its own virtual machine instance, but this was not done. We used the instance type Standard_D64s_v3, which according to Azure documentation is either based on the 2.4 GHz Intel Xeon E5-2673 v3 (Haswell) processor or 2.3 GHz Intel Xeon E5-2673 v4 (Broadwell) processor. Since we have no direct control over the physical hardware, it is only meaningful to compare eficiency (i.e., performance metrics such as query latency) across diferent images running on the same virtual machine instance. Nevertheless, our evaluations focused solely on retrieval efectiveness. This is a shortcoming, since a number of images packaged search engines that emphasize query evaluation eficiency.
The results of running the jig on the submitted images comprise the “oficial” OSIRRC 2019 image library, and is available on GitHub.6 We have captured all log output, run files, as well as trec_eval output. These results are summarized below.
For the TREC 2004 Robust Track test collection, 13 images generated a total of 49 runs, the results of which are shown in Table 1; the specific version of the image is noted. Efectiveness is measured using standard rank retrieval metrics: average precision (AP), precision at rank cutof 30 (P30), and NDCG at rank cutof 20 (NDCG@20). The table does not include runs from the following images: Solrini and Elastirini (which are identical to Anserini runs), EntityRetrieval (where relevance judgments are not available since it was designed for a diferent task), and IRC-CENTRE2019 (which was not designed to produce results for this test collection).
As the primary goal of this workshop is to build community, infrastructure, and consensus, we deliberately attempt to minimize direct comparisons of run efectiveness in the presentation: runs are grouped by image, and the image themselves are sorted alphabetically. Nevertheless, a few important caveats are necessary for proper interpretation of the results: Most runs perform no parameter tuning, although at least one implicitly encodes cross-validation results (e.g., Birch). Also, runs might use diferent parts of the complete topic: the “title”, “description”, and “narrative” (as well as various combinations). For details, we invite the reader to consult the overview paper by each participating team.
We see that the submitted images generate runs that use a diverse set of retrieval models, including query expansion and pseudorelevance feedback (Anserini, Anserini-bm25prf, Indri, Terrier), term proximity (Indri and Terrier), conjunctive query processing (OldDog), and neural ranking models (Birch and NVSM). Several images package open-source search engines that are primarily focused on eficiency (ATIRE, JASS, JASSv2, PISA). Although we concede that there is an under-representation of neural approaches, relative to the amount of interest in the community at present, there are undoubtedly replication challenges with neural ranking models, particularly with their training regimes. Nevertheless, we are pleased with the range of systems and retrieval models that are represented in these images.
Results from the TREC 2017 Common Core Track test collection are shown in Table 2. On this test collection, we have 12 runs from 6 images. Results from the TREC 2018 Common Core Track test collection are shown in Table 3: there are 19 runs from 4 images.
Our eforts complement other concurrent activities in the
community. SIGIR has established a task force to implement ACM’s policy
on artifact review and badging [
From the technical perspective, we see two major shortcomings of the current jig implementation. First, the training hook is not as well-developed as we would have liked. Second, the jig lacks GPU support. Both will be remedied in a future iteration.
We have proposed and prototyped a technical solution to the
replicability challenge specifically for the SIGIR community, but
the changes we envision will not occur without a corresponding
cultural shift. Sustained, cumulative empirical progress will only
be made if researchers use our tools in their evaluations, and this
will only be possible if images for the comparison conditions are
available. This means that the community needs to adopt the norm
of associating research papers with source code for replicating
results in those papers. However, as Voorhees et al. [
Given appropriate extensions, we believe that the jig can be
augmented to accommodate a range of batch retrieval tasks. One
important future direction is to add support for tasks beyond batch
retrieval, for example, to support interactive retrieval (with real or
simulated user input) and evaluations on private and other sensitive
data. Moreover, our efort represents a first systematic attempt
to embody the Evaluation-as-a-Service paradigm [
Finally, we view our eforts as a stepping stone toward reproducibility, and beyond that, generalizability. While these two important desiderata are not explicit goals of our workshop, we note that the jig itself can provide the technical vehicle for delivering reproducibility and generalizability. Some researchers would want to package their own results in a Docker image. However, there is nothing that would prevent researchers from reproducing another team’s results, that is then captured in a Docker image conforming to our specifications. This would demonstrate reproducibility as well as replicability of those reproducibility eforts. The jig also supports mechanisms for evaluations on document collections and information needs beyond those that an image was originally designed for. This aligns with intuitive notions of what it means for a technique to be generalizable.
Overall, we believe that our eforts have moved the field of information retrieval forward both in terms of supporting “good science” as well as sustained, cumulative empirical progress. This work shows that it is indeed possible to coordinate a large, communitywide replication exercise in ad hoc retrieval, and that Docker provides a workable foundation for a common interface and lifecycle specification. We invite the broader community to join our eforts!
We would like to thank all the participant who contributed Docker images to the workshop. This exercise would not have been possible without their eforts. Additional thanks to Microsoft for providing credits on the Azure cloud.
ANT_r4_100_percent.BM25+.s-stem.RF 0.2184 0.3199 0.4211
Anserini Anserini Anserini Anserini Anserini
Anserini-bm25prf
Birch Birch Birch Birch Birch Birch Birch Birch Birch Birch Birch
Indri Indri Indri Indri Indri Indri Indri Indri Indri
OldDog
Terrier Terrier Terrier Terrier Terrier Terrier Terrier Terrier Terrier 0.2531 0.2903 0.2895 0.2467 0.2747 0.2774 0.2916 0.2928 0.3102 0.3365 0.3333 0.3079 0.3232 0.3229 0.3396 0.3438 JASSv2_c17_10 core17-1000 v0.1.1 v0.1.1 v0.1.1 v0.1.1 v0.1.1 v0.1.1 v0.1.1 v0.1.3 v0.1.3 v0.1.1 v0.1.1 v0.1.3 0.4293 0.5093 0.4980 0.4467 0.4827 0.4953 0.5613 0.6347
AP 0.2495 0.2920 0.3136 0.2526 0.2966 0.3073 0.1802 0.2381 core18-1000 0.2384 0.3500 0.3927