This volume contains the papers presented at OSIRRC 2019: The OpenSource IR Replicability Challenge,1 co-located with the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019) held in Paris, France, on July 25, 2019. The importance of repeatability, replicability, and reproducibility is broadly recognized in the computational sciences, both in supporting desirable scienti c methodology as well as sustaining empirical progress. OSIRRC 2019 aims to address the replicability challenge for ad hoc document retrieval. That is, how can we make it easy for others to replicate our results, by building community consensus around a common technical speci cation, with reference implementations. Our vision is to build Docker-based infrastructure for replicating results on standard ad hoc retrieval test collections (newswire, web, etc.). A future research paper, for example, might be paired with a Docker image whose execution yields the results presented in the paper. However, to maximize the impact of these Docker images, the following would be desirable: { These Docker images should follow some common speci cation, with \hooks" for indexing, training, retrieval, etc. The development of this speci cation should involve a community process. { There needs to be evaluation infrastructure that calls the hooks above for multiple images to perform aggregation and analyses, for example, to populate a leaderboard or to evaluate the images on a blind held-out test set.
All papers were reviewed by two members of the Program Committee and we selected 3 position papers and 10 docker papers for presentation at the workshop. In total, we collected 17 docker images from the participants, comprising the OSIRRC 2019 \image library" and available at: This repository captures the runs produced by the images, trec eval output, as well as links to: the image source code itself, the ready-to-use image on Docker Hub, and an archival copy of the image on Zenodo.
We would like to express our special thanks to all the participants in the challenge, who greatly contributed to shape and de ne the common Docker infrastructure, the authors of the position papers, who stimulated the discussion with additional topics and viewpoints, the Program Committee members, who helped us in ensuring the quality of the published papers, and, last but not least, all the attendees who made OSIRRC 2019 a success.
Workshop co-Chairs
Program Committee