Preface


    This volume contains the papers presented at OSIRRC 2019: The Open-
Source IR Replicability Challenge,1 co-located with the 42nd International ACM
SIGIR Conference on Research and Development in Information Retrieval (SIGIR
2019) held in Paris, France, on July 25, 2019.
    The importance of repeatability, replicability, and reproducibility is broadly
recognized in the computational sciences, both in supporting desirable scientific
methodology as well as sustaining empirical progress.
    OSIRRC 2019 aims to address the replicability challenge for ad hoc document
retrieval. That is, how can we make it easy for others to replicate our results, by
building community consensus around a common technical specification, with
reference implementations.
    Our vision is to build Docker-based infrastructure for replicating results on
standard ad hoc retrieval test collections (newswire, web, etc.). A future research
paper, for example, might be paired with a Docker image whose execution yields
the results presented in the paper. However, to maximize the impact of these
Docker images, the following would be desirable:

 – These Docker images should follow some common specification, with “hooks”
   for indexing, training, retrieval, etc. The development of this specification
   should involve a community process.
 – There needs to be evaluation infrastructure that calls the hooks above for
   multiple images to perform aggregation and analyses, for example, to popu-
   late a leaderboard or to evaluate the images on a blind held-out test set.

We solicited two types of contributions from the community:

 – Participation in the replicability challenge and associated “docker paper”:
   we asked participants to contribute Docker images that encapsulate strong
   baselines as well as state-of-the-art techniques; examples include runs based
   on query expansion, term proximity models, and neural networks. In partic-
   ular, we actively encouraged involvement from researchers working on neural
   ranking models.
 – Position papers: we solicited position papers around issues of replicability
   in the IR field, with a special focus on infrastructures to enable them. The
   goal of these papers is to further stimulate discussion at the workshop and
   to help shape the path forward for the community.

  Copyright c 2019 for this paper by its authors. Use permitted under Creative Com-
  mons License Attribution 4.0 International (CC BY 4.0). OSIRRC 2019 co-located
  with SIGIR 2019, 25 July 2019, Paris, France.
1
  https://osirrc.github.io/osirrc2019/




                                      i
All papers were reviewed by two members of the Program Committee and we
selected 3 position papers and 10 docker papers for presentation at the workshop.
In total, we collected 17 docker images from the participants, comprising the
OSIRRC 2019 “image library” and available at:
    https://github.com/osirrc/osirrc2019-library
This repository captures the runs produced by the images, trec eval output, as
well as links to: the image source code itself, the ready-to-use image on Docker
Hub, and an archival copy of the image on Zenodo.
    We would like to express our special thanks to all the participants in the
challenge, who greatly contributed to shape and define the common Docker
infrastructure, the authors of the position papers, who stimulated the discussion
with additional topics and viewpoints, the Program Committee members, who
helped us in ensuring the quality of the published papers, and, last but not least,
all the attendees who made OSIRRC 2019 a success.



                                                                        July, 2019

                                      Ryan Clancy, Nicola Ferro, Claudia Hauff,
                                       Jimmy Lin, Tetsuya Sakai, Ze Zhong Wu
                                                             Workshop co-Chairs




                    Program Committee
Shane Culpepper, RMIT University, Australia
Yixing Fan, Chinese Academy of Sciences (CAS), China
Norbert Fuhr, University of Duisburg-Essen, Germany
Faegheh Hasibi, Radboud University, The Netherlands
Makoto P. Kato, University of Tsukuba, Japan
Diane Kelly, University of Tennensee, USA
Maria Maistro, University of Copenaghen, Denmark
Raffaele Perego, ISTI, CNR, Italy
Martin Potthast, Leipzig University, Germany
Ian Soboroff, National Institute of Standards and Technology (NIST), USA
Andrew Trotman, University of Otago, New Zealand
Arjen P. de Vries, Radboud University, The Netherlands
Justin Zobel, University of Melbourne, Australia




                                      ii