University of Waterloo Docker Images for OSIRRC at SIGIR 2019 Ryan Clancy, Zeynep Akkalyoncu Yilmaz, Ze Zhong Wu, and Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo 1 OVERVIEW Solrini and Elastirini capabilities are exposed via the interact The University of Waterloo team submitted a total of four Docker hook in the OSIRRC jig. Since both Solr and Elasticsearch are de- images to the Open-Source IR Replicability Challenge (OSIRRC) signed as web apps, the user can trigger the hook and then directly at SIGIR 2019. This short overview outlines the functionality of navigate to a URL to access system capabilities. The batch runs each image. As the READMEs in all our source repositories provide provided by the solrini and elastirini images are exactly the details on the technical design of our images and the retrieval same as the anserini image. models used in our runs, we intentionally do not duplicate this The final image submitted by our group packages Birch, our newest information here. open-source search engine2 that takes advantage of BERT [4] for Our primary submission is a packaging of Anserini [11, 12], an ad hoc document retrieval: open-source information retrieval toolkit built around Lucene to https://github.com/osirrc/birch-docker facilitate replicable research. This anserini-docker image resides BERT can be characterized as one instance of a family of deep neu- at the following URL: ral models that make heavy use of pretraining [8, 9]. Application https://github.com/osirrc/anserini-docker to many natural language processing tasks, ranging from sentence The Anserini project grew out of the Open-Source IR Reproducibil- classification to sequence labeling, has led to impressive gains on ity Challenge from 2015 [5] and reflects growing community in- standard benchmark datasets. The model has been adapted to pas- terest in using Lucene for academic IR research [1, 2]. As Lucene sage ranking [7] and question answering [13], and Birch can be was not originally designed as a research toolkit, Anserini aims to viewed as a continuation of this thread of research, alongside other fill in the “missing parts” that allow researchers to run standard recent models such as CEDR [6]. The central insight that Birch ad hoc retrieval experiments “right out of the box”, including com- explores, as detailed in Yang et al. [14], is to aggregate sentence-level petitive baselines and integration hooks for neural ranking models. scores to rank documents. This image allows other researchers to Given Lucene’s tremendous production deployment base (typically replicate the results of our paper with the search hook. via Solr or Elasticsearch), better alignment between research in information retrieval and the practice of building real world search REFERENCES [1] L. Azzopardi, M. Crane, H. Fang, G. Ingersoll, J. Lin, Y. Moshfeghi, H. Scells, P. engines promises a smoother transition path from the lab to the Yang, and G. Zuccon. 2017. The Lucene for Information Access and Retrieval “real world” for research innovations. Research (LIARR) Workshop at SIGIR 2017. In SIGIR. 1429–1430. [2] L. Azzopardi, Y. Moshfeghi, M. Halvey, R. Alkhawaldeh, K. Balog, E. Di Buccio, In addition to our main Anserini image, we built two ancillary D. Ceccarelli, J. Fernández-Luna, C. Hull, J. Mannix, and S. Palchowdhury. 2017. images for the OSIRRC exercise: Lucene4IR: Developing Information Retrieval Evaluation Resources Using Lucene. SIGIR Forum 50, 2 (2017), 58–75. https://github.com/osirrc/solrini-docker [3] R. Clancy, T. Eskildsen, N. Ruest, and J. Lin. 2019. Solr Integration in the Anserini https://github.com/osirrc/elastirini-docker Information Retrieval Toolkit. In SIGIR. Paris, France. [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2018. BERT: Pre- In production environments, Lucene is most often used as a core training of Deep Bidirectional Transformers for Language Understanding. In arXiv:1810.04805. search library that powers two widely-deployed “full stack” search [5] J. Lin, M. Crane, A. Trotman, J. Callan, I. Chattopadhyaya, J. Foley, G. Ingersoll, C. applications: Solr and Elasticsearch. With “Solrini” and “Elastirini”, Macdonald, and S. Vigna. 2016. Toward Reproducible Baselines: The Open-Source we have integrated Anserini with Solr and Elasticsearch, respec- IR Reproducibility Challenge. In ECIR. 408–420. [6] S. MacAvaney, A. Yates, A. Cohan, and N. Goharian. 2019. CEDR: Contextualized tively. The integration is such that we can use Anserini as a common Embeddings for Document Ranking. In arXiv:1904.07094. frontend to index into a backend Solr or Elasticsearch instance. This [7] R. Nogueira and K. Cho. 2019. Passage Re-ranking with BERT. In arXiv:1901.04085. allows unification of the document processing pipeline (tokeniza- [8] M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep Contextualized Word Representations. In NAACL. 2227–2237. tion, stemming, etc.) to support standard TREC ad hoc experiments, [9] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving Lan- while allowing users to take advantage of the wealth of capabilities guage Understanding by Generative Pre-training. Technical Report. [10] E. Sadler. 2009. Project Blacklight: A Next Generation Library Catalog at a First provided by Solr and Elasticsearch. In the case of Solr, users can in- Generation University. Library Hi Tech 27, 1 (2009), 57–67. teract with sophisticated searching and faceted browsing interfaces [11] P. Yang, H. Fang, and J. Lin. 2017. Anserini: Enabling the Use of Lucene for such as Project Blacklight1 [10], as described in Clancy et al. [3]. In Information Retrieval Research. In SIGIR. 1253–1256. [12] P. Yang, H. Fang, and J. Lin. 2018. Anserini: Reproducible Ranking Baselines the case of Elasticsearch, we can gain access to the so-called ELK Using Lucene. JDIQ 10, 4 (2018), Article 16. stack (Elasticsearch, Logstash, Kibana) to provide a complete data [13] W. Yang, Y. Xie, A. Lin, X. Li, L. Tan, K. Xiong, M. Li, and J. Lin. 2019. End-to-End analytics environment, including slick visualization interfaces. Open-Domain Question Answering with BERTserini. In NAACL Demos. 72–77. [14] W. Yang, H. Zhang, and J. Lin. 2019. Simple Applications of BERT for Ad Hoc Document Retrieval. In arXiv:1903.10972. Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). OSIRRC 2019 co-located with SIGIR 2019, 25 July 2019, Paris, France. 1 https://projectblacklight.org/ 2 https://github.com/castorini/birch 36