This manuscript discusses our ongoing work on ranxhub, an online repository for sharing pre-computed runs: the ranked lists of documents retrieved for a specific set of queries by a retrieval model. First, we discuss the many advantages and implications that an online repository for sharing runs can bring to the table. Then, we introduce ranxhub and its integration with ranx, a Python library for the evaluation and comparison of Information Retrieval runs, showing its very simple usage.
there is no other initiative with this specific purpose. To promote an open culture, all the data are available under the very permissive CC BY 4.0 license4. In the following, we motivate our work and describe its usage.
In this section, we introduce the main motivations behind the implementation of ranxhub: improving the time efectiveness of research, promoting transparency, and reducing the environmental impact caused by modern Information Retrieval research.
Speed up Research The evaluation and comparison (statistical tests) of new Information Retrieval models w.r.t. the state-of-the-art is an integral part of the research workflow, which often require time-consuming and error-prone activities, such as implementing, training, and executing baseline models to reproduce their results. Across research labs, those activities are usually carried out independently with limited research artifacts sharing, severely afecting the research process time-wise. A public repository providing pre-computed runs and seamlessly integrated into an evaluation library could enable researchers to find appropriate baselines and conduct comparative evaluations in just a few minutes, thus improving the time-efectiveness of their work.
Transparency Transparency is a fundamental principle in Science. Without the openness of
research artifacts, it is hard to assess the improvements and findings in a research field. We
believe providing tools to support virtuous behavior is as important as promoting research
ethics. In this regard, ranxhub can provide researchers with a quick and easy way to share
the results of their work and demonstrate the trustworthiness of their research papers while
gaining visibility. Although sharing well-documented and working source code for training
and evaluating a new retrieval model should be the final goal transparency-wise, it also comes
with some issues [
In this section, we overview the main features provided by ranxhub following the platform’s workflow. We first describe the browsing system. Then, we introduce the run cards, the collection of metadata enriching the available runs. Finally, we describe the integration with ranx and how to share a run with the community.
Browsing The browsing process5 of ranxhub works as follows: 1) the user chooses a
benchmark, such as MSMARCO [14] or the Multi-Domain Benchmark for Personalized Search
Evaluation [15], 2) the system shows a table for each test set related to the benchmark (e.g., Dev,
TREC DL 2019 [16], TREC DL 2020 [17] for MSMARCO), displaying the available runs, their
IDs, and the metric scores used for the specific test set, 3) the user choose a pre-computed run,
and 4) the system shows the related run card (described in the following section).
Pre-computed Runs and Run Cards A run comprises the results retrieved by a model
for a specific set of queries. Each run is accompanied by a collection of metadata (inspired by
ir-metadata [18]) called run card6. A run card is organized in four sections: 1) run metadata,
2) model metadata, 3) metric scores, and 4) links to the related resources, such as the model
source code, the original paper, and its BibTex. For brevity, we encourage the reader to refer to
footnote 6 to get an idea of how a run card is displayed and what metadata it includes.
Integration with ranx We extended ranx, a Python evaluation library, to integrate with
ranxhub and allow for downloading pre-computed runs. Thanks to this integration, users can
download pre-computed runs with a single function call and perform comparative evaluations
very rapidly. Specifically, once the users have chosen the pre-computed baseline run(s), they
can easily download and import them with ranx, as exemplified in Listings 1 and 2. In just a
few lines of code and a matter of seconds, users can compare multiple runs without the need
for implementing, training, and executing the related retrieval models. A working example can
be found here7. For further details on ranx, please refer to [
Sharing To share a pre-computed run with the community, researchers can rely on ranx to pack their runs in the correct format and upload them and their related run cards using our submission form8. An empty run card can be found here9. Once received, we will upload the run on our server (currently on Amazon AWS S310) to make it available to others and assign it a unique identifier. We believe submissions must be moderated to maintain high-quality standards and avoid cluttering. Therefore, we only accept runs related to already published research articles. Runs that replicate published results but not from the authors of the original papers are equally valid. As we do not intend to be a monopoly, all the available runs are 5https://amenra.github.io/ranxhub/browse 6Run card example: https://amenra.github.io/ranxhub/browse/amdbfpse/cs/bm25 7https://tinyurl.com/yc639v4y 8https://forms.gle/fK6wLS83yZeoS1mL8 9https://github.com/AmenRa/ranxhub/blob/main/files/runcard-empty.yaml 10https://aws.amazon.com/s3/ )
# Model MAP@100 MRR@100 NDCG@10 --- ------ -------- -------- -------a bm25 0.233 0.234 0.239 b bert 0.366 0.367 0.408 c my_run 0.405 0.406 0.451
exportable using ranx to JSON and TREC-style files. Moreover, as ranx is open source, other libraries can copy-paste part of its code to download data from ranxhub.
In this manuscript, we discussed our ongoing eforts to provide the community with an online repository for sharing pre-computed retrieval runs, called ranxhub, and the underlying motivations. Specifically, ranxhub could speed up research by avoiding time-consuming and error-prone activities such as implementing, training, and executing baseline models. Moreover, it could promote virtuous behavior and transparency and reduce the environmental impact of modern Information Retrieval research. We described ranxhub’s browsing system, the run cards, and the integration with ranx, a Python library for Information Retrieval evaluation. By leveraging this integration, users can compare the results of multiple systems in just a few lines of code. To conclude, we believe ranxhub could positively impact Information Retrieval research. However, its success can only be determined by a community efort and the will to pursue transparency and improve the research experience of others. J. Dean, Carbon emissions and large neural network training, CoRR abs/2104.10350 (2021).
URL: https://arxiv.org/abs/2104.10350. arXiv:2104.10350. [12] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, J. Mach.
Learn. Res. 21 (2020) 140:1–140:67. URL: http://jmlr.org/papers/v21/20-074.html. [13] H. Scells, S. Zhuang, G. Zuccon, Reduce, reuse, recycle: Green information retrieval research, in: E. Amigó, P. Castells, J. Gonzalo, B. Carterette, J. S. Culpepper, G. Kazai (Eds.), SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, ACM, 2022, pp. 2825–2837. URL: https://doi.org/10.1145/3477495.3531766. doi:10.1145/3477495.3531766. [14] P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Tiwary, T. Wang, Ms marco: A human generated machine reading comprehension dataset, 2016. URL: https://arxiv.org/ abs/1611.09268. doi:10.48550/ARXIV.1611.09268. [15] E. Bassani, P. Kasela, A. Raganato, G. Pasi, A multi-domain benchmark for personalized search evaluation, in: M. A. Hasan, L. Xiong (Eds.), Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, ACM, 2022, pp. 3822–3827. URL: https://doi.org/10.1145/3511808.3557536. doi:10.1145/3511808.3557536. [16] N. Craswell, B. Mitra, E. Yilmaz, D. Campos, E. M. Voorhees, Overview of the TREC 2019 deep learning track, CoRR abs/2003.07820 (2020). URL: https://arxiv.org/abs/2003.07820. arXiv:2003.07820. [17] N. Craswell, B. Mitra, E. Yilmaz, D. Campos, Overview of the TREC 2020 deep learning track, in: E. M. Voorhees, A. Ellis (Eds.), Proceedings of the Twenty-Ninth Text REtrieval Conference, TREC 2020, Virtual Event [Gaithersburg, Maryland, USA], November 1620, 2020, volume 1266 of NIST Special Publication, National Institute of Standards and Technology (NIST), 2020. URL: https://trec.nist.gov/pubs/trec29/papers/OVERVIEW.DL. pdf. [18] T. Breuer, J. Keller, P. Schaer, ir_metadata: An extensible metadata schema for IR experiments, in: E. Amigó, P. Castells, J. Gonzalo, B. Carterette, J. S. Culpepper, G. Kazai (Eds.), SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, ACM, 2022, pp. 3078–3089. URL: https://doi.org/10.1145/3477495.3531738. doi:10.1145/3477495.3531738.