This manuscript discusses our ongoing work on ranx, a Python evaluation library for Information Retrieval. First, we introduce our work, summarize the already available functionalities, show the user-friendly nature of our tool through code snippets, and briefly discuss the technologies we relied on for the implementation and their advantages. Then, we present the upcoming features, such as several Metasearch algorithms, and introduce the long-term goals of our project.
In this section, we present the main functionalities ranx provides, show its user-friendly nature through some code snippets, and discuss its implementation and the advantages brought by the employed technologies. More details and examples are available in the oficial repository.
First, ranx provides a convenient way of managing the data needed for evaluating and
comparing diferent retrieval models: the query relevance judgments (qrels) and ranked lists of
documents retrieved for those queries by the systems (runs). ranx implements two custom
classes for these kinds of data: Qrels and Run. In particular, data can be loaded from Python
dictionaries and Pandas DataFrames [
ranx provides the most commonly used ranking evaluation metrics4 such as Reciprocal Rank,
Average Precision, and Normalized Discounted Cumulative Gain [
ranx also ofers functionalities to compare runs and perform statistical tests. As shown in Figure 3, by providing the query relevance judgments and a list of runs and defining the desired 2https://trec.nist.gov 3https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/2_qrels_and_run.ipynb 4A complete list of the implemented metrics can be found here: https://github.com/AmenRa/ranx#metrics 5https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/3_evaluation.ipynb metrics, the compare function performs a comparison of the runs. It returns a Report instance, which stores the information produced by the compare function and can be printed as in Figure 3 or exported as a LATEX table, ready for a scientific publication. The code underlying Table 1 was generated by ranx. To learn more about comparing diferent runs, we invite the reader to follow our online Jupyter Notebook6. 6https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/4_comparison_and_report.ipynb
In addition to its user-friendly interface, ranx is also very eficient due to its Numba-based
implementation. Numba[
We are currently implementing several Metasearch [
To conclude our discussion, we introduce the long-term goals of our library. Besides adding more metrics and other Metasearch methods, we plan to build a companion repository for storing runs of state-of-the-art models accompanied by rich metadata for searching and indexing. By integrating this online repository with ranx, we aim to allow researchers to download pre-computed runs and compare the results of their models with those of state-of-the-art approaches in just a few seconds. We think such functionality could help accelerate research in Information Retrieval, allowing researchers to rapidly find appropriate baselines and avoiding time-consuming and error-prone tasks entirely, such as re-implementing or re-training complex retrieval models from scratch. Moreover, sharing runs of state-of-the-art models could promote virtuous behaviors and transparency and reduce electricity consumption and pollution. [27] D. Lillis, L. Zhang, F. Toolan, R. W. Collier, D. Leonard, J. Dunnion, Estimating probabilities for efective data fusion, in: F. Crestani, S. Marchand-Maillet, H. Chen, E. N. Efthimiadis, J. Savoy (Eds.), Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, Geneva, Switzerland, July 1923, 2010, ACM, 2010, pp. 347–354. URL: https://doi.org/10.1145/1835449.1835508. doi:10. 1145/1835449.1835508. [28] A. Mourão, F. Martins, J. Magalhães, Multimodal medical information retrieval with unsupervised rank fusion, Comput. Medical Imaging Graph. 39 (2015) 35–45. URL: https: //doi.org/10.1016/j.compmedimag.2014.05.006. doi:10.1016/j.compmedimag.2014.05. 006. [29] S. E. Robertson, S. Walker, Some simple efective approximations to the 2-poisson model for probabilistic weighted retrieval, in: Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. Dublin, Ireland, 3-6 July 1994 (Special Issue of the SIGIR Forum), ACM/Springer, 1994. doi:10. 1007/978-1-4471-2099-5\_24. [30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017. [31] J. Lin, R. Nogueira, A. Yates, Pretrained Transformers for Text Ranking: BERT and Beyond, Synthesis Lectures on Human Language Technologies, Morgan & Claypool Publishers, 2021. URL: https://doi.org/10.2200/S01123ED1V01Y202108HLT053. doi:10.2200/ S01123ED1V01Y202108HLT053.