We report on the systems that the Math Information Retrieval group at Masaryk University (mirmu) and the team of Faculty of Informatics students (msm) prepared for task 1 (find answers) of the arqmath lab at the clef conference. To study the efects of diferent system settings and hyperparameters, we have prototyped several diverse math-aware information retrieval (mir) systems: both “old” inverted index-based ones and new neural ones. By ensembling the results of the “weak” individual systems into committees, we report on entailments, benefits, and drawbacks of system ensembling. We evaluated the proposed individual systems and ensembles, considering their diversity, hyperparameters, and representations used, and classified their approaches. Our prototypes have helped to understand the challenging problems of question-answering in the stem domain: the key lies in the proper representation of document semantics. Our reproducible evaluation Python library PV211-utils allows to reproduce and further advance mir re-search.
Math Information Retrieval (mir) and math-aware representation of meaning of scientific
documents have been researched at mir laboratory at Masaryk University for decades, as nicely
summarized by Novotný [
Our arqmath reports [
Our systems were mainly developed as part of the Information Retrieval course PV211 and
will allow reproducible research using Python package PV211-utils [
In Section 2 we describe resources and methods used to train and develop our systems. Section 3 describes systems and strategies used to prepare our runs. We report and evaluate our results in Section 4 on page 6. We are summing up our conclusions with Section 5, drawing possible research plans based on computed metrics and availability of collected systems, ensembling techniques, and ground truth datasets.
This section describes the math representations ingested by our information retrieval systems, the corpora used for training the models that power our systems, and the relevance judgments we used for parameter optimization, model selection, and performance estimation.
We used the most straightforward math representation for all our submitted systems: LATEX.
In one submitted run of mirmu group, we studied the efect of the L ATEX encoding of math compared to the sole text to see how the presence of the math representation afects the resulting score.
We described our datasets, ensembling methods, and evaluation measures in detail in our
previous reports [
The following sections describe nine systems students have developed as part of their studies. Their diversity brings diferent ways to represent the meaning of math content and how to pick and rerank the answers for given topics. Table 1 on page 7 summarizes ten submitted runs by both MU teams.
Our systems submitted under the mirmu group consist of the following parts applied sequentially
after one another, inspired by RE3QA architecture [
• Indexer – to assign each document a dense vector representation; • Retriever – to compute the dense vector representation of the input query and to compute the cosine similarity between query and each document representation and sorting all documents by this similarity; • ReRanker – to rerank top- most relevant documents from Retrieving part. We achieved the best results with tiered reranking, taking multiple non-overlapping slices from top- results and reranks each slice separately. This part is computationally expensive; therefore, we cannot do it on the whole dataset in practice.
For implementing our system, we used the Sentence Transformer library, which is the Python
framework for state-of-the-art sentence, text, and image embeddings. [
For the implementation of the Base model, we used the BiEncoder model with the pre-trained
all-MiniLM-L12-v2 model [
We used Sentence Transformers library2 for fine-tuning of all our models. The architecture of both models and their diferences are depicted in Figure 1. The Base model fine-tuned only the ReRanker model, as the fine-tuning of the generic Retriever pre-trained for a retrieval on a vast and heterogeneous datasets did not bring measurable benefits of quality. We submitted ifne-tuned version of the ReRanker as the alternative run.
The ReRanker was fine-tuned using the BCE with logits loss and the AdamW optimizer. The training set consisted of relevance judgments, a train+validation set from ARQMath 2021, extended by additional samples. The additional positive samples, with the label equal to 1, were pairs of a question and an answer from the same thread, where the answer received at least 50 upvotes. The additional negative samples, with the label equal to 0, were the most similar documents given by Retriever but with no overlapping Math Stack Exchange tags. Because ifnding similar documents for all questions would not be feasible, we decided to generate ten negative samples for ten random questions at the start of each epoch.
The model was trained on all training queries and the same number of additional samples every epoch. The ratio of positive and negative samples was 0.5. The training process was stopped as there was no decrease in loss over the test set from ARQMath 2021.
For the mirmu team, we submitted diferent versions of the described system to see the efect of
the diferent components of the system. The variation we used are:
• finetuning / not finetuning of the Retriever model
• using ReRanker model pretrained on the Math texts (RoBERTa vs MathBERTa3) [
• using diferent representation of the input (text vs text + L ATEX)
We submitted the Base model as the primary as we couldn’t outperform its performance. The
other variants we submitted as the alternative runs are:
• Base – model described in the previous section;
• Trained only on text – system using only the text representation of documents and
queries;
• MathBERTa ReRanker – system using MathBERTa model as the ReRanker;
• Retriever fine-tuned – Retriever model adjusted by the relevance judgements collected
over ARQMath2020 and ARQMath 2021, using Negatives Ranking loss [
To quantify the efect of the individual changes in the mirmu system, only one parameter was
changed at time for given run, and all other hyperparameters were left the same.
3.2. tf-idf
As one of the students’ baseline systems, we used the tf-idf model implementation available in
the Gensim library [
In the preprocessing phase, we removed extreme values (below 8 absolute term frequency and higher than 0.7 relative document frequency), where we chose the hyperparameters experimentally on train subset. Then we removed punctuation, repeating whitespaces. Lastly, we tokenized the document by splitting on whitespace and stemmed individual tokens using the snowball stemmer available in snowball_py 4 library.
We used smart (System for the Mechanical Analysis and Retrieval of Text) tf-idf
weighting variant [
4https://github.com/shibukawa/snowball_py
3.3. BM25
BM25+ is an improvement over BM25 introduced by Lv and Zhai [
BM25+(, ) = ∑︁ log ∈ ︂( + 1 )︂ df
⎛ · ⎝
(1 + 1) · tf, 1 · ︁( (1 − ) + avg ︁( d )︁ )︁ + tfd
⎞ + ⎠, (1) where 1, , and are hyperparameters, is the number of documents in the collection, df is the number of documents containing the term , tf, is frequency of term in document , the length of document in words, and avg is the expected length of a document in words.
We represented our answers as a concatenation of its body and title, body, and tags of its parent question. In the next preprocessing stage, we firstly removed punctuation, repeating whitespaces, and English stopwords. Then we transformed the text into lowercase and stemmed it using Porter stemmer. Lastly, we tokenized the text by splitting it on whitespace.
We used the implementation of BM25+ in the rank_bm25 Python library [
As the example of the most straightforward possible ensemble system of two unsupervised systems, we used the ensembling of tf-idf and BM25 models. We constructed the ensemble as the simple sum of the given scores from the individual systems. The BM25 is configured as described in Section 3.3. The tf-idf system uses the same preprocessing as BM25 system and tf-idf implementation available in the Gensim library with smart tf-idf weighting variant, which corresponds to logarithmic term frequency weighting, zero-corrected idf, and no document normalization.
We also submitted the Compubert model prepared and submitted by the mirmu team last year.
The model and hyper-parameters can be found in last year’s report [2, Section 3.4].
This section describes the used ensemble systems. With ensembling and voting techniques, we
can combine the strengths of diferent systems to produce more accurate results. Historically,
there is a long tradition of boosting, [
As we believe that our systems could agree on a small portion of the most relevant documents,
reflecting diferent ‘points of view’ on the search problem. Depending on dozens of parameters,
each individual system will miss the great majority of relevant documents. With ensembling and
voting techniques, we can combine the strengths of diferent systems to produce more accurate
results. [
The ibc is ensemble technique we introduced in arqmath 2020 in our paper [
rrf+() = ∑︁ ∈
1 + () (2) where is set of rankings and () is ranking of document . The hyper-parameter parameterizes the ensemble. We used default value of = 60, suggested in [25].
As the rbc model we refer to trained regression model, which predicts the gain of train judgements from the ranks of the individual.
For the performance estimation of rbc, we produced a result list by taking the 1,000 answers with the highest predicted gain for each topic in the test subset. 3.6.1. IBC 3.6.2. RRF 3.6.3. RBC 3.6.4. WIBC wibc is weighted variant of the ibc algorithm. Instead of electing the candidate with the highest median rating, wibc elects the candidate with the highest weighted median rating. Instead of breaking ties by selecting a random rating out of a uniform distribution of all ratings, we select a random rating out of a weighted uniform distribution.
To compare the submitted systems, we evaluate their performance on the topics from previous arqmath competitions.
For each system, we report the resulted scores as in Table 2 in similar form as in the overview
arqmath 2022 paper by Mansouri et al. [
Performance drop in mirmu 2 run compared to other runs indicate that training and adjusting the system on math is a must.
We further fine-tuned our systems benefiting from more ground truth data and experience we got by previous evaluations.
• MathBERTa ReRanker – MathBerta ReRanker fine-tuned on altered preprocessing; • BM25-based system – BM25-based system msm 3 described in Section 3.3, where we optimized hyperparameters 1, , and using grid search. The values we found to yield the best ndcg′ on our training set are 1 = 1.8, = 0.75, and = 1. • Improved Base – Base system with reduced preprocessing, more precisely trained ReRanker and improved tiered reranking with slices on indexes: 3, 7, 12, 16, 20, 50, 100, 125. • Retriever only – A system with removed reranking phase. Document ranking is based only on cosine similarity on embeddings obtained from Retriever.
We report the results of extended experiments with ensembles in Table 3 on the next page. rrf ensembles deliver best results by far margin. The more diverse systems one combines, the more the quality metric as ndcg′ monotonously grows. As the math information systems have to cope with really complex problems, it is really hard to build one complex system that is capable of learning all the complex stuf: disambiguation of overloaded math symbols, structured ambiguous notation, deduction and long causal dependencies. ens 1: rrf 60 of 4 fine systems ⁓0⁓.4⁓9⁓3 ens 2: ibc of all 0.401 ens 3: ibc of all 4 msm 0.324 ens 4: ibc of all 5 mirmu 0.468 ens 5: ibc of all 4 msm +mirmu 1 0.354 ens 6: ibc of msm 4 and mirmu 1 0.459 ens 7: rrf 60 of all 0.480 ens 8: rrf 180 of all 0.486 ens 9: rrf 60 of all 4 msm 0.328 ens 10: rrf 60 of all 5 mirmu 0.465 ens 11: rrf 60 of all 4 msm and mirmu 1 0.422 ens 12: rrf 60 of msm 4 + mirmu 1 0.465 ens 13: rbc of all 0.476 ens 14: rbc of all 4 msm 0.312 ens 15: rbc of all mirmu 0.468 ens 16: rbc of all 4 msm and mirmu 1 0.475 ens 17: rbc of msm 4 and mirmu 1 0.474 ens 18: wibc of all 0.466 ens 19: wibc of all 4 msm 0.332 ens 20: wibc of all mirmu 0.466 ens 21: wibc of all 4 msm and mirmu 1 0.488 ens 22: wibc of msm 4 and mirmu 1 0.466
The Figure 2 shows the dependence of rrf quality on the parameter . Instead of suggested = 60 from the original paper, the best performance is with = 180. We hypotetize that the more diverse the primary systems are, the higher optimal parameter scores.
It remains to be studied to which extend the performance depends on participating individual systems. Also, how the performance change with diferent hyperparameters and initial setting and choices of pre-trained models, diversity and initial setup of individual systems.
For reproducibility, we are going to publish our notebooks, ensemble implementations and
models in our lab and course repository [
“You must cultivate activities that you love. You must discover work that you do, not for its utility, but for itself, whether it succeeds or not, whether you are praised for it or not, whether you are loved and rewarded for it or not, whether people know about it and are grateful to you for it or not.” Anthony de Mello
We have developed nine mir systems with as diverse approaches and variants as possible. We have evaluated them on available arqmath data from last three years. We have studied the ways how they could be ensembled to gain better performance. We have reported our findings: a) math-aware representation with deep models started to outperform flat token-level based systems b) ensembling done with expertise and insight about merits of individual systems matter.
In the future, we plan to further enhance our neural models and ensembling algorithms by several means: • evaluation of diferent ensembling strategies based on individual systems’ types and hyperparameter settings of neural systems; • evaluation of initial setting and hyperparameters of neural Retriever and ReRanker systems; • deep systems’ adaptation to math specifics by using Adapt r library [26]; • study robustness and out-of-domain performance of mir systems.
We thank all PV211 course students and former members of mir group for their contributions. We thank the two anonymous reviewers for their insightful comments. We extend our gratitude to the arqmath 2022 organizers for keeping the research of math information retrieval aflame.
This work has been partly supported by the Ministry of Education of CR within the LINDATCLARIAH-CZ project LM2018101. [20] Q. Wu, C. J. C. Burges, K. M. Svore, J. Gao, Adapting boosting for information retrieval measures, Information Retrieval 13 (2010) 254–270. doi:10.1007/s10791-009-9112-1. [21] Y. Wang, I.-C. Choi, H. Liu, Generalized Ensemble Model for Document Ranking in Information Retrieval, Computer Science and Information Systems 14 (2017) 123––151. doi:10.2298/csis160229042w. [22] R. Nuray, F. Can, Automatic Ranking of Information Retrieval Systems Using Data Fusion, Information Processing and Management 42 (2006) 595–614. doi:10.1016/j.ipm.2005. 03.023. [23] M. Mosbah, B. Boucheham, Majority Voting Re-ranking Algorithm for Content BasedImage Retrieval, in: E. Garoufallou, R. J. Hartley, P. Gaitanou (Eds.), Metadata and Semantics Research, Springer International Publishing, Cham, 2015, pp. 121–131. [24] A. T. Albaham, N. Salim, Quality Biased Thread Retrieval Using the Voting Model, in: Proceedings of the 18th Australasian Document Computing Symposium, ADCS ’13, ACM, New York, NY, USA, 2013, pp. 97–100. doi:10.1145/2537734.2537752. [25] G. V. Cormack, C. L. A. Clarke, S. Buettcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, in: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, ACM, New York, NY, USA, 2009, pp. 758–759. doi:10.1145/1571941.1572114. [26] M. Štefánik, V. Novotný, N. Groverová, P. Sojka, Adaptr: Objective-Centric Adaptation Framework for Language Models, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL, Dublin, Ireland, 2022, pp. 261–269. URL: https://aclanthology.org/2022.acl-demo.26.