This paper elaborates on our submission to the ARQMath track at CLEF 2020. Our primary run for the main Task-1: Question Answering uses a two-stage retrieval technique in which the rst stage is a fusion of traditional BM25 scoring and tf-idf with cosine similarity-based retrieval while the second stage is a ner re-ranking technique using contextualized embeddings. For the re-ranking we use a pre-trained robertabase model (110 million parameters) to make the language model more math-aware. Our approach achieves a higher NDCG0 score than the baseline, while our MAP and P@10 scores are competitive, performing better than the best submission (MathDowsers) for text and text+formula dependent topics.
{ Achieve a competitive score that beats the best baseline in terms of the
NDCG0 score. { Achieve better results than the best submission for Text and Text+Math dependent posts. { Propose a Masked Language Model1 used for re-ranking candidate answers containing both text and math formulas.
Our paper has the following Sections. Section 2 discusses our two-stage retrieval approach which includes indexing and re-ranking phases. Section 3 describes our experimental setup, system con guration, and a detailed comparison of our results with the best submission. Conclusions are in Section 4. 2
Here we describe our two-stage cascade candidate retrieval and ranking
approach. tf-idf with cosine similarity and an o -the-shelf BM25-based search
platform [
First-Phase: Retrieval The aim of this phase is to get a su cient number of relevant candidates to be re-ranked in the next stage. Past work has used BM25 scoring for this but we add cosine similarity ranking as well.
Indexing: We convert the MSE dataset from XML to JSON format so that it can be ingested by the search platforms we use. Because indexing all answers will potentially lose information in the questions, we generate a document by concatenating the question (Q) post text (title and body) with their corresponding answer (A) posts text (body). As such, the question body and title concatenated with the answer body become a document - one unit of retrieval. This allows us to remove questions without corresponding answers and represent the relevant posts as one large document because the relevant information the user is seeking could be in either the answer or the question post. This results in a total of 1,435,643 Q+A posts to be indexed.
We index the data using two o -the-shelf libraries - Elasticsearch (ES) and Anserini. We use two di erent libraries because each has its strengths and weaknesses. Anserini seems to perform better than ES in terms of relevance ranking 1 https://huggingface.co/shauryr/arqmath-roberta-base-1.5M and recall, which we observed when searching and evaluating the three training queries for the Question Answering Task provided by the organizers. Elasticsearch has a more scalable implementation of tf-idf with cosine similarity which is slow for a large dataset when using Anserini. For the formulas, we use the raw LATEX strings. We re-rank answers based on contextualized text and formulas embeddings using RoBERTa.
Retrieval: Once all posts in Elasticsearch and Anserini have been indexed, we
query the topics/queries for Task-1 to get the top-1000 candidates. For each
task, each index is queried independently and later fused using Reciprocal Rank
Fusion (RRF) [
RRF score(p 2 P ) = X r2R
1
k + r(p)
;
(1)
where the original work [
Second-Phase: Re-Ranking Here we describe how we pretrain our language model and re-rank candidates obtained in the last phase.
BERT [
BERT-based systems [
We choose the RoBERTa model [
ES Anserini
Runtime in seconds 0.5 1.0 1.5 2.0 2.5 the Masked Language Model (MLM) to get contextualized token embeddings averaged over the sequence length to represent the candidates from Phase-1 into a 768 dimension vector. Similarly, we obtain topic/query embeddings and rank the candidates using their cosine distance. 3
Our experiments were conducted on a 24 core machine with Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz with 256GB of RAM with 4 RTX 2080 Ti GPUs. Default con gurations were adopted in ES (cosine similarity; tf-idf) and Anserini (BM25). Anserini runs on a single thread while ES uses a multi-threaded function. In Figure 1 we compare runtimes of BM25 ranking using Anserini and tf-idf based cosine similarity ranking using ES. The size of ES Q+A index is 3.6GB whereas for Anserini the size is 2.2GB.
Pretraining RoBERTa: We use the transformers2 library's implementation of RoBERTa to train the MLM. We start with the original weights released by the authors for roberta-base and then further pretrain the model on the MSE dataset. Fortunately, BASE-vocabulary used in the original was able to cover the whole MSE dataset so did not train from scratch. Further, we reduce the batch size to 4 per GPU and increase step size to facilitate gradient accumulation. We kept the maximum sequence length of 512 to accommodate longer posts.3 Q+A posts are usually longer than 512 tokens so we had to break the posts into chunks before feeding them to our system.
Once we are done with pretraining our MLM, we use it to extract the
embeddings for each token in the candidate posts from the rst-phase. For longer
Q+A posts, we had to nd a way of truncating them so that the sequence can t
2 https://github.com/huggingface/transformers
3 Link to training details
in the 512 token window. We experimented with the head tail approach [
We show in Table 1 how our system compares with other submissions and
the baselines. We only include the best submissions for baselines and other
systems. Our system BM25+tf+tf.BERT clearly beats the best baseline in terms
of NDCG0. Before the challenge, we had only three train topics provided by the
organizers on which we could test our approach. For these three topics tf-idf with
cosine similarity was running better than BM25 scoring, so we did not include
BM25 in our nal submission. Later we added BM25 scoring which showed a
greater increase in the number of relevant retrieved documents. It can be seen
that tf.BERT that selects candidates using tf-idf similarity and re-ranking
using our pretrained RoBERTa model does not achieve the best performance. But
when the rankings of tf.BERT and tf-idf are fused, NDCG0 [
8 4 23 note that the tf runs are not as consistent since ES does sharding and roundrobins between di erent shard searching. Thus, we did multiple runs to achieve the above scores.
Comparison with Other submissions: Our best unsubmitted run,
BM25+tf+tf.BERT was compared with other submissions, among which
MathDowsers is the system that achieved one of the best results in Task 1.
Submissions by Approach0 and MathDowsers leveraged Tangent-S [
(a) Number of topics for which each system had higher NCDG0 than the other system.
(b) Average NCDG0 over total topics depending on di erent types of topics.
In Figure 2 we see that our system is better for the topics that depend on text and text+formula. For Text dependent topics MathDowsers had 4 topics for which there NDCG0 score was higher than our best run, while our system performed better in 8 topics. For the 31 text+formula dependent topics our system had better NDCG0 score for 17 topics. MathDowsers achieves a higher ( 96:4% higher) NDCG0 score for the topics which are only formula dependent. In contrast, our system is better ( 22:42%) at ranking posts containing both formula and text. This is attributed to the contextualized embeddings which our pretrained MLM can produce. It models equations with surrounding text and hence has better performance in text+formula topics. The di erence ( 77:2%) is even more when we compare text dependant topics.
Overall, our participation in the ARQMath Track helped in our understanding of how to improve multi-modal search (text+formula) by exploiting state-ofthe-art text embedding and information retrieval models. In terms of e ectiveness, our most e ective post-hoc run BM25+tf+tf.BERT was able to outperform the baselines and all submissions except Mathdowsers in terms of NDCG0. Our post-hoc system achieved the best results for queries that depend on text and text+formula.
Future work would include the use of the MSE dataset to get question-answer post pairs to train a better ranking model. It would also be helpful to investigate how important a formula is in a question post to retrieve relevant answers.
We would like to thank members of the ARQMath lab at the Department of Computer Science in Rochester Institute of Technology for organizing this track. Special thanks to Behrooz Mansouri for providing the dataset, initial analysis of topics, and starter code to all the participants of the task; it made it easier for us to pre-process the data and jump directly to the experiments which have been presented in this work.