In this paper, we describe the results of the participation of the Information Management Systems (IMS) group at AILA 2020 Task 1, precedents and statutes retrieval. In particular, we participated in both subtasks: precedents retrieval (task a) and statutes retrieval (task b). The goal of our work was to compare and evaluate the eficacy of a simple reproducible approach based on the use of either lemmas or stems with a tf-idf vector space model and a plain BM25 model. The results vary significantly from one subtask/evaluation measure to another. For the subtask of statutes retrieval, our approach performed well, being second only to a participant that used BERT to represent documents.
The remainder of the paper will introduce the methodology and a brief summary of the experimental settings that we used in order to create the oficial runs that we submitted for this task.
In this section, we summarize the pipeline for text pre-processing which has been developed in the last years [4]. In general, our method follows the principles described by [5] where the idea is to mine textual information from large text collections in an eficient and efective by means of organized workflows named pipelines. Pipelines are an efective way to manage the sequential process of text analysis by splitting the source code into steps, where the output of one step is the input for the subsequent step. The R programming language has an interesting set of packages that follow this idea, named tidyverse,2 that we will use in our experiments.
Apart from being a tidy way of organizing software, an important advantage in working with pipelines is that this practice promotes shareability and reproducibility in research workflows which is one of the main pillars in the European Open Science Cloud (EOSC).3
In order to produce the clean dataset, we followed the same pipeline for data ingestion and preparation for all the experiments: • split text into words;4 • remove stopwords; • remove words with less than two characters; • lemmatize/stem words;5 • compute tf-idf for each word; • compute relevance score (BM25) for each word.
In this section, we briefly describe the setting of oficial runs that we submitted for this task and the preliminary results sent by the organizers before the workshop.
The datasets of the two subtasks consisted in:
• Corpus: 3,257 casedocs • Queries: 50 description of situations • Corpus: 197 statutes • Queries: 50 description of situations (same as subtask 1) 2https://www.tidyverse.org 3https://www.eosc-portal.eu 4https://www.tidytextmining.com 5https://cran.r-project.org/web/packages/corpus/vignettes/stemmer.html
For subtask 1, we split each casedocs into the following parts (we add an example for casedoc number 1):
In the runs of subtask1, we used only the last field (that we named ‘text’) for the retrieval of precedents.
The goal of our experiments is to compare the efectiveness of the diferent lexical choices (stem or lemma) with a baseline (BM25).
We submitted three runs for each subtask, and we followed the same procedure for each subtask: • bm25_lemma: this run uses a BM25 retrieval model with lemmas; • tfidf_lemma: this run uses a tfidf document representation and a cosine similarity score on lemmas; • tfidf_stem: this run uses a tfidf document representation and a cosine similarity score on stems;
The organizers of this task provided the results (averaged across topics) achieved by many baselines compared to the runs of each participant. In Table 1 and Table 2, we show a summary of these results.
A preliminary analysis of the results shows that, in terms of standard evaluation measures such as MAP, BPREF, reciprocal rank, and P@10 the use of the BM25 with lemmas performed worse compared to the other two runs.
For subtask 1 (Table 1), the performance was poor compared to the median values for all the measures. This result, compared with the performance of the other subtask, requires a failure analysis to understand how the choice of just on field afected the performance, as well as how additional information in the statistics of the word may be useful or not (see for example team HLJIT2019-AILA of AILA 2019 [6]).
For subtask 2, the use of a vector space model with tf-idf on stems was the second best run overall. It is interesting to see that, despite the type of query (which is a complex description of the situation), and the result in the previous subtask, this approach performed very well compared to the other systems.
The aim of our participation to the FIRE AILA 2020 Task 1 was to test the efectiveness of a reproducible baseline without any learning strategy. The initial results show a completely diferent scenario according which subtask is considered. The approach seems very promising, but a failure analysis and a topic-by-topic comparison is needed to understand when and how the diferent combination in the retrieval pipeline are significantly better/worse than other models.