This report gives an overview of the system developed by Team DAM for Task 1 of the LongEval Lab at CLEF 2024. The team members are students enrolled in the Computer Engineering master's program at the University of Padua. The team developed an information retrieval system which is then used to perform queries on a corpus of documents in French language, or in their translated English version. Nowadays, online searching for all types of information has become part of people's daily routines. Billions of users worldwide expect to find needed information quickly and accurately. A search engine (SE) is a software that helps people satisfy such a need using queries to express an information need. Since the number of web pages has been rapidly increasing, there are considerable challenges that this type of software faces. One of the main challenges is the variability of performance of the system over time. That is why LongEval Lab, organized by the Conference and Labs of the Evaluation Forum (CLEF), aims to solve this problem by encouraging participants to develop information retrieval (IR) systems that can adapt to the evolution of corpus over time. The paper is organized as follows: Section 3 describes our approach; Section 4 explains our experimental setup; Section 5 discusses our main findings; finally, Section 6 draws some conclusions and outlooks for future work.
Sentence Embedders have been extensively used for the reranking phase of information retrieval systems
for many years [
Our approach is based on the usage of finely tuned of-the-shelf components provided by Apache
Lucene. In addition to those, a reranking phase based on Sentence Embedder has been implemented.
Our idea was to use a model fine-tuned mostly on online data (Reddit comments, citation pairs and
WikiAnswer, just to name a few) to try to encode the meaning of topics and documents in an efective
way. Furthermore, we tried to improve one of the works from the previous year which used reranking to
improve the IR system performance (Enrico Bolzonello et al. [
In this section we will cover the methodology that has been used to develop our IR system. In order to better understand the complete IR system we developed, all the system’s components will be explained using the diagram in Figure 1. The diagram shows the main components of a SE as well as the diferentiation between ofline and online components.
To develop our IR system, we used Apache Lucene version 9.10.0 1.
The Apache Lucene project develops open-source search software. It is a high-performance,
fullfeatured SE library written entirely in Java. This library provides a robust and scalable set of tools to
developers building eficient IR systems [
First of all, to create a fast and reliable IR system, it is necessary to parse the data we want to run our queries on. For this task, LongEval releases the corpus of documents in two diferent formats, TREC and JSON. We decided to use JSON files.
The corpus came divided into more files, each of them containing a JSON array of documents. The structure of a document is shown in Figure 2.
It was thus necessary to write a parser able to read the documents eficiently from the disk.
In order to create an eficient parser for reading documents from the LongEval corpus in JSON format,
several key components were implemented:
1Downloaded from: https://lucene.apache.org/core/downloads.html
• File Parser: a file parser is responsible for reading JSON files containing arrays of documents
eficiently. This component iterates through each line of the file and extracts the JSON objects
which represent a single document;
• Document Model: a document model defines the structure of a document in the corpus. In this
case, each document consists of two fields: docno (document number) and text (document text),
as shown in Figure 2. This model is used to deserialize JSON objects into Java objects during
parsing;
• JSON Deserialization: the parser implements a JSON deserialization library, Jackson [
Once the parsing of the documents has been set up, the very next step is analyzing the documents, in general, consisting of: • Tokenization: we split the documents into tokens, that are going to be our "unit" of computation in the system; • Stopword removal: a predefined list of words, considered to be useless in the context of search are removed. An example of these words is articles: they appear in every document so they do not help to discriminate documents; • Stemming: we reduce words to their root or base form to improve search results by capturing variations of the same word.
These are the techniques that have been used in at least one of our experiments: • StandardTokenizer • StopFilter • ICUFoldingFilter • LengthFilter • SnowballFilter
• EnglishPossessiveFilter • KStemFilter
• FrenchLightStemmer • ElisionFilter
While the techniques mentioned above are valid for both English and French analyzing, some techniques specific to a language have been implemented.
3.5.1. BM25 Once the documents analysis is completed, the next step is indexing. It is a crucial phase, since it creates a searchable database, the index, that contains essential metadata about parsed documents. This metadata includes details like the words and phrases within each document, their frequency, and their location within the document. By structuring documents in this manner, we improve retrieval eficiency, enabling users to search for documents based on keywords or phrases with ease. In order to do that, we indexed the parsed documents through an inverted index, using two diferent fields : So basically, every parsed document is saved in the indexer with these two fields, one used to identify the document itself and the other representing its whole content.
The Searcher serves as the component responsible for interpreting input queries and searching through indexed documents to identify those that fit best ( "best match") with the query. Then it retrieves these documents and presents them back to the user.
The next step is to fetch the pertinent documents based on the given queries: this involves identifying the most similar documents to our queries using various scoring functions. So we assign scores to each document in our collection, ranking them from highest to lowest. The highest-ranked document is presumed to be the most relevant to the given query.
The BM25 ranking function, that belongs to the “BM family” of retrieval models (BM stands for Best
Match), in addition of being simple and efective, seems to be very competitive compared to more
modern techniques [
q062228 aeroport bordeaux.
Once TSV queries are loaded and parsed they’re transformed into an Object, that stores num and text.
This is then submitted to the index searcher for retrieval of relevant documents, generating ranked lists
of documents based on their relevance to each query. Before submitting them to the actual searcher,
queries are parsed using Lucene Query Parser [
In order to improve the performance of our IR system, one possibility can be using proximity search, which allows us to search for a document based on how closely two or more search terms of the query appear in the document. The distance between the two terms is given by a parameter k, which depends on the context and the length of the documents. For example, the query "red brick house" could be used to retrieve documents that contain phrases like "red house of brick" or "house made of red brick", while in the meantime avoiding documents where the words are scattered or spread across. Later, we will discuss the improvements given by this kind of search.
Another way to improve the performance of our IR system is using query expansion. Query expansion
is a technique that consists in reformulating the queries to better match relevant documents. There
are several ways to perform it; the one we tried was synonym query expansion. With this approach,
each query term is expanded with its own synonyms. To find all the synonyms of every english word
we used Wordnet, that is a large lexical database of English containing all the words (nouns, verbs,
adjectives, . . . ) grouped into sets of cognitive synonyms [
After the searching process has retrieved the highest ranked documents with respect to the employed criteria, we can apply a second phase of ranking. This phase is not going to look through all the documents again, but instead it is going to work with the documents the first phase retrieved. In our case, we will take only the first documents for eficiency reasons. The value of will be discussed later. The following diagram shows the process flow happening during the reranking phase.
The reranking approach we use is based on machine learning, specifically on sentence embedding models.
Essentially, we employ a pre-trained model that maps text to a vector in a multidimensional space. For each one of the topics we are processing, we compute its vector. Then we take the first documents retrieved by Lucene’s searcher and compute their vector. We then compute a score for each match using the dot product. The resulting value is used to rerank the documents retrieved by the search by increasing their score accordingly.
For each one of the documents, its updated score will be computed as follows: ←
+ · sim (emb(), emb()) .
Where is a coeficient that is used to decide how much to value the output of the sentence embedder;
sim is the function that computes the similarity between the two vectors. In our system we use one
provided by the sentence_transformers python package [
The model we used is all-mpnet-base-v2 and it is freely available from HuggingFace [
The architecture of the model is based on Microsoft’s MPNet, which stands for Masked and Permuted
Language Modeling [
As a similarity function we used the dot_score function from the sentence_transformers
package. This function computes the dot product between the two vector associated with the documents
[
Figure 4 illustrates the reranking algorithm that has been implemented, detailing the number of documents after each phase. To clarify, N represents the number of top-ranked documents retrieved in the initial search phase, while k has already been mentioned. As it can be seen from the diagram, only scores for the first k documents are updated, while the remaining N-k documents aren’t reranked.
As we previously said, to interact with the model we used the sentence-transformers library in
python, and in order to interact with python from the java code, we use a flask HTTP server [
Althought this adds a little overhead, it was much simpler than interacting with the sentence embedder directly from Java.
All the experiments concerning the reranker are available in Section 5.4.
This section briefly explains another technique for improving SE results, called the N-gram word model.
The idea behind it is that phrases are particularly significant in the IR field. Specifically, statistics show
that most two or three-word queries are phrases. Because of this, it is important to consider multiple
words as phrases rather than independent words. However, the impact of using phrases can be complex,
so it is essential to be careful when using them. Let’s consider the following definition of the word
phrase: Phrase is any sequence of n words, also known as an n-gram. Sequences of two words are called
bigrams, while sequences of three words are called trigrams. The greater the frequency of occurrence of
a word n-gram, the higher the probability that it corresponds to a meaningful phrase in the language.
In the context of IR, n-grams are used to index and retrieve documents based on user queries [
The setup used to run the experiments is the following: • The system has been run on the collections available from clef-longeval.github.io/data/. All the evaluation has been performed on the 2024 Training Set; • To evaluate the performance of the SE we used trec_eval 9.0.7, available at trec.nist.gov/trec_eval/; • The repository with the code can be found at bitbucket.org/upd-dei-stud-prj/seupd2324-dam/. We ran most of the experiments on the following hardware:
For reproducibility reasons, the runs were also submitted to TIRA [
In particular for Information Retrieval TIREx (Information Retrieval Experiment Platform) [
Before exploring the results our system is able to obtain, it is useful to list all the possible components we can make use of and assign them a keyword. Such keyword will be used in the name of the run to identify it.
• FR: if the indexing and searching process was done on the French version of the documents. • EN: if the indexing and searching process was done on the English version of the documents. • Snowball: if the Snowball stemmer was used. • Krovetz: if the Krovetz stemmer was used. • FrenchLight: if the FrenchLight stemmer was used. • Poss: if the English possessive filter was used. • Elision: if the ElisionFilter was used. • Stop: if the list of stopwords was used. For the English documents we used a default one provided by Lucene. For the French documents we used a custom list that is available in our repository. It is important to note that for french documents we used both lists. This was done because the documents contain some paragraphs in English language. • ICU: if ICU folding was used. • Prox: if proximity search was used. If this keyword is present, we will write also the distance parameters, as explained in Section 3.5.3. For example, if proximity search at a distance of 50 characters is employed, we will write Prox(50). • Reranking: if reranking was used. This will come with a parameter too: the value we discussed in Section 3.5.5. The parameter instead will take a fixed value of 5. As an example, if the system reranks the first 50 documents, we will write Reranking(50). • Syns: if the query expansion technique using synonyms was used.
• Shingles: if word N-grams are being used. In our case we will generate ngrams of length 2 and 3.
All runs also employ a filter that discards all tokens shorter than 2 characters and longer than 20. Other than that, by default, all the tokens are transformed into lowercase. All the experiments have been run using BM25 with default parameters as the ranking function.
The two main metrics we are going to use to compare runs are the Normalized Discounted Cumulative
Gain (nDCG) and the Mean Average Precision (MAP). The reason why we chose these metrics is that
they are widely used in IR tasks, since they ofer a comprehensive understanding of the efectiveness
of retrieval systems. Additionally, they are simple and easy to understand and interpret. nDCG helps
measure how close the system’s output is when compared to an ideal run, where all the items are
sorted in decreasing order of relevance. Moreover, since nDCG is a normalized metric, it enables fair
comparisons between diferent lists of varying lengths and relevance distributions [
To set a baseline for our next runs, we ran a baseline Lucene configuration on both the French and the English documents. This baseline configuration will just use the standard tokenizer, the Snowball stemmer, the lower case filter and the BM25 as retrieval model.
The results we obtained are shown in Table 1. nDCG
As we can see, the performance on the “original” French dataset is better than the on on the translated English dataset. nDCG and MAP are 23.96% and 38.59% better respectively.
This could be explained by the fact that the automated translation is not very accurate and some information is lost in the process.
As described in Section 3, there are several stemmers that can be used to derive word roots. We made some runs to decide the best one for English and French respectively. For both collections, the corresponding stoplists were used to conduct the experiments.
FrenchLight stemmer, compared with SnowBall, improves the performance of our system, incrementing both nDCG and MAP of 1.47% and 2.06%, respectively.
In this case, SnowBall stemmer improves the performance of our system compared to Krovetz. Both nDCG and MAP increase of 1.86% and 3.18%, respectively.
In this section we are going to talk about synonym query expansion. We tried this approach both for English and French. The experiment setup was: • english stoplist, possessive filter and SnowBall stemmer for English collection.
• french stoplist, elision filter, ICU filter and FrenchLight stemmer for French collection. The synonyms included in the queries are analyzed with the same analyzer used for indexing the documents and, furthermore, are assigned a weight of 0.7 to reduce their significance compared to the words originally present in the queries. The results are reported in Tables 4 and 5. nDCG nDCG
As we can see, in both cases the synonym query expansion worsened the performance of our information retrieval system. One of the reasons why this is happening could be that synonyms that are not related to the queries are still included in them because, maybe, they have the same root. So, in the following experiments we completely discard synonyms to focus more on other methods to improve the efectiveness of our system.
Since, as it was covered in Section 3.5.5, we need to decide a threshold for the number of documents to undergo reranking, we made some runs to decide the best value.
We will use the reranker for both experiments with the English documents and French documents, so we tried it on both the collections. In addition to the techniques used in the previous experiment, we also added a stopword list.
For the French collection the stopword list, ICU folding and the elision filter have been used as well. The runs on the English collection made use of the English possessive filter. nDCG
As we can see, the reranker allow us to improve the performance of the retrieval system. Between the run with no reranking ( = 0) and the one with = 200, we notice an increase in the nDCG of 2.97%.
The best value of MAP occurs instead for = 150, with an increase of 9.25%. nDCG
For what concerns the English collection, the best result is obtained with = 200, with an increase of nDCG and MAP of, respectively, the 7.03% and 17.66%.
In this section we are going to compare the best configurations for various parameters in both French and English languages. These configurations represent the runs we have chosen to submit to the LongEval Conference: • The best system that doesn’t use reranking for English language; • The best system for English language, using reranking and SnowBall stemmer; • The best system that doesn’t use reranking; • The best overall system. This is the system that achieved the best results using reranking and other parameters discussed in the previous sections; • A system using word N-grams.
The results of these systems are displayed in the following Table 8.
EN-Stop-SnowBall-Poss-Prox(50) EN-Stop-SnowBall-Poss-Prox(50)-Reranking(200)
FR-Stop-FrenchLight-Elision-ICU-Prox(50)
FR-Stop-FrenchLight-Elision-ICU-Prox(50)-Reranking(150) FR-Stop-FrenchLight-Elision-ICU-Shingles-Prox(50)-Reranking(150) nDCG
One first notable observation is that our search engine generally performs better on French documents. As we previously noted, this trend may be attributed to the fact that the original corpus was written in French and later translated into English.
A further analysis of the results reveals that re-ranking leads to the best performance for both languages. However, the introduction of N-grams (Shingles) technique significantly decreases the search engine’s performance, nullifying the benefits of re-ranking and resulting in an even worse outcome compared to the best-performing configuration without this approach.
Figure 5 shows the interpolated precision-recall curves of the systems described in Table 8. This type of plot is useful to represent the trend of two diferent measures like precision and recall. In this way we can compare diferent systems to figure out which one is better than the others. System 5 intersects System 3 curve at recall 0.2. Hence, System 5 performs better at the high ranks and worse at the low ranks. Regarding other systems, the curves never intersect, so their performance are more clearly separated. Accordingly to the results reported in Table 8, System 4 is the best.
To conclude how well the submitted runs perform, we analyzed nDCG and MAP measures, and also performed Statistical Hypothesis Testing (SHT). The analysis was done for each of the five submitted runs, for both Short-term and Long-term collections. Short-term represents the collection from June 2023, while Long-term represents the collection from August 2023. This kind of analysis is essential as it shows how well each IR system performs. Moreover, it is particularly significant to compare these results since the purpose of the LongEval task is to find an IR system that can handle changes over time, meaning that we are interested in systems that have low measure drops over time. Table 9 shows the performance drop of the systems between June and August.
Additionally, SHT was performed to determine if the systems performance is statistically diferent or not. This is important to understand whether some configurations improve the system or not, for example, if the usage of reranking has a real efect on the performance or the mean increase is due only to the variance of the test.
In this section, we will further explain SHT, present the results for the previously mentioned metrics as well as for SHT, and provide our conclusions about them.
SHT is a type of statistical analysis used to estimate the relationship between statistical variables. Later in this section, the results produced by performing SHT on both Short-term and Long-term datasets will be shown and explained. SHT is important for determining in a scientifically valid way whether our systems are performing similarly or diferently. In other words, we are interested in knowing if there is a statistical significant diference between them.
In order to perform SHT, the two mutually exclusive hypotheses H0 and H1 must be defined. H 0 is called the null hypothesis while H1 is the alternative hypothesis. Beside these hypotheses a threshold must be defined, representing a significance level. For example, = 0.05 means there is a 5% probability of wrongly declaring that the systems are diferent. SHT uses sample data to determine if H 0 can be rejected. If that is the case, it means that the alternative hypothesis H1 is true [22].
In particular, we will use Two-Way Analysis of Variance (ANOVA2) as a statistical test. It examines the influence of two diferent variables which are, in our case, the systems and the topics. ANOVA2 is used to evaluate the diference between the means of more than two groups, which is useful in our case since we have 5 diferent IR systems. In ANOVA2 the hypotheses are as follows: • H0 - the means of all groups are equal, • H1 - at least 2 groups have diferent means [23].
In this section, results for nDCG and AP measures, as well as for SHT, are presented for Short-term collection. Additionally, conclusions for the given results are provided.
Table 9 displays nDCG and MAP measures of the five submitted systems for the Short-term collection. It is evident that the best-performing system for the training data, based on both measures, remains consistent: FR-Stop-FrenchLight-Elision-ICU-Prox(50)-Reranking(150). However, there is a notable performance drop of 21.12% for nDCG and 19.16% for MAP. This drop was expected, as user queries and preferences have evolved over time. Notably, the systems with English configurations experience the largest drops. This phenomenon could be related to the automated translation process from French to English.
Based on the box plots in Figures 6 and 7, it is notable that the FR systems tend to have higher median values for both measures compared to the EN systems, indicating generally better performance. Additionally, the FR systems do not have as many outliers as the EN systems, which shows a more consistent performance.
Table 10 and Table 11 show the results of the ANOVA2 test.
The SS column shows the total variability for each source. Higher values indicate greater variability. df indicates how many degrees of freedom a source of variation has. For example, since the matrix of the scores contains a system for each column, and 5 systems are being compared, the columns will have 4 degrees of freedom.
The MS column is the average of the sum of squares (SS) for each source, calculated by dividing SS by its corresponding degrees of freedom (df). It represents the variance for each source.
F shows the F-statistic, calculated as the ratio of the mean squares of the source to the mean squares of the error. It is used to determine if the observed variance between groups is significantly greater than the variance within groups.
Finally, the column we are most interested in, Prob>F, indicates the p-value associated with the F-statistic. A lower p-value (in our case less than 0.05) suggests that the diferences between group means are statistically significant.
In this case, both Table 10 and Table 11, allow us to state that the 5 systems have diferent performance with a probability of being wrong very close to 0.
After discovering that the 5 systems are diferent, it is useful to compare systems pairwise in order to understand where the diference comes from. For this purpose, we will employ Tukey’s Honest Significant Diference (HSD) test.
(a) nDCG
(b) AP In this section, results for nDCG and AP measures, as well as for SHT, are presented for Long term collection. Additionally, conclusions for given results are provided.
Table 9 shows the nDCG and AP scores of the five systems submitted for the Long-term collection. It’s clear that the best-performing system, based on both measures, remains consistent: FR-StopFrenchLight-Elision-ICU-Prox(50)-Reranking(150). However, there’s a significant performance decline of 41.45% for nDCG and 44.47% for MAP between training and Long-term data. This decline was expected, given the evolution of user queries and preferences over time.
Based on the box plots in Figures 9 and 10, we can make similar conclusions we made when comparing box plots for the Short-term collection - the FR systems showed better performance for both measures, compared to the EN systems. Moreover, the FR systems have fewer outliers than the EN systems. When comparing the box plots 9 and 10 for the Long-term collection with the box plots 6 and 7 for the Short-term collection, we can conclude the following: all five IR systems show a slight decrease in the median nDCG and AP values over time, which was expected. Another notable observation is that the box plots for the Long-term collection have more outliers than those for the Short-term collection. This behavior could be explained with various factors, such as the impact of diferent queries and documents and changes in user behavior over time.
In the same fashion as what was done for the Short-term dataset in Section 5.6.2, we run the ANOVA2 test and the pairwise comparison using the HSD test. Table 14 and Table 15 show the results of the ANOVA2 test. The output of the systems comparison is reported in Table 16 and Table 17 for AP and nDCG, respectively. The p-values lower than = 0.05 are shown in bold, meaning that we refuse to accept the null hypothesis. One notable thing is that, on these datasets, all the systems, except for System 3 and System 5 are statistically diferent.
(a) nDCG
SS
(b) AP
System 2 System 3 System 4 System 5 System 3 System 4 System 5 System 4 System 5 System 5
P-value
In this section we summarize the main achievements and the conclusions we reached during the development of the SE. Firstly, using a basic configuration (with the standard tokenizer, the Snowball stemmer, and the lowercase filter), we observed better performance on the French dataset compared to the English dataset. We could explain this behavior with the fact that the translation to English is automated.
Secondly, the default performance shown in Section 5 is quite good, but after experimenting with diferent configurations, we managed to find better parameters. It’s important to mention that we always used the appropriate stopword list, as we noticed an improvement in performance when utilizing them. Moreover, the SE performed better when using an appropriate stemmer, while the usage of synonyms had decremental efects on it. Another technique that brought significant progress was the usage of a reranker for both languages.
While analyzing performance drops and SHT results for both Short-term and Long-term collections, we arrived at the following conclusions. There is a significant diference between the systems with French and English configurations in terms of performance. The FR systems consistently outperform the EN systems in both datasets. As already mentioned, this is related to the automated translation from French to English and the fact that the language evolves over time [24]. This demonstrates that languagespecific optimizations play a vital role in the efectiveness of retrieval systems. Moreover, systems with reranking generally perform better than non-reranking systems. Furthermore, the performance drop over time is evident, and it highlights the need of continuous updates to maintain performance over time.
Regarding future work, we could try to improve query expansion with the use of Large Language Models (LLMs), which have shown remarkable capabilities in the IR field, particularly in text understanding [25]. Indeed, in order to better perceive the user’s intent and create a more eficient query, we could reformulate it. For this purpose, we could use a language model to rephrase the query and hopefully increase the performance of the SE. These models can grasp the context and meaning of text, enabling more accurate retrieval of relevant documents [26]. Another idea that could be beneficial to the performance of our system would be to use a sentence embedder model trained specifically on French data. This hypothetically could increase the boost in performance obtained with the use of the reranker even more. [21] Ren Jie Tan, Breaking Down Mean Average Precision (mAP), https://towardsdatascience.com/ breaking-down-mean-average-precision-map-ae462f623a52, 2019. [Online; accessed: 2024-05-02]. [22] Christina Majaski, Hypothesis Testing: 4 Steps and Example, https://www.investopedia.com/ terms/h/hypothesistesting.asp, 2024. [Online; accessed: 2024-05-15]. [23] Will Kenton, What Is Analysis of Variance (ANOVA)?, https://www.investopedia.com/terms/a/ anova.asp, 2024. [Online; accessed: 2024-05-15]. [24] Rabab Alkhalifa, Elena Kochkina, Arkaitz Zubiaga, Building for tomorrow: Assessing the temporal persistence of text classifiers, https://arxiv.org/abs/2205.05435, 2022. [Online; accessed: 2024-0522]. [25] Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zhicheng Dou, Ji-Rong Wen , Large Language Models for Information Retrieval: A Survey, https://arxiv.org/pdf/2308.07107, 2024. [Online; accessed: 2024-05-04]. [26] Vishal Gupta, Ashutosh Dixit, Shilpa Sethi, An Improved Sentence Embeddings based Information Retrieval Technique using Query Reformulation, https://ieeexplore.ieee.org/document/10141788, 2023. [Online; accessed: 2024-05-04].