This report presents the work conducted by our team for LongEval-Retrieval Task 1 [1] in CLEF 2023 [2]. The primary objective of this task is to develop an information retrieval system that can efectively adapt to the temporal evolution of Web documents. Using the Longeval Websearch collection provided by the commercial search engine Qwant[3], our team has built a retrieval system that addresses the challenges posed by the changing nature of Web documents and user search preferences. This paper discusses our approach to the subtasks of short-term persistence and long-term persistence, as well as the evaluation of our retrieval system's performance.
Nowadays, the advent of the internet has led to an exponential growth of the information available online and this brought a more challenging way to find relevant and accurate information. This is the environment where search engines play a vital role that allows us to access the correct information quickly and eficiently just with a few clicks. Information retrieval systems are more crucial than ever, influencing several fields such as healthcare, business and mainly education.
Our objective is to investigate the subject of information retrieval in search engines and its adaptability to changes over time. The necessity to create a retrieval system that can successfully handle the dynamic nature of the web is what stimulates this research’s development. Our team, QEVALS, is participating in this challenge as a student group project conducted in the Search Engines course a.y. 2022/23 at the Computer Engineering master’s degree at the
University of Padua. We’ll be working on the Longeval [
This project helped develop our understanding of the information retrieval systems used in the context of a web search engine. Specifically, we applied ourselves to the task of processing queries and documents in order to get the best possible ranking results to provide users with the most relevant information. The paper is organized as follows: Section 2 describes our approach; Section 3 explains our experimental setup; Section 4 discusses our main findings; ifnally, Section 5 draws some conclusions and outlooks for future work.
The development process was iterative and based heavily on discussions between the group members regarding the subjects covered during the lectures. However, our first results were significantly below the baseline set by the organizers of the LongEval task.
The initial implementations of the classes were very simple and we started adding features to
them day by day. The main purpose of an analyzer in an IR model is to pre-process the input
data in a way that reduces the complexity of the document representation, while still
retaining the relevant information necessary for accurate retrieval. The final models we submitted,
present a tokenizer implemented using LetterTokenizer class from Apache Lucene [
One of the tools that had the biggest impact on our system’s performance was the implementation of a StopFilter. As the name suggests, it is used to remove stopwords (very frequent words that contain little information about the contents of a document or a query) from a stream of text. While testing diferent stoplists and experimenting with query boosting some parts of the queries, we discovered that for some stoplists, individually boosting query tokens that were non-stopwords would improve system performance, even though the queries were subject to a StopFilter later down the pipeline. We tried using the same stoplist for our query boosting and our StopFilter, as well as diferent ones and found that the best combination varied on a case-by-case basis. This particular type of query boosting was efective only on less-performing stoplists, thus it isn’t included in the final system. These are the results compared:
• CountWordsFree [
As we can see from the obtained values, the Google stopwords list is the one that performs best, without stoplist based query boosting. It is worth noting that it is the shortest list that was tested. 2.1.1. Tested but unused filters The systems submitted by our group are the results of numerous trials and runs, and unfortunately many of our attempted approaches proved themselves to be inefective or unfeasible. In our first prototypes, a LengthFilter was used to remove the words with a number of characters below and over certain thresholds. Initially it incremented the model’s scores, but upon further refinements of other parts of the system it started having a negative impact on performance.
A few diferent configurations of a ShingleFilter were also attempted to create tokens from overlapping sequences of n words (token n-grams) from a token stream. We found they weren’t suited to the LongEval task.
Multiple attempts were made to integrate an NLP (Natural Language Processing) library into
our system. It could be used for many tasks, such as tokenization, sentence segmentation,
part-of-speech tagging and named entity extraction. Specifically, we focused on two diferent
libraries: OpenNLP [
The Indexer is a class responsible for creating an index of the terms in a collection of documents. The purpose of the indexer is to speed up the retrieval of relevant documents in response to user queries. It creates an internal data structure, the ’index’, that allows quicker access to data. An indexer reads through the text data, identifies important and searchable entities, and constructs an index of those entities and their location in the data.
Our custom indexer, called LongEvalIndexer, is an update of the initial indexer design user for the Tipster collection discussed during the lectures. The latest version of it, recursively visits each file in the specified document directory and extracts the contained text data through the Parser class. Each document is parsed and javascript or PHP code is removed through the TextFilter class. Subsequently, each document is stored in the index and divided into the following fields: • IDField: this field stores the unique identifier associated with each document; • BodyField: the body of the document is indexed and stored in this field after being processed by the parser. The field type is defined using a FieldType object, which is configured to store the term vectors, positions, and ofsets; • LinkField: the link corresponding to the document is extracted from the URLs file provided in the LongEval dataset and is processed to remove specific characters in order to obtain the words contained in the link. It is then added to the document using a custom LinkField, which is configured to store the term vectors, positions, and ofsets.
The Searcher is a class used to represent the user or the query that seeks information from a collection of documents, so its main function is to express the information need of the user in the form of a query. This is the last class to be used in the information retrieval process since to perform research on the documents they have to be indexed first. The final version of this class takes as input a set of queries from the set provided by LongEval and it builds a BoostQuery object from each query. The BoostQuery class allows to give a boost to the wrapped query. Boost values that are less than one will give less importance to this query compared to other ones while values that are greater than one will give more importance to the scores returned by this query. Combined with this class, we exploited the MultiFieldQueryParser class, which is used to associate queries and documents with multiple fields, also defining the weight they will have in the search. In our case, the fields we researched were "body" and "link". Using diferent weights, we noticed that the weights given to these fields were crucial for the model’s performance.
Other approaches for the implementation of this class were attempted but did not result in
performance gains. We tested two types of query expansion techniques, one based on the top
relevant documents for each query and another one developed with the help of gpt-3.5-turbo
model by OpenAI [
The second query expansion we tested made use of an NLP model developed by OpenAI. The idea was to give the model a single query as input and ask it to return an expanded one. The first problem we encountered was linked to the number of calls we could do, by default OpenAI limits users to 3 calls per minute, but each run was about 700 queries. In order to avoid this bottleneck, we formed a prompt to expand 100 queries in each call. Unfortunately, the model was not particularly accurate and sometimes it diverged too much, expanding the queries incorrectly. For this reason, we had to abandon this idea too, but we believe that with some refinements it could be a good improvement for a future model.
Similarity function is a fundamental component used to measure the similarity, or relatedness, between queries and documents. The function returns a score that reflects how well the document matches the query. During the development process the BM25 similarity model with default parameters was used as a baseline. The systems our team is submitting for the LongEval task have various configurations of similarity models in order to test how they perform with separate but similar datasets and study any trends that emerge. The models being tested are described in the Results 4 section.
The data used in the project are the evaluation collection provided for task 1, "LongEvalRetrieval", consisting of collections of web documents and queries provided by Qwant, a search engine. Both documents and queries were collected in French and then automatically translated into English. This scheme represents the collection process: The data can be described as follows: 3.1.1. Documents The collection includes relevant documents that are selected to be retrieved for each query and potentially irrelevant documents randomly sampled from the Qwant index to better represent the nature of a Web test collection.
• Train data: consists of 1,570,734 web pages, acquired during June 2022. They can be downloaded from the Lindat/Clarin website. • Test data: consist of 1,593,376 documents and 882 queries, collected over July 2022, for the short-term persistence sub-task and 1,081,334 documents and 923 queries, collected over September 2022, for the long-term persistence sub-task. Both can be downloaded from the Lindat/Clarin website. 3.1.2. Topics 3.1.3. Queries The queries are extracted from Qwant’s search logs, based on a set of selected topics, but exact details regarding them were not made available to us.
The queries are extracted from Qwant’s search logs, based on a set of selected topics. The query set was created in French and was automatically translated into English.
The evaluation tools used during development are trec_eval [
(1) where and are the precision and recall at the ℎ threshold.
The git repository containing the source code of the system is available in the repository at the link seupd2223-qevals (https://bitbucket.org/upd-dei-stud-prj/seupd2223-qevals). 3.4. Hardware components • Saverio’s PC: - CPU: Intel® Core™ i5-8600K 3.60GHz × 6 - GPU: NVIDIA GeForce GTX 1060 6GB - RAM: 16 GB DDR4 - SO: Ubuntu 22.04.2 LTS • Enrico’s PC (laptop) - CPU: Intel Core i7-8550U 1.8 GHz Base, 4.0 GHz Turbo - GPU: NVIDIA GeForce MX250 2GB GDDR5 - RAM: 16 GiB DDR4 - SO: Pop!_OS 22.04 LTS
The numerous runs that were executed on the LongEval training dataset provided us with
good baseline analyzer and searcher configurations that we could use to compare how diferent
similarity models performed on the LongEval task. Specifically, we learned that:
• Better results were obtained working on the dataset containing documents in the original
language compared to the English-translated one.
• In the case of French queries and documents the best-performing tokenization is obtained
through Lucene’s LetterTokenizer.
• Stop Lists can improve performance, but most available lists tend to over-filter the data
to the point of performing worse than using no filter at all.
• In case of stop list over-filtering it is possible to regain some performance by query
boosting tokens in the query that are specifically not in a stop list, even though the stop
list tokens get filtered later in the pipeline.
• The best performing Stop Lists had at most 200 words. Specifically, we chose the Google
stop list [
Using the configuration described above five diferent similarity models were tested: • QEVALS_BM25DFLT: BM25 similarity model with default parameters (k1 = 1.2, b = 0.75). • QEVALS_BM25CSTM: BM25 similarity model with the best-performing parameters that were found (k1= 1.2, b = 0.9). • QEVALS_LMDirichlet: Language model based similarity with Bayesian smoothing using Dirichlet priors. • QEVALS_DFR: Probabilistic model that measures the divergence from randomness. The three components chosen were: Geometric approximation of Bose-Einstein, Laplace’s law of succession and Uniform distribution of term frequency. • QEVALS_IB: Information-based model. The three components chosen were: Log-logistic probabilistic distribution, Total Term Frequency Lambda and Uniform distribution of term frequency.
Following is the summary table reporting the MAP and nDCG values obtained in the training set for each run: nDCG
As reported, the values of each run are similar, in fact the variances for both sets of measures are very low. This is due to the fact that the changes across the models are not so relevant, indeed the main structure is almost the same.
Here instead, there is the Interpolated Precision vs Recall graph for the five tested similarity models:
Interpolated Precision vs Recall
QEVALS_BM25DFLT QEVALS_BM25CSTM QEVALS_LMDirichlet
QEVALS_DFR QEVALS_IB 0.4 0.35
0.3 0.25
0.2 0.15
0.1 n o i s i c e r P 5 · 10− 2 0 0
0.5
Recall 0.1 0.2 0.3 0.4 0.6 0.7 0.8 0.9 1
The graph shows that the performances of the chosen similarity models are quite close to each other for the LongEval training dataset. In particular, BM25 performs best overall, and changes to the default parameters can result in further (though limited) gains in performance. The language model is very slightly better than BM25 at low recall, but drops of more steeply at around 30% Recall, while the statistical models have comparable performance to BM25 at most points, but lag behind at low recall.
This section aims to provide a comprehensive analysis of the performance of the five models submitted for evaluation. Various metrics are used to evaluate the efectiveness and eficiency of each model. These metrics include average precision(MAP), precision(P@10), normalized discounted cumulated gain (nDCG) and recall (Recall_1000).
First, we present three tables, each corresponding to one of the test sets. These tables compare the performance of each run based on the metrics listed above.
Furthermore, we reported the boxplots calculated from the average precision (AP) respectively for short and long term. These graphical representations are intended to provide a detailed overview of the variability and distortion of the system’s accuracy values.
Next, there are the comparative graphs of the MAP values obtained with the Tukey’s HSD of the runs for short and long term. This will ofer a comparison of the mean performance of the systems, highlighting their respective strengths and weaknesses.
Subsequently, we reported the ANOVA tables (one-way) respectively for long and short term. These tables show how the sum of squares are distributed according to source of variation, and hence the mean sum of squares.
Finally, we have plotted the topics performance at 5 for short and long term. 0.1329 0.1296
Heldout
Short nDCG nDCG
In Tables 6, 7, 8 the observed values of the diferent "runs" of the system are similar, without significant deviations, and we can observe that the MAP values in both tables remain around 20%. It is interesting to note that the QEVALS_IB run performed well in the heldout set, obtaining the best results (map: 0.2108, p@10: 0.1329, nDCG: 0.3597), indeed it did not scored as expected in the Short and Long Test datasets. The values obtained from run QEVALS_BM25DFLT in the short and long test set seem slightly better than the others.
From the boxplots in Figure 3 we can see that the accuracy values obtained from the runs are very similar, also we see that QEVALS_LMDirichlet and QEVALS_BM25CSTM seem to perform well in one test set and less in the other, instead QEVALS_DFR and QEVALS_BM25DFLT seem to maintain performance in both.
From the Figure 4, even if there are not relevant diferences in performances, as already confirmed from the previous analysis, we can highlight that the best run is QEVALS_BM25DFLT.
From the analysis of the two-way ANOVA, visible in Table 9 and Table 10, it is confirmed how there is no substantial diference between the runs. A low F-value statistic suggests that the variation between the groups is relatively small compared to the variation within them; in addition, a high F-value probability indicates that the probability that the observed diferences are due to chance is very high. This further confirms that the diferences between runs tested are not statistically significant.
Similarly to the training dataset, the diferent runs on the short-term and long-term datasets manifested significant consistency, with similar MAP, nDCG, and Recall values. This indicates a relative stability of system performance and shows the capacity of adaptation to the temporal evolution of the documents.
As expected, the overall performance of the runs showed no significant variation between our training baseline and the short-term and long-term document sets. Figure 5, in particular, shows that our system’s performance is virtually identical for the entire range of topics of the short-term and long-term datasets. The capability to have consistent performance regardless of the contents of a dataset can be considered an advantage of a system such as ours, which does not use past data to train a model for future applications. One notable aspect that we can infer from the test data is that the fine-tuning of the parameters of the BM25 similarity model is quite dependent on the available data. In fact, for both short-term and long-term dataset test runs, the default BM25 model performed slightly better than our fine-tuned one. However, in general, the overall performances don’t change drastically. Indeed, the values of the measurements difer at most ∼ 2%. We can conclude that for a task such as LongEval the similarity model choice alone has little impact on the overall performance of the system.
Finally, QEVALS_BM25DFLT (the default configuration of the BM25 similarity model) proved to be the best performing run, achieving the highest MAP, nDCG and Recall values among the diferent runs evaluated. Our conclusion is that this system is the most suited to the task out of the ones we tested.
While our systems do not perform much better than the baseline provided by LongEvals, the process to arrive at the systems we are submitting ofered us an opportunity to acquire experience and knowledge in dealing with information retrieval systems. Many of our approaches did not perform as hoped, however thanks to our numerous trials we are much better equipped to try new approaches to information retrieval in the future. Moreover, we are aware of many possible enhancements to the project.
Our first challenge was the language. At a baseline, performance is better on the dataset in the original language, so our team decided to focus more on the French dataset early on. Our initial attempts on the English dataset seemed less promising, but it’s possible that we would have had more success implementing the more complex systems for it.
The use of an NLP library such as OpenNLP or CoreNLP would have helped us take a big step forward. We learned that using Natural Language Processing technology to implement the whole model is probably unfeasible since it would be too expensive from a computational point of view, but we believe that a part of part-of-speech tagging or named entity extraction would have been helpful. Therefore, if we had a chance to run the project on more powerful computers, and suficient time to build eficient parallelization algorithms, we believe that the model could gain better results and performances
Another important aspect we were not able to implement properly was a query expansion technique, as discussed in the Methodology 4 section. There are many algorithms that could be used and some of them are already present in some libraries. The main problem was that our dataset was in French and we have not been able to find a good dictionary in order to expand the queries. So, we think that a query expansion technique with a proper dictionary could be a good addition to the project. Moreover, the two algorithms we tried to implement could be improved. In fact, the first one, which expands the query using the top retrieved documents’ words, may already be in a working state, but in order for it to work well, the rest of the system must provide precise enough results at low recall values. This is due to the fact that a better base model leads to more relevant documents and so, to more relevant words with which to expand the query. Due to low precision at low recall we were adding noise to the initial query that, in the end, was performing worse than the original one.
The query expansion technique which was using the gpt-3.5-turbo model, was not a success either, however, it must be noted that compared to our other approaches it is the one we worked on the least because of time limitations and OpenAI’s API access quota limitations for non-paying users. The problem our team ran into was the unpredictability of the answers: the model was asked to expand each query to five words, but in some instances it expanded queries to up to fifteen words, adding useless noise. For such a system to work, some additional experience in prompt formulation for information retrieval purposes would likely bring much better results.