This work reports our participation in the retrieval task of the second LongEval lab iteration at CLEF 2024. As part of this year's contribution, we analyze to which extent prior relevance signals on the document level and term level can be used to improve the retrieval efectiveness. In order to exploit these kinds of signals, we fetch corresponding document identifiers pointing to the same document in the diferent dataset slices of all timestamps. Based on several heuristics, we submit and evaluate a total of five systems that either follow our previous year's methodology or that combine baseline rankings with prior relevance signals. Our evaluations provide insights to which extent these signals can be used but let us also conclude with several recommendations for future work. Most notably, we envision a companion resource that ties together all slices of the dataset by unified document identifiers to have a better understanding of more rigorous data splits and to avoid potential data leakage that might afect the evaluation of (deep) learning-based systems.
The overall goal of Information Retrieval (IR) systems is to assist in finding relevant information for given information needs. These systems are designed to cope with the ongoing flood of information. Since the information landscape is continuously changing, the foundational data basis is ever-evolving, exposing systems to always new and updated information. While these changes directly influence the retrieval efectiveness, they are rarely considered during evaluation. Web search is an especially dynamic search scenario since websites and queries change quickly. This evolving search setting is brought to a test in the LongEval shared task that has the main goal of evaluating how systems cope with changes over time. Therefore, retrieval systems are evaluated on progressing snapshots of a test collection.
While the systems are exposed to changing data, users expect consistent, good efectiveness. To provide and maintain this, it is essential to quantify the efectiveness regularly. However, since the foundational data source, including documents, topics, and qrels is evolving, it is an open question for how long evaluations remain valid or can be generalized. While the LongEval shared task provides a one-of-a-kind test bed for the described endeavors, previous submissions mainly investigate how the changes in the test collection afect systems that remain static across time. The systems submitted last year did not use the temporal aspects of the test collection and instead treated the snapshots as independent search tasks. Exploiting the historical data as prior relevance signals in ranking systems is an important step in learning about the temporal connection between the snapshots and what diferentiates longitudinal evaluations from cross test collection evaluations.
In our contribution, we aim to investigate diferent approaches that exploit prior relevance signals on diferent levels in intuitive and explainable ways. Our goal is to provide initial approaches to adapt the system to the evolving retrieval setup by using past snapshots, including ranked documents and qrels. By that, the system itself becomes an evolving component in this evaluation, like the documents, topics, and qrels. While these methods are mainly limited to topics with a known history, i.g. queries that were already issued previously, the Average Retrieval Performance (ARP) clearly improves.
In short, the contributions of this work are: • Three experimental systems that exploit information of the past snapshots (past rankings and qrels) as prior relevance signals and two baselines are submitted to the LongEval shared task. • An extensive evaluation of the submitted systems regarding their behavior and efectiveness across time through reproducibility and replicability measures. • We outline and discuss in detail three directions of future work, namely the need for intellectual relevance labels, possible data leakage, and a companion resource that makes the snapshots more accessible.
To facilitate reproducibility, we make the code publicly available on GitHub.1
The LongEval dataset is a retrieval test collection introduced in the LongEval2 CLEF lab in 2023 [
The test collection is originally available in French, but an English automatic translation is additionally
provided, on which this work mainly relies on. While we generally consider the translations to be good,
some mismatches and near duplicates occur. This is also reflected in a higher efectiveness for runs
using the French version [
The test collection statistics of the newly added snapshots are summarized in Table 1. They contain between 1.7 and 2.5 million documents and 404 to 1518 topics. Compared to last year’s snapshots, it contains more documents but fewer topics on average. The qrels classify the documents as not relevant, relevant, and highly relevant. The number of qrels is balanced almost evenly. Roughly the same amount of positive and negative labels are present while regarding the positive labels, only one-third are highly relevant. The distribution across topics is highly skewed, as visualized in Figure 1.
The dataset’s diferent snapshots overlap, strengthening the connection between them. Similar to the snapshots investigated last year, some topics appear in multiple or even all snapshots. Likewise, the document collection holds websites present in all snapshots. Additionally, many more topics and documents are added and removed over time. Besides creating and updating operations on the collection, some documents that are present in multiple snapshots also change over time. For components that were already present in a previous snapshot, the history of this component can be constructed by relating the current version to its previous ones. Tracking these diferent changes, even on an abstract
level, remains challenging since each snapshot uses diferent identifiers. Therefore, no direct connection between topics, documents, or qrels is possible. A detour must be made via additional information to connect the components across time. We chose to link the documents by the provided URLs and the topics by exact string matches of the French query string. Linking the components by exact matches on the strings is straightforward but not reliable because this will not identify (near) duplicates for which the URL or query slightly changed. Even though the English version of the dataset is used in general, for the topic linking, the French queries are used to avoid diferences occurring by the translation, especially with regards to the larger time gap between the snapshots from this year and the dataset from last years iteration in which the translation system might have changed. Linking between diferent temporal versions of the same topics and documents allows us to exploit the history of the test collection for the ranking.
The snapshots appear to be rather loosely connected since the overlap in topics is small. While in total
404 to 1518 topics are available, only 98 topics are present in all three new snapshots. In comparison,
124 topics could be found with the same matching method that are available in all snapshots from last
year [
Based on the data analysis of corresponding document identifiers described in the previous section, we can identify the same document in diferent snapshots of the datasets, i.e., the test collection at diferent timestamps. The following approaches are based on the heuristic that prior snapshots or rankings retrieved from earlier snapshots, bear some kind of relevance information that can be exploited for the ranking of the current timestamp. Generally, we derive relevance signals for the ranking at timestamp from the sub-collection or a ranking at one or more earlier timestamps − 1. This history of snapshots can also include the sub-collections from last year (0, 1, 2). Compared to an earlier snapshot, a current ranking at can contain new documents that are added after − 1, documents that were also present at − 1, and documents that were present at − 1 but now have diferent content.
An overview of all runs and their general methodology is given in Table 2.
Given the queries of the timestamp , we run BM25 [
The underlying retrieval method of the run submission CIR_BM25+monoT5 reranks the BM25 baseline
run CIR_BM25 with monoT5 [
The run submission BM25+time_boost is based on the hypothesis that given a document appears in both the ranking at the current timestamp and an earlier timestamp − 1, it is relevant — at least to the degree that it was not removed from the index and is still considered as a potential document in the ranking.
For this approach, besides the baseline BM25 ranking at timestamp , we also determine rankings at the earlier timestamp − 1 with the same set of queries but with the corresponding index at timestamp − 1. Both the scores and the document identifiers are normalized. This procedure results in two runs for the same set of topics that have possibly diferent documents ranked. Then, the documents that are also ranked at − 1 are boosted by ,( ) = {︃ 2 (1 − )2 if ∈ ,− 1 otherwise (1) where ,− 1 is the ranking corresponding to the query at timestamp − 1 and is a free parameter. Figure 2 illustrates the weighting score , based on diferent values. The basic intuition is that we can control the ratio between weights that are assigned to documents that are either contained in a ranking at an earlier timestamp or those documents that are not. 4https://huggingface.co/castorini/monot5-base-msmarco 1.0 0.8
Suppose we assign a value larger than 0.5. In that case, we would emphasize prior, i.e., older documents in the ranking, which is a rather conservative ranking policy in a real-world setting. On the other hand, values lower than 0.5 yield a higher weight to new documents that were not contained in a prior ranking, corresponding to a more experimental, risky ranking policy in a real-world setting. If = 0.5 the ranking remains the same, and the efect of the weighting is negligible.
For the submitted run, = 0.503 was selected based on a grid search on the train set at 3. Notably, this run only relies on the fact that documents were ranked earlier, but neglects any direct relevance information.
The run submission BM25+qrel_boost extends this rather weak heuristic by taking the relevance label directly into account and considering not only the − 1 snapshot but further prior snapshots. In this sense, this run follows a similar ranking policy (conservative vs. experimental) but with a focus on relevance information obtained from prior labels. Independent of any prior ranking, all documents ranked for a query at that were judged at a previous snapshot, e.g. − 1 are boosted by: ⎧ 2 ⎪ ,( ) = ⎨ 2 if , = 1 if , > 1 . ⎪⎩(1 − )2 if , = 0 (2)
This extends the boost described in Eq. 1 to additionally account for highly relevant documents. The additional free parameter can be used to assign a diferent boost to documents with higher relevance labels. The boosting can be repeated for further points in time by using more distanced qrels e.g. − 2. This leads to increasing scores for documents that are labeled relevant at multiple points in time, which is in line with our initial assumption. For the submitted run, = 0.7 was chosen, and all available previous snapshots (3, 2, 1, 0) as history. Due to a bug that was only found after submission, instead of highly relevant documents, all relevant documents were boosted again by 2 with = 2.
In addition to the run submissions described above, we propose one additional approach that aligns with the general idea of exploiting prior relevance signals. BM25+RF follows a query expansion method, making use of the relevance feedback provided by prior documents, i.e., those documents with a relevant label at earlier timestamps.
The relevant documents of all earlier timestamps are normalized and deduplicated based on the URLs. Then, the vocabulary of all remaining relevant documents corresponding to a query is unified. After stopwords are removed, the top ten expansion terms are extracted based on the TF-IDF scores and are used to expand the original query string. Hypothetically, this approach allows us to profit from the prior relevance information independent of a particular ranking or the labels for query-document pairs.
Since not all topics have a history with relevant documents available, the rankings for unknown
topics are reranked by pseudo-relevance feedback based on RM3 [
The ARP measured by P@10, bpref [
The retrieval efectiveness changes over time regarding all systems and measures. For almost all systems, it is decreasing compared to 3. It can be observed that the newer the snapshot, the lower the measured efectiveness. The ΔRI shows more minor changes in efectiveness. In contrast to the ARP trends, it indicates a slightly increasing efectiveness as indicated by the negative ΔRI values.
All runs per snapshot are tested for significantce compared to the BM25 baseline using paired t-tests with = 0.05 and Bonferroni correction. Most runs show significant diferences except BM25+time_boost at 4 and 5 for all measures or BM25+monoT5 at 4 measured by bpref or at 3 by MRR.
The ranking of systems appears to be relatively consistent across time. The BM25 baseline is outperformed by BM25+monoT5, but only by a little. The system BM25+time_boost initially performs the worst but is on par with BM25 for the later snapshots. These rankings are so similar that no significant diferences can be measured. The two systems, BM25+qrel_boost and BM25+RF systems, often outperform BM25+monoT5. In particular, the BM25+qrel_boost system performs substantially better.
The ranking similarity between two points in time measured by Kendall’s Tau and RBO appears to be generally low. Especially the BM25+RF system shows a low similarity regarding Kendall’s Tau, maybe because of the query rewriting that depends on the history. If the efectiveness changes are compared to the ranking similarity as displayed in Table 4, even small changes in efectiveness can lead to vastly diferent rankings. For example, the system BM25+time_boost evaluated at 3 and 4 show only small improvements on all measures but also among the smallest similarities in the rankings for Kendall’s Tau and RBO. Interestingly, the system BM25+monoT5 shows similar efectiveness to the system BM25+qrel_boost considering nDCG but diferent ranking similarities measured by RBO. BM25 has the most similar rankings, followed by BM25+monoT5 and BM25+qrel_boost.
The systems that rely on previous relevance signals, especially BM25+qrel_boost but also BM25+RF most often show more variance over time in the ΔRI and ER values, compared to the other systems. For
RBO@100 1 1 1 1 1 at 4 but only a smaller increase (nDCG ΔRI of -0.0740) at 5. Simultaneously, the nDCG ΔRI for BM25+monoT5 only difers by 0.0001 points.
Based on our experimental results, we basically see three directions for future work, namely:
3. the curation of a novel resource that ties together the diferent snapshots of the dataset (by identifying corresponding documents identifiers that point to the same document).
In general, we can see a performance drop for all systems and all measures over the three timestamps. None of our submitted systems specifically addressed the goal to identify new potentially relevant documents in their methodology. Instead, the submissions rather followed a retrospective approach where the relevance information of prior rankings was exploited.
As expected the baseline method BM25 is outperformed when reranking it in the second stage with monoT5, which is in line with our observations from the lab’s first iteration. Even though the performance drops over time for the BM25 baseline and the monoT5 reranking alike, the latter outperforms BM25 wrt. every measure at each timestamp.
The approach underlying CIR_BM25+time_boost is less efective at timestamp 3 but on par at 4 and 5 with BM25. This lets us conclude that it is not advisable to simply emphasize prior documents for the sake of better retrieval efectiveness without considering any other relevance information.
However, we have to point out that at timestamps 4 and 5 the retrieval efectiveness does not deteriorate, which is an important insight especially when considering that the rank correlation between the rankings at diferent timestamps is low. That means the rankings are quite diferent but have the same retrieval efectiveness in the end. From a user perspective, this could imply tremendous diferences that are simply not reflected by the efectiveness scores.
We assume that this can be explained by the dataset currently only having shallow relevance judgment pools. Suppose that more documents would have been assigned with a relevance label. In that case, we would probably see more diverse outcomes here. In the future, we would strongly recommend to curate a companion dataset or resource with intellectual relevance labels that complements the dataset with deep relevance judgment pools.
A good example of an equivalent is the combination of the TripClick [17] and TripJudge [18] datasets, that cover both click-based and annotator-based relevance labels. Of course, the use of large language models for generating relevance labels could be a viable option if costs for human annotators are too high [19].
However, when comparing the monoT5 results to the other run submissions, we also see that specifically CIR_BM25+qrel_boost and CIR_BM25+RF are on par or even outperform monoT5, which opens up several interesting points for discussion. These systems are the best-performing ones among our submissions.
Regarding CIR_BM25+RF, we can conclude that query expansion with terms from prior relevant documents is a viable option to make use of available relevance signals. While our experimental results show rather minor improvements over the monoT5 reranking, this approach reliably yields better scores. In the future, other language models for generating the expansion terms should be considered and evaluated.
CIR_BM25+qrel_boost opens up several other interesting perspectives for a more in-depth analysis. The approach of this run submission specifically boosts those documents that were judged as relevant earlier. This general approach was rather successful in the OpenSearch TREC and its CLEF predecessor LL4IR [20, 21]. Instead of (derived) relevance judgments, the original implementation used click data to approximate relevance. Nevertheless, it has to be considered that these documents and judgments could have served as training data for deep learning retrieval methods. This is critical as data leakage might be an issue when the datasets are used as provided and we recommend that future work should analyze this circumstance in more detail [22].
We have the concern that this circumstance was not that apparent as the documents with the same URL and the same content have diferent identifiers in the diferent versions of the datasets at diferent timestamps. That finally leads us to our third and final direction for future work, which would be a companion resource to the LongEval datasets that ties together all of the six dataset’s snapshots by unifying the document identifiers and making potential document overlaps, which may cause data leakage in a learning-based scenario, explicit.
A higher variance in ΔRI and ER for the systems that rely on the relevance signals of previous snapshots could be observed. This can be interpreted as validation of the comparison strategy. The runs directly utilize the relevance signals in more or less immediate. However, for both test points in time (4 and 5) the same history of snapshots (0 to 3) is used. This means that the relevance signals for 4 are more up-to-date compared to 5 and a higher improvement can be expected. In addition, the similarity of the rankings further supports this interpretation. Both, BM25+qrel_boost and BM25+RF, indicate especially consistent similar rankings over time, caused by boosting the same documents for both runs.
As part of this year’s contributions to the LongEval lab, we conclude that leveraging prior relevance signals from existing logs for the sake of better retrieval efectiveness is a promising direction. Furthermore — in line with our earlier observations from the first iteration of the lab — we confirm that the retrieval efectiveness changes with diferent snapshots of the dataset and addressing the need for making retrieval systems more robust, reliable, and predictable is an exciting direction for future work that should receive more attention.
In particular, we applied some heuristics to make use of the available relevance information that can be obtained from prior snapshots of the dataset to improve a recent ranking. To do so, it was a requirement to identify the same document across the diferent snapshots. Our preliminary analysis revealed that there are indeed the same documents in diferent snapshots and using these (relevance) signals helps to improve retrieval efectiveness. Based on these outcomes, we envision the contribution of a companion resource that ties together the dataset’s snapshots to identify documents and duplicates that would allow a more in-depth analysis about the document’s relevance over time but also make a more rigorous experimental setup possible.
We would like to express our gratitude to the LongEval Shared Task organizers for their invaluable eforts in constructing the LongEval dataset. Their dedication and hard work have provided an essential foundation for our research. We also gratefully acknowledge the support of the German Research Foundation (DFG) through project grant No. 407518790. [15] M. G. Kendall, Rank correlation methods., Grifin, 1948. [16] W. Webber, A. Mofat, J. Zobel, A similarity measure for indefinite rankings, ACM Trans. Inf.
Syst. 28 (2010) 20:1–20:38. URL: https://doi.org/10.1145/1852102.1852106. doi:10.1145/1852102. 1852106. [17] N. Rekabsaz, O. Lesota, M. Schedl, J. Brassey, C. Eickhof, Tripclick: The log files of a large health web search engine, in: F. Diaz, C. Shah, T. Suel, P. Castells, R. Jones, T. Sakai (Eds.), SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, ACM, 2021, pp. 2507–2513. URL: https://doi.org/10.1145/3404835.3463242. doi:10.1145/3404835.3463242. [18] S. Althammer, S. Hofstätter, S. Verberne, A. Hanbury, Tripjudge: A relevance judgement test collection for tripclick health retrieval, in: M. A. Hasan, L. Xiong (Eds.), Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, ACM, 2022, pp. 3801–3805. URL: https://doi.org/10.1145/3511808.3557714. doi:10.1145/3511808.3557714. [19] O. Zendel, J. S. Culpepper, F. Scholer, P. Thomas, Enhancing human annotation: Leveraging large language models and eficient batch processing, in: P. D. Clough, M. Harvey, F. Hopfgartner (Eds.), Proceedings of the 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval, CHIIR 2024, Shefield, United Kingdom, March 10-14, 2024, ACM, 2024, pp. 340–345.
URL: https://doi.org/10.1145/3627508.3638322. doi:10.1145/3627508.3638322. [20] P. Schaer, N. Tavakolpoursaleh, Historical clicks for product search: GESIS at CLEF LL4IR 2015, in: L. Cappellato, N. Ferro, G. J. F. Jones, E. SanJuan (Eds.), Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum, Toulouse, France, September 8-11, 2015, volume 1391 of CEUR Workshop Proceedings, CEUR-WS.org, 2015. URL: https://ceur-ws.org/Vol-1391/26-CR.pdf. [21] N. Tavakolpoursaleh, M. Neumann, P. Schaer, Ir-cologne at TREC 2017 opensearch track: Rerunning popularity ranking experiments in a living lab, in: E. M. Voorhees, A. Ellis (Eds.), Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017, Gaithersburg, Maryland, USA, November 15-17, 2017, volume 500-324 of NIST Special Publication, National Institute of Standards and Technology (NIST), 2017. URL: https://trec.nist.gov/pubs/trec26/papers/IR-Cologne-O.pdf. [22] S. Kapoor, A. Narayanan, Leakage and the reproducibility crisis in machine-learning-based science, Patterns 4 (2023) 100804. URL: https://doi.org/10.1016/j.patter.2023.100804. doi:10.1016/ J.PATTER.2023.100804.