Temporal dynamics in retrieval settings have shown to carry helpful information for retrieval processes. In this submission to the CLEF LongEval lab we propose five diferent approaches: (1) Finding time-dependent queries with the help of LLMs and to treat these queries diferently by boosting their retrieval scores based on the categorization; (2) Finding time-dependent queries and scoring them on a scale from 0 to 1 and to use that score to influence the final ranking; (3) Using relevance information from older sub-collections and to use relevance feedback on the current sub-collection by using query expansion using tf-idf; (4) Boosting known relevant documents-query pairs from older sub-collections but comparing the similarity of old and recent documents; ifnally, (5) a neural relevance re-ranking based on a topcial semantic clustering. In total we submitted seven runs to the WebRetrieval task of the lab. The results indicate that only four of them could outperform BM25.
The LongEval lab at CLEF is focused on the evaluation of retrieval systems on changing test collections over time. In this lab notebook we summarize our submissions to the CLEF LongEval lab in 2025 that extend on previous work on the lab from 2023 [1] and 2024 [2]. The submissions are the result of a students’ project course with the Cologne Information Retrieval group (CIR) at TH Köln - University Applied Sciences in Cologne, Germany. Five groups participated in the course (see Table 1).
The approaches tested in this submission range from (
CEUR
ceur-ws.org
Queries are not all the same. We know that there are diferent query types in web search, like transactional, navigational, or informational queries [3]. Additionally, we know from studies on temporal retrieval [4] that users make a diference and most often prefer recent vs. old information, and that there are diferent temporal entities like events that should be treated diferently in the retrieval process. We picked up these ideas and tried to first distinguish diferent types of temporal queries and use this classification information to apply a boost to recent documents that were ranked for these queries.
We define the following four query types: time-independent, explicit-time, event, and timeliness. We used GPT-4o mini to categorize each LongEval query into one of these categories. We instructed the LLM to categorize the queries using the following definitions: • time-independent (timeless information not tied to a specific time or event, e.g., definitions, recipes, general rules), • explicit-time (requests with explicit time references, e.g., years, dates, specific periods), • event (requests about specific events, e.g., Named public events, Scheduled institutional events,
Historical events), or • timeliness (time-sensitive or current information where up-to-date info or availability matters e.g., weather, stock prices, live updates, buying intent, tax rates).
We instructed the LLM to use the following categorization process: (
Some examples of labels that this categorization process would assign are listed in Table 2. We see that some of the labels are debatable, like the assignment of World War II to the “explicit time” category instead of the “event” category. In a manual test with 100 random queries, we achieved a Cohen’s Kappa between 0.33 and 0.42 (two separate annotation runs with GPT-4o mini). The LLM struggled the most with the event and explicit-time categories, where it was hard to find any matching queries. Instead of the default English prompt, we also tried a French prompt, which lowered the agreement rate to 0.24.
Based on the categories assigned by the LLM, we applied a boost to the original BM25 scores. As documents in LongEval don’t have a specific timestamp, we applied a simple heuristic: If a document was detected in more than three LongEval sub-collections, it was considered old, and all other documents were considered recent. We then applied the boosting only on the recent document with the following boosting factors: • time-independent: 1.20 • explicit-time: 1.15 • event: 1.15 • timeliness: 1.20 time-independent “definition of gravity”, “chess rules” 23849 explicit-time “World War II”, “US president 1990” 544 event “Cannes festival 2025”, “French Revolution” 1146 timeliness “Apple stock price”, “weather Lyon” 4601
In contrast to the previous approach, we are only interested in how time-dependent a query is. We prompt an LLM to assign a value between 0 and 1 to every query, encoding the time-dependency of a query.
We prompted GPT-4o with the following instruction: “You have this Query. Give a score on how time-dependent this Query: query_text. The Score is between 0 and 1. Don’t answer with anything more than the Score.” and received a score for 47.053 queries. In Figure 1, we see a plot of the distribution of temporal dependency scores. The average score was 0.325, indicating a clear majority of less timedependent queries, with only a few time-dependent ones in comparison. Using a threshold of 0.6, only 22% of the queries were marked as time-dependent.
For our re-ranking, we take the BM25 scores based on our PyTerrier implementation with the default
parameters for each pair of query and document and an additional boost:
score_combined(, ) =
score_bm25(, ) + (
score_time() × score_recency())
(
1 1 + log(1 + |{snapshot| ∈ snapshot AND snapshot ∈ ()}|)
This submission builds upon our relevance_feedback approach as submitted in 2024 [2], where we used a query expansion method, making use of the relevance feedback provided by prior documents, i.e., those documents with a relevant label at earlier timestamps.
We reimplemented the original pipeline with the following modifications: (
The overall idea of this approach is to use relevance information from previous sub-collections by boosting known relevant documents. This approach was already proposed as qrel_boost in our submission 2024 and in 2025 we further refined this approach by including a filter step and a more fine-grained boosting mechanism. In the original approach we boosted all known relevant documents independent of any potential updates. This time we compare old and new documents not only based on the URL but also on the document content itself. It builds on observations from previous studies on pseudo relevance feedback [5]. Only if the old and new document content is the same or similar, we applied a boost on the original BM25 scores.
We developed and implemented three diferent methods to identify, filter, and boost document pairs that appear in two temporally distinct sub-collections of the LongEval dataset: A length matching approach, a document similarity comparison based on Sentence BERT, and a comparison by Jaccard index. The overarching goal was to recognize relevant document versions that remained stable over time, either structurally or semantically, and to integrate this information into a retrieval pipeline. All methods were grounded in a query-document relevance mapping (query_doc_map) derived from the oficial qrels file, which associates each query with its set of relevant documents. The result was a dictionary assigning each query ID to a standardized list of relevant document IDs, which served as the basis for our comparison strategies over periods. For the submitted approaches, only the qrels from the 2023-03 snapshot were used.
Length Matching The first method aimed to identify document pairs whose text content had the same length in both snapshots. We extracted and compared the text lengths for each document and retained only those pairs where the lengths matched perfectly. This strict filtering approach provided a fast and reliable pipeline for unchanged content, ensuring that only structurally identical document versions were boosted by taking the original BM25 score and multiplying it by two.
Length-Based Similarity with Tolerance Buckets Recognizing that minor formatting changes or metadata updates might alter document length slightly without afecting the core content, we introduced a more flexible filtering scheme based on length ratios. For each document pair, we computed the ratio between the shorter and longer version and classified them into predefined buckets and assigned a boosting factor (see Table 4). Separate filtered mappings were created for each category, allowing for graded analysis or boosting strategies depending on the degree of length similarity.
Sentence-BERT Similarity (Hard Threshold) To move beyond structural comparison and capture true semantic stability, we employed Sentence-BERT embeddings using the all-MiniLM-L6-v2 model. Each document version was encoded into a dense vector representation, and cosine similarity was calculated between the two versions. Only those document pairs with a similarity score above a strict threshold of 0.9 were retained and the original BM25 score was boosted with a factor of two. This method enabled us to preserve documents that may difer lexically but convey the same meaning, ofering a more context-aware filtering strategy.
Sentence-BERT Similarity with Buckets) Expanding on the previous method, we categorized document pairs into semantic similarity buckets rather than applying a hard cutof. This allowed us to group documents based on graded similarity levels similar to the string length approach (see Table 4).
Jaccard Similarity As an alternative to embedding-based methods, we also evaluated lexical overlap through Jaccard similarity. By tokenizing and lowercasing the document texts, we computed the ratio of intersecting to union word sets for each document pair. Document pairs exceeding a specified similarity threshold of 0.9 were retained and boosted by a factor of two. This approach was particularly useful for identifying minimal editorial changes.
This approach aims to develop a search engine that goes beyond keyword matching and also evaluates search results based on their content. Essentially, this means assigning new documents to thematic clusters. We used machine learning to identify hidden content-related connections between content clusters and their relevance.
First, we carried out a systematic clustering of all queries from the LongEval database. The queries were encoded with the help of OpenAI embeddings (text-embedding-3-large) into a 3072-dimensional semantic space. We used Uniform Manifold Approximation and Projection for Dimension (UMAP) to reduce the vectors to 50 dimensions, and then applied k-means clustering [6, 7]. The optimal number of clusters of 56 was determined by silhouette score analysis, resulting in thematically coherent groups (e.g. clusters with terms such as 4: “Job/Employment” or 32: “Food/Ingredients”) [8]. These clusters served as the basis for the subsequent modeling. In Figure 2, we see a visualization of ten high-level clusters from the original 56 topics discovered in the queries.
All relevant document were assigned to one or many diferent clusters. After pre-processing
(lowercasing, normalization, stopword removal, SnowballStemmer for French, and punctuation removal),
we extracted term frequencies for the top 10,000 terms and used a multi-hot encoding. For the model
itself, we developed a dense neural network with TensorFlow/Keras. Our aim was to use the text
content to predict which topic clusters a document fits into and how relevant it is. The model was built
as a dual-output network with two separate prediction branches: (
The actual retrieval was a two step approach: (
The retrieval efectiveness of the presented approaches was evaluated using the nDCG@10 metric [ 9], which aligns with the web-search context of this task. We also report the efectiveness of the BM25 [ 10] baseline, as most systems, excluding the Sauerkraut approach, function as re-rankers applied to this initial retrieval stage. The comprehensive results are depicted in Table 5 and Figure 3.
Among the evaluated systems, the SchaeredRetrieval approach, which boosted known documents based on the temporal type of the query, demonstrated the weakest performance across nearly all snapshots, with nDCG@10 scores ranging from 0.20 to 0.25. Similarly, the timeliness-focused approach by SuperTeam123 performed on par with the BM25 baseline for most snapshots. Although the nDCG@10 scores typically difered by only a few thousandths, a substantial drop occurred at the 2023-04 snapshot, where it recorded the lowest score of 0.192 among all systems and snapshots.
In contrast, the relevance feedback approach was the first to outperform the BM25 baseline. It shows similar performance trends as the baseline but consistently achieved better results. It is the second best unique approach. The JMFT team submitted three variations of their approach: Jaccard, Bert, and length. All of which outperform the baseline and the other approaches. Notably, for the first two
Effectiveness across Snapshots by Approach 0.20
2023-03 JMFT_Jaccard
Sauerkraut snapshots, all three JMFT approaches yielded identical nDCG@10 scores, with only minor variances observed thereafter.
The final team, fair_schaer, proposed a neural relevance re-ranking model. This approach achieved an efectiveness that positioned it between the BM25 baseline it was designed to re-rank and the ScharedRetrieval system. This approach also did not outperform BM25. Over time, the performance gap between this system and the BM25 baseline narrowed slightly.
Overall, all submitted approaches exhibited broadly similar trends in retrieval efectiveness. A greater variance in performance among the systems was observed in the initial two snapshots when the training data was most recent. This variance diminished in the later snapshots, with the final snapshot showing the least variance between the systems.
We proposed five distinct approaches for leveraging temporal information within test collections. While some of these methods are further developments of recent submissions, others are novel and previously untested. Ultimately, only two approaches, relevance feedback and qrel_boosting, managed to outperform the BM25 baseline on the test data. These results confirm, once again, that both are efective strategies for improving retrieval efectiveness at a low computational cost. In contrast, our ifndings indicate that the timeliness of a query could not yet be successfully utilized as an efective relevance signal.
We gratefully acknowledge the support of the German Research Foundation (DFG) through project grant No. 407518790.
During the preparation of this work, the authors used ChatGPT and Grammarly in order to: Grammar and spelling check, Paraphrase and reword. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.