In all areas of science, the volume of literature has become overwhelming, and staying current with the bleeding edge of research is increasingly unmanageable. This problem is particularly acute in medicine, where studies are published at a rate of approximately two per minute, and up-to-date information is crucial for decisionmaking processes. Maintaining the evidence base by updating systematic reviews is a costly and time-consuming endeavour, yet there is comparatively little research that tackles this important problem via computational means. We propose a shared task that addresses this challenge in (biomedical) scholarly information access. Specifically, we propose three tasks involved in updating systematic reviews: determining when a review should be updated, retrieving literature for the review update, and classifying whether each piece of retrieved literature should be included in the review. These are dificult problems that have received limited research attention, and improved solutions could have considerable impact.
Systematic reviews, i.e., comprehensive literature reviews on narrowly focused questions, constitute the
highest form of evidence in medicine. They are central to decision-making across health and medicine,
including the regulatory approval of new treatments, the development of clinical guidelines, and the
formulation of institutional and governmental health policies. However, creating a systematic review
cost upwards of e130,000 and may take over a year to complete [
Unfortunately, it is exceedingly common for a systematic review to be outdated by the time it is
published. As medical studies appear at a rate of roughly two per minute [
With our shared task, we propose a diferent approach by focusing on updating existing systematic
reviews. Searching for related work, we found only a single study [
□ Task 1: Review Update Prediction
Ttoheu pfirsdtatteasak creovniceewr.ns Tdheicsiddinegciswiohnenis PReevriioedw Update RIenvitiieawl URpedvaietewd
typically just time-based [
This task corresponds to the phase of the systematic review process in which reviewers decide when
an update is warranted. Since there is little methodological research to guide update timing, methods
developed in this shared task have a good chance to impact future guidelines for systematic review
updates. Participants will develop approaches that, given a systematic review topic and a collection of
studies, predict a point in time such that all new studies would be captured by an updated review.
□ Task 2: Study Retrieval The second task concerns retrieving relevant studies for a review update.
Typically, this work is delegated to an information specialist, namely a librarian who collaborates
with the review team, who crafts a complex Boolean query. Revising the previously used query
when updating a review has been emphasized as an important methodological practice [
This task corresponds to the study retrieval phase of a systematic review. Since the CLEF TAR tracks
featured a related retrieval task, participants familiar with those tracks should have an easy entry point.
However, our setting specifically targets retrieving the additional studies that should be included in a
review update, allowing participants to exploit knowledge about the studies have been included in a
review’s initial version. Participants will develop Boolean queries, either manually or via automatic
approaches [
This task corresponds to the study screening phase of a systematic review. Since the CLEF TAR tracks also included a screening task, participants familiar with those tracks should find the setup familiar. Our variant, however, explicitly allows participants to exploit information about studies included in the initial version of the review. Participants will develop classification systems, either human-in-the-loop or fully automatic, that predict whether a retrieved study should be included in the updated review.
This shared task fits neatly with the goals of the SCOLIA workshop, targeting practitioners in natural language processing and information retrieval who work on the analysis of scientific (biomedical) documents. It also addresses problems in academic literature processing that have received little research attention. We expect participants will find the tasks engaging, while also contributing new methods toward open challenges in the field.
In particular, this shared task directly targets several topics SCOLIA aims to address, including the construction of scholarly information access systems, user models and collections, and quality assurance and scientific integrity (i.e., filtering high-quality research papers for inclusion in reviews).
The following artefacts that we expect participants to submit will be collected via the TIRA platform [15]. TIRA also enables participants to submit their code and working systems in addition to runs, supporting reproducibility. □ □ □
Task
YYYY-MM-DD.
and TREC run file containing retrieval results. Plain text file where each line in the file corresponds to a study classified as included to the review.
trieve all studies published between the initial review and the update by applying a single date filter to the document collection.
We will provide a tool to validate that the submitted queries correspond to the submitted TREC run files.
We will provide a tool to validate that submitted results are compatible with our evaluation setup.
Datasets For this task, we acquired an open-access subset of Cochrane systematic reviews that includes review updates. While we cannot distribute the raw data due to licensing constraints, we can use the reviews’ associated metadata (e.g., bibliographic information) to construct an open test collection tailored to this task.
In total, the dataset comprises 35 systematic review updates (10 more topics than the next-largest
systematic review update dataset [
The document collection will be based on a baseline PubMed dump that we will index for participants. The index can be used locally via pybool_ir [16], which is available as a Docker image and requires minimal setup. The same index will also be accessible through an API, allowing participants to retrieve documents and produce TREC runs directly. If participants prefer not to use pybool_ir or the provided API, the index is a standard Lucene index that can be accessed through many other toolchains, e.g., 1We will actively investigate if frontier models have memorized our data and develop corresponding baselines. PyTerrier.2 Constraining the document collection to this index ensures reproducibility and enables others to continue working on the tasks after the shared task has concluded.
Measures □ Task 1: A TREC run file derived from the submission and evaluated in terms of recall and precision. □ Task 2: The submitted TREC run file will be evalauted in terms of recall and precision. □ Task 3: The result file will be evaluated in terms of accuracy and micro/macro F1
We plan to run the shared task in the second half of 2026, starting in late July and ending in late October. The three tasks will be released in phases, each running for approximately one month. A visual timeline of this plan is shown in Figure 2. This phased setup is intended to let participants incrementally build their systems over the course of the shared task. With the data release for Task 1, we will provide the topics with their initial review versions, along with metadata such as review titles and the list of included studies. For Task 2, we will additionally release the original Boolean queries used to retrieve studies for the initial review. For Task 3, we will provide the set of documents retrieved by the Boolean query for the updated review. Each task can be attempted independently, with no dependencies on earlier phases, so participants may choose to tackle any subset of the tasks.
2026 2027 4 5 6 7 8 9 10 11 12 1 2 3 4
Announcement of UP2DATE task at SCOLIA’26 Advertisement of UP2DATE task Release of data for Task 1 Release of data for Task 2 Release of data for Task 3 Analysis and Preparation of results Presentation of results at SCOLIA’27
The organizing committee, listed below, comprises experienced members, most of whom have coorganized multiple shared tasks in the past, including the PAN workshop series since 2007 and Touché since 2020. Other co-organized tasks include the SCAI’21 task on Conversational Question Answering, the SemEval-2019 task on Hyperpartisan News Detection, and the SemEval-2023 task on Human Value Detection. All members are familiar with, or actively working on, challenges in systematic reviews and information access for scientific literature, and have experience developing test collections for information retrieval and natural language processing.
Pierre Achkar is a PhD student at Leipzig University and Fraunhofer ISI. His research focuses on information access in scientific literature.
Tim Gollub is a postdoctoral researcher at Bauhaus-Universität Weimar. He is one of the main developers of the IR-Anthology project and has co-organized several shared tasks in the past, including at SemEval and CLEF.
Martin Potthast is a Professor at the Unviersity of Kassel. He has a wealth of experience organising numerous IR and NLP shared tasks at CLEF and SemEval.
Carsten Eickhof is a Professor at the University of Tübingen. He is an expert in biomedical IR and
NLP and has experience organising shared tasks at venues like CLEF.
Harrisen Scells is an Assistant Professor at the University of Tübingen. He has a great amount of experience working on problems at the intersection of systematic reviews and IR and has experience organising shared tasks at venues like CLEF.
During the preparation of this work, the authors used ChatGPT in order to: Grammar and spelling check, paraphrase and reword. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.