Systematic reviews are the standard method for synthesizing scientific evidence, but their creation requires substantial manual efort, particularly during retrieval and screening. While recent work has explored automating these steps, evaluation resources remain largely confined to the biomedical domain, limiting reproducible experimentation in other domains. This paper introduces SR4CS, a large-scale collection of systematic reviews in computer science, designed to support reproducible research on Boolean query generation, retrieval, and screening. The corpus comprises 1,212 systematic reviews with their original expert-designed Boolean search queries, 104,316 resolved references, and structured methodological metadata. For controlled evaluation, the original Boolean queries are additionally provided in a normalized, approximated form operating over titles and abstracts. To illustrate the intended use of the collection, baseline experiments compare the approximated expert Boolean queries with zero-shot LLM-generated Boolean queries, BM25, and dense retrieval under a unified evaluation setting. The results highlight systematic diferences in precision, recall, and ranking behavior across retrieval paradigms and expose limitations of naive zero-shot Boolean generation. SR4CS is released under an open license on Zenodo, 1 together with documentation and code,2 to enable reproducible evaluation and future research on scaling systematic review automation.
Systematic reviews are the standard method for synthesizing evidence on focused research questions.
They are valued for their completeness, transparency, and reproducibility, and their results often
serve as a guide for future research, policy, and practice [
In recent years, increasing eforts have been made to automate systematic reviews to reduce the
cost of their creation, both end-to-end and in individual phases such as query formulation, screening,
and synthesis. Work on automatic Boolean query generation ranges from computational adaptations
of expert-designed strategies [
However, evaluation resources for systematic review automation remain heavily concentrated in
the biomedical domain. Datasets such as the SIGIR 2017 SysRev Query Collection [
snowballing screening; citation chasing
systematic reviews, but there is no comparable large-scale resource for other domains. In the absence of domain-specific benchmarks, evaluating retrieval and screening methods often requires costly expert involvement, constraining scale and reproducibility.
To address this gap, we introduce SR4CS, the first large-scale test collection of systematic reviews in computer science, paired with the original Boolean search strategies reported by the authors and curated reference pools. Each review includes the original Boolean query, alongside an approximation normalized to a unified title-and-abstract-only retrieval setting for controlled evaluation. In addition, each review is associated with structured methodological metadata, including research objectives, inclusion and exclusion criteria, databases searched, and temporal constraints. In total, SR4CS contains 1,212 systematic reviews with 104,316 resolved references, of which 89,447 include abstracts. The dataset is publicly available together with documentation and code, enabling reproducible evaluation of Boolean query translation, retrieval efectiveness, and screening methods in computer science.
Existing evaluation resources for retrieval and screening in systematic reviews have largely focused on the biomedical domain. While these resources vary in scale, design, and supported tasks, they all provide reusable test collections for evaluating retrieval efectiveness, screening, and related automation methods. Table 1 summarizes the most relevant datasets and contrasts them with SR4CS.
The SIGIR 2017 SysRev Query Collection [
The Seed Studies Collection [
The CLEF TAR 2019 corpus [
SR4CS extends these eforts beyond biomedicine by providing the first large-scale test collection of systematic reviews in computer science. It preserves the original Boolean search strategies as reported in the reviews, while also providing normalized, approximated variants for controlled evaluation. With 1,212 systematic reviews and resolved reference pools, SR4CS supports reproducible evaluation of
Boolean query translation, retrieval efectiveness across paradigms, and screening methods in a domain where comparable evaluation resources have previously been lacking.
The corpus construction comprises three main stages: data collection and filtering, parsing and information extraction, and resolving references. Figure 1 provides an overview of the pipeline. "Systematic Review" 11,317 results
Filtering
Filtering 104,316 3.1. Data Collection A set of candidate systematic reviews was retrieved from DBLP by searching titles for “systematic review”, yielding 11,317 results. We applied filter rules to retain only genuine systematic reviews that required clear wording (e.g., “systematic review of/on”), peer review procedures, open access, and valid DOIs. Outliers due to extreme page counts were removed. After filtering, 1,339 candidate reviews remained. In the following steps, only reviews that explicitly stated their Boolean queries were considered for inclusion in the final dataset. 3.2. Parsing and Extraction The PDF files of the systematic reviews were downloaded and converted into Markdown format for further processing. To achieve the conversion, Nanonets-OCR-s,6 a modern OCR model that produces semantically enriched Markdown text rather than plain text, was used. Unlike traditional OCR tools, it can recognize LaTeX equations, describe figures and tables, and preserve document structure. This functionality was particularly important because many reviews contain Boolean queries and other important information in figures or tables.
The documents were then processed in a zero-shot settings using GPT-4.1 Mini to extract a series
of structured fields: (i) databases used, (ii) Boolean queries reported, (iii) year range, (iv) language
restrictions, (v) inclusion and exclusion criteria, (vi) main topic, (vii) objective, (viii) research questions,
and (ix) whether snowballing or citation chasing was performed. This approach follows recent work
demonstrating the feasibility of LLM-based zero-shot data extraction in scientific documents [
To assess extraction quality, the extracted fields were manually inspected for errors and
inconsistencies. Additionally, we randomly selected 10% of the reviews and manually extracted the same
information independently. In 85% of the sample cases, the fields extracted by the LLM matched the
manual extraction at the field level, and the errors found were minimal, such as minor discrepancies
in the wording of the inclusion/exclusion criteria or the specification of certain field values (e.g.,
language restrictions) based on indirect clues rather than explicit statements. These findings indicate that
zero-shot LLM-based extraction provides a suficiently reliable basis for large-scale corpus construction,
while still leaving room for more robust, feedback-driven extraction methods in future extensions.
Finally, we excluded reviews that did not explicitly state their Boolean queries, resulting in a final
dataset of 1,212 systematic reviews.
3.3. Reference Resolution
References were extracted from each systematic review via a hybrid pipeline combining AnyStyle7 and
GROBID,8 a configuration proven to be highly efective for structured citation analysis [
The SR4CS corpus is released to support reproducible research on retrieval and screening for systematic reviews in computer science. The dataset is designed for three main tasks: (i) evaluating automatic Boolean query generation against approximations of expert-designed Boolean queries, (ii) comparing retrieval paradigms under realistic systematic review topics, and (iii) investigating screening behavior using curated reference pools. To illustrate its use, we present baseline experiments for Tasks (i) and (ii), evaluating Boolean, probabilistic (BM25), and dense retrieval methods using titles and abstracts only.
0.238 0.083 0.262 0.373 4.1. Approximating Expert-Designed Boolean Queries A Boolean retrieval baseline is constructed by approximating the expert-designed queries reported in the original reviews. Since these queries rely on database-specific syntax and metadata fields, normalization is required to obtain a unified Boolean formulation operating over document titles and abstracts.
Query rewriting. Reported Boolean queries are converted into valid SQLite FTS5 MATCH syntax using GPT-4.1-mini in a zero-shot setting. The query structure is preserved, restricted to the title and abstract fields, and stripped of unsupported metadata filters. A random 25% sample is manually verified, with only seven queries requiring minor corrections, primarily for verbatim phrases.
Index and execution. A SQLite FTS5 index is created using the unicode61 tokenizer with diacritics folded and prefix indexing (2–10) to account for common spelling and formatting variations. All refined queries are executed against this index; when multiple Boolean queries are associated with a review, their result sets are merged via union. 4.2. Alternative Retrieval Baselines Using the Boolean approximation as a reference point, we evaluate three additional baselines operating on the same corpus and fields: (i) zero-shot Boolean queries generated from the review title and objective using GPT-4.1-mini, (ii) a keyword-based BM25 baseline, and (iii) a dense semantic retriever based on the all-MiniLM-L6-v2 Sentence-BERT model.
For both BM25 and dense retrieval, queries are constructed from the review title, the objective, and their concatenation. Results are reported for the best-performing variant (title + objective). Retrieval is capped at 1,000 documents per query to reflect a fixed screening budget and ensure comparability across retrieval methods. 4.3. Evaluation and Results All retrieval outputs are compared against the curated reference pools to compute precision, recall, F1, and F3, macro-averaged across reviews. For ranked retrieval methods, we additionally report MAP, P@10, and R@100.
Although Boolean retrieval is inherently unranked, Boolean result sets are deterministically ranked using SQLite’s internal BM25 scoring function for the purpose of ranking-based evaluation.
Table 4 reports set-based screening efectiveness, while Table 5 reports ranking efectiveness across all evaluated methods. Together, these results provide a consistent and reproducible reference point for evaluating Boolean query generation and alternative retrieval paradigms within the SR4CS corpus.
The development of SR4CS builds on recent advances in document processing and natural language
technologies for large-scale extraction from scientific publications. OCR parsers based on vision–language
models improve handling of complex PDF layouts [
The retrieval results clarify how diferent paradigms behave when evaluated under the same titleand-abstract-only constraints imposed during corpus construction. Approximated versions of expertdesigned Boolean queries achieve the highest precision (0.352), confirming that Boolean logic remains the most efective means of controlling screening efort; the moderate recall (0.342) primarily reflects the approximation process, which strips database-specific operators, controlled vocabularies, and citation-based expansion techniques rather than shortcomings of the original strategies. Dense retrieval, in contrast, delivers substantially higher recall (0.702) and the strongest ranking efectiveness (MAP 0.281, P@10 0.545), demonstrating robustness to vocabulary mismatch and terminological variation in metadata-constrained settings. The zero-shot GPT-4.1- Mini Boolean baseline performs poorly across all metrics (recall 0.099), providing a clear negative result: naive zero-shot Boolean generation with an of-the-shelf model fails to recover either the coverage or selectivity of expert-designed queries, indicating the need for more structured and adaptive query construction approaches. Overall, these results establish SR4CS as a targeted diagnostic benchmark for exposing precision–recall trade-ofs and failure modes in retrieval and automated query generation.
Several directions for future extensions of SR4CS are possible. First, deeper integration with open
scholarly infrastructures such as OpenAlex would support reference coverage, metadata enrichment,
and query execution within an open and reproducible ecosystem, enabling end-to-end studies that rely
on open identifiers and transparent metadata. Second, extending the corpus-construction workflow
to additional domains beyond computer science (e.g., education and social science) would broaden
the benchmark’s scope and help assess how retrieval and query-generation methods transfer across
diferences in terminology and reporting conventions. Third, future work could explore agentic
LLMbased information extraction frameworks that use iterative feedback and explicit grounding in source
text to improve extraction fidelity and consistency [
We introduced SR4CS, a large-scale test collection of systematic reviews in computer science, designed to support reproducible research on Boolean query generation, retrieval, and screening. By providing approximated versions of expert-designed Boolean queries, curated reference pools, and a controlled evaluation setting over titles and abstracts, SR4CS enables systematic comparisons of retrieval paradigms and query formulation strategies beyond the biomedical domain. The dataset is released with documentation and code to facilitate reuse, benchmarking, and future work on systematic review automation, retrieval evaluation, and query generation in computer science and related fields.
During the preparation of this work, the authors used DeepL Write for sentence polishing and grammar correction. 2510.00276. arXiv:2510.00276.