Efective question answering (QA) over scientific literature is essential for accelerating research. However, existing benchmarks often fail to capture the complexity and domain-specific reasoning required. To address this gap, we introduce DBLP-QA, a benchmark dataset of 50 manually crafted question-answer pairs derived from the abstracts of scientific publications. It is designed to test both the retrieval and generation capabilities of large language models (LLMs) in scientific contexts. To demonstrate its utility, we evaluate RAGScholar, a retrieval-augmented generation (RAG) system that integrates external knowledge sources, and compare multiple retrieval and generation strategies. The dataset provides a robust foundation for advancing and objectively evaluating QA systems for scientific literature.
RQ1: How efectively does BM25 retrieve a publication whose abstract contains the correct answer? RQ3: Does a two-step strategy, where concatenated answers from single abstracts serve as context, improve final answer quality compared to using multiple retrieved abstracts directly? RQ4: How does model choice afect performance under identical retrieval and context conditions?
Our contributions are threefold: (1) DBLP-QA, a curated benchmark for abstract-based scholarly QA; (2) RAGScholar, a system integrating BM25 retrieval with instruction-tuned LLMs; and (3) a systematic evaluation of retrieval and context strategies across multiple models.
Scientific question answering has been studied through a variety of benchmarks that difer in domain
coverage and annotation depth. General-domain datasets such as SQuAD [
Several datasets target QA over scholarly publications. Scientific QA datasets, including BioASQ [
In contrast to these large, heterogeneous, or full-paper datasets, DBLP-QA provides a compact, manually crafted benchmark grounded solely in scientific abstracts from computer science. This design enables controlled assessment of retrieval-augmented generation systems, such as RAGScholar, under well-defined and reproducible conditions.
DBLP-QA is a novel benchmark dataset for scientific question answering that consists of 50 questions
and corresponding answers derived from research articles in the computer science domain. To construct
it, we used the dblp corpus dump from April 2025 [
This section presents the architecture, workflow, and experimental setup of RAGScholar, a retrievalaugmented generation (RAG) system designed to answer questions in the domain of computer science. By integrating the system description with the experimental methodology, we provide a concise and coherent overview of the framework and its evaluation.
RAGScholar consists of three main components: a retriever, a generator, and a web-based user interface (Figure 1). Users submit questions through a single-page application, upon which the retriever identifies 1https://www.semanticscholar.org/product/api; note that the abstracts dataset from the corpus of April 2025 is currently unavailable. The current abstracts dataset from January 20, 2026 does not include all the abstracts from the DBLP-QA dataset due to copyright licensing restrictions, but they are still accessible through the web interface. 2https://seafile.rlp.net/f/6581519cdd1d4782bccc/ IR System (Lucene BM25) retrieve update
Index KEY, DOI TITLE AUTHOR YEAR ABSTRACT
Query + Top-5 Documents
LLM - FLA-NT‑Tin5yL(LLaarMgeA, XXL) - Phi 4 - Mistral
Answers + Documents
User Interface
User Query Answers + Documents
A1 A2 A3 A4 A5 DoCTcooupnm-c3e.--nts ACTnoospwn-ce3.r--s DoCTcooupnm-c5e.--nts ACTnoospnw-ce5.r--s relevant scientific publications from a pre-constructed Apache Lucene 3 index. The index is built from the April 2025 dblp dump and contains, for each publication, the dblp key, DOI, title, year, author names, and abstract. User queries are translated into Lucene search expressions over titles and abstracts, and the retrieved publications are ranked using BM25. The top-ranked abstracts are provided as context to the generator, a large language model that produces a natural-language answer. Depending on the configuration, the generator uses either the top abstract, multiple individual abstracts, or concatenated contexts derived from the top retrieved documents. The generated answer and the corresponding source documents are presented to the user.
RQ1 evaluates whether BM25 reliably retrieves a publication containing the answer to a question of the DBLQ-QA benchmark dataset. We use BM25 with default parameters (1 = 1.2, = 2) and evaluate performance via Recall@k and MRR@k.
RAGScholar employs nine context strategies, grouped into three categories: (1) Single-Document Contexts (5 variants): The LLM receives the abstract of each of the Top-5 (A1, . . . , A5) retrieved documents individually; (2) Concatenated Document Contexts (2 variants): Top-3 Concatenated Documents and Top-5 Concatenated Documents, formed by concatenating the top-ranked abstracts; and (3) Concatenated Answer Contexts (2 variants): Top-3 Concatenated Answers and Top-5 Concatenated Answers, where intermediate answers for the Top-3 or Top-5 documents are concatenated and used as context for generating a final answer.
RQ2 compares strategies from the first and the second category to assess the efect of context size and document aggregation. RQ3 compares strategies from the second and the third category to study document- vs. answer-level aggregation.
To evaluate the influence of model scale and architecture, five instruction-tuned causal and sequence-tosequence LLMs (0.78B–14.7B parameters) were tested under identical retrieval and context conditions. All models used consistent generation settings (temperature 0.7, top- = 0.9, maximum output lengths of 512–1024 tokens). Automatic precision and device selection were applied when supported. Table 1 shows the models evaluated and the parameters used.
For all 50 benchmark questions, we generated 2,500 answers (5 models × 10 context variants, including a no-context baseline). Answers were manually rated against the ground-truth answer on a three-point Generated Answer: Compres- Generated Answer: Compressive sensing is a signal processing sive Sensing, also known as Comtechnique that allows for the re- pressed Sensing (CS), is a techconstruction of signals or images nique used in signal processing from a small number of measure- and imaging that allows for the ments, significantly fewer than detection and recovery of signals what is typically required by the that are inherently sparse or lowNyquist-Shannon sampling theo- rank, such as images or signals rem. Explanation: Answered in with noise. Explanation: Related the same way as Ground Truth An- to the Ground Truth Answer. swer.
Example (Score 0) Generated Answer: Compressive Sensing is a method used to improve on what any nonadaptive method can achieve in the context of recursive bisection method. It is a technique that establishes a non-asymptotic lower bound that applies to all methods, regardless of their computational complexity. Explanation: Unrelated to Ground Truth Answer.
Scale: 2 = correct and complete, 1 = correct but incomplete, 0 = incorrect or irrelevant. Table 2 contains
an example question-answer pair with generated answers and their manual scores. In addition, all
answers were evaluated against the ground-truth answers using ROUGE-L [
The retriever ranked a document containing a correct answer at rank 1 for 44 out of the 50 benchmark questions. In four questions, the document at rank 2 was the first document in the ranking to contain a correct answer, and in two questions, the first document containing an answer was ranked third. Consequently, the retriever achieved MRR@3 of 0.93 and Recall@3 of 1. We also examined the ranking position of the document from which the question-answer pair was derived. For 32 questions, the document was ranked at the top, for seven questions, it was ranked at two, for two questions, it was ranked at three, and for one question, it was ranked at four. The document was not in the top 5 for five questions.
The following example shows a case where the top-ranking document was not relevant: Question: What is CtRL-Sim? Ground Truth Answer: CtRL-Sim is a method that leverages return-conditioned ofline reinforcement learning to eficiently generate reactive and controllable trafic agents.
Abstract of retrieved document at rank 1: . . . calibration transformer with line-classification (CtRL-C) . . . demonstrate that CtRL-C outperforms . . .
Comment: The retriever was misled by high-frequency mentions of the query keyword ctrl. Mistral-7B Phi-4 TinyLlama-1.1B FLAN-T5-Large FLAN-T5-XXL
Mistral-7B
Context selection is a central design choice in retrieval-augmented QA because it determines the evidence available to the model at generation time. We therefore evaluate Top- context concatenation for ∈ {3, 5}, where abstracts of the top-ranked retrieved papers are merged into a single composite context to increase coverage and reduce omissions due to ranking errors. The results in Table 3 show that increasing can improve human-perceived answer quality, but the gains are model-dependent: for Mistral-7B, Top-5 context concatenation (Top-5-CD) achieves the best manual score (1.74) compared to Top-3-CD (1.63), while Phi-4 remains stable at 1.66 for both Top-3-CD and Top-5-CD. In contrast, TinyLlama-1.1B slightly drops from 1.58 (Top-3-CD) to 1.52 (Top-5-CD), suggesting that additional context can introduce redundancy or noise beyond what smaller models can efectively use. Across models, however, aggregated evidence is consistently beneficial relative to no context (e.g., Mistral-7B: 1.74 vs. 0.80; Phi-4: 1.66 vs. 0.40), highlighting the overall importance of context selection in balancing coverage, noise accumulation, and explainability in scholarly QA. For single-document contexts, A1 achieves the best results compared to the other single-document variants. For TinyLlama, this variant attains the highest overall score, and for the other models, the manual scores are only slightly lower than those of the respective best-performing variant. Moreover, a slight correlation can be observed between the manual evaluations and the automatic metrics BERTScore and ROUGE. This suggests that these metrics could potentially be used for the automatic evaluation of generated answers for DBLP-QA, although further research is required.
Answer concatenation leverages multiple retrieved documents. The model generates one answer per document and then combines the candidates for Top- ( ∈ {3, 5}). However, concatenating abstracts before generation is consistently stronger in our results. In Table 3, Top-5 concatenated documents (Top-5-CD) outperforms Top-5 concatenated answers (Top-5-CA) for the best models (Mistral-7B: 1.74 vs. 1.49; Phi-4: 1.66 vs. 1.51; TinyLlama-1.1B: 1.52 vs. 1.34), suggesting that direct exposure to aggregated evidence yields more complete answers. Still, answer concatenation improves over no context (e.g., Mistral-7B: 1.49 vs. 0.80; Phi-4: 1.51 vs. 0.40), indicating that output-level fusion remains useful. Interestingly, TinyLlama attains higher ROUGE-L with Top-5-CA than Top-5-CD (0.30 vs. 0.23), implying that answer fusion can increase surface overlap even when human judgments favor context concatenation.
All models were evaluated under identical conditions, so diferences can be attributed primarily to the LLMs. Table 3 shows clear separation: Mistral-7B performs best (Top-5-CD: 1.74), followed by Phi-4 (up to 1.66) and TinyLlama-1.1B (up to 1.58), while FLAN-T5 variants are much lower (0.38-0.61 under Top-5-CD). Retrieved evidence consistently improves quality over no context (e.g., Mistral-7B: 0.80→1.74; Phi-4: 0.40→1.66), confirming retrieval conditioning as a key driver. Context size shows a model-dependent coverage-noise trade-of: Mistral benefits from Top-5 vs. Top-3 (1.74 vs. 1.63), Phi-4 is stable (1.66 for both), and TinyLlama slightly drops with Top-5 (1.58→1.52), indicating higher sensitivity to redundancy. Overall, strong performance is achievable with eficient mid-sized models in a reproducible pipeline, supporting scalable scholarly QA deployments.
Limitations Despite the efectiveness of the framework, limitations remain. The system uses abstracts rather than full texts, limiting fine-grained methodological questions. Retrieval is BM25-only, which may miss semantically relevant evidence under paraphrasing without dense/hybrid methods. We also evaluate only small and medium-sized LLMs due to compute constraints, although results are already strong, highlighting the impact of retrieval and evidence packaging. Finally, DBLP-QA contains only 50 items from one domain, limiting generalizability.
Future Work Several directions can extend this work. First, re-running the pipeline with larger
instruction-tuned, long-context models (e.g., 34B/70B) would allow to estimate the performance ceiling.
Second, retrieval could be strengthened via dense/hybrid methods by adding DPR [
In this work, we introduced DBLP-QA, a benchmark of 50 QA pairs derived from abstracts, and RAGScholar, a retrieval-augmented QA system integrating BM25 retrieval and generative LLMs. The evaluation provides several insights in the design of domain-specific RAG systems for bibliographic corpora.
BM25 retrieval is efective in our scholarly QA setting, achieving a Recall@1 of 0.88 and a perfect Recall@3. Context construction is a crucial factor in determining the answer quality. Providing the abstract of the top retrieved document as context to the LLM results in better answers compared to the baseline scenario without context. Concatenating abstracts of the top-3 retrieved documents can improve the quality of the answer, while concatenating five abstracts may lead to reduced quality due to noise. A two-step context setting that first generates answers per retrieved document and then concatenates them as context does not improve answer quality compared to directly concatenating abstracts. Mid-sized LLMs perform well, while smaller models are more sensitive to context size and noise, indicating that efective scholarly QA depends more on retrieval and context construction than on large models.
DBLP-QA and RAGScholar ofer a reproducible evaluation setup for studying retrieval-augmented methods over scholarly corpora, with future work on extending the benchmark dataset, improving retrieval and exploring automatic evaluation metrics.
The author(s) have not employed any Generative AI tools. and producing summaries, SIGIR Forum 51 (2017) 209–210. URL: https://doi.org/10.1145/3130348. 3130369. doi:10.1145/3130348.3130369.