<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>Mistral AI Team, Mistral small</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <article-id pub-id-type="doi">10.48550/arXiv.2505.15948</article-id>
      <title-group>
        <article-title>Benchmarking Large Language Models on Reference Extraction and Parsing in the Social Sciences and Humanities</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Yurui Zhu</string-name>
          <email>yurui.zhu@odoma.ch</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Giovanni Colavizza</string-name>
          <email>giovanni.colavizza@odoma.ch</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Matteo Romanello</string-name>
          <email>matteo.romanello@odoma.ch</email>
        </contrib>
      </contrib-group>
      <pub-date>
        <year>2026</year>
      </pub-date>
      <volume>3</volume>
      <issue>2025</issue>
      <fpage>2</fpage>
      <lpage>24</lpage>
      <kwd-group>
        <kwd>eol&gt;Reference extraction</kwd>
        <kwd>Reference parsing</kwd>
        <kwd>Benchmarking</kwd>
        <kwd>Large language models</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Reliable citation processing is a prerequisite for large-scale scholarly infrastructure, as it facilitates
the creation of curated bibliographies, citation graphs and citation indexes. In practice, reference
extraction and parsing remain fragile and challenging in Social Sciences and Humanities (SSH) literature,
where citations often appear in footnotes, rely on abbreviations and ellipsis, mix languages within a
document, and follow style conventions that vary across periods, venues, and national traditions. These
characteristics hinder both (i) reference extraction (detecting and delimiting reference spans within
full text) and (ii) reference parsing (recovering structured fields such as author, title, venue/publisher,
and date). In summary, providing robust and scalable citation processing of SSH literature is a major
technical obstacle to overcome for building an SSH citation index [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ], which constitutes the backdrop
and ultimate objective of the work presented in this paper.
      </p>
      <p>A second, closely related issue is evaluation mismatch. Widely used benchmarks often center on clean,
English end-of-section bibliographies, which can inflate perceived robustness and hide SSH-specific
failure modes. Meanwhile, SSH-oriented datasets are rarer and often cover only parts of the pipeline
(e.g., parsing from gold strings rather than end-to-end processing from PDFs). Consequently, it remains
dificult to assess real-world progress—both for supervised pipelines and LLM-based systems—under
the regimes most relevant to SSH digitization and citation indexing.</p>
      <p>In this paper, we address this gap with a benchmark designed around SSH-realistic conditions and
operational constraints, and use it to assess whether contemporary LLMs are suficiently reliable for
reference extraction and parsing. We evaluate three complementary datasets: CEX, EXCITE, and
LinkedBooks, capturing (i) controlled English journal references across disciplines, (ii) German/English
documents with explicit citation-placement regimes (end-section, footnote-only, mixed), and (iii)
humanities references with high stylistic and multilingual variability. We benchmark three tasks of increasing
dificulty: reference extraction, reference parsing, and end-to-end document parsing. For comparability
across heterogeneous systems, we use a schema-constrained evaluation: LLMs must produce strict
JSON outputs, and GROBID predictions are mapped to the same target schema.</p>
      <p>Our empirical focus is twofold. First, we provide a transparent, side-by-side comparison between
a strong supervised baseline (GROBID) and several contemporary LLMs, including DeepSeek-V3.1,
Mistral-Small-3.2-24B, Gemma-3-27B-it, and the Qwen3-VL family. Second, we study two strategies
to boost performance in practice: (i) parameter-eficient LoRA adaptation for SSH-style parsing and (ii)
document segmentation/pipelining strategies that trade global context for shorter, verifiable outputs.
The resulting findings suggest a pragmatic hybrid direction: GROBID remains highly efective when
its operating assumptions hold, but LLMs—especially when adapted—ofer stronger robustness under
distribution shift (multilinguality and footnote-heavy layouts). This naturally motivates routing policies
that allocate easy, in-distribution documents to GROBID and escalate high-risk cases to adapted LLMs.
Contributions.</p>
      <p>• We introduce a unified SSH-focused benchmark spanning CEX, EXCITE, and LinkedBooks,
and covering extraction, parsing, and end-to-end document parsing under a shared schema and
evaluation protocol.
• We benchmark GROBID against a diverse set of LLMs (DeepSeek-V3.1,
Mistral-Small-3.224B, Gemma-3-27B-it, Qwen3-VL), highlighting where LLMs are competitive or superior under
SSH-typical distribution shifts.
• We quantify the impact of two practical levers—LoRA adaptation and
segmentation/pipelining—showing consistent improvements in robustness and end-to-end utility.
• We synthesize these results into actionable guidance for hybrid deployment via routing, balancing
throughput and reliability in large SSH collections.</p>
      <p>Paper outline. Section 2 situates our work in prior datasets and tool chains; Section 3 describes the
three datasets; Section 4 details tasks, systems, and evaluation; Section 5 reports benchmarking results
and error analyses; Sections 5.2 and 5.3 study LoRA and segmentation; and Section 6 concludes with
implications for robust SSH citation processing. Our code is available at: Github</p>
    </sec>
    <sec id="sec-2">
      <title>2. Related Work</title>
      <p>Research on bibliographic reference extraction and parsing has long been shaped by two practical
constraints: the limited availability of high-quality gold standards and a persistent mismatch between
laboratory-style citation data (clean end-section bibliographies in English) and the heterogeneity of real
scholarly documents, particularly in the Social Sciences and Humanities (SSH). This section reviews
the datasets most commonly used to develop and benchmark reference processing systems, alongside
representative toolchains ranging from supervised pipelines to recent LLM-based approaches. This
context motivates our benchmark design, which explicitly targets multilinguality, footnote-heavy
layouts, and long-form documents—settings in which traditional systems are known to degrade.</p>
      <p>
        Existing datasets vary widely in authenticity, scale, and coverage. EXCITE [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] is notable for providing
an SSH-oriented benchmark with a strong German component and an explicit mixture of end-section,
footnote-only, and mixed citation regimes. Beyond reference strings, it includes intermediate artifacts
such as page-layout exports and corrected XML, which enable evaluation of end-to-end robustness
rather than string-level parsing alone. GEOcite [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] similarly contributes German-language scientific
publications with extracted reference strings and field-level segmentation annotations, mainly
supporting supervised training for metadata extraction. In addition, CEX [
        <xref ref-type="bibr" rid="ref4 ref5">4, 5</xref>
        ] ofers carefully aligned PDFs
and Text Encoding Initiative (TEI) annotations compliant with the GROBID citation schema, yielding a
controlled multi-disciplinary English benchmark where citation conventions difer across 27 SCImago
subject categories. Complementary corpora leverage paired full-text markup: the XML Markup
Evaluation Corpus [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] and Open Research Europe (ORE) [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] provide Journal Article Tag Suite (JATS)-aligned
representations that facilitate large-scale harvesting of reference lists with structured metadata. At the
other extreme, GIANT [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] provides massive synthetic supervision by rendering Crossref-derived records
into reference strings under many Citation Style Language (CSL) styles. While this scale is valuable for
training data-hungry models and broad style coverage, the clean generation process underrepresents
the noise, abbreviation practices, multilingual variation, and idiosyncratic source types that are common
in SSH references. Finally, LinkedBooks [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] stands out as a humanities-first resource, ofering tens of
thousands of manually annotated references spanning footnotes and bibliographies, including primary
sources and abbreviated forms, with fine-grained field markup; its stylistic diversity and historical
variability make it particularly suitable for stress-testing parsers under SSH-typical conditions.
      </p>
      <p>
        On the methods side, supervised pipelines remain the dominant baseline for reference extraction
and parsing. GROBID [
        <xref ref-type="bibr" rid="ref10 ref11">10, 11</xref>
        ] is widely adopted for PDF-to-TEI conversion and frequently used as a
comparative standard; recent benchmarking confirms its strong performance alongside systems such
as AnyStyle on heterogeneous evaluation suites [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ]. Nevertheless, multiple studies report brittleness
under distribution shift, especially for multilingual content, complex page geometry, multi-column
layouts, and footnote-embedded citations, where segmentation failures and missed boundaries become
common [
        <xref ref-type="bibr" rid="ref13">13, 14</xref>
        ]. Motivated by these limitations, recent work has begun to apply LLMs directly to
citation parsing. Sarin and Alperin [15] construct paired datasets from JATS sources and show that
small open-weight LLMs can achieve strong field-level accuracy in few-shot settings, often surpassing
previously reported results on similar tasks. However, their studies largely focus on English, DOI-rich
journal references and do not systematically evaluate SSH-specific regimes (footnotes, long monographs,
mixed-language corpora), nor do they provide transparent, side-by-side comparisons against strong
GROBID configurations on the same inputs.
      </p>
      <p>Overall, a consistent gap remains: evaluation conditions are frequently misaligned with SSH reality,
where citations are embedded in footnotes, abbreviated, multilingual, and shaped by diverse historical
conventions. Our benchmarking work is designed to address this gap by jointly testing extraction,
parsing, and end-to-end behavior on CEX, EXCITE, and LinkedBooks, while retaining GROBID as a
strong supervised baseline and extending comparison to contemporary LLMs and parameter-eficient
adaptation.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Datasets</title>
      <p>
        For evaluation, we select three complementary datasets: the CEX project Goldstandard, the EXparser
Gold Standard from the EXCITE project, and LinkedBooks. They difer in domain, language
distribution, and citation style (end-of-document bibliographies vs. footnotes), enabling robust benchmarking
across document- and reference-level tasks. To avoid skewing the dataset towards simple, non-SSH
citation styles, we deliberately exclude additional datasets with overlapping scope. Table 1 summarize
their main characteristics.
3.1. CEX
The CEX project Goldstandard (hereafter CEX) [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] is a curated and corrected expansion of Ciofi [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ],
released as paired PDFs and TEI-encoded reference annotations compliant with the GROBID citation
model. It contains 112 English papers (1,581 pages; 5,160 references) spanning 27 general SCImago
      </p>
      <sec id="sec-3-1">
        <title>3.2. EXCITE</title>
        <p>English-language journal articles from 27 SCImago disciplines
Majority German; rich mix of end-section, footnote-only, and
mixed citation layouts
Monographs and articles; multilingual references: IT (72.3%),</p>
        <p>EN (18.3%), DE (4.3%), FR (3.9%), ES (0.6%), PT (0.5%), NL (0.1%)
disciplines, where reference formatting and citation conventions vary substantially across fields. We
use the full set as a controlled benchmark to probe cross-domain generalization.</p>
        <p>
          The EXparser Gold Standard (hereafter EXCITE) was created within the EXCITE project [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ], a
reference extraction and citation matching toolchain for German-language social-science publications,
including sociology and migration studies, political science, health and social policy, and economics.
EXCITE is designed to support layout analysis, reference identification, and segmentation/metadata
extraction. It comprises 351 SSH papers (8,041 pages; 10,171 references): 251 German documents (219
with end-section references, 12 with mixed end-section and footnotes, and 20 footnote-only) and 100
English documents with standard end-section bibliographies. Beyond layout variation, the
footnoteheavy classes (2 and 3) introduce abbreviated back-references typical of German SSH writing: Ebd.
(ebenda, equivalent to ibid.) and a.a.O. (am angegebenen Ort, equivalent to op. cit.). The gold standard
retains these strings verbatim without resolving them to their canonical referents.
        </p>
        <p>The gold standard provides multiple aligned representations per paper (PDF plus intermediate
layout/annotation files), enabling evaluation of extraction robustness across layout and citation-placement
conditions. We use the full gold standard in our experiments.</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.3. LinkedBooks</title>
        <p>
          LinkedBooks [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ] is a humanities-oriented citation mining project centered on the historiography of
Venice. It provides 40k+ manually annotated references extracted from both bibliographies and footnotes,
with fine-grained field segmentation and substantial variation in humanities citation style, including
primary vs. secondary references and abbreviated forms. The corpus is released with predefined splits
(train ≈ 35k, validation ≈ 2k, test ≈ 2k). For our benchmark, we evaluate on the full test split but restrict
to primary citations only. This yields 1,194 references, predominantly in Italian and English (72.3% IT,
18.3% EN), with smaller portions of German, French, Spanish, Portuguese, and Dutch. This multilingual
and stylistically heterogeneous subset is well suited to stress-testing reference parsing in history- and
humanities-style citation practices.
        </p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Setup</title>
      <p>4.1. Tasks
We evaluate all systems on three tasks of increasing complexity: reference extraction, reference
parsing, and end-to-end document parsing. This setup separates span detection from structured
metadata prediction and assesses their combined behavior in realistic pipelines.</p>
      <p>Reference extraction. A document-level task: given a document (or segment), systems must detect
and delimit all bibliographic references—whether in bibliographies or footnotes—and return them as
plain-text spans without internal structure.</p>
      <p>Reference parsing. A reference-level task with gold reference strings as input. The goal is to
convert each string into a structured record (authors, title, year, venue/publisher, identifiers), isolating
ifne-grained semantic segmentation from extraction errors.</p>
      <p>End-to-end document parsing. A pipeline task that jointly performs extraction and parsing from
raw documents, producing structured reference records. Performance here reflects cumulative errors
and thus approximates practical usability for citation indexing and linking.</p>
      <sec id="sec-4-1">
        <title>4.2. Systems and infrastructure</title>
        <p>We compare GROBID with several LLMs under a unified, schema-constrained setup.
GROBID baseline. We run GROBID with the Deep Learning sequence labelling model released
by the CEX project [16]. The model is trained on citation data closely related to, but distinct from, our
CEX gold standard (Section 3.1). It thus provides a strong supervised baseline for English scientific
articles and has a distributional advantage on CEX; we therefore emphasize cross-dataset performance
on EXCITE and LinkedBooks to assess robustness on multilingual, heterogeneous SSH material. For
the reference parsing task, we use /api/processCitationList endpoint on gold reference strings
directly; for the end-to-end parsing, we use /api/processFulltextDocument endpoint on PDF
documents.</p>
        <p>LLM models. Our pool includes open-weight models that can be self-hosted and one model we access
via API for convenience. DeepSeek-V3.1 [17] represents a very-large, state-of-the-art open-source
LLM; although open weights permit self-hosting, we evaluate it through the public API in this study.
Mistral-Small-3.2-24B-Instruct-2506 (hereafter Mistral-Small-3.2-24B) [18] and Gemma-3-27B-it [19]
cover a practical range for local inference. The Qwen3-VL family (4B, 8B, 30B-A3B, 32B) [20] enables
controlled scaling comparisons within one architecture. We recognize that certain documents within our
datasets are publicly accessible and may be included in the pretraining corpora of the assessed models.
Nonetheless, there is no indication that any of the chosen LLMs were trained on the specific
documentreference pairs utilized in our benchmark; the datasets also encompass closed-access publications that
the models could not have accessed; furthermore, SSH citation processing is an under-represented
domain in LLM training, thereby diminishing the likelihood of systematic bias.</p>
        <p>Prompts, input representation, and decoding. All LLMs use a few-shot prompt with a fixed JSON
schema (authors, full_title, year, venue/publisher, identifiers, etc.) 1. Prompts include brief instructions,
an inline schema block, and 1–2 style-diverse examples. Outputs must be strict JSON (no prose). For
document-level tasks (reference extraction and end-to-end parsing), we first convert each input PDF
into a text-only Markdown representation (including footnote text when present) using the marker
PDF-to-text pipeline,2 selected on the basis of an internal evaluation balancing extraction accuracy and
processing speed, and provide the markdown text as the model input. GROBID, by contrast, ingests
PDF files directly and performs its own layout analysis, so no intermediate conversion is applied.
For reference parsing, the model input is the gold reference string and the PDF-to-Markdown step is
bypassed. We use provider-recommended inference settings for each model (temperature and top_p),
and we do not impose additional output-length caps beyond the serving backend’s defaults/limits.
Generations are validated as JSON; if an output is invalid (e.g., malformed/truncated JSON or empty),
we perform a single retry with identical settings, after which the instance is counted as a failure.
1The prompts we use can be found at: https://github.com/odoma-ch/ssh-citation-index/tree/main/prompts
2https://github.com/datalab-to/marker
Infrastructure. Self-hosted models are served with vLLM on 2× A100 NVIDIA GPUs; DeepSeek-V3.1
is accessed via its public API; GROBID runs on a 24 GB MIG slice. We report accuracy metrics alongside
wall-clock runtime and failures (invalid JSON, empty outputs, truncation).</p>
      </sec>
      <sec id="sec-4-2">
        <title>4.3. Evaluation protocol</title>
        <p>We evaluate three tasks: reference extraction, reference parsing, and end-to-end document parsing.
All tasks use the same soft one-to-one matching: each gold reference is paired with the most similar
prediction using a fuzzy Levenshtein similarity score over the reference string and key fields.
Prior to scoring, we case-fold and normalize whitespace/punctuation; names are canonicalized in
an order-insensitive manner (e.g., “Surname, First M.” ≈ “First M. Surname”). We do not apply hard
similarity thresholds; similarity directly determines partial credit in precision and recall. We
evaluate against verbatim gold standard strings in their original document order, without resolving
abbreviated back-references (e.g., Ebd., a.a.O.). Downstream resolution is left to dedicated entity-linking
components.</p>
        <p>For reference extraction (string level), we match predicted reference strings to gold strings and
report Precision, Recall, and F1. Extra duplicates count as false positives; unmatched gold references
count as false negatives.</p>
        <p>For reference parsing and end-to-end document parsing (field level), we score structured fields
for each matched reference under the same rule. We report micro-F1 (pooled over records and fields),
macro-F1 (averaged over records), and per-field precision/recall/F1 for key fields (Title, Authors, Year),
with additional fields in detailed tables.</p>
        <p>Invalid, empty, or ill-formed JSON yields no valid items (TP = 0) and contributes to errors through
matching. We additionally track failure counts and runtime as operational indicators.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Experiments and Results</title>
      <sec id="sec-5-1">
        <title>5.1. Of-the-shelf LLM baselines</title>
        <p>Our first experiment benchmarks of-the-shelf LLMs against the GROBID baseline in a single-call,
few-shot setting (Section 4), without task-specific fine-tuning. We evaluate reference extraction,
reference parsing, and end-to-end parsing on CEX, EXCITE, and LinkedBooks. Table 2 reports the main
results. We do not report a standalone GROBID score for reference extraction, as GROBID is designed
as a pipeline and does not provide an extraction-only endpoint that isolates bibliography/footnote
segmentation from downstream parsing.</p>
        <p>Task-wise performance. Across tasks, the LLMs and GROBID exhibit distinct performance regimes.
For reference extraction, the larger LLMs largely saturate the problem on both CEX and EXCITE:
once the model has suficient capacity, span identification is highly reliable and invalid outputs are rare.
Diferences between top models are small, suggesting limited sensitivity to modeling choices.</p>
        <p>For reference parsing, outcomes depend more on domain match and supervision. On CEX, GROBID
leads, consistent with its in-domain training advantage and its role as a strong supervised baseline.
On EXCITE and LinkedBooks, however, several LLMs match or surpass GROBID, indicating better
cross-style generalization and greater robustness under distribution shift.</p>
        <p>End-to-end results mirror this trade-of. GROBID performs best on CEX where its advantage
compounds across stages, while on EXCITE the strongest LLMs overtake the GROBID by retaining
more stable parsing under non-standard layouts and multilingual references.</p>
        <p>Scaling efects. After a moderate capacity threshold, model size has limited impact on extraction, but
plays a much larger role in parsing and end-to-end parsing, where systems must maintain structured,
well-formed outputs over long and noisy inputs. This pattern holds both within and across model
CEX
EXCITE
CEX
EXCITE
End-to-end Parsing
families: smaller variants (e.g., Qwen3-VL-4B-Instruct) exhibit more failures and less stable behavior,
while larger models are generally more robust and accurate.</p>
        <p>Within Qwen3-VL, moving from Qwen3-VL-4B-Instruct to Qwen3-VL-8B-Instruct yields a
clear jump in reliability, and the 30B-A3B and 32B models often approach or exceed the frontier-level
performance of DeepSeek-V3.1, matching or outperforming GROBID on heterogeneous SSH datasets.
At the same time, these gains are not strictly monotonic with parameter count: Mistral-Small-3.2-24B
is frequently among the strongest models on EXCITE parsing and can outperform
Qwen3-VL-32BInstruct in some settings, while Qwen3-VL-4B-Instruct occasionally surpasses larger models such
as Gemma-3-27B-it and Mistral-Small-3.2-24B on specific datasets or fields. In practice, scaling laws
still provide a useful aggregate trend, but architectural design, instruction tuning, and model “freshness”
can be at least as important as raw size when choosing a model for structured citation prediction.
SSH factors The EXCITE breakdown in Table 3 highlights how SSH-specific characteristics, i.e.,
language and citation placement shape performance. For LLMs, English vs. German bibliographies
(Class 1) difer only slightly, whereas GROBID shows a language clearer gap under conventional layouts.</p>
        <p>In German, shifting from bibliography-style references to footnote-only and mixed regimes causes
a large drop for all systems, but GROBID deteriorates most sharply. LLMs are consistently more
robust in footnote-heavy settings, with DeepSeek-V3.1 exhibiting the smallest placement-induced gaps,
consistent with stronger generalization at higher capacity. Mixed regimes remain dificult across the
board, likely because they combine segmentation ambiguity with cross-location consistency constraints.</p>
        <p>Overall, LLMs are better suited to heterogeneous SSH citation practices than GROBID, but
footnotedriven layouts remain the dominant failure mode for all systems.
Error analysis. Figure 1 provides a reference-level view of failure modes on CEX, complementing
the F1-based evaluation in the previous section. For this analysis, we first reconstruct a canonical
reference string from each predicted JSON record (by concatenating key fields in a fixed order) and
compare it to the gold reference string using the same fuzzy-matching procedure as in Section 4. The
per-field similarities are then collapsed into four coarse categories: Correct if all fields reach a similarity
of at least 0.95; Minor error if at least one field falls into the [0.60, 0.95) band while none drops below
0.60; Major error if any key field is below 0.60 but the JSON is structurally valid; and Structural error if
the output cannot be parsed into a reference at all (e.g., malformed or truncated JSON).</p>
        <p>Figure 1(a) shows the resulting distribution over these categories, aggregated across all references.
Most open-weight models and GROBID produce a large majority of Correct references, with residual
mistakes dominated by Minor rather than Major errors, i.e., by field omissions or local formatting issues
instead of completely mismatched records. DeepSeek-V3.1 remains an outlier with a comparatively
high fraction of Structural errors, which we attribute primarily to output-length limits in the API: for
documents with long bibliographies, generations are frequently cut mid-output, yielding incomplete
JSON that cannot be evaluated. At the same time, the stacked bars make a scaling efect visible within
the Qwen3-VL line: the smallest variant Qwen3-VL-4B-Instruct and Qwen3-VL-30B-A3B-Instruct
show noticeably more Structural failures than its larger counterparts, whereas structural errors almost
disappear for Qwen3-VL-32B-Instruct. This pattern supports the main benchmark finding that below
a certain capacity threshold models become unstable on long, schema-constrained outputs, even when
their token-level accuracy is competitive.</p>
        <p>Figure 1(b) breaks down the share of Correct references (our strict ≥ 0.95 threshold on all fields) by
document category. A consistent trend is that all systems achieve higher accuracy on STEM-oriented
categories than on SSH/humanities-marked fields (indicated with **), suggesting increased sensitivity to
stylistic variation, multilinguality, and heterogeneous citation conventions in the latter. Two categories
stand out as dificult across models: Health Professions and Neuroscience, which show pronounced
drops in accuracy. For Neuroscience, this correlates with unusually long reference lists in CEX (roughly
75 references per paper, versus an overall average of ∼ 41 ), increasing the risk of truncation and
accumulated formatting errors; both categories also contain highly specialised terminology and
venuespecific citation patterns that appear less well covered by general instruction-tuned LLMs, leading to
ifeld confusions (e.g., identifiers mislabelled as page ranges). These category-level diferences should
nonetheless be interpreted with caution, as each CEX subject area is represented by only four documents,
and some efects may therefore reflect sampling noise rather than stable disciplinary trends.</p>
      </sec>
      <sec id="sec-5-2">
        <title>5.2. LoRA Fine-tuning</title>
        <p>We investigate parameter-eficient fine-tuning, more specifically Low-Rank Adaptation (LoRA)[ 21], to
adapt locally deployable LLMs to SSH-typical reference parsing, using two open-weight backbones:
Mistral-Small-3.2-24B and Qwen3-VL-8B-Instruct. We train LoRA adapters for reference parsing
on a supervised dataset 3 we construct from the same three datasets used in our benchmark (CEX,
EXCITE, LinkedBooks). Since CEX and EXCITE provide no dedicated training corpora, we sample
from their gold references and enforce document-level exclusion from evaluation to prevent leakage.
To balance scale across sources, we subsample roughly 10% of the LinkedBooks training/validation
material and retain only high-quality primary citations. To better reflect production inputs, we also
create “grouped” instances by concatenating references that co-occur in the same gold reference block,
apply our conversational templates to these blocks, and supervise the corresponding JSON lists, yielding
1,832 training conversations in total.
3Training dataset available at https://huggingface.co/datasets/odoma/reference-parsing-finetuning.</p>
        <p>We train with Axolotl[22] on two A100 GPUs, freezing base weights and selecting hyper-parameters
via pilot sweeps. Best settings are: Mistral-Small-3.2-24B4 (=32, =64 , lr 5×10 −5 , 2 epochs) and
Qwen3-VL-8B-Instruct 5 (=64, =128 , lr 5×10 −5 , 3 epochs). To separate parameter adaptation
from prompt efects, we evaluate three inference variants per backbone and dataset (Table 4): (i) base
model with our detailed few-shot prompt; (ii) LoRA model with the same detailed few-shot prompt;
and (iii) LoRA model with the short, zero-shot training-style prompt used during fine-tuning.
Reference parsing results for LoRA fine-tuning (GROBID as baseline). We compare (i) base models with few-shot
prompting, (ii) LoRA-finetuned models with the same prompt, and (iii) LoRA-finetuned models with the zero-shot
training-style prompt. Best Micro F1 per dataset and backbone in bold. (ties included).</p>
        <p>LoRA consistently improves over the base models across all datasets, with the largest gains
on SSH-heavy benchmarks. On CEX, improvements are modest but stable (Mistral: +0.7% relative
micro-F1; Qwen: +2.1%). On EXCITE, both models improve more substantially (Mistral: +2.5% with
few-shot, +3.3% with training-style prompting; Qwen: +0.9% and +1.8%). The strongest efect is on
LinkedBooks, where both backbones gain ≈ +10% relative micro-F1 (Mistral: +9.4% and +10.3%; Qwen:
+10.6%). Prompt–training alignment yields additional but non-uniform benefits: it helps Mistral across
all datasets and Qwen on EXCITE, while Qwen on CEX and LinkedBooks still prefers the richer
few-shot prompt, suggesting that detailed instructions remain useful for highly variable humanities
references even after adaptation. Operationally, failures remain zero in Table 4, indicating preserved
JSON validity; runtime diferences mainly track prompt length, with training-style prompting generally
reducing latency for Mistral.</p>
        <p>Overall, lightweight adaptation with a small SSH supervision set (1,832 instances) substantially
improves local reference parsing—especially under multilingual and style-rich distributions—and in the
best cases enables more compact, schema-aligned prompting.</p>
      </sec>
      <sec id="sec-5-3">
        <title>5.3. Segmentation strategies</title>
        <p>Segmentation controls how much text the model sees per request and therefore drives the trade-of
between accuracy, robustness, and runtime. Rather than tuning segmentation per document, we treat
it as an experimental factor and apply each strategy uniformly on CEX and EXCITE to isolate its
efects under SSH-typical conditions (long documents, multilingual passages, and references in both
end sections and footnotes). We compare three variants (Figure 2):
• Single-call (global context) sends the full document (or one long slice) in a single request when
it fits the context window, preserving global coherence but becoming brittle as inputs and JSON
outputs grow long.
• Adaptive pipeline (local context) reduces task complexity by breaking a single request into
multiple smaller ones: for extraction, we apply per-page segmentation; for end-to-end parsing,
we decompose the task into a sequential extract → parse pipeline. This keeps structured outputs
small and localizes failures.
• Semantic pre-selection (section-aware) retrieves likely reference-bearing
regions and only sends those chunks to the model, reducing token usage; we use
intfloat/multilingual-e5-large-instruct for semantic retrieval.</p>
        <p>Across variants, we keep the prompt and JSON schema fixed, enforce strict JSON validation, and log
wall-clock runtime and invalid/empty outputs as failures.</p>
        <p>Results Table 5 summarizes results for Mistral-Small-3.2-24B on CEX and EXCITE. For reference
extraction, performance is high and diferences are small: single-call is consistently best, the per-page
adaptive setting is slightly weaker and can be slower on long documents, and semantic pre-selection
largely preserves single-call accuracy while reducing runtime (notably on EXCITE).</p>
        <p>For end-to-end parsing, segmentation has a much larger impact. Single-call is the least reliable,
while the two-step extract→parse pipeline performs best on both datasets by keeping structured outputs
short and failures local. Semantic pre-selection is fastest and improves over single-call, but can miss
scattered or stylistically atypical reference regions (e.g., footnote-heavy pages), limiting recall.</p>
        <p>Overall, segmentation is secondary for extraction but critical for end-to-end parsing: routing the
model through short, verifiable units (or selectively retrieving reference-bearing regions) is key to
robust structured output on SSH documents.
4This model is available at: https://huggingface.co/odoma/Mistral-Small-3.2-24B-LoRA-ref-parsing
5This model is available at: https://huggingface.co/odoma/Qwen3-VL-8B-LoRA-ref-parsing</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>6. Conclusion and Future Work</title>
      <p>This paper benchmarked bibliographic reference processing under SSH-realistic conditions, using CEX,
EXCITE, and LinkedBooks across three tasks: reference extraction, reference parsing, and end-to-end
document parsing. We compared a strong supervised pipeline baseline (GROBID) against contemporary
LLMs under a unified, schema-constrained setup, and further studied practical deployment levers
including LoRA adaptation and document segmentation strategies.</p>
      <p>Across datasets, extraction is comparatively stable: beyond a moderate capability threshold, most
models reach similarly high accuracy and diferences are driven more by throughput and failure rates
than by peak scores. In contrast, parsing and end-to-end parsing remain the key bottlenecks, as they
require consistent structured outputs and long-range robustness under noisy layouts. Here, the results
reveal a clear trade-of: GROBID is typically the fastest option and excels on documents close to its
training assumptions in the end-to-end document parsing task, but it degrades more strongly under
distribution shift, especially for languages beyond English and for references embedded in footnotes or
mixed citation regimes, where LLMs, especially after LoRA finetuning, are markedly more robust and
often yield higher utility.</p>
      <p>These findings suggest LLM-routing [ 23] as a practical way forward to combine speed and robustness
at scale. Data produced during evaluation can be used to train a small LLM to function as a router
capable of dispatching the input document to the best-performing LLM or system, based on evaluation
results (triplets of input document, model name, and F1-score). This approach would allow us to send
to GROBID documents that match its operating envelope (e.g., English, well-structured PDFs with
end-section bibliographies), while redirecting documents with known risk factors (multilinguality,
dense footnotes, complex layouts) to task-adapted LLMs.</p>
      <p>Moreover, the stronger generalization of LLMs compared to supervised systems such as GROBID
makes them particularly promising for SSH corpora, where complex layouts and multilinguality are the
norm rather than the exception. This is especially relevant for highly multilingual collections, where a
natural direction is to decompose adaptation into modular components: (i) language-specific continuous
pretraining for low-represented language (e.g., Latin, Polish), and (ii) task-specific LoRA adapters that
capture the structured-output requirements of reference parsing. Composing these adapters separates
linguistic coverage from task structure, enabling targeted updates without retraining the full system.
Looking ahead, while our LoRA experiments target reference parsing, extending parameter-eficient
adaptation to end-to-end document parsing remains an important next step. We will investigate LoRA
for the full pipeline—layout analysis, reference extraction, and structured parsing—to improve overall
reliability and reduce brittleness under SSH-specific shifts.</p>
    </sec>
    <sec id="sec-7">
      <title>Acknowledgments</title>
      <p>This work was carried out in the context of the GRAPHIA project and received funding from the
European Union (grant ID: 101188018, HORIZON-INFRA-2024-TECH-01 call) and from the Swiss State
Secretariat for Education, Research and Innovation (SERI). Views and opinions expressed are however
those of the author(s) only and do not necessarily reflect those of the European Union or the Agency.
Neither the European Union nor the granting authority can be held responsible for them.</p>
    </sec>
    <sec id="sec-8">
      <title>Declaration on Generative AI</title>
      <p>During the preparation of this work, the authors used ChatGPT in order for grammar and spelling
checking, paraphrasing and rewording. After using these tool, the authors reviewed and edited the
content as needed and take full responsibility for the publication’s content.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>G.</given-names>
            <surname>Colavizza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Peroni</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Romanello</surname>
          </string-name>
          ,
          <article-title>The case for the Humanities Citation Index (HuCI): a citation index by the humanities, for the humanities</article-title>
          ,
          <source>International Journal on Digital Libraries</source>
          <volume>24</volume>
          (
          <year>2022</year>
          )
          <fpage>191</fpage>
          -
          <lpage>204</lpage>
          . URL: https://doi.org/10.1007/s00799-022-00327-0. doi:
          <volume>10</volume>
          .1007/s00799-022-00327-0.
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>A.</given-names>
            <surname>Hosseini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Ghavimi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Boukhers</surname>
          </string-name>
          , P. Mayr, EXCITE - A Toolchain to Extract, Match and Publish Open Literature References,
          <source>in: 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL)</source>
          ,
          <year>2019</year>
          , pp.
          <fpage>432</fpage>
          -
          <lpage>433</lpage>
          . URL: https://ieeexplore.ieee.org/document/8791194. doi:
          <volume>10</volume>
          .1109/JCDL.
          <year>2019</year>
          .
          <volume>00105</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>B.</given-names>
            <surname>Birkeneder</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Aufenvenne</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Haase</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Mayr</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Steinbrink</surname>
          </string-name>
          ,
          <article-title>Extracting literature references in german speaking geography - the geocite project</article-title>
          ,
          <source>in: ULITE@JCDL</source>
          ,
          <year>2022</year>
          . URL: https://api. semanticscholar.org/CorpusID:252599864.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>A.</given-names>
            <surname>Ciofi</surname>
          </string-name>
          ,
          <article-title>Data for Testing and Evaluating References Extraction</article-title>
          and Parsing
          <string-name>
            <surname>Tools</surname>
          </string-name>
          (
          <year>2022</year>
          ). URL: https://zenodo.org/records/6182066. doi:
          <volume>10</volume>
          .5281/zenodo.6182066, publisher: Zenodo.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>O.</given-names>
            <surname>Pagnotta</surname>
          </string-name>
          , CEX Project - Dataset and Gold Standard,
          <year>2024</year>
          . URL: https://zenodo.org/records/ 10535653. doi:
          <volume>10</volume>
          .5281/zenodo.10535653.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>A.</given-names>
            <surname>Garnett</surname>
          </string-name>
          ,
          <source>The XML markup evaluation corpus</source>
          ,
          <year>2016</year>
          . URL: https://pkp.sfu.ca/
          <year>2016</year>
          /04/18/ the-xml
          <article-title>-markup-evaluation-corpus/.</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>European</given-names>
            <surname>Commission</surname>
          </string-name>
          , Open research europe:
          <article-title>Full article corpus</article-title>
          ,
          <source>Complete collection of all articles published on Open Research Europe</source>
          ,
          <year>2025</year>
          . URL: https://open-research-europe.
          <source>ec.europa.eu, accessed 19 March</source>
          <year>2025</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>M.</given-names>
            <surname>Grennan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Schibel</surname>
          </string-name>
          , A. Collins,
          <string-name>
            <surname>J. Beel,</surname>
          </string-name>
          <article-title>GIANT: The 1-Billion Annotated Synthetic Bibliographic-Reference-String Dataset for Deep Citation Parsing</article-title>
          [Data],
          <year>2019</year>
          . URL: https: //dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/LXQXAO. doi:
          <volume>10</volume>
          .7910/ DVN/LXQXAO.
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>G.</given-names>
            <surname>Colavizza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Romanello</surname>
          </string-name>
          ,
          <article-title>Annotated References in the Historiography on Venice: 19th-21st centuries</article-title>
          ,
          <source>Journal of Open Humanities Data</source>
          <volume>3</volume>
          (
          <year>2017</year>
          ). URL: https://openhumanitiesdata.metajnl. com/articles/10.5334/johd.9. doi:
          <volume>10</volume>
          .5334/johd.9.
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>D.</given-names>
            <surname>Tkaczyk</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Szostek</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Fedoryszak</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. J.</given-names>
            <surname>Dendek</surname>
          </string-name>
          , L. Bolikowski,
          <article-title>CERMINE: automatic extraction of structured metadata from scientific literature</article-title>
          ,
          <source>International Journal on Document Analysis and Recognition (IJDAR) 18</source>
          (
          <year>2015</year>
          )
          <fpage>317</fpage>
          -
          <lpage>335</lpage>
          . URL: https://doi.org/10.1007/s10032-015-0249-8. doi:
          <volume>10</volume>
          . 1007/s10032-015-0249-8.
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>A.</given-names>
            <surname>Prasad</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Si</surname>
          </string-name>
          , M.-
          <string-name>
            <given-names>Y.</given-names>
            <surname>Kan</surname>
          </string-name>
          ,
          <article-title>Dataset Mention Extraction and Classification</article-title>
          , in: V.
          <string-name>
            <surname>Nastase</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Roth</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Dietz</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          McCallum (Eds.),
          <source>Proceedings of the Workshop on Extracting Structured Knowledge from Scientific Publications</source>
          , Association for Computational Linguistics, Minneapolis, Minnesota,
          <year>2019</year>
          , pp.
          <fpage>31</fpage>
          -
          <lpage>36</lpage>
          . URL: https://aclanthology.org/W19-2604/. doi:
          <volume>10</volume>
          .18653/v1/
          <fpage>W19</fpage>
          -2604.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>T.</given-names>
            <surname>Backes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Iurshina</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. A.</given-names>
            <surname>Shahid</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Mayr</surname>
          </string-name>
          ,
          <article-title>Comparing free reference extraction pipelines</article-title>
          ,
          <source>International Journal on Digital Libraries</source>
          <volume>25</volume>
          (
          <year>2024</year>
          )
          <fpage>841</fpage>
          -
          <lpage>853</lpage>
          . URL: https://doi.org/10.1007/ s00799-024-00404-6. doi:
          <volume>10</volume>
          .1007/s00799-024-00404-6.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>B.</given-names>
            <surname>Ghavimi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Otto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Mayr</surname>
          </string-name>
          ,
          <article-title>Exmatcher: Combining features based on reference strings and segments to enhance citation matching</article-title>
          ,
          <year>2019</year>
          . URL: https://arxiv.org/abs/
          <year>1906</year>
          .04484. arXiv:
          <year>1906</year>
          .04484.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>