Information Technology and Implementation (IT&I-2025), November 20-21, 2025, Kyiv, Ukraine * Corresponding author. These authors contributed equally.
Introduction with downstream use research. There are some good modern benchmarks and resources for this task.
[
The authors have experience in designing linguistic resources and datasets [
Nevertheless, several published resources can serve as practical surrogates.
MKQA provides aligned question answer pairs across 26 languages (including Ukrainian) for ope ]. Multilingual QA pairs not bound to a specific Wikipedia snapshot or For our task, we must mine evidence on Ukrainian Wikipedia and bind each item to page/section IDs to make it usable for training, as an evaluation
UNLP questions (ZNO) [15
1.0 suite publishes Ukrainian evaluation tasks for sanity checks and model
selection [
TRWU contributes Ukrainian news messages with stance/sentiment/bias labels for robustness
experiments [
The Wikimedia Dumps portal documents the pages-articles.xml.bz2 files, directory structure, and checksums. We pinned the corpus to a single Wikimedia dump snapshot of the target Wikipedia (e.g., ukwiki-20250401-pages-articles.xml.bz2) to guarantee reproducibility and leakage control. We pulled the dump from https://dumps.wikimedia.org/ukwiki/20250401/ using wget with resumed transfers and verified integrity against the published checksums. We recorded the snapshot date, dump file names, file sizes, and checksums in a machine-readable manifest (snapshot.json). We chose the pages-articles dump (namespace 0) so only encyclopedic articles are included by default. We also archived the stub-meta-current.xml.gz for quick metadata lookups. The manifest stores {project: "ukwiki", snapshot: SNAP, dump: filename, md5} and is checked into version control.
Grounding all tasks in a single Wikipedia snapshot is the standard practice that underpins reproducibility, comparability, and leakage control in knowledge-intensive benchmarks (e.g., KILT). Binding our pipeline to a specific snapshot ensures that retrieval sets and generated examples are stable across reruns and across research groups. Compared to pipelines that mix API reads with live content, our snapshot discipline yields deterministic corpora, mirroring best practice in KILT and BEIR while focusing on Ukrainian. This improves auditability (we can always point to the exact article revision) and allows fair head-to-head retriever evaluations.
We converted wikitext to cleaned plain text using WikiExtractor[
Although pages-articles already targets namespace 0, we applied additional page-type and quality filters before indexing: 1. Redirects & disambiguation: dropped pages beginning with #REDIRECT and those containing common disambiguation templates, we also filtered pages whose entire body is a bulleted list of links (list-pages and set-index pages). 2. Minimum content , we removed articles dominated by tables/lists (>70% non-prose). 3. Structural sanity: excluded pages with extreme template density (ratio of markup tokens to words) and pages with no alphabetic tokens in the first paragraph.
Retrieval systems are sensitive to noisy or weakly structured pages, filtering out non-prose content improves dense retriever effectiveness and reduces hallucination in RAG. Benchmarks such as BEIR and MIRACL curate Wikipedia content with quality controls (native judgments, explicit exclusions), which supports our conservative filtering stance. Relative to English-centric pipelines (KILT/BEIR), our Ukrainian-first filtering aggressively removes list-pages/disambiguation pages that often poison top-k retrieval. This is expected to lower the proportion of non-answerable passages retrieved by >10% while retaining coverage of long-tail topics.
Next sub-step is provenance & audit trail. Precise provenance enables error analysis and
deterministic rebuilds key requirements echoed across KILT, BEIR or FEVEROUS. Maintaining
revision IDs supports future diff-based updates without full re-extraction. Now every passage
carries strong provenance: {page_id, revision_id, title, url, section_path, snapshot_id, timestamp,
lang}. We computed a stable document hash (SHA-1 of normalized wikitext) and stored it in both
the JSONL[
Our segmentation design converts article-level text into retrieval-ready passages with the following requirements: indexability under subword token limits of modern encoders, contextual coherence sufficient for sentence-level factual questions, and consistency across languages (Ukrainian primary, multilingual optional) if possible. ={100, 200, 300}) with overlap O (default O=0.2L) using the tokenizer aligned to the downstream encoder, so consecutive windows start at token indices.
We adopted the tokenizer of the encoder actually used for passage representation to eliminate
train test mismatches in subword boundaries. Our default tokenizer is BERT-family
WordPiece[
The window family is motivated by prior retrieval-augmented systems that operate on short passages (approx. 100 200 tokens), and by recent analyses of long-context behavior indicating sensitivity to the position of evidence. Overlap protects discourse continuity across boundaries, away from the beginning or end of the prompt, by ensuring that the same sentence can occur near retrieval and generation. For sections shorter than L it is accepted that they form a single passage, very long sentences are allowed to straddle boundaries to preserve token-level determinism. However, we ensure at least one boundary falls at whitespace. We record token_span and char_span for every passage to retain reproducibility and facilitate highlighting.
We performed simple test on a small subset of our corpus with bag-of-words representation and queries derived from section leads. We compared different values of L (100,200,300) with different overlaps O (0.1, 0.2, 0.3) under a simple retriever. Shorter windows (100 200) consistently showed better retrieval precision at short queries, while 300-token windows slightly improved recall for multi-sentence questions. Overlap at 0.2 was found as optimal point between index growth and recall. Going for 0.3 offered limited additional benefit. These patterns accord with prior literature that emphasizes short, self-contained passages for retrieval-augmented QA.
Fast tokenizers provide deterministic normalization and alignment maps, enabling exact reconstruction of text spans for audits. This matters for retrieval because span misalignment degrades dense similarity and complicates evidence highlighting. Finally, overlap increases index size, we mitigate this with deduplication keyed by (page_id, token_span) and by pruning windows that are near-empty after cleaning.
There is a certain risk in such approach since fixed windows can split discourse phenomena (anaphora, enumerations). There are other options but we decided against using them in spite of mentioned risks due to their disadvantages.
Sentence-aware splitting uses sentence boundaries to avoid mid-sentence cuts, optionally packing adjacent sentences until a token budget is reached. It improves readability and fewer dangling coreferences, but there are disadvantages: reliance on sentence segmenters that may degrade on titles, lists, or scientific notation and introduce language-specific errors.
Semantic (embedding-aware) splitting provides adaptive boundaries chosen by embedding similarity between adjacent sentences, chunks expand or contract to keep intra-chunk coherence high. It often produces fewer irrelevant tokens per chunk, but it has disadvantages: higher
Hierarchical or hybrid splitting organizes passages at multiple granularities (section to paragraph to token windows) to enable coarse-to-fine retrieval. It is flexible with recall/precision trade-offs, but it has such disadvantages as index complexity and fusion logic at query time.
We adopted fixed token windows as the primary method for their determinism, speed, and However, our code path keeps sentence-aware and semantic splitters as pluggable components for ablations.
We adopted a line-oriented JSONL corpus as the basis store for all passage-level artifacts produced -granularity supports incremental processing, graceful degradation (malformed lines quarantined without corrupting neighbors), and easy diffing across releases. It also simplifies re-tokenization experiments: the same JSONL payload can be re-tokenized into alternative chunkings without re-extracting Wikipedia content, preserving provenance fields verbatim across regenerations.
Although JSONL lacks intrinsic indexing, line boundaries permit offset indexing without parsing the entire file. During writing we captured tell() positions and compressed-block boundaries, at finalize time we produced an index table (SQLite) keyed by doc_id. This way, retrieving k passages from disparate shards requires only k random seeks and decompression of at most k gzip members.
Each passage is recorded as a self-contained JSON object on a separate line, enabling stream processing and line-granular recovery. We bound every passage to an immutable, 64-bit doc_id and preserved both char_span and token_span derived during segmentation, so that auditors can reconstruct the exact context byte-accurately from the source text. For efficient random access, we generated a compact offset index that maps doc_id into tuple of (file_path, byte_offset, byte_len) and persisted it as a SQLite container provide supporting functionality (sidecar). This yields O(1) document lookup while keeping the storage human-inspectable.
The extraction pipeline produced gzipped shards (approx. 1 lang/snapshot_id and by the first digits of doc_id. We normalized Unicode (NFC), compacted whitespace, and retained minimal yet sufficient metadata: page_id, revision_id, title, url, section_path, lang, snapshot_id, tokenizer, encoder, tokens, token_span, char_span, and text. The resulting JSONL serves simultaneously as: ground truth for all later layers, as an audit log of segmentation decisions, and a portable interchange format requiring only a text reader.
Schema (abbreviated). {doc_id:int64, page_id:int64, revision_id:int64, title:str, url:str, section_path:list<str>, lang:str, snapshot_id:str, tokenizer:str, encoder:str, tokens:int32, token_span:list<int32>, char_span:list<int32>, text:str}. We deliberately avoided nested blobs that duplicate the raw article, keeping each passage atomic and append-only. When a subsequent snapshot changes an article, we allocate fresh doc_ids, ensuring immutability of previously released shards.
We constructed the initial retrieval subsystem entirely from the JSONL corpus, omitting any
text field was embedded using basic BERT [
For this purpose JSONL shards were streamed in batches, texts were prefixed with no special markers (BERT does not require instruction prefixes), tokenized with the exact tokenizer pinned in Part 2, and encoded on GPU where available. We retained exact tokenizer/model revisions in the shard manifest to guarantee reproducible embeddings. The embedding matrix and a parallel doc_ids.npy vector (int64) were written per shard, with simple statistics (mean norm is approx. 1.0, zero NaNs) logged for each batch.
While specialized retrieval encoders exist, BERT remains a robust baseline for multilingual semantics when paired with an appropriate pooling strategy. Using the same tokenizer/wordpiece inventory as segmentation removes a source of drift between segmentation and retrieval and simplifies ablation: any retrieval gains can be attributed to indexing rather than tokenization mismatch.
For a query string, we apply the same BERT tokenizer and mean-pooling, normalize the vector, and compute cosine similarities against the passage matrix using blocked matrix multiplication to keep CPU caches warm. We then select top-k by partial sorting and return the corresponding doc_ids. Empirically, on a single 16-core server, brute-force cosine over approx. 250k passages remains within interactive latencies (approx. 10 60 ms) due to efficient BLAS kernels. Beyond that scale, sharding the matrix across processes provides linear throughput gains.
In parallel to embeddings, we distilled a small SQLite sidecar keyed by doc_id holding title, url, section_path, lang, and char_span. Given retrieved doc_ids, we reconstruct human-readable evidence by reading the passage from JSONL (via the offset index explained in Part 4) and highlighting the char_span. This minimal path preserves strict provenance while keeping the indexing stack lightweight.
The JSON-only, BERT-based retrieval provides a transparent and reproducible baseline that reaches acceptable Recall@k for definition and entity-centric queries on our snapshot while maintaining deterministic evidence reconstruction. It establishes a floor against which later improvements (alternate encoders or ANN structures) can be measured without confounds.
We transitioned from the JSON-only, brute-force cosine baseline to a graph-based approximate
nearest-neighbor (ANN) index using Hierarchical Navigable Small Worlds (HNSW)[
We trained no additional model: with BERT-derived, L2-normalized passage embeddings produced before we adopted inner product as equivalent to cosine on unit-norm vectors and constructed HNSW indexes per shard (approx. 1 5 M passages). We set default hyperparameters to M = 32, efConstruction = 200, efSearch = 64, with adaptive increases of efSearch for queries exhibiting low score margins. Indexes were wrapped in IndexIDMap2 to preserve 64-bit doc_ids and serialized alongside a manifest that records (dim, metric, M, efConstruction, efSearch, checksum).
HNSW yields a favorable latency recall frontier on sentence-level embeddings and supports incremental additions. Memory overhead scales roughly linearly with M and the number of nodes; thus, M = 32 balances recall and RAM, while efConstruction = 200 improves connectivity without prohibitive build time. Empirically (on our Wikipedia-like corpus), this configuration reduced median query time by an order of magnitude relative to brute force, with high Recall@20 preserved. We verified that these trends match established evaluations reported in independent benchmarking environments and vendor-neutral documentation.
We parallelized construction across shards. Each worker (CPU-bound for HNSW) mapped embeddings_bert.npy read-only, built a IndexHNSWFlat(d, M, METRIC_INNER_PRODUCT), set efConstruction, and added vectors in contiguous ID order to favor cache locality. After add_with_ids, the index was reparameterized with efSearch defaults and written to disk. We performed post-build sanity checks: degree distribution histograms, connectedness probes (BFS sample), and a smoke-test query set with known neighbors.
Queries encoded by BERT are normalized and searched with HNSW k-NN. We monitor margin -1 and top-2) and empty-hit rate. If a margin is small or the hit set is nearly duplicate, efSearch is raised (e.g., 64 is changed to reranker cost in downstream generation.
Search returns doc_ids, which we join to the SQLite sidecar to recover title, url, section_path, lang, and exact char_span. The index and metadata are version-locked via manifest hashes to guarantee reproducibility.
HNSW is robust to moderate noise and local clustering; however, performance can degrade on purely random vectors. We therefore encode with stable BERT embeddings and normalize vectors to mitigate norm drift. We also reject shards whose post-build checks show abnormally high disconnected components or degree pathology and recompute with higher M. The HNSW deployment produced an interactive retrieval layer with stable p90 latency and competitive Recall@k, while preserving strict, file-based reproducibility. This forms the first production-ready search substrate atop our JSONL corpus.
We evaluated several variants of templated with understanding that they are not mutually exclusive.
QA style (Question/Context/Answer) - replaces Instruction with Question. atural for
factoid QA and pairs well with DPR retrievers[
ReAct-style RAG interleaves Thought and Act(Search) steps to fetch more passages during inference. It improves factuality and reduces hallucination by allowing targeted retrieval. But it requires a controller and tool integrations (increasing development complexity) and is costlier and harder to cache.
Demonstrate-Search-Predict bootstraps few-shot demonstrations with retrieval in the loop approach is strong for tasks that benefit from pipeline-aware exemplars. However, it is even more complex and manpower intense than ReAct RAG and one must be careful with curation needed to avoid leakage which rises manpower requirements even more.
XML-constrained RAG wraps segments in tags:
It is machine-readable and plays well with validators, also it is helpful for structured output requirements. Disadvantages are: sensitivity to novel tags and a bit reduced human readability.
JSON schema RAG when
directly machine-readable and simplifies downstream scoring (same as XML-constrained). However, it can over-constrain free-form exposition and is fragile if the model produces trailing commentary.
We decided to use the Instruction/Context/Answer (I/C/A) for ground generation in retrieved evidence because it is sufficient for the task at hand. We adopted triple-quotes for marking the context block to minimize boundary errors and support verbatim spans, while keeping machine parsing trivial. Each context passage is accompanied by [doc_id] and char_span for provenance. Our post-processor verifies that the final answer is attributable to identified sources and/or satisfies RAG-specific metrics.
System: You are a careful assistant. Use only the Context. If the Context is insufficient, say so. Instruction:{instruction} Context: """ [{doc_id_1}:{char0_1}-{char1_1}]{passage_1} [{doc_id_2}:{char0_2}-{char1_2}]{passage_2} """ Answer:
This format decouples Instruction (task control) from Context (evidence) and exposes explicit provenance tags that our evaluator can match against citations.
On the technical side we implemented a context packer that assembles the top-k passages returned by HNSW into a bounded-length window. Passages are ranked by a convex combination of similarity, recency (revision date), and entity overlap with the instruction. We de-duplicate near-identical passages by doc_id and Jaccard overlap on token sets to reduce redundancy.
We used Mistral-7B-Instruct [
Having established a high-recall nearest-neighbor index over BERT embeddings, we designed a family of adaptive prompt templates question in a normalized slot, optionally elicit reasoning traces for more demanding tasks, and standardize the answer channel to facilitate evaluation. We additionally support a Template insertion control (e.g.,
) that toggles the latent reasoning mode without enforcing a particular trace length.
Structured prompts reduce ambiguity and provide latent affordances for reasoning strategies (decomposition, plan/execution separation). Empirical studies report large changes from format choices alone, even when semantic content is held constant. Reasoning-eliciting templates consistently improve success on compositional and arithmetic tasks. Meanwhile, formatting sensitivity implies that stable, parseable layouts (XML/JSON) can mitigate variance and ease downstream evaluation.
We tried several forms to test robustness and accommodate different inference regimes (not templates):
Minimalist (Question - Answer): #Question:{question} #Answer: <final answer here>.
Such approach allows for lowest token overhead, strong for simple factoid QA when retriever is accurate. However, it is fragile on multi-step reasoning, there is no explicit affordance to plan.
Self-Ask #Question:{question} #Sub-question 1: <question> #Answer 1: <answer> #Sub-question 2: <question> #Answer 2: <answer>
It encourages explicit decomposition and can integrate search actions between sub-questions. The disadvantages are longer prompts and it requires reliable gating to avoid over-decomposition.
Plan-and-Solve header #Plan # # #Solve <reasoned answer>
This approach separates planning from execution and reduces missing-step errors. But, it also adds 10 25% token overhead and is sensitive to header labels.
Least-to-Most #Problem:{question} #Easier subproblems: [s1,s2, #Solve in order and combine.
Such approach helps with compositional generalization. Though it has high manpower cost for subproblems and there is a risk of leakage if subproblems reveal answers.
Tree-of-Thought controller #Goal:{question} #Think in branches of up to B alternatives per step; depth D; evaluate with heuristic H. #Return only the final answer.
It enables breadth exploration and backtracking, but it has highest inference cost and requires custom controller and stopping rules.
Schema-constrained / XML-tagged #<q>{question}</q><r/> #<format>final-only</format>
It has best advantages: it is parseable, it aligns with downstream extraction and plays well with safety filters. The only drawback is that LLMs are sensitive to unknown tags, so continuity is paramount.
In our ablations, minimalist and Plan-and-Solve variants yielded the best latency accuracy balance under tight budgets, while Self-Ask and Tree-of-Thought offered higher ceilings on multi-hop tasks at increased token cost. The XML-tagged variant proved most robust to noisy context windows and is our default for production datasets because it simplifies evaluation and redaction.
We formalize an Instance as a 4-tuple (question, reasoning, answer, template_label) with optional reasoning and template_label. Each instance is serialized into two orthogonal views: • • a human-readable (for this manuscript), and a machine template with explicit delimiters for injection into LLM contexts.
Our default machine template for adaptive prompts: <task> <question>{question}</question> <reasoning>{reasoning}</reasoning><!--optional--> <answer>{answer}</answer> </task> <controls> <template>{template_label}</template> </controls> <!--e.g.,
step by step: [{question}]-->
The XML-like markers act as sentinel delimiters to reduce boundary errors during concatenation and to ease post-hoc parsing. We preserve the underlying JSONL record as ground truth by mirroring these fields as keys .
Prompt-format sensitivity is substantial, so structure and delimiters are not cosmetic. Our ablations showed double-digit swings in exact-match when toggling separators and tag names, formatting strongly affects LLM behavior.
For each question we normalized the user questions by lowercasing only function words and preserving named entities. This prevents capitalization-based retrieval drift while keeping entity surface forms intact. Each question is linked to its retrieval bundle (top-k passages + doc_ids) for provenance.
For easy questions (factual, single-hop), we leave <reasoning> field empty and rely on evidence citations during generation. For compositional or multi-hop questions, we generate a concise, model-internal outline during dataset construction (two to four sentences) that expresses salient relations or decompositions (e.g., -wavelength ). We use the <template_label> to switch on an instruction phrase (e.g., Explain step by step), without prescribing the full chain-of-thought at inference -this lets the serving model decide how much intermediate text to expose.
We keep <answer> empty at prompting time (for test splits) or fill it for training instances used in instruction tuning. In both cases the template reserves the slot and the post-processor knows
This kind of templates naturally support hidden reasoning or outline-level hints but allows to evaluate only the final answer against evidence.
By consolidating on the XML- -byachieved consistent improvements on multi-hop questions without exposing long reasoning traces, lowered annotation overhead than with Self-Ask and Tree-of-Thought, and created robust parsing in our evaluation harness. The template also integrates seamlessly with our RAG context, as tags cleanly delimit query, evidence, and expected outputs.
After all is done the remaining bottleneck for dataset quality is example hygiene: overly general queries must be removed, near-duplicate instructions must be collapsed, and examples whose answers are not attributable to the retrieved context must be rejected.
We adopt a three-gate policy executed in the following order: (G1) query specificity, (G2) instruction de-duplication, (G3) context-to-answer consistency. The ordering is motivated by cost: pre-retrieval processing is (G1) cheapest, near-duplicate detection (G2) amortizes costs over entire shards, and faithfulness checks (G3) is the hardest. All gates leave a machine-readable audit trail (JSON) that explains the decision with scalar scores and thresholds.
All gates produce structured audit records that log features, scores, thresholds, and the reason for accept/reject. This allows re-running the validator across snapshots with different hyperparameters without re-encoding texts. We deliberately bias toward precision (filtering too much) in G1 and G2, because failures escalate to a rewrite/curation queue; in G3 we balance precision/recall so as not to discard legitimate paraphrastic answers.
Of course there are some issues. Entity-free queries that are still specific in closed domains (e.g., mathematics) may be over-filtered. Lexically diverse paraphrases may evade MinHash. Embedding proximity misses subtle contradictions. Nevertheless automation greatly reduces amount of required manual filtering.
The objective is to identify and remove prompts whose information need is under-specified (e.g., evaluation. We formalize under-specification as low lexical specificity.
We combine pre-retrieval predictors from [
Stopword ratio and entity density (named entities per token), to penalize boilerplate phrasings while retaining entity-anchored requests.
Length controls (minimum 3 content tokens after stopword removal) to rule out one-term commands.
We compute these signals per query and apply a learned decision rule trained on a small, hand-labeled dev set. The classifier is a calibrated logistic regression over standardized features (avgIDF, stopword ratio, lexical specificity, content length), chosen for transparency.
Applying G1 removes a consistent tail of vague prompts while preserving entity-centric and compositional questions. This aligns with reports that pre-retrieval specificity predictors are strong proxies for query quality and reduces the frequency of ungroundable generations in our RAG evaluation.
The objective is to collapse near-duplicate Instruction and Question variants to avoid training and test leakage, evaluation bias, and over-representation of popular intents.
We combine lexical features MinHash to capture surface-form duplicates and semantic cosine with the same BERT mean-pooled embeddings used elsewhere. So, each normalized instruction is shingled (word 5-grams), hashed into a MinHash signature, and indexed. The BERT embedding matrix of instructions is searched with HNSW to collect cosine neighbors. We merge close items into clusters. This hybrid approach catches both trivial paraphrases and deep semantic duplicates. It directly follows evidence that deduplication improves generalization and lowers memorization in LLMs, and mirrors web-scale dataset construction practices that rely on MinHash at scale.
The objective is to detect and reject examples whose Answer is not supported by the Context returned by the retriever, thereby enforcing attribution and reducing hallucinations.
We combine a vector-proximity test with a textual alignment score: • • •
Attribution compliance. If the template requires citations, we verify that cited doc_ids are a subset of the retrieved doc_ids. This is formal procedure.
Embedding proximity test. We split the answer into sentences, encode each with BERT mean-pooling, and compute its maximum cosine similarity to any sentence from the packed context. An example passes if sufficient number of scores of answer sentences exceed a threshold. This test is indexing-agnostic and efficiently batched.
For curated subsets we compute a factual consistency score with an alignment model between the context and the answer. Low alignment flags unsupported claims even when embeddings yield false positives.
We first run the proximity test, failing items are dropped immediately. Passing items are passed to the alignment scorer. The two-stage test rejects obviously unsupported answers cheaply and reserves expensive alignment modeling for ambiguous cases. In practice this reduces hallucination rate and raises attribution scores in our RAG evaluations without inflating latency during dataset construction.
While BERT-compatible WordPiece tokenization is our default for interoperability with BEIR-style retrievers, we considered several alternatives. Each of them has strong advantages and they are listed below, but disadvantages made them untenable at this stage of development.
XLM-R [
mDeBERTaV3 [
MPNet[
Advanced embedding models such as E5-v2[
As for performance, there is another option for memory-sensitive applications. IVF+PQ [
We successfully designed and implement a reproducible Ukrainian language centered Wikipediato-Tasks pipeline that can do the following: • • • • • robustly extracts and cleans content from MediaWiki snapshots; performs adaptive chunking that respects section/paragraph boundaries and augments queries with retrieval context; builds lexical, dense, and hybrid indices with explicit provenance; generates and normalizes instruction and QA pairs supported by retrieval evidence; and filters answer mismatches by embedding similarity) with targeted human review.
We are very grateful for support we received during this research from Department 165 of Institute of Information Technologies and Systems and Department of Mathematical Informatics from Faculty of Computer Sciences and Cybernetics of Taras Shevchenko National University of Kyiv. The research was funded by the National Research Foundation of Ukraine.
During the preparation of this work, the authors used ChatGPT-5 in order to: Text Translation, Grammar and spelling check. After using this tool, the authors reviewed and edited the content as .