This study evaluates the feasibility and performance of deploying open-source large language models (LLMs) on CPU-only systems using the llama.cpp runtime with 4-bit GGUF quantization. The analysis covers seven instruction-tuned models-Phi-4-mini-instruct (3.8B), DeepSeek-R1-Distill-Llama-8B (8B), Llama-3.2-1B-Instruct (1.2B), Qwen2.5-7B-Instruct (7.6B), Gemma-3-12B-it (12B), Mistral-Small-3.1-24B-Instruct-2503 (24B), and Gemma3-1B-it (1B)-across three CPU environments: a self-hosted Intel Xeon E5-2695 v2 VM, a cloud-based Intel Xeon Platinum 8480+ (“New Xeon”), and a modern Intel Core i7-13700H laptop. Benchmarks report throughput (tokens/s), latency, and memory footprint. On the New Xeon, Llama-3.2-1B reaches ∼ 120 tokens/s, Qwen2.5-7B ∼ 45 tokens/s, and Mistral-24B ∼ 15 tokens/s; on the older E5-2695 v2, typical rates are ∼ 25, ∼ 8, and ∼ 2 tokens/s, respectively. Memory usage spans ∼ 0.5-13.4 GB at 4-bit, enabling multi-agent deployments under modest RAM budgets. A case study, MedLocalGPT, demonstrates CPU-only bilingual (EN↔UK) translation with Phi-4-miniinstruct and domain Q&A with Gemma-3-1B-it, grounded by a RAG pipeline over an EBSCO rehabilitationmedicine corpus. The findings provide practical guidance on model choice, threading, and deployment trade-ofs for cost-sensitive or privacy-preserving scenarios.
The selection and usage of CPU-based LLMs are investigated in the context of the MedLocalGPT
project [
To make LLMs feasible on commodity hardware, many studies have focused on quantization – reducing
model precision to shrink memory and compute requirements. Shen et al. [
Frantar et al. [
Xiao et al. [
Lin et al. [
Kim et al. [
Beyond quantization, researchers have optimized the Transformer architecture itself for CPUs. Zhang
et al. [
The open-source llama.cpp project [
Another relevant trend is distilling larger models into smaller ones to better suit CPU deployment. The
DeepSeek-R1 [
To summarize, the preceding two years have seen rapid progress in: quantization techniques (int8, int4, FP4) that radically reduce model size with minimal accuracy loss; CPU-specific optimizations in attention algorithms and math kernels that improve inference speed by 2× or more; and model compression and fine-tuning (distillation, data-centric training) that produce smaller models with capability closer to their large predecessors. These advancements set the stage for deploying multi-billion-parameter LLMs on commodity CPUs. Subsequent sections will leverage these developments – specifically 4-bit quantization via llama.cpp – to evaluate real-world LLM deployments on CPU-only environments.
A diverse set of open-source LLMs, ranging from 1B to 24B parameters, was selected, all of which have publicly available weights.
• Phi-4-mini-instruct [
The instruct variant, designed for conversational use, is employed in this study.
• Gemma-3-12B-it [
Gemma-3-12B can handle text and images natively and supports over 140 languages (including
Ukrainian) with a 128K context window. It is instruction-tuned for chat and agent tasks. This
model represents a state-of-the-art open LLM that pushes the envelope in size and capabilities
while remaining deployable (12B parameters) on CPU with quantization.
• Gemma-3-1B-it [
In this study, three CPU-only environments are considered: • Intel Xeon E5-2695 v2 CPU environment (self-hosted server).
The Intel Xeon E5-2695 v2 [31] is a 12 cores / 24 threads CPU with a base clock speed of 2.4 GHz (3.2 GHz turbo), and a thermal design power (TDP) of 115 W. Based on the Ivy Bridge-EP microarchitecture, this CPU supports AVX (256-bit) instruction sets but lacks AVX2/AVX-512 support.
Benchmarking was conducted on a virtual machine (VM) provisioned via libvirt with Kernel-based
Virtual Machine KVM, hosted on an HP ProLiant DL380p Gen8 server [
The host server configuration included: CPU – 2x Intel Xeon E5-2695 v2, RAM – 400 GB DDR3 Advanced ECC memory; storage – 2x 400 GB SSDs in RAID 1, and 8x 400 GB SSDs in RAID 10; network connection – 1 Gbps Ethernet; power supply – 2x 460 W; UPS – Eaton 5SC 1500VA; host operating system (OS) – Ubuntu 22.04.5 LTS.
The VM used for benchmarking was configured with the following specifications: vCPUs – 16 cores; RAM – 32 GB DDR3 RAM; Storage – 256 GB SSD; guest OS – Alpine Linux v3.18. • Intel Xeon Platinum 8480+ CPU environment (AWS EC2 VM server).
Intel Xeon Platinum 8480+ [32] is a 4th Gen Sapphire Rapids CPU featuring 56 cores / 112 threads, with a base clock frequency of 2.0 GHz, and a TDP of 350W. This CPU represents a modern, high-core-count Xeon configuration commonly available in cloud environments, such as Amazon Elastic Compute Cloud (AWS EC2) [33]. It supports advanced instruction sets, including AVX512, VNNI, and AMX, which significantly accelerate matrix and vector operations relevant to deep learning inference. In the subsequent results, this CPU is referred to as the “New Xeon”. With its high core count and substantial memory bandwidth, it exemplifies an ideal scenario for CPU-based LLM inference.
Benchmarking was conducted on AWS EC2 C7i-flex VM instance [ 34] – c7i-flex.4xlarge with the following specifications: vCPUs – 16 cores; RAM – 32 GB DDR5 RAM; Storage – 256 GB SSD; guest OS – Alpine Linux v3.18. • Intel Core i7-13700H (modern laptop CPU).
Intel Core i7-13700H [35] is a 13th Gen Intel CPU featuring 14 cores (6 Performance + 8 Eficient cores), up to 5.0 GHz turbo (45W TDP, 115W PL2). This is a consumer-grade laptop processor with AVX2 and VNNI support (but no AVX-512 on consumer chips). The test laptop has 32 GB DDR5 RAM. This setup reflects on-the-go or edge deployments where only a laptop/PC is available. Thermal throttling is a factor here, and the hybrid architecture means not all cores are equal in performance.
Benchmarking was conducted on ThinkBook 14 G6 laptop with the following specifications: RAM – 32 GB DDR5 RAM; Storage – 512 GB SSD; Host OS – Ubuntu 22.04.5 LTS; guest OS – Alpine Linux v3.18.
All models were converted to llama.cpp’s GGUF format with 4-bit quantization. This study used Q4_K (g4) quantization which quantizes weights to 4 bits and groups them in blocks with some extra scaling metadata. This yields approximately 4× smaller model size in memory compared to float16. For example, the 7B models occupy about 3.7–4.0 GB at 4-bit (depending on grouping overhead) versus 20+ GB in half-precision. All tests were run on Ubuntu Linux with llama.cpp (commit v1.5.3) using CPU BLAS backend. Multi-threading was enabled and threads were pinned to physical cores where applicable. For the Xeon environments, a single model was executed at a time, utilizing all available cores to maximize per-model throughput. For the hybrid laptop, llama.cpp uses all 14 threads; it tends to schedule heavier work on P-cores, but the exact thread scheduling is left to the OS.
Deployment Environment: llama.cpp library compiled and executed inside a Docker container built on the Intel oneAPI Base Toolkit, specifically using Intel oneAPI Math Kernel Library (oneMKL) for CPU-optimized linear algebra routines. No GPU acceleration was used at any point.
Table 1 summarizes the throughput (tokens/s) and corresponding per-token latency on the three CPU platforms for each model. As expected, the newer 56-core Xeon outperforms the older 12-core Xeon dramatically, and the laptop i7 falls in between (closer to the older Xeon on larger models, but competitive on smaller models due to its high turbo frequencies).
New Xeon vs. Old Xeon. The 56-core Sapphire Rapids Xeon is about 3× to 6× faster than the old 12-core Ivy Bridge across these models. For instance, on Llama-3.2-1B, the new Xeon reaches 120 tokens/s, vs 25 on E5-2695 v2. On the largest 24B model, it sustains 15 tokens/s(67 ms/token) whereas the old Xeon struggles at 2 tokens/s(half a second per token). This gap is due to a combination of more cores, higher per-core IPC, and newer instructions. Notably, the old Xeon lacks AVX2/AVX-512, so it processes 4-bit dot-products much less eficiently (likely in 128-bit chunks) compared to the AVX-512 VNNI support on the new Xeon. The new Xeon also has 8 memory channels of DDR5 feeding the cores, helping with the heavy memory bandwidth demands of attention and feed-forward network layers.
Laptop vs. Old Xeon. The Core i7 laptop actually outperforms the old Xeon on smaller models. For 1B and 3.8B, the i7 is 2× faster (e.g. 40 tokens/svs 15 tokens/son Phi-4-mini). This is because the workload is not perfectly parallelizable across dozens of cores – the 1B model, in particular, cannot eficiently use all 12 cores of the old Xeon due to its limited layer and matrix sizes. The i7’s few high-clock (5
GHz) cores complete the work quickly, whereas many of the old Xeon’s cores sit idle or waiting when running a tiny model. However, on larger models (7B+), the old Xeon’s additional cores begin to catch up. By 12B and 24B, the E5 and i7 have similar throughput (e.g., 5 vs 9 tokens/sat 12B, and 2 vs 4 tokens/sat 24B) – in these cases the laptop is constrained by thermal limits and its eficiency cores. Overall, the laptop CPU is roughly equivalent to a 2015-era dual-socket server in LLM inference. This is encouraging for edge deployments: a high-end laptop can handle a 7B model at 15 tokens/s(about 4 tokens generated per second after prompt), which is suficient for a basic chatbot with modest latency.
Latency to First Token. The table above focuses on per-token latency during generation. It’s important to mention the initial prompt processing latency. This depends on prompt length. In our tests with a 256-token prompt, the time to process the prompt was about 6.5 seconds on E5-2695 v2 for a 7B model, 1.5 s on the new Xeon, and 3 s on the laptop. This front-loaded cost means that for short user queries the “time to first token” can be dominated by prompt encoding. Techniques like caching prompts or using shorter context can mitigate this. For instance, if re-using a long conversation history, llama.cpp can leverage the KV cache so that only new tokens are processed. Still, on CPU one should expect an initial wait on the order of a few seconds for the model to “think” before starting to stream output, especially with long contexts. After that, tokens stream at the rates shown in Table 1 (which for the new Xeon can be faster than real-time reading speed, 5-20 tokens/sneeded for fluent speech).
Multilingual and Multimodal Overhead. Models like Gemma-3 and Mistral 24B have additional capabilities (140 languages, vision features). Images were not provided during testing (all evaluations were text-only); consequently, the vision encoders in Gemma and Mistral remained inactive.However, their presence and larger vocabularies do have some overhead. Gemma-3-12B, for example, has to handle a 128k vocabulary and local+global attention mechanism, which is slightly more compute per token than a standard 12B model. This may explain why its throughput (30 tokens/son new Xeon) is a bit lower than one might expect compared to, say, OPT-13B int4 results in other work. In general, specialized architectures can afect speed: the Qwen-7B model uses GQA with 28 query heads and 4 key heads, efectively reducing attention computations by sharing keys/values. This likely helped Qwen achieve 45 tokens/son the new Xeon, slightly beating a non-GQA 7B. Mistral 24B likewise uses multiquery attention (one head for all keys/values per attention block) similar to LLaMA-2 architecture, making it more eficient than a naive 24-head design. These nuances show up in throughput diferences of a few tokens/s.
All models comfortably fit in RAM with 4-bit quantization (GGML Q4_K). Table 2 lists the memory footprint of each model in 4-bit, alongside their full FP16 size for reference. Even the largest model (Mistral 24B) uses about 13.4 GB in 4-bit, which is within the 32 GB memory of a decent laptop (though leaving little room for other processes). The smallest (Llama-3.2-1B and Gemma-3-1B) are around Model Phi-4-mini-instruct DeepSeek-R1-Distill-Llama-8B Meta Llama-3.2-1B-Instruct Qwen2.5-7B-Instruct Gemma-3-12B-it Mistral-Small-3.1-24B-Instruct-2503 Gemma-3-1B-it
0.5–0.6 GB, trivial for any modern system. This illustrates the 25× size reduction from full precision – e.g., LLaMA-2 7B is 13 GB in FP16 but 3.7 GB in 4-bit. This reduction is what makes CPU inference feasible.
One caveat: models quantized to 4-bit do incur a small accuracy penalty. The literature reports <1% loss on average benchmarks, but certain tasks (especially ones requiring precise numerical calculations or uncommon tokens) might degrade a bit more. In our experience with these instruct models, the quality remained very good for chat and reasoning; all models produced coherent and relevant answers to prompts. If absolute fidelity is required, 8-bit quantization is another option – it roughly doubles memory usage but can be a sweet spot. Some of the new Xeon CPUs support INT8 acceleration (via AVX512-VNNI and AMX instructions) which can make 8-bit as fast as 4-bit or faster in some cases. However, 4-bit quantization was prioritized to maximize memory savings.
This study highlights the feasibility and trade-ofs of deploying LLMs on CPU-only environments. Key considerations and practical recommendations for diferent scenarios are outlined below.
Modern CPUs with many cores can achieve impressive throughput on quantized LLMs—on the order
of tens of tokens per second for models up to 13B, as observed with the 56-core Xeon. This aligns
with other reports; for example, Intel’s study reported ∼ 20 ms/token (50 tokens/s) for a 7B model
on a Sapphire Rapids 8480+ using INT4 quantization [
Diferent tasks may favor diferent models, and the best model choice can depend on the CPU power available.
Retrieval-Augmented Generation. In RAG, the model is provided with retrieved context (e.g., documents) and asked to compose an answer. This reduces the need for the model to have vast knowledge parameters. Hence, smaller models can work well since they mainly need language fluency and the ability to integrate given facts. For a CPU-bound RAG system (say a QA bot that fetches from a knowledge base), one might choose Llama-3.2-1B or Phi-4-mini (3.8B) on an older CPU. These tiny models are extremely fast (20–30 tokens/son old Xeon) and can still produce coherent answers, especially if the needed facts are in the retrieved text. If the documents are long, the 1B Llama-3.2’s 128K context length is a huge advantage – it can ingest very large retrievals (with some performance penalty for the long attention, but it’s manageable). For more complex RAG where reasoning over facts is needed, the DeepSeek 8B or Qwen 7B would be a good middle-ground if the CPU can support them, as they bring more reasoning capability. On the new Xeon, one could even use the 12B model to possibly get better answers – e.g. Gemma-3-12B’s broad knowledge could fill gaps if retrieval misses something. But the cost is speed: on our new Xeon, Gemma 12B was 30 tokens/svs Llama-1B at 120 tokens/s(4× slower). So for high-throughput RAG (like answering many queries per second), a smaller model is preferable. One approach is to use two-tier models: a small model quickly answers simple or factoid questions, and a larger model (maybe on a separate machine or thread) handles only the complex queries that need more reasoning or a second pass.
Language Translation (English↔Ukrainian). Translation quality and bidirectional consistency are paramount in our setting. Among the evaluated models, Gemma-3-12B-it and Qwen2.5-7B-Instruct are the most suitable for English ↔ Ukrainian due to broad multilingual training and long-context support. On the Xeon 8480+, Gemma-3-12B achieves 30 tokens/s( 33 ms/token), enabling near-real-time sentence-level translation ( 0.4–0.7 s for a 15–25-token sentence). Qwen-2.5-7B attains 45 tokens/s( 22 ms/token), ofering slightly lower latency with competitive quality. On resource-constrained systems, Phi-4-mini (3.8B) and Llama-3.2-1B-Instruct provide usable throughput (e.g., 40–80 tokens/son modern laptops) and can serve short, domain-focused segments, albeit with greater risk of lexical omissions and style drift. For older CPUs (e.g., E5-2695 v2), Qwen-7B ( 8 tokens/s) remains practical for low-volume workflows, while 1-4B models are preferable for interactive use.
Recommended practice for English↔Ukrainian includes: deterministic decoding (temperature 0.1–0.3, top-p 0.9–0.95, repeat-penalty 1.1) to reduce stylistic variance; terminology control via in-prompt glossaries (e.g., medical or legal lexicons) and explicit style directives (formal vs. conversational Ukrainian; preservation of proper names and abbreviations); sentence-level chunking with UTF-8–safe segmentation to avoid context dilution. Specialized MT systems (e.g., NLLB/M2M100) may exceed general LLMs on BLEU/chrF for EN ↔ UK, but the models evaluated here ofer competitive quality with superior instructionability (style, register, terminology constraints), which is advantageous in CPU-only deployments. In practice: select Qwen2.5-7B when latency and cost are tight; prefer Gemma-3-12B-it when translation fidelity and robustness to domain terminology are prioritized; and reserve Phi-4-mini / Llama-3.2-1B for lightweight, interactive translation of short segments or as fallbacks on constrained hardware.
Chatbots and Generative QA. This is the canonical use case for instruction-tuned LLMs. Response quality (coherence, helpfulness, reasoning) is the primary objective, but latency must remain acceptable to sustain a natural dialogue. Model selection therefore involves a trade-of: larger models tend to answer better but respond more slowly. On an E5-2695 v2–class server, deploying a 24B model (e.g., Mistral-24B) for interactive chat is sluggish—individual sentences can take 10+ s. In contrast, 7–8B models such as DeepSeek or Qwen are markedly more practical, yielding short-query responses in roughly 2–3 s ( 8 tokens/s). In the present evaluation, DeepSeek-8B on the E5-2695 v2 produced multi-sentence answers in under 15 s and showed noticeably higher accuracy on challenging prompts than 1–3B models. Consequently, 6–8B emerges as the sweet spot for CPU-based chatbots on older hardware—balancing acceptable answer quality (via instruction tuning and, in some cases, distillation) with tolerable latency.
With a more capable CPU (e.g., a new Xeon or a modern 16-core desktop), stepping up to the 13B class can enhance depth and nuance. On a 56-core Xeon, for instance, Mistral-24B produced high-quality answers with 4 s delay for short prompts, and the outputs exhibited more context and subtlety than typical 7B responses. The compute budget should therefore guide model choice: • Low budget (<16 CPU threads): prefer 1–4B models. • Mid budget (16–32 threads): prefer 7B models.
• High budget (64+ threads): 13B or even 30B becomes viable.
Use instruction-tuned variants whenever possible. Base (pretrained) models generally require careful prompting or few-shot scafolding to achieve comparable reliability, whereas the evaluated instructiontuned models (Phi, DeepSeek, Qwen, Llama-3.2, etc.) provide robust conversational behavior out of the box.
Cost Considerations. Running LLMs on CPUs can be cost-efective in certain cases. Many organizations have spare CPU capacity or existing CPU servers, whereas GPU resources are scarce or expensive. By quantizing models, one can use cheap standard VMs to deploy AI features without the steep pricing of GPU instances. There is also the aspect of software cost – some GPU deployments require proprietary frameworks or cloud services, whereas llama.cpp is free and MIT-licensed, easily integrated into any environment. That said, for very large-scale deployments (like serving thousands of requests per second), CPUs might end up using more electricity and more machines than an equivalent GPU cluster, so the calculus can change. It often comes down to what assets you already have and the scale of usage. For moderate usage or internal tools, repurposing commodity CPUs for LLM inference is extremely attractive.
The MedLocalGPT system [
Models selection. Phi-4-mini-instruct ofers strong instruction following and competitive smallmodel translation quality under CPU constraints; at 4-bit it fits in 1.9 GB RAM and sustains 15 tokens/son an E5-2695 v2–class host and 40–80 tokens/son modern laptop/new-Xeon CPUs (Table 1). This makes sentence-level EN↔UK translation responsive while preserving headroom for concurrent services. Gemma-3-1B-it ( 0.6 GB at Q4) is selected for Q&A to maximize throughput tokens/sand minimize latency for short, fact-seeking prompts; typical rates are 30 tokens/s(E5-class) and 70–100 tokens/s(laptop/newXeon), adequate for interactive clinical-education use. The small memory footprints allow both agents, the embedding service, and the database to co-reside on a 16-vCPU/32 GB KVM VM without swapping, while the Intel oneAPI (oneMKL) build of llama.cpp provides consistent BLAS performance in the containerized runtime.
RAG over EBSCO dataset. The ingestion pipeline extracts text from EBSCO articles, normalizes typography, and applies domain-aware chunking (target 512–768 tokens with 15–20 % overlap) to preserve local cohesion around methods and results sections. Each chunk is embedded with an Instructorlarge model; the store maintains cosine similarity vectors and metadata (journal, year, DOI, MeSH-like tags). Prompts given to the Q&A agent include: the user question, top-k chunks with citations, a grounding instruction (answer strictly from the provided passages; otherwise say - insuficient evidence), and formatting constraints suitable for clinical or educational contexts. For bilingual sessions, the translation agent post-processes answers to the target language and preserves clinical terminology via a prompt-level glossary.
Latency and concurrency on CPU. On the 16-vCPU VM, practical allocation dedicates 6 vCPUs to the Q&A agent and 7 vCPUs to the translation agent, leaving the remainder to the embedding/query layer and the web API. With this partitioning and Q4 quantization, median end-to-end latency for short, grounded answers ( 80–120 generated tokens, k=8 context chunks) is typically: 2.5–4.5 s on an E5-class host and <2 s on a newer Xeon. For translation, sentence-level round-trips (15–25 tokens) complete in 1–3 s depending on CPU class. To sustain multi-user access, the deployment favors multi-process concurrency (one llama.cpp server per agent) with CPU afinity and NUMA pinning; for small prompts, running two parallel Q&A workers can increase throughput more than a single worker using all cores.
The CPU-only design simplifies procurement and on-premises deployment for sensitive data, while the small-model agents lower energy draw and cost. Overall, the architecture delivers grounded, bilingual assistance for rehabilitation medicine with predictable latency on commodity CPUs, matching the project’s privacy and cost constraints.
This work was designed to report quantitative performance (primarily speed) of selected LLMs running with llama.cpp on defined CPU-only environments, and to demonstrate the feasibility of operating LLM-based agents on low-end hardware. It forms one component of a broader program, Development of an AI-Based Decision-Support Expert System for the Medical Domain. By design, qualitative assessment of reasoning quality, factuality, safety, and usability lies outside the present scope and will be addressed in subsequent publications.
• Task scope. Measurements emphasize tokens/s, latency, and memory. No human evaluation, BLEU/chrF (for EN↔UK translation), or extrinsic task metrics (e.g., answer groundedness, clinical usefulness) are reported here. • Model and tooling coverage. The study focuses on a limited set of open LLMs and a single runtime (llama.cpp) with 4-bit quantization. Results may difer with other models, quantization schemes (e.g., INT8/FP4), or inference stacks. • Hardware specificity. Findings reflect the tested CPUs and a particular virtualization stack (KVM/libvirt) and containerized build (Intel oneAPI/oneMKL). Diferent microarchitectures, memory configurations, or BLAS backends can materially change throughput and power profiles. • Operating conditions. Benchmarks were taken under controlled loads with fixed decoding settings; background contention, thermal throttling can introduce variance in real deployments. • Domain limitation for RAG. Retrieval experiments are grounded in a specialized EBSCO rehabilitation-medicine corpus [36]; generalization to other medical subdomains or non-medical text is not established. • Cost and reliability. A full cost–performance analysis (cloud pricing, energy economics) and robustness testing (fault tolerance, long-running stability) were not performed.
These boundaries are intentional to isolate CPU inference characteristics. Future work will expand to qualitative evaluation of reasoning and translation quality, safety and alignment audits, broader model/runtime comparisons, and end-to-end assessment within the target decision-support workflow.
This work shows that CPU-only deployment of open-source LLMs with llama.cpp is practical when models are quantized and the runtime is tuned for the target microarchitecture. Across three representative CPU environments, smaller models (1–4B) deliver near real-time interaction, mid-size models (6–8B) ofer a favorable balance of quality and latency on legacy servers, and larger models (12–24B) are usable on high-core-count cloud CPUs with predictable delays. Measured performance spans ∼ 2–120 tokens/s(depending on model size and CPU class), while 4-bit memory footprints of ∼0.5–13.4 GB enable multi-agent designs under modest RAM limits.
For task fit, instruction-tuned models remain essential for conversational reliability; 6–8B emerges as a sweet spot for chat on older hardware, while 1–4B models are well suited to lightweight understanding and fast utilities. For English↔Ukrainian translation, Qwen2.5-7B favors lower latency on constrained budgets, whereas Gemma-3-12B-it prioritizes fidelity; Phi-4-mini-instruct and Llama-3.2-1B-Instruct are efective for short, interactive segments. Operationally, one-thread-per-physical-core scheduling, multi-process concurrency (per-agent workers) with CPU afinity/NUMA pinning, and deterministic decoding improve throughput, stability, and output consistency.
The MedLocalGPT case confirms that a CPU-only, privacy-preserving architecture can support bilingual translation and domain Q&A using two small models and a RAG pipeline over an EBSCO rehabilitation-medicine corpus. Future work will extend this line of research with qualitative assessments of reasoning and translation, cost–energy analyses, broader runtime/quantization comparisons (e.g., INT8/FP4), eficient attention for long contexts, and integration of retrieval-interleaved generation over SPARQL and ontology-driven knowledge to enhance clinical decision support.
The source code for the MedLocalGPT project, including deployment scripts and example configurations for CPU-only inference with llama.cpp, is openly available at: https://github.com/knowledge-ukraine/ medlocalgpt.
The retrieval corpus used for RAG comprises a curated EBSCO rehabilitation-medicine dataset. A packaged release of this dataset is available via Zenodo at: https://doi.org/10.5281/ZENODO.8308214. Use of these materials is subject to the licensing and terms of use specified in the respective repositories and records (and, where applicable, the original EBSCO access/license terms).
This research was funded under a contract with I. Ya. Horbachevsky Ternopil National Medical University, Ministry of Health of Ukraine, for the R&D project “Development of an AI-Based Decision-Support Expert-System Prototype” (https://prozorro.gov.ua/uk/contract/UA-2025-05-13-012069-a-c1). The study forms part of the broader national R&D initiative “Development of a Personalized Tele-Diagnostic Platform with AI for Physicians and Patients (TD + AI),” carried out under contract between the Ministry of Education and Science of Ukraine and I. Ya. Horbachevsky TNMU (State registration No. 0125U001036).
Additional support was provided by the R&D projects “To develop theoretical foundations and a functional model of a computer for processing complex information structures” (State registration No. 0124U002317; https://nrat.ukrintei.ua/searchdoc/0124U002317/) and “Develop Means of Supporting Virtualization Technologies and Their Use in Computer Engineering and Other Applications” (State registration No. 0124U001826; https://nrat.ukrintei.ua/en/searchdoc/0124U001826), funded by the National Academy of Sciences of Ukraine.
All projects were conducted at the V. M. Glushkov Institute of Cybernetics, National Academy of Sciences of Ukraine (Kyiv, Ukraine).
During the preparation of this work, the author used gpt-oss-20b (OpenAI’s medium-sized openweights model) solely for grammar and spelling checks. arXiv (2025). doi:10.48550/arXiv.2501.12948. [28] Meta AI, Meta-llama-3-70b, 2024. URL: https://huggingface.co/meta-llama/Meta-Llama-3-70B. [29] M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kaufmann, J. R. Lee, Y. T. Lee, Y. Li, W. Liu, C. C. T. Mendes, A. Nguyen, E. Price, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, X. Wang, R. Ward, Y. Wu, D. Yu, C. Zhang, Y. Zhang, Phi-4 technical report, arXiv (2024). doi:10.48550/ARXIV.2412.08905. [30] Gemma Team, Gemma 3 technical report, arXiv (2025). doi:10.48550/ARXIV.2503.19786. [31] Intel Corporation, Intel xeon processor e5-2695 v2, 2013. URL: https://www.intel.com/ content/www/us/en/products/sku/75281/intel-xeon-processor-e52695-v2-30m-cache-2-40-ghz/ specifications.html. [32] Intel Corporation, Intel xeon platinum 8480+ processor, 2023. URL: https://www.intel.com/content/ www/us/en/products/sku/231746/intel-xeon-platinum-8480-processor-105m-cache-2-00-ghz/ specifications.html. [33] Amazon Web Services, Amazon elastic compute cloud, 2025. URL: https://aws.amazon.com/ec2/. [34] Amazon Web Services, Aws ec2 c7i-flex instances, 2025. URL: https://aws.amazon.com/ec2/ instance-types/c7i/. [35] Intel Corporation, Intel core i7-13700h processor, 2023. URL: https://www.intel.com/content/ www/us/en/products/sku/232128/intel-core-i713700h-processor-24m-cache-up-to-5-00-ghz/ specifications.html. [36] K. S. Malakhov, D. Vakulenko, V. V. Kaverinskiy, Ebsco articles dataset (domain knowledge: rehabilitation medicine) + json of every article, 2023. URL: https://zenodo.org/record/8308214. doi:10.5281/ZENODO.8308214.