Large Language Models (LLMs) can enhance analytics systems with powerful data summarization, cleaning, and semantic transformation capabilities. However, deploying LLMs at scale - processing millions to billions of rows - remains prohibitively expensive in computation and memory. We present IOLM-DB, a novel system that makes LLM-enhanced database queries practical through query-specific model optimization. Instead of using general-purpose LLMs, IOLM-DB generates lightweight, specialized models tailored to each query's specific needs using representative data samples. IOLM-DB reduces model footprints by up to 76% and increases throughput by up to 3.31× while maintaining accuracy through aggressive compression techniques, including quantization, sparsification, and structural pruning. We further show how our approach enables higher parallelism on existing hardware and seamlessly supports caching and batching strategies to reduce overheads. Our prototype demonstrates that leveraging LLM queries inside analytics systems is feasible at scale, opening new possibilities for future OLAP applications.
LLMs have demonstrated exceptional capabilities in
natural language understanding and generation [
review_summary FROM product_reviews;
prompt('summarize in 5 words: ' || review) AS This approach opens up new possibilities for generating, summarizing, cleaning, and transforming structured and unstructured data directly within the database.
However, applying LLM prompts to each row of data
presents significant challenges. A typical query, such as
a transformation, classification, or schema extraction,
requires a separate LLM invocation involving tokenization,
context encoding, and autoregressive decoding. This
perrow inference results in high computational overhead, as
even simple queries can trigger millions or billions of LLM
calls, leading to excessive latency and resource
consumption, especially for large tables. While running local
models [
We propose a practical approach to mitigate these
challenges: instance-optimized LLMs for databases (IOLM-DB).
By tailoring models to the specific workloads and data
distributions of a given query and database instance, we reduce
the cost of LLM inference, making it more practical to use at
scale. We find that the OLAP setup is an ideal environment
for creating optimized models because it operates in a
controlled setting where the workload and data are predictable.
IOLM-DB combines multiple model compression techniques,
including quantization (reducing numerical precision) [
We have developed an initial prototype of IOLM-DB that
targets Python’s pandas library [
• We propose an end-to-end system for prototyping LLM prompting in OLAP scenarios and a series of workloads to assess its performance. • We introduce the first method for generating instance-optimized LLMs in database environments. • We evaluate the eficiency of our method on the proposed workloads, showing significant performance improvements. These results indicate a promising ifrst step toward demonstrating the feasibility of
in this paper are publicly available at https://github.com/ © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License mpi-dsg/IOLM-DB.
CEUR
ceur-ws.org
This section briefly overviews key concepts and related work that motivate our approach.
LLM Prompting in Databases Augmenting databases
with the ability to prompt LLMs can facilitate data
wrangling and analysis by enabling natural language queries and
transformations [
powerful new capabilities for database systems, especially when working with unstructured or semi-structured data. These capabilities include summarization, sentiment analysis, data extraction, error correction, and semantic transformations. LLMs can also enable more flexible abnd intuitive operations, such as fuzzy matching and semantic joins beyond exact string matching.
Other Approaches Existing approaches to adding these
capabilities have notable limitations. Code generation
techniques [
anything short of directly invoking the LLM for such tasks
is inherently limiting, as it would restrict the system’s
expressiveness. Therefore, eficiency is paramount to making
LLMs practical at scale, especially for large-scale
analytics. Many anticipated use cases require invoking the LLM
at a fine-grained level, such as once per row or more
frequently [
OLAP environments are particularly well-suited for such optimizations, as their controlled setting often allows query patterns and data characteristics to be inferred or extracted in advance, enabling model optimization tailored to these patterns. Similarly, interactive queries frequently exhibit recurring or predictable patterns that can be leveraged to create instance-optimized models, improving eficiency and accuracy as these patterns evolve over time. Although our current method may introduce too much overhead for some use cases, we expect it to be highly efective for long-running queries, where significant performance gains outweigh the upfront cost of optimization during execution.
LLM Compression Techniques Recent advances in LLM
optimization provide a foundation for our approach. We
leverage three key techniques from the literature:
quantization, which reduces numerical precision to decrease memory
requirements [
IOLM-DB is an OLAP system that integrates LLM invocation directly into its execution pipeline while ensuring that this integration is eficient, scalable, and cost-efective. 3.1. Overview We assume a setup where the user executes queries over one or more database tables (in our prototype, pandas DataFrames). These queries can include calls to an LLM, for example, to summarize free-form text columns, transform semi-structured fields, or provide semantic annotations. Rather than invoking a general-purpose LLM repeatedly and incurring high computational and memory overhead at runtime, IOLM-DB creates a specialized, instance-optimized LLM explicitly tailored for the query and dataset. For example, a query performing text summarization on product reviews requires a diferent optimization than a data correction query. To achieve this specialization, the model is compressed and pruned based on the query type and data characteristics. This approach is necessary because a onesize-fits-all model is ineficient; diferent queries demand varying levels of understanding, precision, and computational cost.
IOLM-DB works in tandem with the underlying analytics engine. While the engine handles standard relational operations eficiently, we introduce custom operators that intercept LLM prompts in the query. These operators trigger a workflow that generates a specialized LLM for that particular query and data distribution. This process leverages a suite of techniques – quantification, sparsification, and pruning – to minimize both the memory footprint of the LLM and the inference costs, ensuring that at runtime the optimized LLM can be invoked with minimal latency and resource usage.
We drastically reduce per-row inference overhead by producing a specialized LLM for each query instance. This approach enables running LLM-based transformations and analyses at scale, supporting massive parallelism and eficient resource utilization. Ultimately, the goal is to handle large workloads and large numbers of concurrent queries while providing near-interactive response times. 3.2. Techniques to Generate
Instance-Optimized Model Our system combines multiple compression techniques to reduce memory consumption and compute costs at the LLM level. By carefully integrating these methods, we maintain model accuracy while achieving significant resource savings. We combine the following techniques in IOLM-DB. Quantization By lowering numerical precision (e.g., using 8-bit weights and sometimes 8-bit activations), we reduce both the GPU memory footprint and computational overhead. Quantization retains the model’s core capabilities but significantly increases inference throughput. Sparsification Imposing structured or unstructured sparsity patterns in model weights reduces the number of active parameters, leading to fewer operations during inference and, in some cases, allows for hardware-level support for sparse computations.
Structural Pruning Removing entire components such
as attention heads or entire layers that contribute little to
the task at hand reduces model depth and complexity. Full
structural pruning, aided by tools such as LLM-Pruner [
Batching We batch multiple requests to the LLM together to amortize invocation overhead. By grouping multiple rows or operations, we minimize switching costs and achieve higher throughput.
Cascading (Future Work) We plan to explore cascading strategies in the future, where an initial coarse-grained LLM invocation feeds into more specialized or higher-precision models only where needed. This approach could further refine trade-ofs between speed, cost, and accuracy.
We evaluate our approach on three representative
workloads, each designed to highlight a diferent aspect of
integrating LLM invocations into OLAP queries. All these
workloads operate on real-world datasets (e.g., Amazon
Review [
We conduct a preliminary evaluation of IOLM-DB through experiments on the three workloads and associated datasets described in Section 4. Our key objective is to verify the viability of our approach for scaling LLM invocations per row in a realistic setting.
Our evaluation centers on the following metrics: • Throughput: The number of rows processed per second by the system, which reflects its overall eficiency and scalability. • Model Size: The size of the model, which serves as a proxy for GPU memory usage and its capacity to parallelize execution efectively. Smaller models typically reduce memory pressure and improve resource utilization. • Accuracy: The proportion of rows where the system produces correct results. For this evaluation, we assume the baseline model achieves perfect accuracy (accuracy = 1), and we compare the optimized models by normalizing against this standard.
Models We conduct our evaluation using Meta’s Llama
3.1 Instruct model with 8 billion parameters
instructiontuned language model (Llama-3.1-Instruct-8B) [
ing Llama-3.1-Instruct-8B, which serves as our baseline for all metrics. Building upon this foundation, we evaluate two optimized variants: Workload Summarization Data Correction Fuzzy Join
Model Baseline IOLM-DB-Perf IOLM-DB-Acc Baseline IOLM-DB-Perf IOLM-DB-Acc Baseline IOLM-DB-Perf IOLM-DB-Acc
These two variants highlight an interesting trade-of between performance and accuracy enabled by IOLM-DB which we aim to explore in future work, allowing the user to select whether to spend more time for better results or to produce slightly less accurate results faster.
ments on a machine equipped with an NVIDIA Hopper H100 80 GB SXM GPU, two AMD CPUs (EPYC 9654), and 2 TB of DDR5. The operating system is 64-bit Debian Linux (kernel version 5.15). We use CUDA 11.8, vLLM 0.6.3, and HuggingFace 4.46.1 transformers. 5.1. Performance, Model Size, and Accuracy Table 1 compares the baseline with our IOLM-DB instanceoptimized models. We show the overall model size as measured on the GPU, the resulting accuracy in our dataset (normalized to the baseline), and the throughput achieved in each case (in rows processed per second).
These results underscore the benefits of instanceoptimization for LLMs across all workloads considered. In all cases except summarization, both IOLM-DB-Perf and IOLM-DB-Acc achieve an accuracy score of 1, demonstrating that our approach maintains baseline-level accuracy. This outcome highlights that our optimizations preserve model quality while reducing resource requirements.
Our method achieves significant reductions in model size, with compression factors of up to 76%, through quantization, sparsification, and pruning. These reductions lower memory requirements and improve computational eficiency. The throughput improvements are substantial: 2.78× for data correction and 2.52× for fuzzy join workloads. For summarization, despite a slight accuracy decrease in IOLM-DBPerf, we observe a 3.31× throughput improvement. These performance gains stem from the compressed models and reduced model size, enabling higher parallelism.
The results demonstrate that instance-optimized LLMs can efectively balance accuracy and eficiency, making them suitable for performance-critical applications. 5.2. Discussion Although these results are preliminary, they highlight the potential of instance-optimization to enable eficient LLM querying in analytics workloads. Our evaluation also revealed several areas for improvement that could further enhance the performance and scalability of our solution.
One key bottleneck identified in our initial prototype is the space consumed by the vLLM cache. Optimizing the cache design could further reduce memory usage and enable increased parallelism, particularly for workloads with high concurrency demands. Also, the interface between the pandas library and our instance-optimized operators presents another area for improvement. Reengineering this interface to reduce overhead and improve integration could significantly boost overall performance. We expect to gain one or two orders of magnitude performance improvements from these optimizations.
An interesting observation is that the resulting quality, particularly for IOLM-DB-Acc, often improves compared to the baseline. However, the normalization of the accuracy score prevents this improvement from being reflected in Table 1. These findings suggest that our instance-optimized approach not only preserves accuracy but can, in some cases, enhance it. Further investigation is needed to understand the underlying factors driving these improvements and to explore how they can be consistently leveraged.
Finally, we plan to explore additional techniques in the LLM compression space, such as advanced quantization methods and knowledge distillation. The time to generate an instance-optimized LLM is in the order of single-digit minutes, which we believe is acceptable for most table sizes. However, reducing that further could unlock additional use cases outside the OLAP space.
This paper presented IOLM-DB, a proof-of-concept system for integrating LLM operations directly into OLAP query pipelines. The core innovation lies in instance-optimizing LLMs based on the specific query and dataset at hand. IOLMDB can generate specialized models that drastically reduce inference costs without sacrificing accuracy by leveraging quantization, sparsification, and pruning techniques. Our preliminary results suggest that this approach makes rowby-row invocation of LLMs at scale both practical and eficient, bridging the gap between powerful linguistic transformations and large-scale analytics.
This line of work opens up promising research directions, encouraging the community to refine these techniques further, explore new instance-level optimizations, and ultimately bring high-performance, query-aware LLM capabilities into mainstream analytical workflows.