With the emergence of large language models, the long studied field of the Text-to-SQL problem was elevated into new spheres. In this paper, we test how our LLM fine-tuning approach performs on two relational databases (small vs. big) and compare it to a default setting. The results are convincing: using in-context learning boosts the performance from a merely 35% (default) to over 85%. Furthermore, we present a detailed architectural framework for such a system, emphasizing its exclusive reliance on open-source components.
The underlying idea for this paper is the following (real-world) case: Imagine a business analyst who needs certain information to answer business questions such as ”Which customers should we contact in our next marketing campaign?” or ”Which customers qualify for a discount?” based on data. Furthermore, the business analyst only has very rudimentary knowledge in SQL programming. So, is there a way to create a system that can help this person to gain insights from the data and answer their business questions? With the novel introduction of large language models (LLMs) and a PostgreSQL database, we create a system that builds a bridge between the LLM and the database — with the usage of in-context learning. Moreover, we test the system based on diferent database sizes and the application of in-context learning vs. default LLM prompting to test the following hypotheses: • H1: In-context learning should perform better than the default. • H2: In-context learning should possess a higher execution time due to having more complex inputs.
• H3: The more complex a database is, the lower the accuracy should be. research.
CEUR Workshop ISSN1613-0073
Within this section, we provide an overview of the central terminology used in this paper: LLMs, ifne-tuning, and the Text-to-SQL problem.
With the introduction of the first commercially usable large language models in 2022, the whole world around artificial intelligence changed drastically — today the buzzword ”AI” (short for artificial intelligence) is everywhere. With the release of GPT-3.5 by OpenAI in late 2022 the trafic and usage of their chatbot with the remarkable name ChatGPT exploded overnight. Since then, many new players have entered the market with their own LLMs for which some need to be paid for and some are open-sourced.
Table 1 provides a broad overview of the current LLM market with respective models and their
capability scoring called ELO or arena score. This number is derived from a multitude of diferent tests a
LLM is encountered with, e.g., coding-, math-, creative writing-, reasoning-tasks. [
To improve the accuracy and general performance of LLMs many methods have been established
throughout the last years. The most prevalent is retrieval augmented generation (RAG) [
These approaches have all one thing in common: enriching LLM systems with external knowledge sources to give the model more context (”to make it see”) — see Figure 1.
First attempts to solve the Text-to-SQL (or NL2SQL) problem were introduced in 2015 and were developed
ever since. [
In the next section, we dive deeper into important publications that highlight these three terms LLM, in-context learning, and Text-to-SQL from diferent angles.
LLMs [
As mentioned, improving accuracy and therefore also reliability of LLMs is currently one of the
biggest topics in the AI system research area. In-context learning was one of the first approaches after
LLM were publicly available in 2022. Pourreza and Rafiei [
Retrieval augmented generation systems also can improve LLM performance. Vichev and Marchev
[
Currently, there is significant hype and enthusiasm surrounding AI agents which is a more direct
problem-oriented approach. Cooperative SQL Generation framework based on Multi-functional Agents
(CSMA) from [
Other mentionable, related work comes from Zaharia et al. [
To validate our initial hypotheses from Section 1, we present an experimental set-up that will be explained in more depth within the following subsections.
As described in Sections 2.3 and 3, Text-to-SQL is a wide field and also the focal point here in this experiment. Figure 2 gives a high level overview of the general approach (referred to as in-context learning): We combine the user’s question (input) which is of course in natural language and retrieve the current database schema. Consequently, we use both inputs (db_metadata and user_input) and create a prompt for the LLM. Then, based on this information the LLM creates a SQL query and runs it automatically on the database. The output is generated. To compare the performance of the in-context learning approach, we hold it against our so called default LLM prompting technique (”default”). For this default technique, we only give the LLM the information that it is a customer database with the respective tables and of course the question. For the experiment, we use two sizes of databases db_small (Figure 3) and db_big (Figure 4) with dummy data based on a classical, fictional customer database.
The conceptual data models are depicted in the entity relationship diagrams in the appendix. Furthermore, we created 50 real-world questions (Table 3) for the experiment to test accuracy and execution time to compare results based on database size and the usage of in-context learning vs. default. For the questions, we were vigilant about including simple and also more complex questions to test how the system reacts to e.g., simple select statements, over more dificult joins, to complex nested queries. The default tests are done with a simple prompt without the database schema input.
The technical setup basically consists of two systems interacting with each other through an API: the relational database management system (RDBMS) and the intelligent AI system.
• RDBMS • AI system
We selected PostgreSQL as our relational database system to run this experiment as it is one of the most used database systems with a big and active community. Furthermore, it is open source and adheres to ACID properties.
The AI system is a LLM from Meta AI called ’meta-llama/Llama-3.1-8B-instruct’. This experiment is conducted on a local machine with a total of 16GB of RAM (8GB GPU + 8GB DDR5 RAM). To calculate which size a certain model must not exceed to be deployed on a machine, the following formula helps to give an estimate: [13] [14] P Q 8 _ = _ = ∗ 8
∗ (1 + ) 8 26∗4 ∗ 1, 2 = 15, 6
bufer capacity (20% is a common estimate)
So, the upper theoretical boundary to run the system locally, is a 26B model (assuming 4-bit quantization). Ceteris paribus, the required RAM on the local machine is 15,6 GB (<16GB). It is important to mention that models with > 8GB of RAM usage will be slower as it will access the DDR5 RAM after the GPU’s RAM is maxed out — for trial purposes, a 24B model ran extremely slow on this machine. However, taking the strong ELO-Score of 1213 (see Section 2.1) into account, the 3.1:8B Meta model is a reasonable selection for this experiment while it only needs 4,8 GB of RAM.
Overall, the results from our experiments (see Table 2) support our initial hypotheses:
The first hypothesis H1 ”in-context learning should perform better than the default.” holds for both db_small and db_big with accuracy values for the default vs. in-context learning of 33.33% vs. 90.91% (small) and 36.00% vs. 86.00% (big), respectively. Taking the average execution time into account, the second hypothesis H2 ”in-context learning should possess a higher execution time due to having more complex inputs.” is also fulfilled: in-context learning possesses an average execution time of 3.1987 seconds (small) and 3.2652 seconds (big) versus 2.9052 seconds (small) and 2.7780 seconds (big). This comes as no surprise as the LLM must process the metadata from the database which logically takes longer than not applying this step. We were also able to observe some diferences in database sizes. Bigger databases mean longer execution times and lower accuracy which is partially in line with H3 ”The more complex a database is, the lower the accuracy should be.” For the default tests this hypothesis does not uphold as the results are reversed: the accuracy is higher for db_big. The accuracy outlier can be explained due to the randomized nature of the queries meaning that e.g., the right column name (total_amount vs. amount) is returned randomly from the LLM as it does not know the column names. For the in-context learning tests the hypothesis upholds with 86,00% (big) being lower than 90,91% (small). Another noticeable finding is also a relative simple one: Spelling. The LLM does not know from the provided metadata how certain things are spelled meaning e.g., question 27 ”Retrieve a list of customers who have opted out of the newsletter.” — from first glance a relative simple question (query) — failed only because the condition in the query for status = ’active’ was misspelled as ’Active’. The same observation also occurred for the definition of the payment_method (credit card vs. Credit Card). In these two specific cases the ENUM() data type (instead of TEXT() or VARCHAR()) would have solved the problem, but if the possible values are not limited, the correctness of the results of these types of queries are random based on how the LLM decides to spell the word. This simple error represents the majority of the errors in our experiment — the LLM was not able to handle these type of questions. Another observation comes from question 25 ”How many percent of customers come from Europe?”: This one returned always an error independent of database size and method due to the fact that there is no information in the dataset which country belongs to Europe. An easy solution would have been to add a column ”continent” in the customers table. To summarize the experiment results, it is important to mention that the in-context learning prompting technique worked surprisingly well with accuracy values around 90% even with some shortcomings when it comes to spelling or metrics definitions. The mentioned solutions to these shortcomings would have increased the accuracy even further.
The technical implementation of this system comes along with a few limitations that are listed below:
As mentioned earlier in Subsection 4.2, we run this experiment on a local machine with a total RAM of 16 GB (8GB GPU + 8GB DDR5) that limits us to the usage of 26B-models (assuming 4-bit quantization). Alternatively, such experiments can also be run in the cloud where much more performant GPUs or even GPU clusters are available, but of course more costly — currently a memory-optimized virtual machine with ca. 500 GB of RAM (to run the most sophisticated open source LLMs) costs roughly 10$ per hour at the major cloud providers.
The computing resources directly impact the models that can be selected. As a rule of thumb, the more parameters a model has, the better it performs (see Table 1). But in our case it might not have changed the outcome as much as the errors arose from the structure of the data and not the created SQL code.
For the experiment, we used 50 questions to test the system. The questions attempt to be as close to the real-world as possible. One could argue why not more questions were used, but further questions would have been a derivation of the already existing ones resulting in similar SQL statements only with some very small changes.
To come back to our initial, real-world notion about the business analyst from the introduction: ”So, is there a way to create a system that can help this person to gain insights from the data and answer their business questions?”: Short answer — yes, but with some limitations. We showed that especially the in-context learning performs much better than default prompting. In addition, the more complex a database is, the lower the accuracy will be. Apart from this, we also compare execution times of the queries and find that in-context learning possesses higher execution times due to having to process more complex inputs. The most mistakes made by our system result from the fact that it simply cannot query what it does not know, e.g., spelling diferences in the data values (”Active vs. active”) or definitions (”Europe”). Future research should focus on solving these gaps. Also, the application of such systems to bigger and more complex data ecosystems like data warehouses or even data lakes might be interesting to address in the future. In addition to this, it might be interesting to take a deeper look into the security aspects: as this system runs automatically queries on a database, there is room for errors, e.g., unwanted deletion or unauthorized change of data.
Gonzalez, M. Zaharia, Optimizing LLM queries in relational workloads., CoRR abs/2403.05821 (2024). [12] J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, X. Zhou, M. Chenhao, G. Li, K. Chang, F. Huang, R. Cheng, Y. Li, Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-sqls, Advances in Neural Information Processing Systems 36 (2023) 42330–42357. [13] Q. Anthony, S. Biderman, H. Schoelkopf, Transformer math 101, https://blog.eleuther.ai/transformer-math/, 2023. [14] C. Chen, Transformer inference arithmetic, https://kipp.ly/blog/transformer-inference-arithmetic, 2022.
The authors hereby declare that no GenAI was used to generate text etc. following the guidelines and policy by CEUR-WS Policy on AI-Assisting Tools (https://ceur-ws.org/GenAI/Policy.html).
49 50
Question Find the average time taken to resolve support tickets.
Retrieve a breakdown of revenue by country.
Find the employee who has resolved the most support tickets.
Find the average delivery time for all shipped orders.
Find the percentage of support tickets that were resolved within 24 hours.
How many support tickets are currently open? What is the average order amount placed by customers who subscribed to the newsletter? What is the average order value for orders placed on weekends vs weekdays? Which countries have the highest proportion of customers subscribing to the newsletter? Which top 5 countries have the most customers subscribing to the newsletter? How many orders were placed in each week of the year? What is the average sales amount for each day of the week? How many support tickets are currently ’In Progress’ and were created in calendar week 1? How many products are supplied by suppliers with ’gmail.com’ in their contact email? How many orders used a payment type that is NOT ’CREDIT’? List the first and last names of employees with the word ’Manager’ in their position? Which orders from which customers took longer than 7 days to deliver? List the first and last names, and E-Mail address of customers whose email addresses end with ’.net’? List the customers’ first and last name, and order dates for orders with a total amount between $100 and $200? Provide a table with the employee’s first, last name, and the customers they are associated with. Sort them by employee.