The text-to-SQL task aims to covert natural language questions into SQL queries. Large Language Models (LLMs) have demonstrated remarkable performance on this task, which relied on in-context learing or Supervised Fine-Tuning (SFT). However, the heterogeneity of database and the complexity of the knowledge acquisition process pose significant challenges in previous works. To address these, we propose a novel text-to-SQL framework that enhances the performance of LLMs through Knowledge Graphs (KGs). We construct the KGs based on schemas, which are structured representations of the relationships and attributes within the databases. Then, we utilize LLMs to extract descriptions and dependencies from historical queries, which are used to complete contextual knowledge in KGs. We leverage retrieval model to recall benefit nodes and edges from KGs and then employ LLMs to generate task-specific evidence. Based on the evidence and retrieved information, we define a unified KGs-based schema for LLMs to generate SQL queries. Our paper conducts experiments on public datasets BIRD and Spider, and the results indicate that our framework significantly improves the text-to-SQL performance.
types Target question: Give the names of the schools with the percent eligible for free meals in K-12 is more than 0.1 and test takers whose test score is greater than or equal to 1500? KGs-based schema: # Table: frpm (Free Meal Count (K-12) : type is REAL,. .), (Enrol ment (K-12): . .,K-12 means 1st grade - 12nd grade,. .), (NumGE1500: . .,Number of Test Takers Whose Total SAT Scores Are Greater or Equal to 1500.) (CDSCode: type is TEXT, primary key, Value examples. .) # Table: satscores (cds: . .,California Department Schools,. .), (sname: . .,school name. Value examples: [ 'Middle Col ege High', . . ), (NumGE1500: . .,Number of Test Takers Whose Total SAT Scores Are Greater or Equal to 1500,. .) . . . # relationships frpm.`CDSCode` = satscores.`cds` eligible free rate for K-12=frpm.`Free Meal Count (K12)`/frpm.`Enrol ment (K-12)` SQL query
Instructions Target question KGs-based schema Generated evidence
Few-shot learning Generated evidence: The schools with the percent eligible for free meals in K-12 more than 10% refers to `Free Meal Count (K-12)`/ `Enrol ment (K-12)` > 0.1 in the table frpm; test takers whose test score is greater than or equal to 1500 refers to `NumGE1500` > 0 in the table satscores; school names are mapped to `School Name` in the table frpm.
Use GPT4 to generate evidence, which maps natural language questions to database schemas. (SFT). However, practical applications encounter several challenges. Firstly, diferent database structures and application scenarios, leading to significant variations in data styles that complicate schema representation. Secondly, SQL queries often involve large
However, the evidence they provide is manually annotated, which limits its applicability to other datasets.
Nonetheless, evidence is extremely scarce across databases.
To address this deficiency, we utilize
generate evidence through
KGs-based schema, which is highly efective
for SQL generation. 2.
text-to-SQL framework.
As illustrated in
Figure 1, our framework comprises four stages. The first stage is KGs construction, where we design two types of nodes and three types of predefined relationships based on database components. The second stage is kno wledge completion, where we utilize
LLMs to extract im plicit kno wledge from user historical questions and SQL pairs, in order to expand the attributes and dependencies in generation. We design a retrieval model to identify and select the most relevant knowledge from the expanded KGs, and then use this knowledge to generate evidence and context for SQL query formulation. The last stage is SQL generation, where we combine the user question and the previously obtained schema knowledge as context and use LLMs to generate SQL queries using a least-to-most method.
Knowledge Graph Construction. Our approach aims to construct the KGs based on tabular structures and columns. The graph mainly consists of two key entity types, i.e.,Table and Column. The former is defined by the attributes “Table Name” and “Table Description ”. The latter is achieved by the attributes “Column Name”, “Column Description”, “Data Type”, “Example Value”, and “Value Explanation”. We design three predefined relational categories: inter-table relationship, inclusion relationship between tables and columns, and foreign key relationship. Overall, we use the distinct operation of databases for extracting metadata (i.e., the above mentioned entity types and relational categories) in order to obtain structured database information for constructing KGs. Figure 1 illustrates our approach of using value examples as attributes for columns to manage the diversity in data storage formats among diferent databases. This diversity stems from the diferent conventions for data representation.
Knowledge Completion with LLMs. We propose a knowledge completion strategy based on LLMs that helps to enrich the attributes of nodes and the relationships between nodes in the knowledge graph. Our method collects tuples {, } comprising user historical questions and corresponding SQL queries. These tuples are processed using LLMs to extract column knowledge as = {, } and functional dependencies as = {ℎ, , }. denotes the column name and represents its description in the question. ℎ and respectively denote the head and tail columns extracted from SQL, while describes the relationship between ℎ and from question. Both and serve as supplementary updates to the graph. For reducing the addition of redundant information, we assess the relevance of existing column knowledge before updating and employ LLMs to determine the necessity of update.
Knowledge Selection and Generation. As illustrated in Figure 1, we design a retrieval model
to recall columns from KGs, which is proved efective in Dense Passage Retrieval (DPR)[
∑︁ ∈[||] max · ∈[||] (1) (2) and are token embeddings indices of and . By setting a threshold of score, we disambiguate the candidates to select the final columns.
Through knowledge selection, we extract general knowledge from KGs. And then, we input the KGs-based schema, user question, and instructions into the LLMs to generate evidence, which is crucial for establishing the correlation between questions and database. We specifically design the evidence generation process which comprises three steps. First, we utilize LLMs to extract key terms from user questions. These key terms typically include important entities, attribute names, and query conditions. In practice, these terms may not always be explicit and clear, and can sometimes be ambiguous or non-standardized. Next, based on these terms, we use LLMs to enumerate normalized schema blocks such as related tables, columns, and database function expressions. LLMs map these terms to the database schema. During this mapping process, LLMs not only correspond the terms to standardized expressions but may also elucidate the dependencies in the query. For instance, they identify relevant tables and columns, determine logical relationships between these columns (such as ratios or comparisons), and may infer necessary data connections (such as joins) and filtering conditions. This detailed mapping by LLMs facilitates the precise extraction of results that meet the specified conditions, ensuring the accuracy and completeness of the query process. We summarize the results of this process into a concise sentence as evidence.
SQL Generation. We employ LLMs with in-context learning to address SQL generation[
As indicated in Table 1, the row for GPT-4 indicates using the origin database schema directly, rather than leveraging schema KGs. In comparison, by organizing schemas components into KGs and utilizing retrieval models to recall tables and columns, we observe a notable enhancement of 3.23% , 0.97% and 2.21% on BIRD and Spider. Secondly, we employ generated evidences for SQL generation. It contributes to additional improvement of 3.13% , 2.62% and 0.90%.