Many real-world applications in medicine, finance or other domains need to combine tabular and textual data. In this paper, we present a new approach called hybrid database operations which is a new class of learned database operations that allows users to seamlessly execute SQL queries over text and tabular data. As a main contribution to enable hybrid database operations, we show how state-of-the-art pre-trained language models such as BERT can be used to implement hybrid database operations such as joins or unions. In our initial evaluation, we report first promising results on real-world data sets which indicate that highly accurate hybrid operations can be realized with minimal training overhead.
Relational databases are the predominant approach used today in business and science for
managing tabular data. One of the features contributing to their success is the query language
SQL which allows structured data to be queried in a simple manner. However, today many
applications need to deal with data sources beyond tabular data such as textual sources. While
several extensions for textual data such as full-text search or pattern matching [
To illustrate this by an example, think of a hospital database, which stores structured data of the patients such as age or gender together with medical reports that contain information about diagnoses and treatments. If all this data would be stored in structured tables, a data analyst could simply author a SQL query that finds out how many patients are currently in the hospital with the same diagnosis. Unfortunately, this is not possible if the information about the diagnoses is only available in textual reports about the patients. In this case, the data analyst today has to set up a complex data extraction pipeline to retrieve all diagnoses. That quickly becomes tedious, especially when the data that needs to be extracted changes over time.
In this paper, we thus present a new approach called hybrid database operations which is a
new class of learned database operations that allows users to seamlessly execute SQL queries
over text and tabular data. The basic idea of hybrid database operations is that we build on
the recent breakthroughs of large pre-trained language models in NLP [
Hence, as a main contribution in this paper, we study how large pre-trained language models
can be used to realize hybrid database operations. To be more precise, we show how hybrid
database operations such as a hybrid JOIN or hybrid UNION can be realized as downstream
tasks based on recent pre-trained language models that can jointly represent text and tabular
data, like TaBERT [
To summarize, at the core, we present two major contributions to enable hybrid database
operations: (1) We show how hybrid database operations can be realized as downstream tasks on
top of TaBERT [
In this paper, we investigate hybrid database operations on a table and a collection of documents . For hybrid JOIN s, each tuple ∈ is linked to documents ⊂ with a foreign-key relationship. As shown in Figure 1, we assume there is a latent tabular representation of the document collection. Performing a hybrid JOIN means joining with . A hybrid UNION on the other hand adds the rows of to .
For this short paper, we want to mention that we introduce a couple of restrictions we aim to relax in the future: For the hybrid JOIN , we assume there is only one document linked to each tuple (i.e. = { }), the latent tabular representation is a single table only, and for each tuple , only a single row needs to be extracted from the linked document . For UNION s, we assume ... ... ... [CLS] Carol was [CLS] The diagnosis of Alice ... [SEP] name | text | Alice [SEP] diagnosis | text | [MASK] [SEP] date of birth | real | 1995 [SEP] ... [SEP]
...
Transformer (TaBERT) ... ... ... ... ... ... ... ... diagnosed with atopic eczema ... Carol [MASK] 1962 ... ...
O
O
O classifier
O
O
B classifier
I
SELECT p.name, p.date_of_birth, r.diagnosis
FROM patients p HYBRID JOIN reports r only a single row per text is added to the table.
As the basis of our model we choose TaBERT [
Figure 2 shows an overview of the model as used in a hybrid JOIN . To perform a hybrid JOIN s with TaBERT, we pair each tuple with its join partner . Additionally, we add the query attributes from the user-provided SQL query to the tuple and add a mask token as the value. The task of the model is to recover the masked values by extracting them from the text and thereby computing the join. To perform UNION s with TaBERT, we pair the text with a tuple, where all values are masked. To realize these hybrid database operations as a downstream task, we formulate them as a masked-attribute reconstruction problem. As explained before, the information that needs to be extracted from the textual source is represented as a masked attribute in the model input. To recover the masked attribute from the text, we extend TaBERT with a downstream model that can compute the answers spans in the text using so called in-out-between (IOB) tags. These answer spans are sequences in the text that are a possible value for the masked cells. This approach allows multiple answers per query and avoids so called hallucination (i.e., the model can not simply generate answers that are not existing in the text which is a typical problem of sequence-to-sequence language models).
To be useful in practice, the model should work on new domains even with little to no training
data. Intuitively, the model should learn the essential skills to perform database operations
during pre-training. For both presented hybrid database operations, it is important to use signal
from table context to extract queried information from text. As our main pre-training objective,
we thus pair table rows and texts, mask random cell values and ask the model to reconstruct
them from the text using IOB tags. A secondary objective aligns column embeddings with text
embeddings. Due to the space constraints, we omit the details here. For these pre-training
objectives, we constructed a new dataset using T-REx [
(a) nobel JOIN (b) nobel UNION (c) countries JOIN (d) countries UNION Figure 3: F1 scores on the JOIN and UNION workloads of the nobel data, and for the countries workload. We vary the number of training samples for fine-tuning on the unseen domain.
In our initial experiments, we are interested how our model performs in unseen domains. For
the evaluation, we use also data extracted from T-REx [
In our initial results, we fix the model architecture using the new model extensions and
compare our pre-training scheme (green) to using the pre-trained weights of BERT [
We show that large language models can be used to perform hybrid database operations having only limited training data. In future work, we aim to support more complex database operations and have a more thorough evaluation on non-Wikipedia texts.
This research was funded by the Hochtief project AICO (AI in Construction). Moreover, we also want to thank hessian.AI at TU Darmstadt as well as DFKI Darmstadt.