Tabular data are made available from diferent sources and this data can be used for exploratory analysis or be integrated with other data sources. However, searching for such data and integrating tables from diferent sources with little information about the schema has been a challenge. Our vision is to develop tools and algorithms to make this process easier. This paper CEUR primary sources of information available to the special- in the formatting of property addresses between the open Joint Workshops at 49th International Conference on Very Large Data ∗Corresponding author.
presents some of our recent in this direction.
this by taking an exploratory approach to search with a
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License about the underlying table schemes. BareTQL achieves
A business intelligence specialist at a bank is in charge of collecting data on local businesses to assess their market value, which is used for loan approvals. There are two ist: property assessments and company profile databases. Property assessments (referred to as A) are made available, as part of open government data, by the municipal government, which conducts these assessments annually for tax purposes. In addition to property assessments, the specialist can access two company profile databases that provide diferent information about the companies. One database contains details about the properties owned by each company (referred to as P), while the other focuses on stock valuations (V). The stock valuations include the company symbol, number of shares outstanding, and the share price among other information. The specialist’s task is to gather the relevant records from these sources and present them in a clean and concise format for the loan approval oficers. By consolidating the property assessment data, company property information, and stock valuations, the specialist creates a comprehensive overview of each business, enabling more informed decision-making during the loan approval process.
The task involves a number of challenges. Firstly, the open government data consists of an extensive collection of tables, numbering in the tens of thousands. To navigate through this vast amount of data, the specialist requires appropriate search tools to identify the relevant tables and records. The data does not provide the names
We have developed Bare Table Query Language (in short
BareTQL) an interactive framework for querying large
collections of tables [
While string-based transformations are humanreadable and interpretable, there have two general limitations: (1) these transformations are syntactic and they usually miss semantic mappings (e.g. synonyms) between sources, (2) learning transformations is a resourceintensive process. Our next work attempts to address these challenges. 4.2. Transformations in a latent space focus on what is known already and building on top.
BareTQL supports four classes of operations as shown in Figure 1. A keyword search may be used when there is little information about the tables being queried and their structures, whereas a table search may be invoked when the user has a table and wants to find more related tables. Through a set of table expansion operators, one can expand a table by adding more rows (xr), adding more columns (xc) and filling missing values (fill). These operations may invoke table search to find the relevant tables, but the relevance ranking can vary for each table expansion operator. In our given scenario, the specialist may search open government data for tables using keywords from property addresses, and filter the results using follow-up operations. The specialist may also provide some examples of addresses and their valuations to find more a more comprehensive list. BareTQL supports joining tables from diferent sources through the use of transformations. We next discuss our work on automatically finding such transformations.
Consider our scenario again where companies in P are
identified by their names, while the same companies
in V are identified by their stock symbols.
Transforming company names to their stock symbols using string
operations is less trivial or meaningful. We have been
recently studying the power of transformer models in
mapping tabular data. In particular, our Deep Tabular
Transformer (DTT) framework [
To train such a model though, one usually needs a large
4. Transforming Tables for Join set of labeled data. Our study shows that string
transformations can be learned from synthetic data which can be
When tables are gathered from multiple sources, they generated in large volumes. DTT employs a decomposer
rarely conform to the same formatting. We study the and an aggregator module to deal with disparity in table
problem of transforming tables for joinability under two sizes and to handle large tables.
settings: (1) each transformation is expressed as a se- Although DTT is not fine-tuned on real-world data,
quence of string operations, and (2) transformations are our experiments indicate that it delivers outstanding
perlearned in a latent space. Unlike previous works relying formance on both real-world and synthetic datasets,
comon similarity functions for matching (e.g., [
Consider joining tables in sources A and P in our scenario based on property addresses. An arbitrary address