In this panel session I address two research questions in the area of the optimization of data integration (DI) processes (a.k.a. ETL processes), which (in my opinion) still need substantial research. The questions include: (1) how to eficiently push down executions of DI tasks to non-relational data sources and (2) how to handle user-defined functions (especially treated as black-boxes) in optimizing the performance of DI processes. The discussion to be initiated during the panel is whether sound answers to these questions can be found by the support of machine learning techniques.
For years, the widespread of complex, data-driven systems has been observed, e.g., medical systems, smart agriculture, and smart cities. These systems produce huge volumes of highly heterogeneous data (a.k.a. big data) that need to be integrated to feed various applications providing descriptive analytics or prediction models. Thus, data integration (DI) architectures are inevitable in modern information systems and they are constantly facing new challenges caused by complex, fast arriving, and ample data as well as emerging data engineering technologies.
A common goal of DI is to make heterogeneous and
typically distributed data available for an end user in a unified
format. Research and development works resulted in a few
standard DI architectures, namely: (1) federated [
DI processes are core elements of all DI architectures. DI processes are complex workflows composed of dozens to thousands of tasks. These tasks are responsible for extracting data from DSs, transforming data into a common model and data structures, cleaning data, removing missing, inconsistent, and redundant data items, integrating data, and loading them into a central repository (i.e., DW, DL, or DLH) or making them available in virtual integration architectures (i.e., federated, mediated, polystore, or data mesh). DI processes are managed by a dedicated software, called a DI engine (an ETL engine in a DW architecture).
Most of the DI engines support a set of predefined (out of
the box) tasks [
Even though, methods for developing DI processes have
been researched and developed for decades (see [
In this context, with the fast advances of machine
learning (ML) techniques, the application of such techniques to
designing and optimizing DI processes may sound attractive.
However, research works on DI focus mainly on mappings
between values [
ML techniques have already been successfully applied to
optimizing system performance, e.g., [
In this panel talk, I will focus on selected challenges related to the performance of ETL processes. My subjective point of view on the presented open issues/challenges results from a cooperation with IBM Software Lab in Kraków (Poland) on a data integration project. 2. Performance optimization of DI processes In order to reduce the execution time of a DI process, a few classes of solutions have been proposed. First, a business approach is to scale-up or scale-out a DI server. Second, DI engines existing on the market support parallel processing of DI tasks. This is also a trend in research. Third, some
DI engines support moving the execution of some DI tasks close to storage. One technique of this class is called the push-down optimization. Fourth, re-ordering of DI tasks has been well researched and resulted in a few approaches.
Scaling refers to adding computing power into a DI architecture. Two types of scaling are common, namely: (1) vertical scaling of a DI server, i.e., by increasing the number of CPUs, the size of RAM, adding specialized hardware like FPGAs and (2) horizontal scaling of a DI architecture, i.e., adding new computing nodes.
Parallel processing consists in computing tasks by
parallel OS processes or threads. This technique was well
researched as well, e.g., [
Data processing close to storage - in the context of databases, there are numerous implementations supporting moving data-intensive processing from an application layer to storage. Examples of such systems are IBM Pure Data for Analytics and Oracle Exadata. Both of them use a dedicated hardware to perform operations on data read from disks, like decompression, filtering, and projection.
Push-down - the principle of the push-down optimization is to move some DI tasks into a data source, to be executed there. Push-down is available in IBM InfoSphere Data Stage and Informatica, but only for relational DSs.
Task reordering is the most researched technique for DI
process optimization. A group of approaches draws upon
the idea of changing the order of tasks in an original DI
process, such that a reordered process is more eficient than the
original one. Finding a (sub-)optimal order of tasks is
computationally complex, and for this reason, some heuristics
have to be used [
This section outlines my subjective view on two still open research challenges in the optimization of DI processes. They include: the push-down optimization on non-relational data sources and DI processes with black-box user-defined functions (BBUDFs).
As stated above, the push-down technique in commercial systems was made available to work only with relational DSs and it is typically applicable to tasks at the beginning of a DI process, i.e., filtering or simple pre-processing. Moreover, push-down may also be applicable to tasks that migrate large volumes of data between systems. The first example of such a task is data anonymization. According to GDPR, sensitive data cannot ’leave’ a source system before being anonymized. It means that a DI engine cannot run the anonymization and this task has to be pushed-down into the source system. The second example is enforcing data access policies. Sensitive data that cannot be accessed in the source system must be filtered out directly in the system. In this case, a data access policy originally included in a DI process must be pushed-down into the DS.
With the widespread of big data storage systems, a natural
step is to extend push-down to non-relational DSs. To the
best of our knowledge, the applicability and eficiency of
this technique for non-relational (a.k.a. NoSQL) DSs has not
been studied yet (with the exception of [
DI processes use not only predefined tasks (e.g., [
The research hypothesis stated in this panel talk threefold.
First, we expect that the push-down technique applied to non-relational DSs will allow to increase performance of DI processes (i.e., reduce their execution time). Based on the developed execution cost models and implementation skeletons, it will be possible to push-down typical DI tasks into non-relational DSs. The question, however, is how pushdown could benefit from machine learning (ML) techniques in the course of: (1) deciding whether a given task should be pushed down into a non-relational DS and (2) providing an eficient implementation of the task in the DS.
Second, we expect that it will be possible to build
performance models of basic and complex BBUDFs (like those
listed in [
Third, we expect that it will be possible to build semantic models of basic BBUDFs by means of machine learning techniques - possibly by applying deep neural networks. Here, open question are: whether ML techniques could be used to build the models; what kind of techniques would be suitable; what input data would be required?
The discussion on the aforementioned challenges should
be extended towards a broader scope of DI: (1) whether ML
techniques can revolutionize the development and
deployment methods of eficient DI pipelines, (2) how to build
an end-to-end DI pipeline with the support of ML, (3) how
to assure and verify the quality of data produced by such
pipelines, (4) how to leverage the ML techniques for
building complex DI architectures with appropriately designed
software and hardware, and (5) how to mitigate bias in the
ML techniques used to build DI pipelines. Furthermore,
another question is whether ML could help solving the still
unsolved challenge of the ETL evolution, e.g., [