Most of all data science projects involve a time-costly data preparation process aimed at enriching the working dataset with additional information to improve the sturdiness of resulting trained models. How to ease the design of the enrichment process for data scientists is defying, as well as supporting the enrichment process at large scale. This document introduces and describes a research proposal for addressing such problem, which focuses on harnessing the semantics as the key factor, by providing users with semantics-aided tools to design transformations, along with a platform to execute pipelines at business scale.
From storages From sensors 0
Data Acquisition DB dumps, JSON files, log files
Data Engineer
Data Analyst
Company
Manager DATA CLEANING AND ENRICHMENT
Data Data Extraction Cleaning
Data Enrichment
ACTUAL DATA ANALYSIS
Data Data
Analytics Visualization open and accessible, but there are few attempts to use it to enrich tabular data. The semantic data enrichment harnesses the semantics to support the user in the enrichment task, relying on two main sub-tasks: the reconciliation, where values in the main source are linked to entities in external KBs, and the extension, where the identi er of linked entities are used to fetch data from external sources (which adopt the same space of identi ers) and extend the information in the main source.
As a matter of fact, a typical data science pipeline is mostly focused on the
data preparation stage (Fig. 1), which cleans and enriches the data taking up to
80% of the time required by a project; only the remaining 20% is spent on data
analysis [
Current approaches do not provide comprehensive solutions to the problem
of data enrichment. Some solutions provide users with data-scientist-friendly
tools to data preparation, supporting the data extension at a limited extent
(i.e., requiring code scripts to link external sources), but without accounting the
development/deployment of production-ready pipelines, hence they o er limited
scalability. Otherwise, tools that support pipelines execution have been often
designed for users with programming skills, but who are usually inexperienced
in the particular domain to which the data pertain [
This research proposal aims to deliver a scalable cloud-based solution to provide data scientists with the tools to (i) design transformation pipelines on tabular datasets that include data enrichment, harnessing the semantics to bring knowledge bases information (generic or speci c) to non-expert people, and (ii) record and manage these pipelines in a repeatable form over large amounts of data, taking full advantage of the potential for scalability and distributed computing o ered by the cloud. All people involved in data science projects could bene t from this solution, because at the same time it (i) eases the enrichment task, and (ii) make it repeatable in a scalable environment, reducing the overall data processing time. Besides, the gap between data engineers and analysts will be bridged: a single gure, the data scientist, will be able to design the transformation process and deploy it by herself, avoiding the continuous handover between engineers and analysts. At last, the proposed solution should ease the access to public knowledge bases also for non-expert people; in fact, so far knowledge bases can be mostly explored by people with a background in Semantic Web. 3
Providing support for the design and execution of data transformation based on scale enrichment requires addressing di erent topics that have been investigated in the literature, leading to the provision of some tools dedicated to the transformation, reconciliation, and extension of tables.
Table transformation. We brie y discuss the table transformation
literature because, even if it is not strictly related to this work, in many cases the
table transformation is preliminary to the table interpretation tasks (almost all
the table interpretation proposed methods perform a pre-processing phase to
automatically clean the values in table). The table transformation deals with
preliminary pro ling and transformation of the tabular data and aims to
identify and address possible data anomalies, eventually resulting in data that is
shaped in a way that is easier to work with for further tasks (e.g., data
reconciliation). A number of approaches and tools have been developed to deal with
data anomalies, ranging from spreadsheet software (e.g., Microsoft Excel1), to
programming languages and libraries for statistical data analysis (e.g., Agate2
Python library for data analysis), to complex systems designed to be used for
interactive data cleaning and transformation in ETL process (e.g., Pentaho Data
1 https://products.office.com/en/excel
2 https://agate.readthedocs.org/en/1.3.1
Integration3 and Trifacta Wrangler4). These tools di er in the target audience
and the capabilities they o er (e.g., coverage of data anomalies, mechanisms to
address the anomalies). Recent research in this area focuses on intelligent and
semi-automated mechanisms to detect and address data anomalies to simplify
and automatize the data preparation process. A detailed discussion on the state
of the art of this eld is discussed in [
Table interpretation. Most of research work has traditionally focused on
schema-level alignment so [5, 7{10, 14, 15, 18]. Most of them create matches only
for columns containing mentions of a real-world entity, while [
To summarize, the application spotlighted in this work di ers from the existing tools because (i) it aims at supporting all the above functionalities (table manipulation, reconciliation, and extension), bridging the gap left from existing tools that tend to focus on one or a subset of them; (ii) it foresees an automatic deployment of the pipelines created at design time, providing the user with an executable model that allows her to re-apply the same pipeline several times (i.e., applying the same pipeline to di erent datasets, which share the same schema but have di erent row values), while all the above proposals require manual preparation of the executions, and do not consider the repeatability of a task over di erent datasets. 4
The research project aims at addressing the following research questions: RQ1: Are the current state of the art reconciliation approaches executable in a
Big Data environment? RQ2: Is it possible to insert the human in the enrichment process, in such a way to use her feedback to improve the system performance? RQ3: Is it achievable to use the history of enrichment pipelines to give suggestions to the user about creating a new pipeline? 5
The research questions stated in Section 4 will be addressed driven by the following hypotheses: (RQ1) H1: State of the art reconciliation approaches can be executed in a Big Data environment, but they do not scale as the dataset size increases. (RQ1) H2: Making state of the art reconciliation approaches scalable requires to rely on a smaller set of information, leading to worse performance. (RQ2) H1: Users feedback can be learned with machine learning models (e.g., neural networks) and reused for subsequent executions to improve the performance. (RQ2) H2: Enrichment steps can be executed in near-realtime on in-memory datasets to support the human-in-the-loop interactively. 5 https://github.com/rmalla1/OpenRe ne-HD (RQ3) H1: Deep learning architectures can be used to learn how users design enrichment pipelines. Once the network is trained, it can be netuned to support this task in di erent domains (similarly to transfer learning approaches). 6
To obtain preliminary results that demonstrate the soundness of this project proposal, we started by extending Grafterizer, a tool that supports data transformation pipelines design and execution, with a new embedded application, ASIA (Assisted Semantic Interpretation and Annotation Tool), which supports users in semantically extending their data. ASIA has been designed as a set of micro-services, namely reconciliation and extension services, each one dedicated to reconcile/extend data to/with a speci c KB. Those services are put behind a gateway, which allows users to use di erent KBs at a time to enrich their data. To test the exibility and scalability of such architecture based on micro-services, we executed three experiments on real datasets; within all experiments, we designed an enrichment pipeline that extends the dataset by querying a weather KB, by reconciling and extending city toponyms to a geospatial service (based on GeoNames (GN)). Both GN and the weather KB have been exposed as distributed services, thus ASIA must perform a series of HTTP requests for reconciling and extending the dataset.
First, we simulated the scenario where the data scientist executes the enrichment pipeline on a commodity machine (small-scale). The main objective was to assess how much the performance is a ected by HTTP requests in a distributed environment, and how much it boosts the system performance by adding di erent levels of cache. We started by testing the reconciliation performances with no caching strategy whatsoever: 200 thousand rows from a real company dataset featuring 2227 di erent toponyms (from Germany and Spain) have been extracted and a pipeline featuring only reconciliation executed. The measured average execution time per row was 12.927ms. The same test has been then repeated enabling a cache implemented at the reconciliation service level. This cache system improved the performances achieving on average 2.558ms per row (5x times faster with respect to the previous baseline). At last, a second cache layer has been enabled, which is implemented locally on ASIA. The objective is to avoid the network latency, which is substantial even in a local setup (via the loopback interface). The pipeline, in this case, ran 770 times faster than the baseline (0.0168ms/row on average).
In order to analyze the behavior of the cache over time, a second experiment has been designed extending the rst one as following: a more complex pipeline is considered, which reconciles city toponyms to GN, extends reconciled entities with their rst administrative level (i.e., regions), and fetches weather information about regions (i.e., temperature for a speci c date and the day after) generating a new dataset with 25 columns. This pipeline has been employed rst to enrich a dataset derived from one of the rst experiments ltering out the 120 )s100 cond 80 se i lilm60 ( iem40 T 20 0 Reconciliation KB-based extension Weather extension Reconciliation KB-based extension
Weather extension 120 100 ) snd 80 coe llism60 i( iem40 T 20 0 0 5
10 20 (a) Without duplicates. 0 20
40
80 (b) With 4 duplicates. duplicates in the reconciliation target column (i.e., each value occurs at most once); thus, resulting in 2227 unique cities (and rows). The outcomes of this experiment, where the cache did not signi cantly improve the performance (as it was built but never used), are depicted in Fig. 2a6. Afterward, a synthetic dataset was built where each line from the previous dataset is repeated four times, allowing to exploit the local cache. As reported in Fig.2b, spikes are still visible due to cache building, but the cache reuse speeds up the process progressively (4x on average), reducing the execution time (which tends to be purely cache access time) considerably.
The nal experiment was devoted to investigate the system scalability. First, a commodity machine is used (this experiment like the previous ones have been performed on a multi-tenant machine with 4 CPUs Intel Xeon Silver 4114 2.20GHz, and 125GB RAM) and ASIA deployed singularly. The same pipeline was used to enrich datasets of di erent size: 100MB, 1GB, 5GB, and 10GB, divided into 10 chunks of equal size and assigned to 10 agents (i.e., components that execute the pipeline). Performance results (in blue), reported in Fig. 3 as dataset size/total completion time, show a linear trend, which highlights the scalability of the proposed solution. Finally, the enrichment of a 100GB dataset ( 500 million rows, 21 columns) was performed; the pipeline was run on the Big Data Environment deployed on a private cloud infrastructure featuring an 8-node cluster of heterogeneous hosts. Five of the nodes have 4-core CPUs and 15.4GB RAM and three nodes with 12-core CPUs, 64GB RAM, with six 3TB HDDs holding a GlusterFS distributed le system (shared across the whole cluster). The enrichment agents were deployed on the three servers. The transformation accessed a load-balanced (using round-robin load balancing) set of 10 ASIA services deployed on the same stack. The linear trend with R2=0.998 is maintained also when the 100GB experiment is considered (the orange point in Fig. 3), despite the di erent context in which the experiments have been carried out. This is mainly due to similar access and reconciliation times between the two con gurations used. 6 Initial spikes are due to the system startup (e.g., database connectors initialization). The approach proposed in this research project to facilitate data processing and enrichment at scale is inspired by a small-scale design/full-scale execution principle, harnessing the semantics to support reconciliation operations that are required for data enrichment. The high-level description of this principle can be sketched as in Fig. 4. The overall design decision is to separate the transformation process in two phases: the design phase, where the user de nes the transformation pipeline working on a sample of the original dataset, and produce an transformation model (i.e., an executable representation of the transformation pipeline), and the processing phase, where the model is executed against the original dataset to obtain the enriched dataset to feed the analytical activities. Both phases rely on distributed datasets and services to support reconciliation and extension activities. In the data science context, this separation brings another advantage in terms of privacy and security: pipelines can be designed on a small sample, keeping private the entire company dataset. At last, since the design phase exploits only a sample, the limited quantity of the data to be transferred and manipulated simplify the infrastructure required to execute the enrichment tools (i.e., they can run in browsers on regular laptops). Instead, the execution phase can be enacted on-premise, thus exploiting the actual corporate infrastructure and tools, without moving the original dataset.
The data enrichment solution has been conceived to obtain large volumes of value-adding data to train analytical models in the easiest way (i.e., without requiring comprehensive programming skills). Most of all approaches available in state of the art are fully-automated approaches that are not suitable for this purpose, even if they represent the easiest way to transform the data, since they would entirely remove the human control over the process and results. Consequently, the above principle allows to give the human control over the quality of the results (e.g., she can x errors introduced by the reconciliation) during the design phase, and then her choices are packaged within an enrichment pipeline runnable in batch over a di erent (possibly massive) dataset. 7.1
In the design phase the user is involved in the design of the pipeline, by interactively performing three steps: (i) the enrichment pipeline design, where the transformation pipeline is built by means of a graphical interface that facilitates and automates interactions with reconciliation and extension services, (ii) the pipeline execution (small-size processing ), where each step of the enrichment pipeline is performed over the small-sized data, and the (iii) quality insights evaluation, where the user is provided with a few statistics that give a general understanding of the overall quality of the result (e.g., number of missing values). This interactive process is executed every time the user edits the pipeline de nition (e.g., adding a new step in the pipeline); thus, when the user is required to enrich considerable volumes of data (or at least larger than can be interactively managed), this approach envisages that the user carries out the design phase using a representative sample of the original dataset. It is essential to point out that the end-user might not necessarily proceed to the processing stage in the case when she needs to enrich tables with a few thousand rows, which can be achieved in the design phase.
Finally, the user-de ned enrichment pipeline is turned in an executable transformation model (e.g., packaged in a Java archive (JAR)), which can be downloaded from the user interface and reused as the main step within the processing phase. 7.2
The goal of this phase is to execute the pipeline designed and tested during the previous stage on a smaller (loadable in memory) dataset on a large (Big Data) dataset through parallelization.
As in the previous phase, three steps can be identi ed: (i) the data ow de nition (Stack con guration), which includes the enrichment pipeline, (ii) the batch execution (possibly in parallel) of the pipeline, and (iii) the quality assessment of the resulting dataset. If the result does not achieve an acceptable quality level, the user could go back and design a new pipeline.
As data ows we refer to an extension of the enrichment pipeline with preand post-processing actions, such as decompression of the input table from user archives, creation of chunks for parallelizing the process, and so on.
It is important to underline that the services implementing the reconciliation and extension functionalities must be available to be invoked (as a service) as the KBs essential to their operations can be massive, making impracticable to encapsulate them within the executable transformation model. This architectural choice, which derives from precise requirements such as modularity and exibility, constitutes the main limitation preventing the scalability of the entire enrichment process. 8
The outcome of this research project will be a tool for supporting data scientists in enriching their datasets with third-party sources, at large scale. Many features must be evaluated to understand the success of the proposed approach: { reconciliation performance: the reconciliation services should be compared with state of the art approaches in terms of precision and recall; a lower performance is expected, due to the high complexity of state of the art approaches that is not sustainable at large scale. The best tradeo between performance and execution time should be found (RQ1 H1). { system scalability: the preliminary results discussed in Section 6 show that the current system can scale, but this condition must be veri ed again when more sophisticated data reconciliation service will be adopted (RQ1 H2). { human-feedback learning capability: quality insights given to the user at design time should be compared with the quality assessment results made after the large-scale execution phase, in order to assess the learning capability of the adopted machine learning models (RQ2 H1). { execution time: the transformation and enrichment pipeline steps must be performed in near-realtime in the small-scale case, in such a way to guarantee interactivity in the design phase (RQ2 H2). { transfer learning: transformation models can be used to train a network and learn how users design enrichment pipeline. Given a set of enrichment pipelines performed over a dataset that pertain to a speci c domain (e.g., marketing), new pipelines for enriching datasets from a di erent context (e.g., e-commerce) can be automatically created using such network. The suggested pipeline must be compared with a human-designed pipeline designed for the same dataset (RQ3 H1). 9
The objective of this research project is to provide a scalable system able to support users with semantic enrichment. Unlike other state of the art approaches, which start by designing the most e ective algorithm, and consider scalability issues only later, we act in the opposite way; we proceeded bottom-up by designing a scalable architecture, based on micro-services, and then we increment the usage of semantic techniques until the scalability property of our system holds. We believe that the scalability property is the most important if we want to push this system towards a di erent user base, i.e., data scientists, who often operate on very large datasets. For this reason, in this section, the lessons learned and the current limitations are discussed, along with some aspects that could be improved to bring a not marginal improvement to the performance of the entire process.
Data Locality. In the initial release of the Big Data Environment, the data locality principle is limited to the life-cycle management of data of the KB employed for enrichment that are brought into the environment; thus, reconciliation algorithms have direct access to the KB. Similarly, the working data are stored in a distributed le system and accessible through the network. This architectural choice uniforms access times to data, but also raises the average read/write times by twice the network latency. There is the chance to improve the performance by distributing the chunks among the machines that execute the agents' containers.
Distributed Caching. The current caching system has the main issue of being
local, i.e., attached to each ASIA replicated deployment. Since the load
balancer runs a round-robin dispatching policy, it occurs that identical requests
are assigned to di erent replicas of ASIA, causing preventable cache misses. A
better solution entails the use of a distributed cache shared among the various
instances of ASIA and among the agents that carry out the pipeline in parallel.
Such a service (for example Ehcache [
E cient API interaction. At present, for both the design and processing
phases, reconciliation and extension are invoked for every single row of the
working table. As a consequence, for each line the agent running the pipeline waits
a time equal to the Round-trip Delay Time (RTD) for each line, forcing the
system to wait time equal to twice the network latency for each line.
Grouping the invocations to the service would improve the performance considerably.
The processing times of the input dataset could be further improved if light
network protocols (such as Websocket [