Self-service data preparation enables end users to prepare data by themselves. However, the plethora of possible data preparation steps can overwhelm the user. We introduce a score-based preparator ranking approach to propose preparator candidates in a context-speci c manner. To this end, we give scoring functions for a selected set of preparators and outline future work towards a full- edged data preparation system.
Data preparation (DP) is the process of cleaning and transforming raw data
before serving them to downstream applications. This process iteratively applies
a series of preparators on data. The traditional way of obtaining prepared data
requires collaboration between IT specialists and end users. The former prepare
data according to some speci cations, whereas the latter, who are usually domain
experts, consume these prepared data in the subsequent analytic applications. If
questionable outcomes occur, the preparation pipeline is revisited and adapted
accordingly. This modus operandi is often far from e cient, rst because it
produces communication overhead: the two sides have to go back and forth until
eventually understanding each other. Second, the preparation itself is not trivial:
it is reported to account for up to 80% of the time spent in the whole data analysis
lifecycle [
However, domain experts may not have enough technical skills or time to prepare their data, especially when the data comes with many dimensions and complicated schemata. For example, facing a dataset with hundreds of columns, one might hesitate about which columns might be merged into one, or not understand which parameters of a preparator to choose to consistently format a column, even provided with elaborate pro ling information. We believe modern data preparation systems should be equipped with a preparator suggestion mechanism that guides users through data preparation steps. We describe a Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). score-based preparator suggestion framework that proposes to the user a ranked list of suggested preparators for the next step in the pipeline. We present a selection of preparators and the corresponding heuristics to allow the framework to determine the applicability of these preparators in the current data context. 2
Only few research projects attempt data preparator suggestion. Heer et al.
proposed a framework for predictive interaction, which captures data transformation
intentions by requiring users to make a few clicks on the content of interest [
An obvious approach to preparator suggestion is to train a model based
on previous choices on similar data. Presistant exploits a meta-learning
approach that trains a learner to suggest a set of preparators that improves the
performance of classi ers on particular use cases [
Other works focus on traditional data cleaning tasks, such as repairing data
errors [
We propose a scoring-based approach that suggests a ranked list of preparators based on the data and interaction context at present. These preparators come from the search space of parameterized preparator candidates. A parameterized preparator is a preparator whose parameters are pre-set with values. We introduce a heuristic metric to measure the applicability of each preparator. These metrics are designed under two general guiding ideas: after applying the suggested preparator, datasets (i) become more homogeneous and (ii) have metadata that is closer to the prede ned target, if it is known. Given a data context, each preparator computes an applicability score based on the corresponding heuristic. The decision engine collects the scores and outputs those with highest scores. In future work, we will explore to use a user-de ned target data schema to assist the decision engine.
A preparator is an atomic operation to transform data. We de ne a preparator as a tuple of precondition and signature. The precondition is the required metadata that validate a preparator. The signature represents the preparator with all its parameters. The signature of a selection of preparators is shown in Table 1.
We design an applicability measure for each of the preparators by using
pro les and statistics derived from the data context. These measurements vary
according to the particular characteristics of preparators. For example, we use
the percentage of empty cells in a column to measure the applicability of the
preparator Fill missing value. More exemplary scoring functions can be found in
Table 1. To make these measurements comparable to each other, we normalize
them to [
Preparators with di erent parameter signatures are essentially di erent candidates. As potentially many values can be assigned to these parameters, the search space of suitable preparators may be large. Calculating all these candidates may cause latency, and is therefore not acceptable in interactive self-service data preparation systems.
We introduce three pruning rules to reduce the search space. First, we check whether the metadata precondition of a preparator satis es the current valid metadata, and lter out it if any of the metadata do not match. For example, given a piece of metadata that indicates no empty cells in a column, the Fill missing values should be ltered out and not shown to the user. Second, the preparators have been conducted may a ect the applicability of a preparator candidate for the next step. For example, if we have conducted Change value format, it is not likely to perform it again. We are seeking various heuristic measurements of using this type of pipeline contexts to reduce the search space. Last but not least, we use the previously computed scores to avoid unnecessary checks. For that, we need to store the score of each candidate that was calculated in a previous step. After computing the score of a candidate, we may directly skip the cached candidates with a lower score, as long as the object data part of these candidates were not a ected by the previous performed preparators. 4
Self-service data preparation aims at reducing DP expense and boosting e ciency of data-driven applications. Suggesting suitable preparation steps can further reduce time consumption. In this paper, we propose a score-based ranking approach for the preparator suggestion problem. We introduce a handful of preparator candidates that could be suggested by our approach, using heuristic scoring measurements.
A future solution to this problem may be to employ machine learning techniques: When provided with enough samples of input/output datasets, and corresponding DP pipelines, we may train ML models to predict appropriate preparators for the preparation situation at hand.