Data wrangling (DW) is the complex process associated with preparation of raw data for analysis. It is typically performed as a craft in an ad hoc manner and its level of complexity and success is highly dependent on the data analysis task at hand, quality of input data, and skill set of the data analyst. This makes the reuse of DW pipelines dificult, forcing data analysts to often devise a new pipeline for each combination of data and analysis tasks, a process that is not only complex but also expensive, consuming between 50 to 80% of even the most experienced data analyst's time. In this paper, we investigate a number of DW pipelines in the form of workflows to find commonalities or patterns in the way DW is performed in practice, considering a multitude of data analysis tasks and data sets, devised by data analysts with varying levels of experience. We present our investigation as a methodology that, from selection of workflow sources to workflow mining techniques, describes how we dealt with the challenges of finding patterns in the way people prepare data for analysis, given the general lack of guidelines for best practices and standards. The obtained results provide insights into the most commonly used DW operations, solution patterns, redundancies and, not only optimisation opportunities, but also opportunities for reuse of experience and best practices in data engineering. We believe that the obtained insights can be useful in facilitating the construction of DW solutions to inexperienced data analysts via the reuse of patterns and best practices in DW.
significant efort that needs to be made when choosing
suitable tools for the task at hand, based on the user’s
Data Wrangling (DW) is regarded as a tedious and com- understanding of the functionality associated with each
plex process, consuming 50-80% of data analysis time available DW operation, and the time taken to develop a
[
While such tools may share, to some extent, common Finding ways to help users build good quality DW DW supporting functionality, there is a lack of standards pipelines without burdening them is essential to the sucin terms of requirements, representations and outcome in cessful adoption of DW tools. This raises questions, such their oferings of DW operations [ 7]. This causes a severe as the following: given a wealth of Web-available reposburden on the user of these tools, resulting from the itories of DW pipelines, what solution patterns can be found in existing pipelines that could allow reuse of exDMaatnaaPglaemt’2e3n:t,2annddIOntpetrinmaitzioantiaoln,WMoarkrcshho2p8,o2n02D3a, tIaoaPnlnaitnfoar,mGrDeeecseign, isting solutions to DW problems? Can good quality so* Corresponding author. lutions to common DW problems be developed through " abdullah.almasaud@postgrad.manchester.ac.uk the use of patterns found in existing solutions? A good (A. K. A. Almasaud); s.sampaio@manchester.ac.uk (S. Sampaio); quality solution can be one that minimises waste in the p.sa0m00p0a-i0o0@02m-7a6n5c0h-e5s0t2e7r.a(Ac..uKk. (AP.. ASalmmapsaaiou)d); 0000-0002-3751-4182 use of computational resources. (S. Sampaio); 0000-0002-5341-1327 (P. Sampaio) In search for answers to these questions, an investiga© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License tion into existing DW pipelines was performed, enabling CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) identification of how analysts develop DW solutions in user provided examples, and the one by Sutton et al. [15], practice. In this paper, we present the methodology we which is inspired by the UNIX Dif command. Although used in this investigation, which combines ideas from tool functionality for direct and automated reuse of existprevious work in fields, such as ETL and Software En- ing solutions is not a contribution in any of these earlier gineering (SE), to discover patterns in DW workflows. research eforts, they bear evidence to the fact that soluMore specifically, we collect a set of workflows to search tions to common issues can be devised. The focus of this for DW practices followed by analysts in diferent data paper is on identification of design patterns in available domains, aiming at identifying common traits present DW pipelines, represented as workflows, considering in their DW pipelines. In this process, we conceptualise that a significant number of them are found in this forDW representations in a workflow as a way to standard- mat. For that, a review of available workflow mining ise them and, through the use of well-established graph contributions is provided in the following. mining solutions, we identify frequently-occurring pat- Workflow mining focuses on business process mining terns of DW tasks and operations. We present usage using event logs for the purpose of redesign and optimisainformation of identified patterns of operations and dis- tion [16, 17], which can’t be used to mine patterns in data cuss how our findings can facilitate reuse of existing DW pipelines. Previous works identifying patterns used logisolutions and optimisation of DW pipelines for common cal modelling of workflows into graphs to enable their DW problems. We believe that identification of reuse mining, where Tosta et al. [18] uses a path approach to and optimisation opportunities in ad hoc processes, such identifying patterns in scientific workflows (Figure 1), as DW, can reduce burden and increase productivity of and Theodorou et al. [19] used Frequent Subgraph Minthe professionals developing these processes, e.g., data ing (FSM) to identify frequent patterns of operations in analysts/scientists performing DW. ETL workflows. While the approaches used by both are
The rest of the paper is organised as follows: Section applicable to DW workflows, [ 18]’s approach would lead 2 presents a brief background and related work, the re- to identifying a set of short frequent paths of operations search methodology is presented in Section 3 followed which may lead to misleading conclusions, e.g, tasks C,D by the findings and discussion in sections 4 and 5 then and E in the figure appear 4 times (once in each path) ifnally our conclusion and future work in Section 6. as opposed to once in the original representation. The approach of [19] was successful in identifying patterns in ETL, however, the repository used was created from 2. Background and Related Work well-formed ETL standard workflows.
Widely established ETL and DW approaches may involve similar subsets of operations and steps, however, the two present considerable diferences. For example: At a time when data is often analogised to gold, diamonds [8] and oil [9], DW is predicted to become an increasingly critical process preceding data analysis, capable of determining the quality of the outcome of data analysis tasks [8]. To facilitate DW, a variety of tools have been proposed, ranging in capability and level of technical skill required from users [7, 10], e.g., from programming tools requiring high-level technical skills, such as R and Python programming, to visual tools requiring lower levels of technical expertise, a category that Talend Data preparation [11] and Trifacta Wrangler[12] fall into.
Both categories of tools, however, are limited in the sophistication of mechanisms to facilitate reuse of existing solutions. This is a shortcoming that Software Engineers have worked hard to overcome and often tackle through the use of patterns found in software design, as a way of standardising and passing down knowledge/experience of expert designers to non-experts, as readily-availablefor-reuse constructs[13]. While the factors motivating creation of design patterns in SE are similar to those in DW, no clear attempts to identify patterns in DW can be found in the literature. However, reuse of previous DW solutions via other means is found in previous work, such as the one by He et al. [14], which shows that suggestions of applicable operations can be made based on • ETL’s main focus is on operations that perform data integration from multiple data sources into a unified schema, and data storage in a consolidated data repository (e.g., data warehouse) whilst DW involves tailored data transformations to prepare a dataset for an analytical task such as machine learning-based prediction. • ETL involves a smaller subset of operations/data transformations applied in a quasi-sequential structure whilst DW involves a wider range of operations with potentially iterative cycles and interactive tasks (e.g., visual data profiling) • ETL is typically performed using specialized software tools (e.g., Talend and Oracle Data Integrator) that automate the majority of the ETL process steps, whilst DW combines GUI-based tools (such as Trifacta) and programming languages such as
Python and R to code user defined functions.
problems faced by analysts, that are not traditionally common in ETL workflows, but the similarities between the two and their required outcomes is the reason for our adaptation of ETL-based approaches to our research purposes. 3. Research Methodology itory were generated using the KNIME tool [23]. While the approach followed is not limited to KNIME, we chose it because it contains builtin operations covering a wide range of purposes and DW functionality, as well as the tool being reported as a leader in data science [24] and aimed at multiple data analysis applications.
data manipulation workflows, the workflows found in [20, 21, 22] were narrowed-down, due to their focus on DW. Following a thorough check, the MyExperiment workflows [ 20] were selected for their high-level of completeness and complexity. The vast majority of the higherquality workflows that were of interest to us in the repos1. Loading Data (L): Obtaining the data and trans
forming it into a format usable by the tool. 2. Exploring Data (D): Tasks related to data discov
ery and validation (e.g profiling). 3. Cleaning (C): Includes two buckets of tasks: record-based cleaning (e.g removing empty records) and value-based cleaning (e.g modifying column datatypes). 4. Intra-record Structuring (S1): Transformations 4. The node’s prevalent trait (e.g., Date&Time Diferthat manipulate individual fields or records at a ence calculates the diference between two dates time and include: removing, reordering and re- in a row and creates a new column with the renaming columns as well as creating new columns. sults; as such, it can be classed under the ’append 5. Inter-record Structuring (S2): Transformations column’ task).
that operate over multiple records and fields at once and are divided into: filtering and reordering records, shifting granularity of dataset via aggregation or pivots, and splitting the dataset into multiple datasets. 6. Integrating Data (I): Enriching a dataset by combining it with other dataset(s) using unions or joins. 7. Enriching Data (E): Transformations that add value to the dataset by deriving new fields us- Figure 5: Node classification process flow ing equations or domain specific operations e.g deriving date of week from date. An example of benefits of applying the taxonomy is 8. Transforming Data (T): Transforming data values illustrated in Figure 6 which can also be extended to within fields (e.g binning or normalisation). unify pipelines from multiple tools. In the example, the 9. Publishing Data (P): Encompasses all tasks that repository initially contains 95k nodes with 1.8k unique indicate an end of a DW portion of a pipeline signatures, which got reduced to 385 signatures, by using such as start of an analysis portion or exporting the 90 labels in the taxonomy. The 90 created labels the dataset in a format of choice. contain 37 DW tasks spanning 60% of the node instances. 1. The (KNIME, in this case) node repository classi
ifcation tree, illustrated in Figure 3. 2. The node’s functionality (e.g., Row Splitter is classified as a row filter operation according to its functionality). 3. The node’s description (e.g., Substructure Search is classified as a row filter operation since it implements domain-specific ’row split’ functionality).
DW pipeline, an initial set of conceptual operation labels 3.3. Parsing of Repository Workflows was also created, based on the transforms presented by into Graphs Raman et al. [28] and the Relational Algebra (RA) operations, eliminating duplicates found between the two. Preparation of workflows to be parsed using the PAFI Further, for each label, a DW task flag was created, as algorithm, developed by Kuramochi et al. [30], is sumpresented in [28, 10, 6, 27, 26, 7, 29]. The factors sum- marised in Table 2. Note that, while the target workflows marised in Figure 5 were used to map each operation (i.e., are, logically, direct acyclic graphs (DAG) [23], the diworkflow node) into a label or to create a new label for an rection of edges had to be neglected during the parsing operation, if needed, in the following order. It is worth process to adhere to PAFI’s requirement of ID order in pointing out that, the dictionary of steps helps mapping edge specification. Additionally, a concurrent parsing operations and labels into its most generalised form. process was performed to produce steps graphs using the steps dictionary (described in Section 3.2). In other words, each node in every repository workflow was mapped into its step representation, where workflow duplicate step occurrences and branching were eliminated to produce a linear graph, to be used in the identification of frequent step patterns. ✓ ✓ ✓ ✓ ✓
Purpose t n e m e r lo iuq oT re n them, which we informally refer to as intruder operations, and slight variations in the order in which DW operations appear. These observations have brought about the idea of making changes to our original approach, which considered obvious patterns appearing in the workflows (i.e., with high frequencies), as suggested by Theodorou et al. [19]. The modified approach aims at identifying lower frequency patterns found in the mining results that could potentially represent high frequency ones, if exceptions are dealt with diferently. This approach is detailed in the next sections but, in essence, variations in operation ordering found in operation groupings that appear with a certain frequency are disregarded, allowing identification of paths of DW operations within these groupings that are frequently traversed, which we call the Most-commonly Traversed Paths (MTPs).
3.4. Mining of Workflows The aim of the workflow mining performed in this research is twofold: (i) to find patterns of DW steps in pipelines, and (ii) to identify the frequency of DW tasks within the pipelines. For that, parsing of the workflows using the dictionary of steps, described in Section 3.2, was performed to produce a simplified graph of each pipeline while still preserving the original relevant content, as shown in Table 3.
Following transformation of DW pipelines into graphs through the application of the algorithm (and tool) proposed by Kuramochi et al. [30], the sets of graphs were divided into buckets based on the % of DW operations contained in them, e.g., (≥ 80%, ≥ 60%, ≥ 40%, ≥ 20%and > 0%), via multiple iterations performed with diferent support 1 (sup) thresholds.
graph to be counted as frequent dataset requiring dealing with null values (imputation or removal) and column based operations (e.g creation of new columns) they would use pattern Ps1. The results of DW steps mining confirmed that the lack of rules on operation placement was not simply an interchange of adjacent tasks which would not appear if generalised but was a more critical issue that appeared even when generalised to the level of steps. An example of the high variation of order can be seen in Pm3 which has 5 core wrangling steps and appeared in 200 diferent order permutations. entific workflows which are not natively supported by Figure 7: Most interesting patterns, parallelogram represent DW tools introduced the need of community to develop Exploration (D) which is not always captured by workflows custom nodes that are not always updated to support the and the double circle used because the cleaning (C) step is latest releases of KNIME. To solve issues related to lack optional and data dependant. of support to community developed nodes, users seem to prefer using UDFs in the form of scripts and code snip- The highest sup value with results in the task level pets. While reviewing these UDFs, it was noted that the pattern mining was 50% when the database of Source1 majority of them perform domain specific functions that was used which is the group that resulted in the largest are highly idiosyncratic and with limited impact on the number of patterns in other sup values. The highest sup approach of identifying general DW patterns. The use of value producing patterns for all databases was 15% as community developed nodes as well as the UDFs some- illustrated in Figure 8. The option to retrieve maximal times dictate the placement of specific nodes which were subgraphs2 only was used when running the algorithm, major contributors to issues identified in varied order of however, the inconsistent order of operations required operation placements. further manual processing to remove duplicate patterns
In the DW steps pattern mining, 31% of pipelines (out and those never independently appearing in workflows. of the 1,787 pipelines investigated) were omitted because The DW tasks were present in 575 subgraphs of which they included less than two DW steps. The remaining 63% contained repetition of tasks (Table 6) and 50% of pipelines resulted in multiple usage patterns from which the remaining subgraphs consisted of 6 tasks (not all DW the 7 most interesting ones are presented in Figure 7. The tasks). Although the produced subgraphs were unique presented patterns are divided into two groups based on to the graph mining algorithm, they were not unique in their inputs a) single-input patterns (Ps1-3) and b) multi- terms of functionality when analysed visually. input patterns (Pm1-4). The most frequent occurring We define a DW pattern as a frequently occurring pattern Pm3 was implemented in 32% of the workflow combination of DW tasks with distinguishing behaviour instances followed by 16% for Ps2. The step patterns leading to its classification. The 15 most interesting patare implemented by users based on factors such as num- terns extracted are presented in Figure 9. Other patterns ber of inputs, structure of input(s) and subsequent analysis requirement as seen in Table 5, e.g: if one has a
while only processing a single output branch which could be achieved with a simple RF operation. Combining the subgraphs as paths to create the DW traversed paths by removing the repetition of operations and disregarding the order of their appearance in the frequent subgraphs unveiled potential significant DW patterns. Figure 10 represents one of the MTPs departing from Row Filter which contain significant insights which didn’t appear in the produced subgraphs. The figure contains 5 nodes, the path A appeared in 45 diferent subgraphs, the path B appeared in 44 and the path C appeared in 27. While the number of times these paths of operations appear is relatively high, they were missed by the mining algorithm or returned with lower frequency because of the various placement order and repetition. Considering the results from the MTPs identified, lower frequency patterns in Figure 9 should have been produced by the mining algorithm with higher sup values had a consistent arrangement of operations been followed, e.g path B represent the patterns "Join with summary" and "join Summaries". On the other hand, path C did not clearly appear in the patterns resulting from the mining although it represents a relatively valid scenario in DW of shifting shape of a summary table before integrating datasets. in the results were either subsumed within the 15 patterns or have no clear distinguishing behaviour that could lead 5. Discussion to its definitive classification. Issues faced in the mining of patterns was mainly resulting from the varied order The findings related to reuse of scientific workflows comof tasks as well as the intruding tasks appearing in fre- bined with general challenges of reuse in workflows add quent combinations which hinder the identification of to the burdens faced by users. While tools such as KNa pattern or results in them appearing with a low fre- IME attempt to develop nodes based on usage statistics quency. From the results, there were interesting insights and user feedback, this doesn’t address reuse of existing that can be confirmed with the appearance of highly re- workflows since users are required to fully understand a peated operations (Table 7) which as well as producing workflow to be able to modify it. The heavy use of UDFs, irrelevant frequent subgraphs, indicated issues related to can be attributed to users reusing existing code to perincorrect utilisation of operations, e.g using a RS node ber of workflows that were authored by a small group of users analysing data in similar domains. Additionally, the operations in (Table 7), indicate lack of planning as well as knowledge of the tool’s capabilities. Although in some cases it may be valid to have a sequence of operations e.g to do selective processing, it is hard to find an Figure 10: MTP in the map of paths departing from RF argument which makes the sequence of multiple RF operations valid. Since RF operations evaluate a predicate by performing a full scan of the dataset, its repetition inform common tasks and in the most extreme scenarios dicates lack of knowledge and/or skill to create a proper use these interactive workflow design tools merely for predicate even with GUI support, which results in bad their orchestration capability. workflow designs and unnecessary execution of opera
Issues of placement of operations resulting in multiple tions. This in addition to the improper use of constructs order permutations of DW steps as well as tasks can be mentioned earlier in the case of operations such as RS. seen in all types of data-centric workflows implemented The workflows included in the repository are not using interactive tools can be attributed to the users and purely DW workflows, nor were they necessarily created tools. The lack of technical knowledge and DW experi- by experienced wranglers or ETL engineers, they repreence of the wider population of non-technical analysts sent a subset of real data pipelines containing DW steps combined with the usage nature of interactive tool can be in a workflow setting. While the subset of DW operaattributed to poor design outcomes in investigated DW tions used is not big, identical combinations of operations pipelines. The interactive tool design tempts a user to were not produced by the mining algorithm indicating only use an operation to fulfil the requirement of a sub- the lack of rules of thumb in DW which also explains the sequent operation including but not limited to loading lack of significant highly frequent patterns of operations and wrangling additional dataset(s) for integration. Al- similar to those found in the works of [19] where the though the patterns presented in Figure 7 had numerous mining was performed on ETL workflows built by ETL permutations during discovery, the order presented in experts using specification of TPC-DI, a benchmark for the figure is influenced by rules of RA where operations Data Integration [31]. It can also be deduced that even with lower-cost and highest impact on reducing size of though a visual interface makes it easier for non-experts the data are applied as early as possible and before more to attempt to wrangle by making the operations easier to expensive operations such as integration. ifnd through diferent interfaces it doesn’t guarantee they
With regard to DW patterns, it was observed that the will be used efectively or eficiently without standards
highest number of patterns appeared in the databases of or design rules to follow. The findings and deductions
"source1" which can be attributed to having the least num- made, resemble the basis of the motivation of the works
of the Gang of Four (GoF) in SE when they started the by specialised personnel in an organisation [26]. We
creation of their patterns catalogue [13], however, the agree with the latter and believe that an ETL pipeline is
diference in mentality of the two sets of users must be a well-defined DW pipeline constructed for reuse. The
considered. On the other hand, the set of paths identi- well-defined practices in ETL were aided by many related
ifed from the mining, present solutions that solve a wide IT fields including adoption of techniques from SE and
set of wrangling use-cases such as filtering, integration, DE domains as well as relying on DB technologies and
structure transformations and value conversions. E.g its optimisations when performing simple tasks.
Fig. 10 includes very expensive operations in terms of The self-service DW tools adopting GUIs and
menutime, memory and CPU resources that are used in varied driven tool along with a spreadsheet approach have
reorder in practice but can be optimally arranged using duced the burden of programming DW recipes by using
query optimisation techniques to produce a semantically familiar interfaces, however, they do not reduce all the
equivalent solution using the same operations, which is burdens of building a DW pipeline and in times they
eficient to run and easy to follow and apply. Using de- increase the dificulties by overwhelming users with
sign principles from SE and applying them to basic DW choices of DW operation with diferent implementations
operations and MTPs would create the standards and that increases the search space of possible operations to
patterns DW lacks which would be a seed to a catalogue be used by the non-technical analysts. The clear lack
to support the design of DW solutions. of standard in implementations of DW operations, their
requirements and outcomes also adds an overhead to
5.1. Recommendations user adoption to new tools. We believe standardising
operations and patterns in the field of DW and building
Other Data engineering (DE) approaches such as query on works such as [28] combined with clear cost
modbuilders (QB) applied to support the creation of SQL els would enable optimisations to be carried in any DW
statements without the burden of learning a new lan- pipeline. Our future work would use findings in this
guage using smart GUIs can reap benefits to building research to standardise the set of commonly used
operaDW pipelines eficiently. These approaches have been tions building on the idea of programming language and
applied in commercial ETL tools, and considering the tool agnostic conceptual DW operations [34], formal DW
similarities between DW and ETL, they can be used to pattern specification, and optimised MTPs using design
facilitate efective DW solution design. While in ETL, sim- principles enabling the creation of tools that reduce the
pler processes are handled using SQL [
diferent [ 6], others view ETL as a type of DW overseen [6] J. M. Hellerstein, J. Heer, S. Kandel, Self-service [19] V. Theodorou, A. Abelló, M. Thiele, W. Lehner, Fredata preparation: Research to practice., IEEE Data quent patterns in etl workflows: An empirical apEng. Bull. 41 (2018) 23–34. proach, Data & Knowledge Engineering 112 (2017) [7] M. Hameed, F. Naumann, Data preparation: A 1–16.
survey of commercial tools, ACM SIGMOD Record [20] D. De Roure, C. Goble, R. Stevens, The design and 49 (2020) 18–29. realisation of the myexperiment virtual research en[8] M. Muller, I. Lange, D. Wang, D. Piorkowski, J. Tsay, vironment for social sharing of workflows, Future Q. V. Liao, C. Dugan, T. Erickson, How data sci- Generation Computer Systems 25 (2009) 561–567. ence workers work with data: Discovery, capture, [21] NodePit, Nodepit, https://NodePit.com/, 2021. curation, design, creation, in: Proceedings of the [22] KNIME, Knime-hub, https://hub.knime.com/, 2021. 2019 CHI Conference on Human Factors in Comput- [23] M. R. Berthold, N. Cebron, F. Dill, T. R. Gabriel, ing Systems, CHI ’19, Association for Computing T. Kötter, T. Meinl, P. Ohl, C. Sieb, K. Thiel, Machinery, New York, NY, USA, 2019, p. 1–15. B. Wiswedel, Knime: The konstanz information [9] W. M. P. van der Aalst, Data scientist: The engineer miner, in: Data Analysis , Machine Learning and of the future, in: K. Mertins, F. Bénaben, R. Poler, Applications : Proceedings of the 31st Annual ConJ.-P. Bourrières (Eds.), Enterprise Interoperability ference of the Gesellschaft für Klassifikation e. V., VI, Springer International Publishing, Cham, 2014, Albert-Ludwigs-Universität Freiburg, March 7 9 , pp. 13–26. 2007, Springer, New York, 2007. [10] G. Convertino, A. Echenique, Self-service data [24] KNIME.com, Knime recognized by gartner as preparation and analysis by business users: New a leader in data science and machine learning needs, skills, and tools, in: Proceedings of the 2017 platforms, https://knime.com/about/news/knimeCHI Conference Extended Abstracts on Human Fac- recognized-by-gartner-as-a-leader-in-datators in Computing Systems, 2017, pp. 1075–1083. science-and-machine-learning-platforms-2019, [11] Talend, Talend data preparation, 2019. Accessed Dec 1, 2019.
https://talend.com/products/data-preparation/, [25] N. Weskamp, Workflow mining, in: Encyclopedia 2021. of Database Systems, 2009. [12] Trifacta, Trifacta wrangler, https://trifacta.com/, [26] T. Rattenbury, J. M. Hellerstein, J. Heer, S. Kandel, 2021. C. Carreras, Principles of data wrangling: Practical [13] E. Gamma, R. Helm, R. Johnson, J. Vlissides, D. Pat- techniques for data preparation, " O’Reilly Media, terns, Elements of reusable object-oriented soft- Inc.", 2017. ware, Design Patterns. massachusetts: Addison- [27] data wrangling handbook 0.1 documentation, 2013.
Wesley Publishing Company (1995). Accessed Dec 1, 2019. [14] Y. He, X. Chu, K. Ganjam, Y. Zheng, V. Narasayya, [28] V. Raman, J. M. Hellerstein, Potter’s wheel: An inS. Chaudhuri, Transform-data-by-example (tde): teractive data cleaning system, in: VLDB, volume 1, An extensible search engine for data transforma- 2001, pp. 381–390.
tions, Proc. VLDB Endow. 11 (2018) 1165–1177. [29] O. Azeroual, Data wrangling in database systems: [15] C. Sutton, T. Hobson, J. Geddes, R. Caruana, Purging of dirty data, Data 5 (2020) 50.
Data dif: Interpretable, executable summaries of [30] M. Kuramochi, G. Karypis, An eficient algorithm changes in distributions for data wrangling, in: Pro- for discovering frequent subgraphs, IEEE transacceedings of the 24th ACM SIGKDD International tions on Knowledge and Data Engineering 16 (2004) Conference on Knowledge Discovery & Data Min- 1038–1051. ing, KDD ’18, Association for Computing Machin- [31] M. Poess, T. Rabl, H.-A. Jacobsen, B. Caufield, Tpcery, New York, NY, USA, 2018, p. 2279–2288. di: The first industry benchmark for data integra[16] W. van der Aalst, B. van Dongen, J. Herbst, tion, Proc. VLDB Endow. 7 (2014) 1367–1378.
L. Maruster, G. Schimm, A. Weijters, Workflow [32] M. Francia, E. Gallinucci, M. Golfarelli, A. G. Leoni, mining: A survey of issues and approaches, Data S. Rizzi, N. Santolini, Making data platforms & Knowledge Engineering 47 (2003) 237–267. smarter with moses, Future Generation Computer [17] M. Hammori, J. Herbst, N. Kleiner, Interactive Systems 125 (2021) 299–313. workflow mining—requirements, concepts and im- [33] R. Mall, Fundamentals of Software Engineering, 4th plementation, Data & Knowledge Engineering 56 ed., Prentice-Hall of India Pvt.Ltd, 2014. (2006) 41–63. Business Process Management. [34] S. Sampaio, M. Aljubairah, H. A. Permana, P. Sam[18] F. E. Tosta, V. Braganholo, L. Murta, M. Mattoso, Im- paio, A conceptual approach for supporting trafic proving workflow design by mining reusable tasks, data wrangling tasks, The Computer Journal 62 Journal of the Brazilian Computer Society 21 (2015) (2019) 461–480. 1–16.