This paper presents a survey of existing tools for Big Data pipeline orchestration based on a comparative framework developed in the DataCloud project. We propose criteria for evaluating the tools to support reusability, flexible pipeline communication modes, and separation of concerns in Big Data pipeline descriptions. This survey aims to identify research and technological gaps and to recommend approaches for filling them. Further work in the DataCloud project is oriented towards the design, implementation, and practical evaluation of the recommended approaches.
The availability of massive amounts of data has tremendously changed the data
collection process and analysis over the last few years. The concept of Big Data
and the supporting solutions have allowed dealing with potentially unlimited
heterogeneous data in diferent formats within a practically acceptable time.
However, the growth of data has increased both opportunities and challenges.
In terms of opportunities, data processing is being heavily invested in to
empower the decision-making process of organizations in possession of Big Data.
Furthermore, the data analytics process is becoming complex due to the
characteristics of Big Data, the sophisticated tools and technologies involved, diferent
interests among stakeholders, often changing business needs, and the lack of a
standardized process for the lifecycle of Big Data pipelines [
Because of the complexity of Big Data analysis tasks, the software
supporting such analysis requires a combination of a broad spectrum of trusted software
components. Such a combination involves integrating components into pipelines
that take care of the pipeline execution and data transfer. The design and usage
of Big Data pipelines increase the eficiency of data analysis, while at the same
time require support for designing and managing the pipelines. Many
organizations recognize the significance of Big Data pipelines, however there are still
critical challenges in their implementation, such as the heterogeneity of involved
stakeholders and limited knowledge reuse [
The problem of combining diferent components into an executable process is
not new. Workflow systems are systems that support the integration of steps of
a semi- or fully automated procedure into a manageable process. Business
worklfows are workflow systems oriented towards automating business processes [
The approach in this paper is based on (1) extracting requirements for Big Data pipelines from the business cases of the DataCloud project as well as existing scientific literature and software tools around Big Date pipelines, and (2) analyzing which of the existing solutions (if any) can satisfy the identified requirements. For the requirements extraction, we defined the following Research Questions (RQ), which guided our work: – RQ1. How to bridge the technological gap between diferent experts involved in the Big Data pipeline design, implementation, and management? – RQ2. How to support the reuse of previously developed knowledge and solutions in designing and implementing Big Data pipelines? – RQ3. How to support debugging of Big Data pipelines? In this paper, we survey existing Big Data pipeline orchestration tools, identify research and technological gaps, and suggest approaches to filling the gaps. This work is done in the context of the HORIZON 2020 project DataCloud3. The rest of the paper is organized as follows. Section 2 provides a brief overview of the DataCloud project, including its objectives and expected results. Section 3 identifies the requirements and the classifiers for building a comparison table of the existing tools relevant to the DataCloud project perspective, and includes the comparison tables to identify gaps in the current solutions. The final Section 4 summarizes the main findings and refers to ongoing work in the DataCloud project to solve the identified gaps. 2
DataCloud Project Perspective In this section, we provide a brief overview the DataCloud project and present some requirements for Big Data pipeline orchestration tools identified while working on the project. 2.1
The DataCloud project, which runs between 2021 and 2023, aims to develop a novel paradigm for Big Data processing over heterogeneous resources, including the Cloud/Edge/Fog Computing Continuum. The core concept of the project is Big Data pipelines, whose complete lifecycle is supported by several processing capabilities. The DataCloud project utilizes this paradigm to solve issues for a broad set of business cases coming from Small and Medium-sized Enterprises (SMEs) and large organizations with dificulties in capitalizing on Big Data due to the lack of technical expertise and suitable processing capabilities.
In the DataCloud project, we develop a set of new languages, methods, infrastructures, and software prototypes for discovering, simulating, deploying, and adapting Big Data pipelines on heterogeneous and untrusted resources. The project underlines the separation of concerns and separates the design from the run-time aspects of their deployment. This separation allows domain experts without significant technical/programming knowledge to participate in the definition and management of Big Data pipelines. Moreover, the DataCloud solutions allow the incorporation of Big Data pipelines in organizations’ business processes more eficiently and make them more accessible to a broader set of stakeholders regardless of the hardware infrastructure. DataCloud assumes a typical Big Data pipeline lifecycle involving a set of high-level processing steps executed in a loop (see Figure 1). The project aims to deliver its solutions to support the Big Data pipelines lifecycle as a set of interoperable tools that form the DataCloud toolbox. The toolbox includes the following components (see Figure 2): – DIS-PIPE: Provides integration of process mining techniques and Artificial Intelligence (AI) algorithms to learn the structure of Big Data pipelines. The learning is based on extracting, processing, and interpreting huge amounts of event data collected from heterogeneous data sources. – DEF-PIPE: Provides support for the visual design and description of Big Data pipelines based on a Domain Specific Language (DSL). The tool includes the means to store and load the pipeline definitions that enable the reuse of previously developed solutions. It also enables data experts and domain experts to define the content by configuring individual steps and injecting code or customizing generic predefined step templates. – SIM-PIPE: Simulates the enactment of Big Data pipelines and provides pipelines testing functionality, including a sandbox for evaluating individual pipeline step performance. Furthermore, SIM-PIPE provides a simulator to analytically predict the performance of the overall Big Data pipelines across the Computing Continuum resources. – R-MARKET: Deploys a decentralized backbone resource network based on a hybrid permissioned and permissionless blockchain. This component provides a marketplace for resources and enables transparent provisioning of resources that increase the overall trust. – DEP-PIPE: Enables elastic and scalable deployment of Big Data pipelines with real-time event detection and automated decision making. – ADA-PIPE: Provides a data-aware algorithm for intelligent and adaptive provisioning of resources and services across the Computing Continuum as well as intelligent resource reconfiguration.
We evaluate the DataCloud solutions on five business cases provided by the DataCloud business partners, which cover a broad spectrum of Big Data pipeline applications: – Smart mobile marketing campaigns. – Automatic live sports content annotation. – Digital health system. – Predicting deformations in ceramics.
– Analytics of manufacturing assets. 2.2
The diversity and complexity of modeling data, processing Big Data pipelines, and the heterogeneity of Computing Continuum platforms require a multidisciplinary efort using expert knowledge of the domain, data, and technical knowledge of the computational environment. However, the collaboration among domain, data, and technical experts requires repeated communication cycles introducing significant overhead and barriers to success. Therefore, it is crucial and challenging to provide tools to bridge the technological and knowledge gaps between all relevant experts and enable them to collaborate while keeping the separation of concerns. Nevertheless, there are many design solutions for Big Data pipelines, so reusing them can boost the design and the development of new pipelines. The recent development of containers (as a technology allowing eficient and reliable installation of software components on diferent platforms) provides a good foundation for implementing reusable solutions. This is why containerization is a promising approach to support reusable solutions.
In designing Big Data pipelines, we need to consider the computational resources and decide about the deployment of the pipelines. However, it is not easy to make such a decision in the design phase in many cases, and running pipelines in the production computational environment might be expensive. Therefore, to make such a process more eficient, we need a simulation and debugging environment for pipelines that operates without deploying the pipeline. This is why combining pipeline descriptions with simulation tools is an essential requirement for modern systems.
By summarizing the above aspects and taking into account the properties of the DataCloud project described in Section 2.1, we outline the following requirements for Big Data pipeline orchestration tools: – Req1: Provide separation of concerns between design and run-time aspects. – Req2: Provide convenient means for describing pipelines including visual and textual (DSL-based) interfaces. – Req3: Support reusability of the previously developed steps and pipelines in designing new pipelines. – Req4: Provide flexible data transfer between steps in pipelines. – Req5: Support containerization for nodes and pipeline descriptions. – Req6: Provide smooth integration of description and simulation of components. 3
Overview of Existing Solutions In this section, we describe a set of criteria that we consider important for Big Data pipeline orchestration tools and compare current solutions with respect to those criteria. Most of the criteria refer to the requirements identified in Section 2.2. 3.1
The traditional way of representing pipelines is DSL that is incorporated into an orchestration environment. We performed an analysis of existing tools based on the following classifiers reflecting the requirements identiefid in the previous section.
Type of Workflow/Pipeline. We consider three types of workflows/pipelines: business workflows , scientific workflows, and Big Data workflows (defined in Section 1). Each tool falls into one of these three types according to its main applications.
Workflow/Pipeline Model. These categories are not mutually exclusive.
Different possible workflow models can be categorized as follows:
– Script-based: In these type of workflows the composition of nodes is described
using scripting languages. These workflows are useful for expert users to
design complex applications more flexibly and concisely [
A transition from one state to another happens on the occurrence of events
emitted asynchronously or by an external trigger. Hence, in event-based
workflows, users define event rules to declare under what circumstances state
transitions should occur. This provides a responsive orchestration process [
Type of Language. This criterion is related to the Req2 requirement. We consider the following types of languages: – General-Purpose Language (GPL): GPL is a highly applicable language (e.g.,
Python, R, Java) across a variety of application domains. – Domain-Specific Language (DSL): DSL is a language that is specially designed for a specific problem domain. The main advantage of using a DSL is that domain experts, who have little knowledge outside of their domain, can eficiently design relevant parts of the pipeline logic. On the other hand, its main drawbacks are the limited portability across diferent environments and low applicability outside the discrete domain.
Input Supported. This criterion is related to the Req2 requirement. Here, we consider two possible input types for designing and implementing Big Data pipelines: text-based input and graphical or visual input. We assume that tools with visual input types may also support text-based input types. Ease of Use. This criterion is related to the Req2 requirement. The possible values for this classifier are Hard, Medium, and Easy, which refer to the level of expertise a user needs to have to be able to use the tool. For example, if a tool allows designing a workflow through a clear graphical interface, the tool is considered to be Easy to use. However, if the tool relies on using a lightweight, human-readable text format, such as YAML, XML, and JSON, the level of easeof-use is considered Medium. Otherwise, if programming knowledge and skills are needed for using the tool, it is concluded that the tool is Hard to use. Sometimes it is dificult to place a tool into one only category and we allow a combination of values, for example, Medium/Hard or Easy/Medium.
Focus on Reusability. This criterion is related to the Req3 requirement.
Reusability refers to the characteristic of a designed workflow, whereby it can
be used to create another similar workflow. The reusability is a tool’s focus if
(i) it provides a visual drag-and-drop feature for reusing the previously-designed
workflows, or (ii) it supports a search for previously-designed similar solutions
to be imported as text-based workflows for reusing in designing the current one.
Otherwise, it is concluded that reusability is not a focus [
Reusable elements. This criterion is related to the Req3 requirement. In this research, the reusability of each tool is evaluated in three aspects: – Whether or not it is possible to reuse the definition of the entire workflow. – Whether or not it is possible to reuse the definition of each step. – Whether or not it is possible to reuse the step implementation.
The possible values for each of these aspects are Yes, No, and Partial, which means that although no specicfi way is designed to share assets, sharing can be done by manually copying scripts.
Nested Step Definition This criterion is related to the Req3 requirement. This is a binary classifier, the value of which would be Yes, if the tool has a specific way for defining and using a step inside and as a part of another more granular step. Otherwise, its value would be No.
Configurable Data Transmission Medium Definition. This criterion is related to the Req4 requirement. This classifier has two possible values; Yes and No, depending on whether or not the tool allows users to choose between multiple data transmission mediums (such as shared file system, web services, RPC, file transfer, HTTP, and FTP) for transferring data between steps during the pipeline execution.
Configurable Communication Medium Definition. This criterion is related to the Req4 requirement. This classifier has two possible values; Yes and No, depending on whether or not the tool supports choosing the medium of control flow communication between steps during the pipeline definition, invocation, and execution. For example, this could be done using RESTful API, message queues, or RPC.
Containerization. This criterion is related to the Req5 requirement. In
diferent workflow types, containers can be used to automate the deployment process,
enabling better scalability. We consider three possible approaches for
containerization [
Integrates a Simulation Tool. This criterion is related to the Req6 requirement. Here, we consider if a tool integrates a simulation tool. The value of this classifier is either Yes or No.
Monitoring. If the tool provides means for monitoring the execution of the pipeline. Depending on how the execution can be monitored, it can be further classified into the following three sub-categories: – Runtime: If the monitoring is available real-time. – Logging : If the run logs are available at the end of the execution. – No: If the tool ofers no monitoring support. 3.2
In Tables 1-4, we compare a representative number of the most popular Big Data pipeline orchestration tools with respect to the classifiers described in the previous subsection. In these tables denotes Yes and denotes No. to rs wflokro WeritnE in in g e
a p t wflokro WeritnE i a ifc s s es Ufo esaE C r d a . t u p n
I m le b 7 l e d o M w lfo k r o W
, ]4 ] ]6 , ] lfireohwAcap ,[]303 lfrookwWrog][4 [rcea40ydhm ]42 []59lreepK i[]ftrcaa65T [irsee5apPm l[eo32Fwak [-reo1CTAG [eea28akknm ]25 [,sse12ag43u A A P tS M C S P
T T T T L L L L S S S S
L L L L
S P S S 5 L S D p n a o wflokro WeritnE i a c i f s s gniroti no M t m i n u es Ufo esaE C d r a m l u
i d . e a m u le p n I t u y r
R a 2 a ta ific ific ific ific ific
t t t t t D n n n n n g ie ie ie ie ie i c c c c c R R L L R R R R l l a a i i t t r r a a
P P t x e P L
L L
L L L
L P P S P P S
l t t t t t a x x x x x u e e e e e s
i
T T T T T T
T
T
V
V
D G G D G G D G G D s l o o T ifc i t n e i c S ] 5 3 [ w o l F t x e ] 9 2 [ 2 7 d y r s a a
P m m Pm m d iu iu iu iu y r d d d d s a e e e e a M H
H E
E H M M M M noiti fine DpetS n s
S b a u io e t R noitatne mel p mI petS t y r l a a
i m r a wflokro WeritnE iif ita a
l c s r s - a es Ufo esaE C -sy iud id rd id sy sy rd a l
u u . a e e a e a a a m u le lfo k r o W T L
L
L R
R
R R 3 7 E
E
E M
M V
V
V D D D D D
D sed lra8iev sed lra sed eiv eitv sed sed lra sed ev itev lra sed ev itev lra sed sed lra sed sed sed ev itev lra i-trcapbS ceduroP ltrceaaD i-trcapbS ceduroP i-trcapbS tadpA lrceaaD i-trcapbS -teanbvE rceoduP i-trcapbS itadpA lrceaaD rceoduP i-trcapbS itadpA lrceaaD rceoduP i-trcapbS -teabnvE rceoduP i-trcapbS i-trcapbS -teavbnE itadpA lrceaaD rceoduP y s a l a u s i L S a t a g i , 8 3 [ D
] E 9 R 3 e y s a l a u s i L S a t a g i ] 7 2 [ E M I N K m u i d e t x e T L S s s e D
D
B S B noitatne mel p mI petS
m noiti fine DpetS n su
S b
a io e t R a wflokro WeritnE iif ita c l s r s a
P t 4
t ex ex T t u lfo k r o W T a t a g i ] 1 5 , 8 4 [ r e t t i k S a t a g i ] 1 1 [ r e t s g a D e
e 4 7 Pm P M H P ] 7 4 [ w lfo e R y s a l a u s i V L S s s e o [9 C L P
G T
T a t a g i ] 6 4 , 5 4 [ t c e f e r
D ] 6 3 [ e h c a p d d d l d d e l e l d d e l d d e l se ev se se ev ra se se ev itv ra iv ra se se ev itv ra se se ev itv ra i-trcabp itapdA i-trcabp -teabnv itapdA rceoduP i-trcabpS -teanbvE itapdA lrceaa rceodu ltrceaa rceodu i-tabp -teanb itapd lrcaaD ceoduP i-trcabp -teavbn itapdA lrceaa rceodu
D P D P rcS vE A e r
S S E S E D P
Conclusions In this paper we investigated existing Big Data pipeline orchestration tools. By analysing them (in Section 3.2) we show that important requirements defined in the DataCloud project are not (or are only partially) satisfied by the currently available tools. In particular, only a few tools support a graphical input language for the description of pipelines. While several tools allow some levels of reusability, most reusability aspects (such as support for searching available suitable solutions) are not implemented. Moreover, integration with other tools (including simulation tools) is not supported by many of them. However, we can mention that Apache Airflow, Argo Workflow, and Snakemake are the closest tools to our requirements among the reviewed ones. Nevertheless, although some aspects of some requirements are considered in some tools, no tool supports all required aspects.
To address these problems in the DataCloud project, we are developing a
DEF-PIPE component (see Section 2.1) for designing Big Data pipelines. This
component will provide means for the description and manipulation of pipelines
and the environment. It will also support a complete graphical and textual
interface, accumulation and reuse of solutions across diferent applications,
flexible integration with a simulation tool, and separation of concerns between the
description of design-time and run-time aspects. Preliminary results in this
direction are reported in [