In this paper, we explore diferent deployment models for data engines and elucidate their implications on data pipeline behavior. Specifically, we examine the impact on data sharing, data protection, pipeline uptime and latency, and the feasibility of moving segments of typical data engines to the edge. Our work demonstrates the consequences of various deployment strategies on non-functional properties of data pipelines, focusing on availability, performance, and privacy. By considering the interplay between data engine deployment and data pipeline requirements, stakeholders can make informed decisions to optimize the eficiency and efectiveness of data-driven systems.
CEUR ceur-ws.org
The last decades have been characterized by multiple ICT revolutions, from service to cloud-edge
computing, from mobile systems to 5G and Internet of Things (IoT), and from big data to machine
learning (ML). These technological enhancements brought to a scenario where data production,
collection, and analysis are carried out at an unprecedented rate [
Data engines (aka data platforms) are yet another type of modern system and consist of many components for data management often implemented as micro-services. Diferent architectural solutions for data engine deployment address the peculiarities of complex data-driven environments. In the past, the choice of the specific deployment model typically hinged on technical feasibility, while not considering the impact of a specific deployment on the non-functional posture (e.g., performance, privacy) of the data-driven systems.
The research community has recently started considering data engine deployment as yet
another dimension of data pipeline validation and verification [
Our reference architecture incorporates a common data engine [
Initially, the architecture collects data from various sources, directing it to a central data storage repository accessible for processing and analytics. The resource manager coordinates these elements, ensuring structured and eficient resource allocation. Data processing and analytics components can write back to the central storage, which is then accessed for data visualization to present and interpret the processed data.
The centralized deployment deploys the entire stack in a single location (e.g., in the cloud). It is well-suited for scenarios where data proximity is not a critical factor, and the primary concern is the eficient management of resources and data processing.
Figure 1 presents the architecture of a centralized deployment built on two main blocks: i) data sources, ii) data engine. Data sources, positioned at various layers of a distributed infrastructure, gather data from sensors, devices, and network (edge and cloud) nodes. These data are transferred to the data engine via communication queues or APIs, and stored in the cloud-based data engine. The data then undergoes processing steps like preparation, analysis, and processing. Centralized deployment integrates key components in a single stack, typically located in a data
Data Storage
Data Processing center or cloud environment, to simplify management and enhance function accessibility for eficient data handling and analysis.
The centralized deployment holds several advantages that contribute to its appeal in various contexts, where simple management and resource availability stand out: i) single point of management, the centralized deployment streamlines administrative tasks, reducing complexity and facilitating more eficient oversight of the entire centralized system; ii) resource availability, built on cloud functionalities with particular reference to scalability and elasticity. The final user has the impression of having infinite resources at its disposal, further empowering its ability to eficiently execute resource-intensive data processing; iii) end-to-end data pipeline control, giving the final user the possibility of managing the entire data pipeline in a single point, leading to a cohesive and orchestrated approach. iv) centralized authentication and authorization, where access to data is centrally regulated reducing the complexities associated with user syncing and access management across multiple locations.
The centralized deployment, however, introduces important challenges that call for careful consideration: i) single point of failure, posing a significant risk, as any malfunctions or outages in the centralized infrastructure can disrupt the entire system. It can be relieved by adopting disaster recovery and high availability protocols; ii) transfer costs and scalability limitations, continuous streams of data towards a centralized cluster for processing can result in significant data transfers, leading to high costs in terms of time and resources. Additionally, scalability limitations may impede the adaptability of the architecture to growing data volumes and increasing demands; iii) increased latency, impacting on the ability of the user to carry out real-time processing when data need to be moved from data sources to the cloud; iv) increased risk of data breaches and data leaks, when data traveling from distributed edge locations to the centralized cluster, traverse various untrusted network points, where they can be intercepted or accessed by unauthorized parties.
The decentralized deployment deploys the entire stack closer to the data at the edge. It is well-suited for scenarios where data proximity is a critical factor, and the main focus is on Data Storage eficient resource management and data processing with privacy in mind. It can be complex and costly to implement in certain contexts.
Some of the advantages of a centralized deployment (i.e., single point of management, end-to-end pipeline control, and centralized authentication and authorization) are also valid in a decentralized deployment. The decentralized deployment provides two additional benefits: i) increased data protection, since sensitive information does not have to travel to external servers, minimizing the risk of data breaches; ii) reduced data transfer costs by deploying processing components in proximity of data sources.
The decentralized deployment, however, introduces important challenges that call for careful consideration: i) increased complexity, due to the shift of the entire stack to the edge; ii) resource limitations, where the system performance is constrained by the computational resources at the edge, which are typically less powerful than those available in a data center; iii) decreased security, because data at the edge lacks the protection provided by a typical data center with no resource limitations, making it more vulnerable to physical attacks; iv) environmental factors (e.g., temperature and humidity) that can impact the performance of the system.
The hybrid deployment combines centralized and decentralized deployments to fully unleashes the potential of microservices technologies. It deploys the building blocks of the data engine where convenient. Temporary Storage
Data Processing
Cloud
Figure 3 presents the architecture of a hybrid deployment. It is built on two main parts, one deployed in the cloud and the other deployed at the edge. The same stack discussed for the centralized deployment (Section 3) is used in the cloud. We note that the resource manager might be compatible and support the communication with its counterpart at the edge. A minimal version of the stack with i) a temporary storage to manage ingestion and ii) a resource manager that receives instructions from the one located in the cloud is used at the edge. A distributed resource manager permits some autonomy at the edge, enabling it to independently manage its own resources and tasks, such as job scheduling or data transformation operations.
The hybrid deployment provides a balanced solution that merges the advantages of both centralized and decentralized deployment models: i) optimized resource allocation and monitoring, where resources are dynamically allocated based on workload demands, maximizing utilization and performance across the distributed environment. By partially processing data at the edge, computational tasks can be ofloaded from the centralized cloud infrastructure, reducing latency and bandwidth usage; ii) increased fault tolerance, because the system remains operational, albeit potentially with limited functionality, in the event of central server failure; confining data within explicitly defined processing boundaries to reduce the likelihood of accidental data exfiltration.
The decentralized deployment, however, introduces an additional challenge: i) increased complexity, due to the need to support data partitioning, fault tolerance, coordination, resource management, and troubleshooting. 6. Mapping Non-Functional Properties on Architectural
Deployments We discuss how architectural deployments can impact the non-functional properties availability, performance, and privacy of the specific data engine and the data pipelines executed on it. Each property is assigned a level of strength in {low, medium, high}, as presented in Table 6.1.
High Multiple replicas, multiple
zones Medium Multiple replicas, single
zones Low Single replica, single zone a) Availability
High Resources can scale without
any limits Medium Resources have a limited
ability to scale Low Resources do not scale b) Performance
Low
No data are transferred Only secondary data are transferred Both primary and secondary data are transferred c) Privacy
It is the availability that a specific deployment model can guarantee to a data engine and, in turn, to the data pipelines executed on it. We define property availability as follows. Definition 1 (Property Availability). Property Availability models the system uptime as a function of the number of system replicas (i.e., single, multiple) and their deployment across diferent zones (i.e., single zone, multiple geographically-distributed zones).
The levels of strength associated with property availability are as follows. Low availability refers to a single replica with data stored on a single zone. If a node fails, the data may become inaccessible until the node is repaired or replaced, resulting in significant downtime. Low availability is generally not suitable for critical systems, but may be acceptable for non-critical data or systems, where cost savings are a priority. Medium availability refers to multiple replicas with data distributed across a single zone. While there is some level of fault tolerance, the system may experience downtime or slower response time if a node fails, as the remaining nodes may become overloaded with requests, or the entire zone may experience downtime. Medium availability may be acceptable for systems where occasional downtime or slower response times can be tolerated. High availability refers to multiple replicas with data distributed across diferent geographically-distributed zones. Even if one or more nodes or zones fail, the system can still serve the data from the remaining nodes or zones. This ensures that the data are always accessible, providing a high level of fault tolerance. High availability is often associated with systems that cannot aford downtime and where data loss is unacceptable.
It is the performance that a specific deployment model can guarantee to a data engine and, in turn, to the data pipelines executed on it, as follows. We define property performance as follows.
Definition 2 (Property Performance). Property Performance models the system performance as a function of the amount of available resources and the time required for moving data from the sources to the data engine. The former aspect considers both static scenarios where resources are assigned a priori and dynamic scenarios where resources can elastically scale. The latter aspect depends on the volume of data to be moved and the corresponding network resources.
The levels of strength associated with property performance are as follows. Low performance refers to resources that do not scale. This means that the amount of resources is fixed and cannot be increased to handle higher demand. If the demand exceeds the capacity of the resources, the system may experience significant performance issues. Medium performance refers to resources that have a limited ability to scale. The system can handle moderate increases in demand, but may struggle or experience performance degradation if the demand increases significantly. High performance refers to resources that can scale virtually with no limits. The system can add more resources to meet an increasing demand. This is often seen in cloud-based systems where resources can be added or removed as needed.
It is the level of privacy that a specific deployment model can guarantee to a data engine and, in turn, to the data pipelines executed on it. We define property privacy as follows. Definition 3 (Property Privacy). Property Privacy models the level of data protection guaranteed by a specific deployment model as a function of the amount and type (either primary or secondary) of data that are exchanged between diferent systems.
The levels of strength associated with privacy are as follows. Low privacy refers to a scenario where data transfer is requested for both primary data collected from the source and secondary data that have been previously pre-processed. These data can include sensitive information, so it is important to have robust security measures in place to protect them during transfer. Medium privacy refers to a scenario where data transfer is requested for secondary data only. Primary data collected from the source are not transferred, reducing the risk associated with sensitive information leakage. Security measures are still important to protect data during transfer. High privacy refers to a scenario where no data transfer is requested. This is the highest level of privacy, as it eliminates the risk of sensitive information being intercepted or misused during data transfer. However, it also means that the benefits of data sharing, such as collaboration and data analysis, are forbidden.
Table 2 describes the correlation between non-functional properties in Section 6.1 and the architectural deployments in Sections 3–5. We use the following symbols to denote the support provided by a deployment model for a specific property level: 3 to denote full support, ~ to denote that the property level can be supported with certain limitations, 7 to denote no support. The subsequent analysis delves into the details of this mapping.
Centralized Deployment takes full advantage of cloud capabilities, ensuring optimal availability and solid performance, while introducing several privacy challenges. It supports all availability levels due to the native support for geographically distributed replicas provided by the cloud. We note that the support for high availability (i.e., multiple replicas stored across geographically distributed zones) mitigates the impact of a single point of failure. It streamlines resource allocation and monitoring, facilitating system scalability and elasticity without
Centralized Decentralized
Hybrid constraints. However, performance could degrade (high latency) when large volumes of data must be transferred from the sources to the cloud. For this reason, while the low and medium performance levels are fully supported, the high level might not be always feasible. Finally, the frequent transfer of raw and unprocessed data to the central server in the cloud raises concerns about unauthorized access or interception, limiting privacy to the low level. Decentralized Deployment executes on a restricted set of resources at the edge, which negatively afects availability and performance. On the other hand, data locality guarantees high privacy. It lacks support for geographically distributed zones. Nevertheless, it can accommodate multiple replicas, sustaining low and medium levels of availability. Operating within a decentralized environment entails coping with limited and less powerful resources. However, proximity to the data source can mitigate latency, thereby increasing the eficiency of data processing. Decentralized deployment thus supports performance levels low and medium. Finally, leveraging decentralized data storage can fortify privacy measures by reducing exposure to a single point of attack and minimizing data transfer. Decentralized deployment ensures privacy across all levels, from low to high.
Hybrid Deployment ofers better properties on average. It enjoys the benefits given by cloud resources availability, as well as the protection/anonymization of sensitive data at the edge. Hybrid deployment breaks the monolithic approach followed by centralized and decentralized deployment models, distributing critical components across the cloud-edge continuum. It supports availability across all levels, from low to high. Although this approach may ensure performance at all levels, from low to high, potential bottlenecks in data transfer can still survive. Finally, it enhances privacy by minimizing exposure to a single point of attack and reducing data transfer. Source data can be first pre-processed at the edge and then transferred to the cloud. Hybrid deployment ensures privacy at all levels, from low to high .
We propose three reference scenarios that evaluate the application of our deployment models in varying contexts. Throughout this section, we also present selected code excerpts illustrating the deployment processes using Docker Compose or Kubernetes. 7.1. Centralized Deployment Scenario - E-commerce Platform Analytics Context: An e-commerce platform that analyzes users’ behavior, sales data, and product trends to optimize its oferings and marketing strategies.
Use Case: Using the centralized deployment model, the platform can aggregate data from various sources into a single data center or cloud environment, facilitating complex analytics and machine learning processes to derive actionable insights.
Benefits: The centralized deployment model ofers high analytics performance and availability. It handles large volumes of data and supports intensive computational tasks, which are crucial for real-time analytics and decision-making in a dynamic e-commerce environment. Analytics performance comes at the cost of an increasing cost in data transfer.
Technological Architecture: The centralized deployment model for e-commerce analytics employs Hadoop Distributed File System (HDFS) as data storage, Yet Another Resource Negotiator (YARN) as resource manager, Apache Spark for data processing and analytics, Apache Superset for data visualization, and Hive as query engine.
Deployment Configurations: This section presents the configurations for a basic centralized setup. For brevity, we propose a Docker-Compose that only includes HDFS, YARN, and Spark. We use YARN as resource manager, because the availability of (potentially unlimited) resources in the cloud makes it a robust and widely adopted solution. Modern orchestrators like Kubernetes can be used, while introducing additional maintenance overhead and complexity. services: namenode: image: apache/hadoop command: ["hdfs", "namenode"] ports:
- 9870:9870 env_file:
- ./config datanode: image: apache/hadoop command: ["hdfs", "datanode"] env_file:
- ./config resourcemanager: image: apache/hadoop command: ["yarn", "resourcemanager"] ports: - 8088:8088 env_file:
- ./config nodemanager: image: apache/hadoop command: ["yarn", "nodemanager"] env_file:
- ./config sparkmaster: image: spark entrypoint: ["bash", "-c", "$$SPARK_HOME/sbin/start-master.sh --host sparkmaster && sleep inf"] sparkworker: image: spark entrypoint: ["bash", "-c", "$$SPARK_HOME/sbin/start-worker.sh sparkmaster:7077 && sleep inf"]
The Docker-Compose file creates a Namenode, which manages the metadata for the HDFS ifle system, and a Datanode, which contains the actual user data. Regarding YARN, the DockerCompose file creates a Resourcemanager and a Nodemanager to eficiently manage the resources across the cluster nodes. Finally, it configures a Spark cluster with two containers: the first one becomes the master upon executing the start-master.sh script, while the second one becomes a worker upon executing the start-worker.sh script, passing the master endpoint as a parameter. 7.2. Decentralized Deployment Scenario - Patients’ Health Monitoring Context: A healthcare system that monitors patients’ health data in real-time across various devices and locations to provide immediate care and intervention.
Use Case: The decentralized deployment model allows patient data to be processed locally at each healthcare facility or via patient monitoring devices, ensuring quick response times and reducing the need to transfer sensitive data over the network.
Benefits: The decentralized deployment model enhances data protection and privacy, both of which are critical in the healthcare sector. It also facilitates real-time monitoring and decisionmaking by processing data close to their sources.
Technological Architecture: The decentralized deployment model for the patients’ monitoring system employs Minio as data storage, Kubernetes as resource manager, Spark for data processing, Apache Superset for data visualization, and Trino as query engine. Deployment Configurations: This section outlines the configurations for a decentralized setup utilizing Kubernetes, Minio, and Spark. Initially, Kubernetes establishes a pod for the Minio server, which is then accessible as a service named minio-service. Subsequently, Spark is deployed on Kubernetes to leverage the dynamic resource allocation it provides. However, integrating Spark with Kubernetes isn’t straightforward and requires the Spark-Operator, an operator designed specifically for managing Spark applications within the Kubernetes ecosystem. The most eficient installation method for the Spark-Operator is through Helm. 1 Following this, a Docker image that contains the Spark code to be executed is created. The final step involves defining a Kubernetes resource type SparkApplication, introduced by the Spark-Operator, for deploying the applications. Below is an example YAML configuration used in this setup. apiVersion: "sparkoperator.k8s.io/v1beta2" kind: SparkApplication metadata:
name: spark-with-minio spec: type: Python pythonVersion: "3" mode: cluster image: "spark-app" mainApplicationFile: local:///app/main.py sparkVersion: "3.3.1" driver: cores: 1 memory: "1024m" labels:
version: 3.3.1 executor: cores: 1 instances: 2 memory: "1024m" labels: version: 3.3.1
The location of the Spark application is specified in the mainApplicationFile property, while driver and executor specify the system requirements for the driver and the executors, respectively. 1https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/quick-start-guide.md# installation 7.3. Hybrid Deployment Scenario - Financial Services Risk Analysis Context: A financial institution that analyzes transaction data for real-time fraud detection, while also conducting deeper, historical risk analysis to refine its fraud detection algorithms. Use Case: The hybrid deployment model can be used to initially process transaction data at the edge (local bank servers) for immediate fraud detection. Simultaneously, data are sent to a centralized cloud server for more complex, long-term risk analysis and model refinement. Benefits: The hybrid deployment model leverages low-latency processing at the edge for immediate fraud detection and robust computational resources in the cloud for deep analytics. It ofers a solution that ensures real-time responsiveness and advanced analytical capabilities. Technological Architecture: The hybrid deployment model for the risk analysis system employs Minio as data storage at the edge and HDFS as data storage in the cloud, Kubernetes as resource manager, Spark for data processing, Apache Superset for data visualization, and Hive as query engine. The configurations for a hybrid deployment resemble and extend those presented for centralized and decentralized deployments. The configurations of the decentralized deployment are used as is; the centralized deployment is migrated to Kubernetes with the need to define all Kubernetes configuration files. The hybrid model introduces several complexities in the setup process due to the orchestration layer shared between cloud and edge nodes. A detailed explanation of how to eficiently deploy hybrid architectures using Kubernetes is beyond the scope of this paper.
Distributed systems have been studied from several angles across the ICT evolution, focusing
on their design, development, deployment, and the evaluation of their non-functional behavior.
Recently, particular emphasis has been given to big data architectures, focusing on the design
and implementation of big data systems, their deployment on the cloud-edge continuum, as well
as the evaluation of their performance and scalability (e.g., [
We explored diferent deployment models for data engines in the cloud-edge continuum and shed light on their impact on data analytics pipeline behavior. Critical aspects, such as data sharing and protection, pipeline uptime and latency, have been explored considering non-functional properties availability, performance, privacy. Our results highlighted the significance of data engine deployment as a critical dimension of data pipeline validation and verification.
Research supported, in parts, by i) project “BA-PHERD - Big Data Analytics Pipeline for the Identification of Heterogeneous Extracellular non-coding RNAs as Disease Biomarkers”, funded by the European Union - NextGenerationEU, under the National Recovery and Resilience Plan (NRRP) Mission 4 Component 2 Investment Line 1.1: “Fondo Bando PRIN 2022” (CUP G53D23002910006), ii) project MUSA - Multilayered Urban Sustainability Action - project, funded by the European Union - NextGenerationEU, under the National Recovery and Resilience Plan (NRRP) Mission 4 Component 2 Investment Line 1.5: Strengthening of research structures and creation of R&D “innovation ecosystems”, set up of “territorial leaders in R&D” (CUP G43C22001370007, Code ECS00000037), iii) project SERICS (PE00000014) under the NRRP MUR program funded by the EU - NextGenerationEU, iv) Università degli Studi di Milano under the program “Piano di Sostegno alla Ricerca”. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the Italian MUR. Neither the European Union nor the Italian MUR can be held responsible for them.