Data platforms are state-of-the-art solutions to implement data-driven applications and analytics, since they facilitate the ingestion, storage, management, and exploitation of big data. Data platforms are built on top of complex ecosystems of services answering diferent data needs and requirements; such ecosystems are ofered by diferent providers (e.g., Amazon AWS and Microsoft Azure). However, when it comes to engineering data platforms, no unifying strategy and methodology is available yet, and the design is mainly left to the expertise of practitioners in the field. Service providers simply expose a long list of interoperable and alternative engines, making it hard to select the optimal subset without a deep knowledge of the ecosystem. A more efective approach to the design starts from the knowledge of the data transformation and exploitation processes that should be supported by the platform. In this paper, we sketch a computer-aided design methodology and then focus on the selection of the optimal services needed to implement such processes. We show that our approach lightens the design of data platforms and enables an unbiased selection and comparison of solutions even through diferent service ecosystems.
A data platform is a centralized infrastructure that facilitates the ingestion, storage, management, and exploitation of large volumes of heterogeneous data. It provides a collection of independent and composable services meeting the end-to-end needs of data pipelines, where: centralized means that a data platform is conceptually a single and unified component; independent means that changes in the implementation of a service do not afect other services; composable means that services have interfaces that enable an easy and frictionless composition; and end-to-end means that services cover the entire data life cycle. Data platforms foster collaboration and shared governance (being centralized, data is unified following some integration and it is easier to ensure compliance with data protection and privacy laws through shared security and access control), and scalability (being implemented on a distributed infrastructure, it is easy to add storage and computing resources as needed).
When it comes to building data platforms, no unifying strategy and methodology is there yet: building data platforms is mainly left to the expertise of practitioners or consultancy companies.
Data Flow Diagrams
Service provider Designer Questionnaire
(Clients)
On the one hand, cloud service providers1 ofer ecosystems of services composed of many engines interoperable with each other; however, choosing the optimal set of services is hard since multiple solutions could fulfill the desiderata (e.g., whether disjoint databases and data warehouses or a single Lakehouse [1] should be used) and could require vertical knowledge on the design of data pipelines. On the other hand, several abstract big data architectures have been introduced (e.g., NIST [2], Lambda [3], and Kappa [4]). However, while they provide the necessary functionalities to enable big-data applications, their implementation and adoption require understanding which services should be used.
Neither providers of service ecosystems nor abstract architectures answer a crucial question: given an ecosystem of services and the data-driven processes to support, which is the optimal subset of services enabling such processes? We believe that to ease the data platform design, the description of data-driven processes should drive such activity. Indeed, data pipelines are the backbone of a data platform as they determine information-rich representations, and are congenial to designers since they encode many constraints on the choices to be made.
This paper discusses the following contributions (its extended version can be found at [5]): (i) a methodology to guide the selection of blueprints for (cloud) data platforms driven by clients’ processes; (ii) the formulation of such selection as an optimization problem where preferences, afinity, and further constraints can be applied; and (iii) the extraction of blueprints from real case studies and validation with experts from consultancy companies. The paper is organized as follows: Section 2 sketches the overall methodology; Section 3 and Section 4 describe and formalize the design steps; Section 5 evaluates the efectiveness and eficiency of the contribution; and Section 6 analyzes the related literature. Finally, Section 7 draws the conclusions.
Our methodology (Figure 1) involves three types of users: cloud service providers (IT experts with in-depth knowledge of services available in the ecosystem); data platform designers (consultants or people with expertise in designing data flows but with no vertical knowledge on cloud/service ecosystems) and clients asking for the design of the data platform blueprint (e.g., partners involved in the same European project with expertise on the applicative domain —such as precision agriculture— but no expertise on IT and architectures).
The methodology aims to assist designers in selecting the services necessary to implement the clients’ data pipelines out of the “unstructured” lists of services provided by cloud providers.
(
(
(
Example 1. Given a data repository, we ask the question: “What are the main types of collected data?” □ Sensor data □ Images □ Videos □ Satellite observations □ Tables Answering “Satellite observations” tags the repository with the properties (Volume, Big), (Data Model, File), and (Data Nature, Raster) (since earth observations are around 1 GB for 100 2) and tags the process to download such data as (Collection, Pull) since files are downloaded from an FTP server [7].
(
Once the DFD and service graphs are characterized by tags from the same taxonomies, it is possible to join them in a matched graph. A DFD process or repository matches (i.e., can be implemented by) a service only if the service has the same or more functionalities to fully implement it.
(
(
Case Study: Agritech Within the Agritech spoke of the NRRP European project [8], we deploy a data platform supporting prescriptive analytic tasks from 8 clients in the field of precision agriculture. We will use one of the partners as a working case study throughout the paper. Here follows a qualitative description of one of the clients’ data flows that the platform must support.
Example 2. The analysis entails a flow that is structured as follows. (i) Data comes from soil moisture sensor grids, weather stations, and SENTINEL-2 satellites; (ii) Sensor and weather data is uploaded every 15 minutes to the platform, while satellite data is periodically downloaded; (iii) Soil moisture data is interpolated using mathematical (e.g., bilinear interpolation) and machine learning (e.g., neural networks) techniques; (iv) Vegetation indexes are computed out of the raw satellite observations and integrated with the enriched sensor data; (v) Reports are periodically generated out of the integrated data; (vi) Given an optimal soil moisture matrix, the integrated data is used to decide how much to irrigate the soil.
{(Data Zone, Landing), (Data Model, Relational)} {(Data Volume, Small), (Computing, Batch)}
{(Computing, Batch), (Functionality, Machine Learning)} Consume
After gathering questionnaires from the clients, we (designers) iteratively refined the interview. Figure 2 depicts the tasks from Example 2 using the DFD formalism. More details are described in [9, 10].
Tags (Table 1) are organized as a collection of hierarchies that can be fueled bottom-up from the documentation of cloud services providers (i.e., the set of tags that are attached to each service or that are inferrable through the service description), and top-down from the literature; e.g., the big data V’s.
Definition 1 (Hierarchy). A hierarchy ℎ is a taxonomy of categorical values (ℎ). ≥ ℎ is the partial order that defines the taxonomy; ≥ ℎ indicates whether a value is more specific than another. We denote with the set of hierarchies.
Example 3 (Tags and hierarchies). Table 1 depicts an example of hierarchies = {Data Model, Volume, ...}. In the partial order of the hierarchy ℎ = Data Model it is, for instance, Relational ≥ Data Model Structured ≥ Data Model Data Model (All). Relational is more specific than Structured in the Data Model hierarchy.
The services available on a specific ecosystem difer for each provider, and no single and shared organization of services is available. Also, it is necessary to consider the relationships between services (e.g., whether the adoption of a service requires another) and preferences in their choice (e.g., due to diferences in performance reasons). That is why we map the services into a directed property graph.
Definition 2 (Directed property graph). A directed property graph is a tuple = (, , , ) where = {..., , ...} is a finite set of nodes, = {..., , ...} is a finite set of arcs connecting nodes and , = {..., (ℎ, ), ...} is a set of key-value properties with ℎ ∈ and ∈ Data Nature (All)
Spatial n4
EMR n2 GeoServer IsCompatible
Requires n1 S3 n3 EC2 Structured Semi-structured Unstructured Temporal
Relational Multidimensional Document Wide-column Key-value Graph File (text, binary) Vectorial
Raster {(Data Model, Structured), (Data Model, File), (Volume, All)} n5
Lakehouse {(Data Model, File),
(Volume, All), (Preferred, True)} (ℎ), and is a set of labels. : ( ∪ ) → returns the properties of a node or arc. : ( ∪ ) → returns the labels of a node or arc.
Definition 3 (Service graph). A service graph is a directed property graph where nodes are labeled as Service, while arcs are alternatively labeled as {Requires, IsCompatible, IsAkin}. The semantics of the labels is the following.
Service: is any engine from the service ecosystem that can be optionally tagged with the property (Preferred, True) to specify whether it should be considered more than others;
Requires: represents whether a service mandatorily relies on another;
IsCompatible: represents whether a service natively interfaces with another (i.e., their interaction is supported by default and does not require custom/additional libraries or connectors);
IsAkin: represents whether a service not only is compatible with another but also if their adoption together is encouraged.
An excerpt of the service graph from the AWS ecosystem is depicted in Figure 3.
We now describe three steps composing our approach. Match extracts the services that can support the execution of the data pipeline, Augment refines the potential candidates and, finally, EC2
Kinesis Post GIS RDS Lambda Sage Maker Churn Pred.
Athena
MSeonisstourrse WSetaattihoenr Satellite
Optimize picks the optimal (minimal) subset. We mainly focus on the optimization of the platform design rather than on how, starting from questionnaires, designers refine the DFD of such flows.
Since data pipelines are the backbone of data platforms, everything starts from the DFD describing the processes of clients’ analysis.
Definition 4 (Data Flow Diagram). A Data Flow Diagram (DFD) is a directed property graph where nodes are alternatively labeled as {Agent, Repository, Process}, while arcs are labeled as Flow. Arcs must connect with at least one Process node. The semantics of the labels is the following. Agent: an external entity that communicates with the system and stands outside of the system; Process: transform inputs to outputs; Repository: store data for later use; Flow: transfer data from one part of the system to another.
Matching the DFD and Service Graphs Given the service graph and the DFD, we can automatically join them by matching their tags. A match represents whether a DFD node can be implemented using a specific service (solid arcs in Figure 4); i.e., services that have the same or more generic values for the properties specified in the DFD node. For instance, a DFD repository tagged as Relational could be implemented by services supporting Relational or all Structured data.
Definition 5 (Matched graph). Given a DFD = ( , , , ) and a service graph = ( , , , ), a matched graph = ( ∪ , ∪ ∪ , ∪ , ∪ ∪ {ImplementedBy}) is a directed property graph obtained as the union of and . is an additional set of arcs, one for every match between nodes in and . Arcs in are labeled as ImplementedBy and can have an optional property (Mandatory, True) to specify that such implementation must be selected in the final blueprint.
Example 4 (Matched graph). Figure 4 depicts an excerpt of the matched graphs for the DFD from Figure 2; for the sake of clarity, not all the arcs have been presented. Nodes from the DFD have a ∑︁ such that
∈ {0, 1} for all ∈ , () = Service ∈ {0, 1} for all ∈ , ( ) = ImplementedBy ≥ for all
∑︁ = 1 for all ∈ , () ∈ {Process, Repository}
∈,()=ImplementedBy
≥ for all ∈ , ( ) = Requires
+ ℎ ≤ 1 for all ∈ , ℎ ∈/ , () = Flow, (ℎ) = IsCompatible
= 1 for all ∈ , (Mandatory, True) ∈ ( )
· ℎ ≥ ℎ for all , ℎ ∈ , () = Flow, (ℎ) = IsAkin
(
Augmenting and Pruning the Matched Graph Designing a platform is not just a technical task but encompasses various organizational, business, and political aspects that must be carefully considered. The successful creation of a platform requires the expertise and experience of designers to manipulate the matched graph and aid in the subsequent optimization. This ensures that the platform is not only technically sound but also aligns with broader strategic goals and user needs. Designers have several tools at their disposal to influence and refine the design process: (i) force the selection of specific services; (ii) prefer certain services over others; and (iii) refine the match by preliminarily removing services that do not meet specific criteria or adding services that were initially unmatched.
Example 5 (Mandatory and preferred services). With reference to Figure 4, imagining that our methodology is used to lift-and-shift an on-premises solution that already used GeoServer to AWS, GeoServer could be tagged with the property (Mandatory, True) since clients already have experience with it (and not with alternative engines such as GoogleEarth). Also, since S3 is cheaper than other storage services (such as EBS and EFS), S3 could be tagged with the property (Preferred, True).
Finally, this step supports the recognition of design patterns, detailed in the extended version [5], to ensure that the data platforms adhere to best practices and maintain a high standard maintainability.
Optimizing the Blueprint We use the minimization of the number of services as an objective function since this indicator correlates with both cost minimization and management simplicity.
More formally, given a matched graph = (, , , ), the selection of the optimal blueprint can be modeled as a linear programming problem inspired by the standard facility location problem. Services are mapped to the potential locations while DFD nodes are mapped to the demand points.
The formulation in Figure 5 reads as follows. (
Example 6 (Optimization). Given the matched graph from Figure 4, the optimal blueprint of the data platform is represented by the services highlighted in bold (with a cost of 7.5), namely Kinesis, Lambda, RDS, EC2, SageMaker, GeoServer, S3 (tagged as preferred), and Redshift. GeoServer has been selected since it is the only available implementation for Raw Images, EC2 and S3 have been selected since they are required by GeoServer. SageMaker has been selected and preferred to EMR and Glue since three DFD processes can be implemented on it.
Validation with real use cases We applied our approach to two real companies: one of the largest Italian consulting firms specialized in data analysis and cloud data platforms and an international company leading in the production of smart and digitalized sports equipment. We (i) asked their experts for two use cases of data platforms, (ii) generated a blueprint for each use case, and (iii) asked the experts to evaluate and compare our blueprints against the ones they implemented.
Blueprints efectiveness. The blueprints returned by our system are compatible with the ones actually adopted by the companies and their overlap is 85%. Although the blueprints are not perfectly overlapping, experts claim that our blueprints are correct from a technical and technological point of view. However, what could be improved in our approach is the integration of more organizational, political, or even familiarity constraints that could bias the service selection.
Methodology efectiveness. The requirement analysis was smooth and the staf from the two companies had no problem describing the data-processes and answering questions needed for tagging. Both companies acknowledged that selecting services for a project is highly complex and that staying up to date about service provider oferings requires considerable time and efort. Overall the experts were enthusiastic about the approach and highlighted the following strengths: (i) it can support and train young designers for the creation of blueprints, (ii) it can remove bias in the selection of services, and (iii) it is an objective approach to compare diferent blueprints.
Eficiency We test how our approach scales up to bigger cloud ecosystems and data pipelines by implementing synthetic DFDs with increasing nodes | | that share a service ecosystem of 200 services and we ensure that every node in the DFD is implemented by at least one service. With | | = 250 the total computational time is 7.76 seconds, of which 7.23 for match and 0.53 for selection. Matching could be improved by further optimizing its implementation on GraphDB.
Understanding which “building blocks” should be part of a data platform is nontrivial. NIST designed the Big Data Reference Architecture [2] which comprised vendor-, technology- and infrastructure-independent logical functional components that are necessary for managing and processing big data (data provider, data consumer, system orchestrator, big data application provider, and big data framework provider). Lambda [3] is an architecture designed to handle massive quantities of data by taking advantage of both batch and stream-processing methods. Kappa [4] overcomes some limitations (e.g., “redundant” batch and streaming implementations) of Lambda by using a pure streaming approach with a single code base. While these architectures define functional components necessary for (big) data platforms, they are abstract and do not address the problem of selecting the optimal services necessary for deploying working data platforms.
Cloud service providers such as Amazon [11], Google [12], and Microsoft [13] provide ecosystems of independent and interoperable services. While ecosystems enable easy deployment of data pipelines, choosing the optimal services is hard: (i) each provider ofers diferent services, (ii) a shared categorization and organization of services is missing, and (iii) it is hard for the users to understand which services are needed based on their data-driven processes [14]. Some multi-objective optimization techniques have been proposed, where users express requirements and preferences (e.g., minimal QoS) about single services [15, 16, 17], cloud deployment models (public and private) [18], and service providers [19, 20]. Finally, domain languages and ontologies have been introduced (e.g., [21]) to enable some service composition (e.g., [22]).
In this paper, we have discussed a methodology to aid designers in selecting the optimal services (out of a service ecosystem) supporting data-driven processes from multiple stakeholders (e.g., partners in a research project), and we have addressed such selection as a facility location optimization problem. Our approach can match and select single services as well as design patterns (e.g., Lakehouse to replace both data lakes and warehouses), it supports additional constraints (e.g., mandatory and afinity), and the produced blueprints have been successfully compared with the ones recommended by expert designers.
We believe that the expressiveness of the approach can still be improved as follows. Expressiveness: introduce more advanced constraints to support organizational and political factors in the design of the blueprints. Resource provisioning: additionally to selecting the services, a complete approach should also consider how many instances of a service are required (to do so, a cost model should be studied). Metadata integration: while catalog and meta-data management services do not directly introduce functionalities for data transformation and exploitation, the design should also recommend services helping in the management of the platform itself [23].
[12] Google cloud platform, https://cloud.google.com/, 2024. Accessed 2024-07-26. [13] Microsoft azure, https://azure.microsoft.com/en-us, 2024. Accessed 2024-07-26. [14] L. Sun, H. Dong, F. K. Hussain, O. K. Hussain, E. Chang, Cloud service selection: Stateof-the-art and future research directions, J. Netw. Comput. Appl. 45 (2014) 134–150. doi:10.1016/J.JNCA.2014.07.019. [15] E. Cavalcante, F. Lopes, T. V. Batista, N. Cacho, F. C. Delicato, P. F. Pires, Cloud integrator: Building value-added services on the cloud, in: NCCA, IEEE, Toulouse, France, 2011, pp. 135–142. doi:10.1109/NCCA.2011.29. [16] M. H. Nejat, H. Motameni, H. Vahdat-Nejad, B. Barzegar, Eficient cloud service ranking based on uncertain user requirements, Clust. Comput. 25 (2022) 485–502. doi:10.1007/ S10586-021-03418-W. [17] L. Lu, Y. Yuan, A novel TOPSIS evaluation scheme for cloud service trustworthiness combining objective and subjective aspects, J. Syst. Softw. 143 (2018) 71–86. doi:10.1016/ J.JSS.2018.05.004. [18] R. Garg, Mcdm-based parametric selection of cloud deployment models for an academic organization, IEEE Trans. Cloud Comput. 10 (2022) 863–871. doi:10.1109/TCC.2020. 2980534. [19] M. B. Kar, R. Krishankumar, D. Pamucar, S. Kar, A decision framework with nonlinear preferences and unknown weight information for cloud vendor selection, Expert Syst.
Appl. 213 (2023) 118982. doi:10.1016/J.ESWA.2022.118982. [20] A. Tomar, R. R. Kumar, I. Gupta, Decision making for cloud service selection: a novel and hybrid MCDM approach, Clust. Comput. 26 (2023) 3869–3887. doi:10.1007/ S10586-022-03793-Y. [21] M. Noura, M. Gaedke, Wotdl: Web of things description language for automatic composition, in: Int. Conf. on Web Intelligence, ACM, Thessaloniki, Greece, 2019, pp. 413–417. doi:10.1145/3350546.3352558. [22] H. B. Mahfoudh, A. Caselli, G. D. M. Serugendo, Learning-based coordination model for on-the-fly self-composing services using semantic matching, J. Sens. Actuator Networks 10 (2021) 5. doi:10.3390/JSAN10010005. [23] M. Francia, E. Gallinucci, M. Golfarelli, A. G. Leoni, S. Rizzi, N. Santolini, Making data platforms smarter with MOSES, Future Generation Computer Systems 125 (2021) 299–313. doi:10.1016/j.future.2021.06.031.