Data platforms are state-of-the-art solutions to implement data-driven applications and analytics, since they facilitate the ingestion, storage, management, and exploitation of big data. Data platforms are built on top of complex ecosystems of services answering diferent data needs and requirements; such ecosystems are ofered by diferent providers (e.g., Amazon AWS and Apache). However, when it comes to engineering data platforms, no unifying strategy and methodology is there yet, and the design is mainly left to the expertise of practitioners in the field. In particular, service providers simply expose a long list of interoperable and alternative engines, making it hard to select the optimal subset without a deep knowledge of the ecosystem. A more efective approach to the design starts from the knowledge of the data transformation and exploitation processes that should be supported by the platform. In this paper, we sketch a computer-aided design methodology and then focus on the selection of the optimal services needed to implement such processes. We believe that our approach lightens the design of data platforms and enables an unbiased selection and comparison of solutions even through diferent service ecosystems.
security and access control), and scalability (being implemented on a distributed infrastructure, it is easy to add Digital transformation is one of the most disruptive storage and computing resources as needed). trends of recent years. Digitization is perceived as the When it comes to building data platforms, no unifymost efective solution to innovate every industrial sec- ing strategy and methodology is there yet: building data tor thanks to process automation and the (automatic) platforms is mainly left to the expertise of practitioners exploitation of the value hidden in data. Such evolution in the field. On the one hand, cloud service providers 1 is pushing information systems towards complex ecosys- ofer ecosystems of services composed of many engines tems of data-oriented services answering diferent data interoperable with each other; however, choosing the opneeds and requirements. timal set of services is hard since multiple solutions could
A data platform is a centralized infrastructure that facil- fulfill the desiderata (e.g., whether disjoint databases and itates the ingestion, storage, management, and exploita- data warehouses or a single Lakehouse [1] should be tion of large volumes of heterogeneous data. It provides used) and could require vertical knowledge on the design a collection of independent and well-integrated services of data pipelines. On the other hand, several abstract meeting the end-to-end needs of data pipelines, where: big data architectures have been introduced (e.g., NIST centralized means that a data platform is conceptually a [2], Lambda [3], and Kappa [4]). However, while they single and unified component; independent means that provide the necessary functionalities to enable big-data changes in the implementation of a service do not afect applications, their implementation and adoption require other services; well-integrated means that services have to understand which services should be used. interfaces that enable easy and frictionless composition; Neither providers of service ecosystems nor abstract and end-to-end means that services cover the entire data architectures answer a crucial question: given an ecosyslife cycle. Data platforms foster collaboration and shared tem of services and the data-driven processes to support, governance (being centralized, data is unified following which is the optimal subset of services enabling such prosome integration and it is easier to ensure compliance cesses? Answering this question is hard since (i) each with data protection and privacy laws through shared provider ofers many (overlapping) services, (ii) diferent providers ofer diferent service categorizations that can hardly be mapped together, (iii) the evolution of cloud ecosystems is fast and stakeholders without vertical technical knowledge can hardly keep up with the pace, and
puting, cloud computing is proving to be a winning business model since deploying and maintaining such a variety of computational resources and assets requires advanced technical skills that companies hardly have “in-house”.
ology, and we mainly focus on steps (4) and (5) in the remainder of the paper due to space constraints.
Service provider Tags (1) Define the service ecosystem. Designer A cloud service provider identifies una tantum: (i) the alternative candidate services to compose the blueprint Formalize Enrich Match of the data platform, and (ii) a taxonomy of tags that describe and characterize such services. The identified Questionnaire services are organized in a service graph describing the FDloawtas Select preferences and dependencies between them as well as the tags characterizing each service. Following the assumption that cloud service providers do not provide two Figure 1: Overview of the methodology identical engines, our guideline in designing the taxonomy of tags is to have enough expressiveness (tags) to distinguish all services (i.e., no two services have exactly (iv) simply selecting “all” engines is not viable due to cost the same tags). and management reasons. (2) Formalize the requirements through a Data
We believe that to ease the data platform design, the Flow Diagram. description of data-driven processes should drive such When building the blueprint of a new data platform, activity. Indeed, data pipelines are the backbone of a data clients compile questionnaires that collect information platform, determine information-rich representations, about their data-driven processes and the main steps, suband are congenial to designers since they encode many jects, and goals of their analysis. Note that the data platconstraints on the choices to be made. form should support (possibly independent) processes
The paper is organized as follows: Section 2 sketches from multiple clients. the overall methodology; Section 3 describes the case Designers then refine the answers to the questionstudy used along the paper; Section 4 and Section 5 de- naires and formalize the processes into a Data Flow Diascribe and formalize the design steps; and Section 6 ana- gram (DFD); during the process, a few interviews with lyzes the related literature. Finally, Section 7 draws the the clients might be necessary. We choose the DFD forconclusions and future directions. malism since it represents flows of data through an information system at a high level of abstraction, emphasizing the movement and transformation of data while 2. Methodology Overview hiding details such as decision points and interactions: knowing which repositories and processes compose the Engineering a data platform is nontrivial. A data plat- data-driven processes is enough to return a blueprint. form must be designed upon a principle of modularity To build the DFD, designers decompose the data flows to be extensible with new components and functional- into agents, processes, and repositories. Starting from an ities. While it is true that reference architectures exist aggregated overview, our guideline is to recursively split (Section 6), not all their modules and services might be candidate processes and repositories until each of them is necessary. Indeed, running useless functionalities and characterized by homogeneous tags (e.g., if a repository services results in a waste of resources and money (e.g., contains both unstructured images and relational tables, in cloud environments where users pay for the running split it into two homogeneous repositories). time of the allocated resources). (3) Enrich the DFD with service tags.
To efectively build a data platform, we propose a In order to match the DFD with the services deployed methodology (Figure 1) that involves three types of users: in a cloud ecosystem, the two must share the same charcloud service providers (IT experts with in-depth knowl- acterization. Each process and repository in the DFD is edge of services available in the ecosystem); data plat- enriched with the tags from the taxonomies previously form designers (consultants or people with expertise in identified by the cloud service provider. To properly chardesigning data flows but with no vertical knowledge on acterize each process/repository (agents are not subjects cloud/service ecosystems) and clients asking for the de- of enrichment, since they are out of the scope of the platsign of the data platform blueprint (e.g., partners involved form), clients answer an additional set of questions; we in the same European project). recall that clients are not required to have vertical knowl
The methodology aims to assist designers in select- edge about computer and data science or engineering. ing the services necessary to implement the clients’ data Such questions are defined by the cloud service providers pipelines out of the “unstructured” lists of services pro- and driven by the tag taxonomies. Note that since our fivided by cloud providers. nal goal is selecting the most appropriate services, we are not interested in identifying and tracking the data/processes but rather their types and their flows.
Example 1. Given a repository from the DFD, we ask the question: “What are the main types of collected data?” □ Sensor data □ Images □ Videos □ Satellite observations □ Tables Answering “Satellite observations” tags the repository with the properties (Volume, Big), (Data Model, File), and (Data Nature, Raster) since earth observations are data-heavy ifles (e.g., around 1 GB for 100 2) and tags the process to download such data as (Collection, Pull) since files are downloaded from an FTP server [5].
(4) Match the DFD and service graphs.
Once the DFD and service graphs are characterized by Field valve DWH tags from the same taxonomies, it is possible to match them. A DFD process or repository matches (i.e., can Figure 2: DFD from Example 2 be implemented by) a service only if the service has the same or more functionalities to fully implement it.
(5) Select the optimal services. Example 3 (DFD). Figure 2 depicts the tasks from Exam
it is necessary to select the minimal blueprint of the data platform that covers all the DFD entities (the fewer the • Moisture Sensors and Weather Stations stream services, the lower the cost and management eforts). data into the platform, such data is Consumed and Furthermore, dependencies and compatibilities between stored in the original format into Sensor Data; services have to be taken into account. Consume
Within the Agritech spoke of the NRRP European project [6], we deploy a data platform supporting prescriptive analytic tasks from 8 clients (mainly research institutes) in the field of precision agriculture. We will use this as a working case study throughout the paper. Here follows a qualitative description of one of the clients’ data flows that the platform must support.
Example 2 (Analytics solutions to manage crops for optimum quality and sustainability). The analysis entails a lfow that is structured as follows. (i) Data comes from soil moisture sensor grids, weather stations, and SENTINEL-2 satellites; (ii) Sensor and weather data is uploaded every 15 minutes to the platform, while satellite data is periodically downloaded; (iii) Soil moisture data is interpolated using mathematical (e.g., bilinear interpolation) and machine learning (e.g., neural networks) techniques; (iv) The interpolated data is stored in a relational database with a spatial extension (PostGIS); (v) Vegetation indexes are computed out of the raw satellite observations and integrated with enriched sensor data; (vi) Reports are periodically generated out of enriched data; (vii) Given an optimal soil moisture matrix, the enriched data is used to decide how much to irrigate the soil.
After gathering questionnaires from the clients, we (designers) iteratively refined the interview into a DFD. • Satellite images are periodically Downloaded and stored into Raw Images; • Sensor Data are Enriched, interpolated, and stored into Integrated Data; • Raw Images are used for the computation of vegetation indexes and integrated with sensor data; • Integrated Data are used to fuel a Data Warehouse through ETL; • The Irrigation Optimization algorithm controls the
Field Valves installed in Emilia Romagna, Italy. Following the guidelines, the data collection processes cannot be merged into a single one since (i) Moisture Sensors streams data into the platform while Satellite images are periodically downloaded, and (ii) Sensor Data and Raw Images cannot be grouped into a single repository since they contain heterogeneous data types. More details on such tasks are described in [7, 8].
that can be fueled bottom-up from the documentation of cloud services providers (i.e., the set of tags that are attached to each service or that are inferrable through the service description), and top-down from the literature; for instance, the big data V’s [9], such as volume (small {..., , ...} is a finite set of nodes, = {..., , ...} is a finite set of arcs connecting nodes and , = {..., (ℎ, ), ...} is a set of key-value properties with ℎ ∈ and ∈ (ℎ), and is a set of labels. : ( ∪ ) → returns the properties of a node or arc. : ( ∪ ) → returns the labels of a node or arc.
a directed property graph where nodes are labeled as Service, while arcs are alternatively labeled as {Requires, IsCompatible}.
example of hierarchies = {Data Model, Volume, ...}. (41) = IsCompatible In the partial order of the hierarchy ℎ = Data Model it is, (1) = {(Data Model, File), for instance, Relational ≥ Data Model Structured ≥ Data Model
(Volume, All), (Preferred, True)} (2) = {(Data Model, File), (Data Nature, Raster)}
for each provider, and no single and shared organization GeoServer requires EC2 since it is deployed on it, EMR of services is available. Table 2 shows an example of is compatible with S3 since it can natively read from and services from the AWS website (as of November 2023). write to the object storage.
necessity to consider all of them at once. Indeed, it is 5. Process-Driven Match and important that the considered services cover the “basic” functionalities necessary to run a data platform such as Selection the ones proposed by NIST [2], among them, storage and processing engines. Given the DFD describing the clients’ analysis2, it is first
Also, it is necessary to consider the dependencies be- necessary to match such graph with the service graph tween services (i.e., whether the adoption of a service and then select the optimal subset of services. We recall requires another) and preferences in their choice (e.g., that the DFD is enriched with tags identified by the cloud due to diferences in performance reasons). This is why service provider (Table 1). we map the services into a directed property graph.
erty graph is a tuple = (, , , ) where =
design rather than on how, starting from questionnaires, designers refine the DFD of such flows. 5.1. Match the DFD and Service Graphs
Definition 5 (Node Match). Given a DFD and a service graph, a node of the DFD matches a node of the service graph if each property in () matches a property in ( ). A property (ℎ, ) matches another property (ℎ , ) if ℎ = ℎ and ≥ ℎ . EC2
Sensor Data Raw Images Churn Pred.
Athena
Lambda Kinesis Post
GIS Lambda Sage Maker
A match represents whether a DFD node can be implemented using a specific service (solid arcs in Figure 3); i.e., services that have the same or more generic values for the properties specified in the DFD node. For instance, a DFD repository tagged as Relational could be implemented by services supporting Relational or all Structured data. If no match is found for a DFD node, we force the match to a default (e.g., to a virtual machine where any functionality can be implemented) ensuring that all DFD nodes have at least one match.
Example 7 (Matched graph). With reference to Figure 3, given the node Raw Images from the DFD, and the nodes S3 and GeoServer from the service graph (Raw Images) = {(Data Model, File),
(Data Nature, Raster)} (GeoServer) = {(Data Model, File), (S3) = {(Data Model, File), (Data Nature, Raster)} (Volume, All)} two DFD entities (e.g., we use a Lakehouse to replace both a data lake and a data warehouse). Additionally, the optimization must be compliant with the following constraints.
Raw Images can be implemented by GeoServer but not in S3 since the former is natively capable of managing geographical raster images (while S3 would only provide storage for the images).
Definition 6 (Matched graph). Given a DFD = ( , , , ) and a service graph = ( , , , ), a matched graph = ( ∪ , ∪ ∪ , ∪ , ∪ ∪ {ImplementedBy}) is a directed property graph obtained as the union of and . is an additional set of arcs, one for each node match between ∈ and ∈ . Arcs in are labeled as ImplementedBy.
the union of the nodes, and the union of the arcs plus additional arcs that represent candidate implementations for the DFD processes/repositories. The label ImplementedBy represents whether a DFD process or repository can be implemented by a specific service.
Example 8 (Matched graph). Figure 3 depicts an excerpt of the matched graphs for the DFD from Figure 2; for the sake of clarity, not all the arcs have been presented. Nodes from the DFD have white background, while services are represented in blue. Solid arrows represent arcs labeled as ImplementedBy (e.g., Consume can be implemented by either Lambda or Kinesis). Dotted arrows represent arcs labeled as IsCompatible (e.g., SageMaker reads from and writes to Redshift). Dashed arrows represent arcs labeled as Require (e.g., GeoServer requires EC2 since it is deployed on it). Churn Prediction and Athena can be discarded a priori since they are not reachable from any DFD entity (i.e., they are neither candidate implementations nor required by other services). 5.2. Select the blueprint Out of the set of all possible matching services, only some of them must be selected such that (i) all the DFD repositories and processes are covered (agents are out of the scope of the platform) and (ii) the amount of services is minimized. The latter is an optimization goal that captures both the needs to minimize the economic cost of the platform and its management complexity. However, selection is not an easy task. For instance, some DFD entities can be implemented atop alternative services (e.g., a data lake that is implemented either on S3 or HDFS), and some services are meaningful only if they cover at least
services must be (recursively) selected too (e.g.,
Geoserver is deployed on EC2). 3. Compatibility: a service can be selected only if it is compatible with the services selected for the previous (if existing) and following (if existing) nodes in the DFD. 4. Preference: services marked as preferred should have more chances to be selected (for instance, preferences can be expressed for services entailing lower costs or higher performance). 5. External pattern injection: additional (architectural) constraints could be injected to force some service selection (e.g., use only services supporting Java language in compliance with legacy services); this can be implemented by pruning the unfeasible services from the matched graph.
Given a matched graph = (, , , ), the selection of the optimal blueprint can be modeled as a linear programming problem inspired by the standard facility location problem. The formulation in Figure 4 reads as follows.
(1) The optimization function minimizes the weighted sum of the selected services. Weights specify preferences for services . (2) are binary variables modeling services. = 1
if the service is selected, 0 otherwise. (3) are binary variables modeling the services implementing DFD processes and repositories. = 1 if ∈ s.t. () ∈ {Repository, Process} is implemented by ∈ s.t. ( ) = Service. (4) Binding variables and : service is selected if it implements ( = 1) a DFD node ∈ s.t. () ∈ {Repository, Process}. (5) Coverage constraints: every repository and pro
cess must be implemented by exactly one service. (6) Dependency constraints: if a service is selected, all the services it depends on must be selected. ∑︁
.. ∈ {0, 1} for all ∈ , () = Service ∈ {0, 1} for all ∈ , ( ) = ImplementedBy ≥ for all ∑︁
= 1 for all ∈ , () ∈ {Process, Repository} ,( )=ImplementedBy, ∈ ≥ for all ∈ , ( ) = Requires + ℎ ≤ 1 for all ( ∈ s.t. () = Flow), (ℎ ∈/ s.t. (ℎ) = IsCompatible) (1) (2) (3) (4) (5) (6) (7)
github.com/big-unibo/DataPlatformDesign) in Python and we leverage the CPlex library to efectively solve the optimization. As to preferences, we set
{︃0.5 if (Preferred, True) ∈ () Example 9 (Selection). Given the matched graph from Figure 3, the optimal blueprint of the data platform is represented by the services highlighted in bold (with a cost of 6.5), namely Kinesis, PostGIS, EC2, SageMaker, GeoServer, S3, Redshift. PostGIS has been selected since it is the only available implementation for Sensor Data, while EC2 has been selected since it is required by PostGIS (similarly for GeoServer). SageMaker has been selected and preferred to Lambda, EMR, and Glue since four DFD processes can be implemented on it.
(7) Compatibility constraints: if two services are in- “redundant” batch and streaming implementations) of compatible, they cannot be selected to implement Lambda by using a pure streaming approach with a sintwo consecutive processes/repositories linked gle code base. While these architectures define functional through a data flow in the DFM. components necessary for (big) data platforms, they are abstract and do not address the problem of selecting the optimal services necessary for designing, implementing, and deploying working data platforms.
Cloud service providers such as Amazon [10], Google [11], and Microsoft [12] provide ecosystems of independent and interoperable services. While ecosystems enable 1.0 otherwise easy deployment of data pipelines, choosing the optimal services is hard: (i) each provider ofers diferent services, (ii) a shared categorization and organization of services is missing, and (iii) it is hard for the users to understand which services are needed based on their data-driven processes [13]. Some multi-objective optimization techniques have been proposed, where users express requirements and preferences (e.g., minimal QoS) about single services [14, 15, 16], cloud deployment models (public and private) [17], and service providers [18, 19]. Finally, domain languages and ontologies have been introduced (e.g., [20]) to enable some service composition (e.g., [21]).
proach are (i) the introduction of a methodology to enable data platform design, (ii) a computer-aided approach to support designers in their choices, and (iii) processdriven optimization to select the optimal blueprint out of many data flows, even from multiple stakeholders.
tions that focus on its functions and their interactions) have been proposed. NIST designed the Big Data 7. Conclusion and Future Works Reference Architecture [2] which comprised vendor-, technology- and infrastructure-independent logical func- The design of data platforms is a nontrivial task. In this tional components that are necessary for managing and paper, we have introduced a methodology to aid designprocessing big data (data provider, data consumer, sys- ers in selecting the optimal services (out of a service tem orchestrator, big data application provider, and big ecosystem) supporting data-driven processes from multidata framework provider). Lambda [3] is an architec- ple stakeholders (e.g., partners in a research project), and ture designed to handle massive quantities of data by we have addressed such selection as a facility location taking advantage of both batch and stream-processing optimization problem. methods. Kappa [4] overcomes some limitations (e.g., Although our implementation delivered a valuable blueprint, the result lends itself to improvement in multi- cedia computer science 48 (2015) 319–324. doi:10. ple aspects. Expressivenes: matching and selection should 1016/j.procs.2015.04.188. consider more complex architectural patterns (Lakehouse [10] Amazon Web Services, https://aws.amazon.com/, to replace both data lakes and warehouses) as well as 2023. Accessed 2023-10-31. support additional constraints (e.g., consider only some [11] Google Cloud Platform, https://cloud.google.com/, service vendors or programming languages). Resource 2023. Accessed 2023-10-31. provisioning: additionally to selecting the services, a com- [12] Microsoft Azure, https://azure.microsoft.com/ plete approach should also consider how many instances en-us, 2023. Accessed 2023-10-31. of a service are required (to do so, a cost model should be [13] L. Sun, H. Dong, F. K. Hussain, O. K. Hussain, studied). Metadata integration: while catalog and meta- E. Chang, Cloud service selection: State-of-thedata management services do not directly introduce func- art and future research directions, J. Netw. Comput. tionalities for data transformation and exploitation, the Appl. 45 (2014) 134–150. doi:10.1016/J.JNCA. design should also recommend services helping in the 2014.07.019. management of the platform itself [22]. User evaluation: [14] E. Cavalcante, F. Lopes, T. V. Batista, N. Cacho, F. C. the produced blueprints should be compared with the Delicato, P. F. Pires, Cloud integrator: Building ones recommended by expert designers. value-added services on the cloud, in: NCCA, IEEE, Toulouse, France, 2011, pp. 135–142. doi:10.1109/ NCCA.2011.29.
References [15] M. H. Nejat, H. Motameni, H. VahdatNejad, B. Barzegar, Eficient cloud service [1] M. Zaharia, A. Ghodsi, R. Xin, M. Armbrust, Lake- ranking based on uncertain user requirehouse: A new generation of open platforms that ments, Clust. Comput. 25 (2022) 485–502. unify data warehousing and advanced analytics, in: CIDR, www.cidrdb.org, 2021, p. 8. URL: http:// [16] dLo.Li:1u,0Y..1Y0u0a7n/,SA10n5o8v6el-T0O21P-SI0S3e4v1a8lu-aWt.ion scheme cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf. for cloud service trustworthiness combining objec[2] W. Chang, N. Grady, N.-P. NIST, NIST big data tive and subjective aspects, J. Syst. Softw. 143 (2018) interoperability framework: Volume 2, big data 71–86. doi:10.1016/J.JSS.2018.05.004. taxonomies [version 2], 2018. doi:https://doi. [17] R. Garg, Mcdm-based parametric selection of org/10.6028/NIST.SP.1500-2r1. cloud deployment models for an academic organiza[3] M. Kiran, P. Murphy, I. Monga, J. Dugan, S. S. Baveja, tion, IEEE Trans. Cloud Comput. 10 (2022) 863–871.
speed big data processing, in: BigData, IEEE, Santa [18] dMo.i:B1.0.K1a1r,0R9./KTCriCs.h2an0k2u0m.2a9r,8D05.3P4a.mucar, S. Kar, Clara, CA, USA, 2015, pp. 2785–2792. doi:10.1109/ A decision framework with nonlinear preferences BIGDATA.2015.7364082. and unknown weight information for cloud vendor [4] J. Kreps, Questioning the Lambda Architecture, vol- selection, Expert Syst. Appl. 213 (2023) 118982.
ume 2, O’Reilly, 2014. [5] Earth Observation Guide, https://business.esa.int/ [19] dAo.iT:1o0m.a1r0,1R6./RJ..KEuSmWAa.r,2I0.2G2u.p1t1a8,9D82e.cision maknewcomers-earth-observation-guide, 2020. Ac- ing for cloud service selection: a novel and hybrid cessed 2023-10-31. MCDM approach, Clust. Comput. 26 (2023) 3869– [6] Agritech, https://agritechcenter.it/, 2023. Accessed
2023-10-31. [20] 3M8.8N7.oduoria:1,0M..1G00a7ed/kSe1,05W8o6t-d0l:2W2-e0b3o7f9t3h-inYg.s de[7] M. Francia, J. Giovanelli, M. Golfarelli, Multi- scription language for automatic composition, in: sensor profiling for precision soil-moisture mon- Int. Conf. on Web Intelligence, ACM, Thessaloniki, itoring, Computers and Electronics in Agriculture Greece, 2019, pp. 413–417. doi:10.1145/3350546. 197 (2022) 106924. doi:10.1016/j.compag.2022. 3352558.
106924. [21] H. B. Mahfoudh, A. Caselli, G. D. M. Serugendo, [8] E. Baldi, M. Quartieri, G. Larocca, M. Golfarelli, Learning-based coordination model for on-the-fly M. Francia, J. Giovanelli, E. Xylogiannis, M. Toselli, self-composing services using semantic matching, J. Smart irrigation system for precision water man- Sens. Actuator Networks 10 (2021) 5. doi:10.3390/ agement: efect on yield and fruit quality of yellow fleshed kiwifruit in northern italy, in: ECPA, [22] JMS.AFNra1n0c0i1a,0E0.0G5a.llinucci, M. Golfarelli, A. G. Leoni, Wageningen Academic Publishers, 2023, pp. 59–66. S. Rizzi, N. Santolini, Making data platforms doi:10.3920/978-90-8686-947-3_5. smarter with MOSES, Future Generation Com[9] J. Anuradha, et al., A brief introduction on big data puter Systems 125 (2021) 299–313. doi:10.1016/j.
future.2021.06.031.