With the growth of the Internet of Things and the rapid progress of social networks, everything appears to generate data. The ever-increasing number of connected devices is accompanied by a growth of the volume of data, produced at an ever-increasing rate, and this massive flow includes data types that are dificult to process using standard database techniques. One of the most critical scenarios is healthcare, whose activities need to store and manage a variety of data types - reports written in natural language, medical images, genomic data and waveforms of vital signs - which do not have a well-defined structure. In order to benefit from this large amount of complex data, Data Lakes have recently emerged as a solution to grant central storage and flexible analysis for all types of data. However, there is no Data Lake architecture that fits all the possible scenarios, since the architecture depends heavily on the application domain and, so far, there are no Data Lake architectures that support the specific needs of the healthcare domain. This work proposes HEALER: a Data Lake architecture that efectively performs data ingestion, data storage, and data access with the aim of providing a single central repository for eficient storage of diferent types of healthcare data. The architecture also enables the analysis and querying of the data, which can be loaded into the Data Lake regardless of their format and type. To verify the efectiveness of the architecture, a proof-of-concept of HEALER has been developed, that allows ingestion of various data, performs waveforms processing to make them more interpretable to researchers and analysts, grants access to the saved data and allows the analysis of natural language reports. Finally we studied the performance of the system in each of its main phases: ingestion, processing, data access and analysis. The results lead us to some important considerations to be taken into account when using and configuring the system components.
for precision treatments [1]. Valuable data about patients and medications are usually digitized and saved as ElecWith the evolution and growing popularity of social me- tronic Health Records (EHR). EHR on a large scale enable dia platforms and, in particular, with the birth of the researchers to identify opportunities to move healthcare Internet of Things (IoT), the data trafic has now reached organizations toward personalized healthcare, that conthe zettabyte threshold. For an increasing number of or- sists in using diagnostic tests to determine which medical ganizations, it is therefore critical to ingest, process and treatments will work best for each patient [2]. Typically, save the most important aspects of the generated data only a portion of EHR contain data in a structured forwhile removing the unnecessary parts. mat that data scientists can easily use, while most of
One of the areas that contribute significantly to the them consist in unstructured and semi-structured data generation of this data is healthcare. In fact, for the (i.e., with a form of structure that does not follow the progress of medical research, and healthcare in general, classical tabular one of relational models). It is thereit is of paramount importance that the data collected from fore necessary to develop an eficient data administration patients are saved for immediate and future batch anal- pipeline to assist scientists in their eforts to provide yses, facilitating the identification of diseases allowing customized prescriptions. An answer to this problem is represented by Data Lakes, that, according to [3] are a DataPlat’23: 2nd International Workshop on Data Platform Design, current and increasingly emerging trend for Big Data Management, and Optimization, March 28, 2023, Ioannina, Greece storage and management. However, despite the presence * Corresponding author. of well-designed examples of Data Lake architectures in $ carlo.manco@mail.polimi.it (C. Manco); the literature [4, 5], none of them has been specifically (tFo.mAmzazasoli.ndio);lcein@ripcool.ibmari.bitie(rTa.toD@olucin)i;cfaatbti.iot.a(Ezz.aBlianrib@iepraotloim);i.it designed to meet the requirements of the healthcare scemarco.gribaudo@polimi.it (M. Gribaudo); letizia.tanca@polimi.it nario. (L. Tanca) The goal of this work is to implement a proof-of0000-0002-1403-7766 (T. Dolci); 0000-0003-0631-2120 concept of a Data Lake for the healthcare scenario, that (F. Azzalini); 0000-0003-1466-0248 (E. Barbierato); can ingest and process data from diferent types of (0L0.0T0-a0n0c0a2)-1415-5287 (M. Gribaudo); 0000-0003-2607-3171 sources (medical devices, clinical trial datasets etc.). Data © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License must be saved in their raw format inside the Data Lake CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) storage area, and subsequently be processed and trans- Denmark, with integrated AI and machine learning tools formed in an easily usable way by scientists and ana- to analyze them. FEDDL proposes a solid architecture lysts. The resulting system, named HEALER (HEalthcare and provides a technical overview of the whole structure. dAta LakE aRchitecture), focuses on allowing analyses us- In fact, their findings played an important role for the ing both machine learning algorithms and conventional definition of our architecture. queries. Since healthcare data are very heterogeneous, In the healthcare field, many cloud-based solutions for the moment we decided to limit our analysis to wave- ofered by cloud providers such as AWS (Amazon Web forms of various signals extracted from patients and to Services) and Azure are emerging [11] and some Data medical reports written in natural language, in addition Lake have been implemented leveraging their services. to structured data management. These types of data, al- A medical Data Lake is proposed in [12], whose authors though being a limited part of the ones usually considered suggest a design that relies on cloud services. Similarly, in healthcare, allow to characterize a complete workflow [13] proposes an IoT-Cloud–based framework for realand thus to exemplify the proposed architecture. time processing of Big Data in the healthcare domain. It
The contributions of this work include: a) the design is developed implementing AWS. Also [14] describes a and definition of HEALER, a Data Lake architecture that cloud architecture that make use of Azure functionalities. can efectively manage both structured and unstructured Here, huge amounts of data are eficiently stored using medical data; b) the implementation of a proof-of-concept Azure Data Lake, to support physicians in detecting heart of HEALER on the basis of the proposed architecture, diseases. focusing on the management of waveforms and natural However, the main challenge of cloud technology aplanguage reports; c) testing and performance analysis of plications in the healthcare sector is the ownership of the various components of the system. sensitive data. Proper security measures must be implemented to protect data at each level of data management.
Despite cloud providers ofering various security ser2. Related Work vices, it is not always clear how to integrate them with the proposed architecture. Moreover, working on proprietary and not on open source software greatly limits the customization of the solutions, and in addition cloud providers usually require monetary payments based on software usage and the hardware made available.
Recently, despite the absence of a full-fledged Data Lake for healthcare, researchers are working towards this direction, for instance by developing query engine components for clinical data [15, 16]
the literature, but most of them do not qualify as comprehensive solutions for the described scenario. The Data Lake is defined in both the scientific and industrial worlds as a repository storing raw data in their native format; however, diferent definitions have diferent emphases, for example in [6] governance and metadata management are highlighted, while [7] places importance on the users, presenting an architecture focused on researchers and analysts. Furthermore, the design of a Data Lake is strongly influenced by the kind of scenario in which it is 3. Goals and Requirements placed.
Many Data Lakes designed for non-healthcare scenar- In order to retrieve valuable insights and helpful
knowlios present a robust architecture, but disregard certain re- edge easily from data, healthcare researchers require a
quirements that are essential for healthcare. For instance, unified system [17] that allows to:
CoreDB [8] is an open-source Data Lake to organize,
index and query data and metadata, but does not guar- • manipulate, process, and analyze diferent types
antee the storage of data in their native format. Hydria of data, from fully structured to unstructured,
[9] is a Data Lake for cultural heritage data: it provides allowing their storage inside the system in raw
an integrated framework that enables easy deployment native format;
of data acquisition services, dataset sharing with other • handle diferent types of data efectively, with
stakeholders, search, filtering and analysis of data via vi- open possibilities for integrating information
sualization tools. Constance [
Storage Layer Transformation Layer
Interaction Layer • meet clinical requirements by allowing fast and
eficient analysis of new combinations of data; • give the rights to design how to access the data: access must be regulated for protection of personal data and privacy.
As a consequence, a Data Lake is the most appropriate framework to adequately manage all these factors. In fact, Data Lakes can provide a unified platform for all relevant data generated by healthcare systems, operating as a repository for both structured data collected from traditional databases and unstructured data derived from various other sources. Data Lakes are extremely fast and versatile because they implement a scalable architecture to store data in their native form, while remaining easily accessible and centralized for end users. Furthermore, Data Lakes can be fully equipped with various security layers to ensure data integrity and compliance with privacy and regulations, which is particularly important in the healthcare context.
This work focuses primarily on the definition of the physical components of a data lake architecture, and conceptual aspects such as security and privacy are left for future work since their components are outside the main data management pipeline [18]. cleaning and data transformation, in order to achieve a predefined final form. In this layer, diferent data representations are created in order to transform raw data into easily accessible data formats, based on algorithms’ and users’ needs.
Finally, the Data Interaction layer provides end-users 3.1. Multi-layered Architecture with access to the data created in the transformation layer. Users can access data in order to perform exploration In our case study, we want to define, as a first step, a tasks, create and apply analytical queries and visualize functional implementation similar to [19], which pro- the stored data using various visualization tools. poses a multi-layered approach based on incorporating layers with separation of concerns. In fact, this type of architecture has the advantage of clearly highlighting 4. HEALER’s Architecture and the functions to implement for a given Data Lake, which Implementation allows for an easy matching to the corresponding required technologies: the key concept is that each layer 4.1. Overview of the Architecture communicates with the adjoined ones, and the data follow a pipeline over all the layers. Figure 1 illustrates the Data Lake architectures are usually classified into Data specific layers considered in the proposed architecture. Pond Architectures and Zone Architectures [20, 21]. In
In particular, starting from the top of the diagram, the the healthcare domain, as suggested in [22], a Zone ArData Ingestion layer has the task of ingesting heteroge- chitecture is preferred. In fact, data should go through neous data in raw format from various data sources and diferent levels of refinement while maintaining a copy into the Data Lake. The two main options to load data of the data in their raw format, characteristics that Pond into a Data Lake are batch and streaming. The choice of Architecture does not provide [23, 20]. These functionimplementing a batch-oriented or stream-oriented data alities are essential for healthcare researchers to allow ingestion layer depends very much on the context to them and analysts to perform more than a single transwhich the Data Lake is applied: in fact, as the context formation and analysis on the same data. More in detail, changes, we may have more or fewer data sources pro- the proposed architecture is composed of five diferent viding streaming data. zones: Transient Landing Zone, Raw Zone, Process Zone,
The Data Storage layer is the core of the Data Lake. It Refined Zone and Consumption Zone (Figure 2). contains raw-data repositories, but also transformed data; Figure 3 shows a high-level view of the proposed it provides support for diferent forms and structures of system, illustrating the main layers of the architecture, the data, including file storage and raw-record storage. where each layer is represented by its corresponding
The Data Transformation layer provides the poten- component. Data are first ingested by the Ingestion comtial for the scalable execution of operations such as data ponent in the Transient Landing Zone, and then permanently saved in the Raw Zone by the Data Storage ticular, it contains two zones, namely the Raw Zone, i.e., Component. Then, they are processed inside the Process the area where the data are permanently stored in their Zone in diferent ways according to their format. More- native raw format, and the Refined Zone, where data are over, the output of the transformation performed by the stored in a form suitable for the end-users and where Processing Component, known as "refined data", is stored processed data are saved. In order for scalability to be in the Refined Zone. The Interaction Component is the guaranteed, this component should be able to increase module that deals with data analysis and querying; it has its storage capacity when needed. free access to both the Raw and Refined zones of the Data The Processing Component represents the Process Storage, and produces data in the Consumption Zone. Zone. This is where data are brought to be prepared
From Figure 3 it can also be observed that each com- and processed. In our case, we manage time series data ponent is associated with at least one zone. In particular, before saving them in the Refined Zone, exposing them the Data Ingestion Component represents the Tran- to a transformation process that makes them more easily sient Landing Zone: batch and streaming data land here interpretable by the final user. after being extracted from the data sources and pushed Finally, the Interaction Component contains the into the Data Lake. Data are transferred without any Consumption Zone that is in charge of granting access transformations, in accordance with the first steps in to the data in the Refined and in the Raw Zone. Here the Extract-Load-Transform (ELT) paradigm. Vital signs users can perform queries on structured data by directly from hospitalized patients, reports written in natural lan- using Python libraries, or apply various data analysis guage, and structured data on patients and lab events are techniques. For instance, reports written in natural lanall ingested regardless of their format. In this work, we guage can be subject to various language-understanding focused on a batch-type data ingestion. tasks. Results obtained from this component can be re
The Storage Component represents the central repos- turned to the Raw Zone for later use. Saving results is itory where large volumes of medical data are stored. very important to allow further analysis: for example, This module must provide a distributed file system while keywords extracted from reports can later be mined to guaranteeing high availability and fault tolerance. In par- find patterns or apply clustering techniques.
a Data Lake [24]. In HEALER, two are the key tools: a ifle system responsible for managing data collection in a distributed storage, and a tool for performing ELT-type data ingestion that can easily connect to various data sources and the distributed storage.
In particular, to fulfill the file system requirements expressed in Section 3, we choose HDFS (Hadoop Distributed File System) [25]. In fact, HDFS provides highperformance access to data across highly scalable Hadoop clusters, and it can handle big volumes of both structured and unstructured data. Additionally, it is possible to specify the number of copies of a file that should be maintained by HDFS, guaranteeing the replication of data. In each cluster there is a single main node, called Namenode, and a variable number of secondary nodes which are called Datanode.
Secondly, we selected Apache NiFi [26] as tool for data ingestion. NiFi has many advantages compared to its competitors [27]. Among them, it provides both batch and real-time data transfer, it can easily interact with the Hadoop ecosystem and particularly with HDFS, it supports the most widely used communication protocols, and it scales linearly to many nodes. Additionally, every node of NiFi provides the same functionality as every other node: this makes the system very robust to node failures.
the Transient Landing Zone. This is where raw data are extracted from the sources to start their path inside HEALER. This phase ends when the raw data is placed within the storage area provided by HDFS. The entire process is handled by Apache NiFi. The flow diagram in Figure 4 shows the two NiFi processors used to manage the ingestion phase: the GetFile processor fetches Scenario 1 In this scenario, the data requested by the data from the data source and then hands them to the user have already been processed and therefore they are PutHDFS processor; PutHDFS establishes the connection already present in the Refined Zone. In this case, the to HDFS and inserts data into the distributed storage. request is immediately addressed and the data are directly Once distributed memory is reached, data are divided retrieved from the Refined Zones. into blocks and replicated in diferent Datanodes for a number of times equal to the replication factor, in our Scenario 2 This scenario involves real-time processing case equal to three. HDFS ensures that this operation of the data requested by the user, due to the fact they is performed automatically to increase data availability have not been processed yet. In this case, the data are and make the system more robust to individual Datanode searched in the Refined Zones; however, some, or all of failures. the data are not available in the Refined Zone and thus
At this point, the raw data saved within the distributed the request must be forwarded to the Raw Zone. If the storage are ready to be processed. This task is performed requested data are found in the Raw Zone, they are sent inside the Process Zone, and the processing procedure to the Process Zone, ready to be processed in real-time. depends on the data format and type; in this context, After the processing, following the normal flow of data, we discuss the processing of time series extracted from
MIMIC-III [28], a medical dataset that contains both structured and unstructured data, such as information on patients’ vital signs in waveforms and discharge notes in natural language. In particular, we define a processing tasks for waveforms, meant to transform them into a more easily interpretable format. This task involves the addition of information - such as the identifier of the patient and the starting date and time of the measures to the file containing the signal measurements. Details on time series processing are presented in Appendix A.
The process takes advantage of the hdfs Python library1, used to handle communication with the storage, not only when the processor node needs to take data to be processed, but also when there is the need to rewrite data into the Refined Zone after processing. For what concerns the execution of data requests in the system, the following two scenarios were studied.
• a single analysis node, with access to the GPU, equipped with useful libraries to perform analysis and to access data (Consumption Zone).
We developed a proof-of-concept of HEALER, implement- Database [28]. MIMIC stands for Medical Information ing the considered nodes on several container based vir- Mart for Intensive Care, and is a large, freely-available tual machines, running on a single physical machine database comprising de-identified health-related data asusing a quad core CPU Intel Core i7-4790S 3.2 GHz, 16 sociated with over forty thousand patients who stayed GB RAM, GPU NVIDIA GeForce GTX 745, and 2 TB Hy- in critical care units. brid HDD. Despite the hardware limitations, the system It includes a structured clinical database and a related was perfectly suited for our tests. The technical overview waveform database [31]. The clinical database includes of the system is shown in Figure 6. The system was tested 26 structured tables, with one of them (named NoteEvents) to ensure the architecture’s efectiveness and also to eval- including medical reports written in natural language. In uate the components’ compatibility and communication. the waveform database, records typically contain ten or Listed below are the specific nodes implemented: more time series of patients’ vital signs sampled once per second or once per minute. Each of this records is composed of a header file, which shows information about the measurements (number of samples, start time of the measurement session, description about the observed signals) and a matching signal file. • a single ingestion node implementing Apache
NiFi and representing the Transient Landing
Zone; • a single node implementing an HDFS Namenode
for managing data storage; • a cluster of five nodes, each implementing an
HDFS Datanode instance, representing the Storage Component and incorporating both Raw Zone and Refined Zone;
of HEALER, focused on the execution of the main stages. • a single process node to execute waveform pro- 2https://www.docker.com
cessing (Process Zone); 3https://kubernetes.io
The ingestion phase transfers data to the HDFS clus- Table 2 ter using Apache NiFi. In particular, we evaluate the Execution time and transfer rate of the processing phase. ingestion time of structured data (from MIMIC III clinical database) and waveforms (from MIMIC III waveform Batch (MB) Process Time (s) Rate (MB/s) database). The experiment begins when the flowchart 9 9.0 1.00 described in Figure 4 is executed, and it ends when all 127 120.0 1.06 the files saved in the Apache NiFi node are ingested into 305 290.0 1.05 the system. Data of diferent batch sizes are transferred, 1024 1034.0 0.99 repeating measurements 20 times for each batch size. Table 1 shows that the average transfer rate goes from 2.20 MB/s for a batch size of 11 MB, to 13.35 MB/s for 3.7 data batch size. Therefore, we reach the conclusion that GB. Thus we infer that, as the batch size increases, the the batch size of has minimal efect on the processing transfer rate also becomes higher. rate.
The processing node in our simulation is in charge of Concerning the access phase, the main objective is to running a Python script that takes a batch of waveforms allow users to access data from the Storage Component. in order to perform the transformation process. HEALER Our simulation involves the evaluation of both scenarios makes use of the pandas library to perform processing on described in Section 4.3, i.e., we evaluate the amount of waveforms and the hdfs library to handle communication time that the analysis node takes when data is already with the Storage Component. Our evaluation in this case in the Refined Zone, and when it is not. We consider involves measuring the time taken by the node to fetch data of diferent sizes and assess the eficiency of the batches of waveforms from storage, transform them, and system based on response time. Table 3 clearly shows ifnally save them into the Refined Zone. We perform the that requests of data not yet processed are much more processing of diferent batch sizes, obtaining the results time-consuming than those that have already been prodescribed in Table 2. Evidently, the system always per- cessed and stored in the Refined Zone. In fact, the data form with an average rate of 1 MB/s, regardless of the access phase is very dependent on the speed of real-time processing: since the processing phase is the slowest step inside the pipeline, accessing data in the Raw Zone is also very slow.
Finally, we evaluated the extraction of keywords from natural language reports. In this experiment, the goal is to compare two diferent Python libraries: Spacy [ 29] and KeyBERT [30]. The main diference between the two libraries is that KeyBERT leverages the GPU, while Spacy does not. In our proof-of-concept, it is the analysis node that performs the extraction operation from the reports. The extraction performance of the two libraries is analyzed on diferent samples of reports extracted from MIMIC-III NoteEvents table. We perform the tasks first on a set of 20 reports, then on a set of 200, using both libraries. Each experiment is repeated ten times. Spacy takes, on average, 16s to analyze 20 reports and 157s for 200 reports, while KeyBERT takes 27s to analyze 20 reports and 233s for 200 reports. With KeyBERT, GPU utilization peaked at over 50%, therefore, with a higher performing unit, it might be advantageous to use this library. On the contrary, if the system lacks a GPU, Spacy will be more eficient in terms of execution time.
to a variety of types of data in the context of healthcare. HEALER allows for analysis and query, and can manage data regardless of their generation speed or volume. In HEALER, the main emphasis is placed on the storage of non-structured data, such as medical waveforms and natural language reports, which are seldom considered in traditional systems.
Moreover, we implemented a proof-of-concept of HEALER, composed by a Data Ingestion Component, a Storage Component, a Processing Component and a Data Access and Interaction Component. We tested the system for a general evaluation of its various stages: ingestion, processing, data access, data analysis. From our experiments, we discovered that both HDFS and Apache NiFi perform well at high batch sizes. In addition, the study showed how the processing phase represents the bottleneck of HEALER and how it also impacts the data access phase in responding to user requests.
For future work, the plan is to expand the system by introducing horizontal layers, such as a layer for Data Governance, to manage metadata and enable the system to be more organized and faster when accessing data. Furthermore, an additional layer to manage user access and ensure data security is also planned, considering the extreme importance of securing the large amount of personal information present in medical data.
Last but non least, in a more realistic context, it is appropriate to test HEALER with a streaming-type ingestion of data, and to expand the Process Zone to a cluster of nodes performing data transformations at a distributed level. To perform distributed operations, we will use an appropriate tool, such as Apache Spark [32].
Finally, we plan to perform workload tests on HEALER with both real-world and simulated data, to obtain more realistic results on response time in diferent workload situations.
the simulation, we observe how Apache NiFi and HDFS Acknowledgments work very well with large batches of data. On the other hand, it is not recommended to use small batch sizes due This work has been partially supported by the Health Big to the risk of lower transfer rates. Moreover, regarding Data Project (CCR-2018-23669122), funded by the Italian HDFS, due to its approach of dividing data into fixed-size Ministry of Economy and Finance and coordinated by blocks, it is suggested to not ingest a large amount of the Italian Ministry of Health and the network Alleanza data with a much smaller size than the block size. This is Contro il Cancro. to avoid flooding the Datanodes with unfilled blocks. Additionally, we are grateful to Giuseppe Serazzi for
Lastly, for what concerns the access phase, it is appro- his advice during the definition of this work and the priate to process larger data asynchronously with respect support in the revision of this paper. to user requests. Instead, real-time processing should be left for accessing data of smaller size.
Data Lake that allows for eficient data storage and access [1] V. Jagadeeswari, V. Subramaniyaswamy, R. Logesh,
V. Vijayakumar, A study on medical internet of
things and big data in personalized healthcare
system, Health information science and systems 6 [14] E. Maini, B. Venkateswarlu, A. Gupta, Data lake-an
(2018) 1–20. optimum solution for storage andanalytics of big
[2] N. V. Chawla, D. A. Davis, Bringing big data to data in cardiovascular disease prediction system,
Inpersonalized healthcare: a patient-centered frame- ternational Journal of Computational Engineering
work, Journal of general internal medicine 28 (2013) & Management (IJCEM) 21 (2018) 33–39.
660–665. [15] T. Heinis, A. Ailamaki, Data infrastructure for
med[3] P. P. Khine, Z. S. Wang, Data lake: a new ideology in ical research, Now Publishers, 2017.
big data era, in: ITM web of conferences, volume 17, [16] G. M. Weber, S. N. Murphy, A. J. McMurry, D.
MacEDP Sciences, 2018, p. 03025. Fadden, D. J. Nigrin, S. Churchill, I. S. Kohane,
[4] H. B. Hamadou, T. B. Pedersen, C. Thomsen, The The shared health research information network
danish national energy data lake: Requirements, (shrine): a prototype federated query tool for
clinitechnical architecture, and tool selection, in: 2020 cal data repositories, Journal of the American
MedIEEE International Conference on Big Data, IEEE, ical Informatics Association 16 (2009) 624–630.
2020, pp. 1523–1532. [17] C. Cappiello, M. Gribaudo, P. Plebani, M. Salnitri,
[5] P. Liu, S. Loudcher, J. Darmont, C. Noûs, Archaeo- L. Tanca, Enabling real-world medicine with data
dal: A data lake for archaeological data manage- lake federation: a research perspective, in:
Heteroment and analytics, in: 25th International Database geneous Data Management, Polystores, and
AnaEngineering & Applications Symposium, 2021, pp. lytics for Healthcare, 2022. To appear.
252–262. [18] C. Giebler, C. Gröger, E. Hoos, R. Eichler,
[6] C. Walker, H. Alrehamy, Personal data lake with H. Schwarz, B. Mitschang, The data lake
archidata gravity pull, in: 2015 IEEE Fifth International tecture framework, BTW 2021 (2021).
Conference on Big Data and Cloud Computing, [19] T. Hlupić, D. Oreščanin, D. Ružak, M. Baranović, An
IEEE, 2015, pp. 160–167. overview of current data lake architecture models,
[7] C. Madera, A. Laurent, The next information archi- in: 2022 45th Jubilee International Convention on
tecture evolution: the data lake wave, in: Proceed- Information, Communication and Electronic
Techings of the 8th international conference on manage- nology (MIPRO), IEEE, 2022, pp. 1082–1087.
ment of digital ecosystems, 2016, pp. 174–180. [20] P. Sawadogo, J. Darmont, On data lake architectures
[8] A. Beheshti, B. Benatallah, R. Nouri, V. M. Chhieng, and metadata management, Journal of Intelligent
H. Xiong, X. Zhao, Coredb: a data lake service, in: Information Systems 56 (2021) 97–120.
Proceedings of the 2017 ACM on Conference on [21] C. Giebler, C. Gröger, E. Hoos, H. Schwarz,
Information and Knowledge Management, 2017, pp. B. Mitschang, Leveraging the data lake: Current
2451–2454. state and challenges, in: Proceedings of the 21st
In[9] K. Deligiannis, P. Raftopoulou, C. Tryfonopoulos, ternational Conference on Big Data Analytics and
N. Platis, C. Vassilakis, Hydria: An online data lake Knowledge Discovery (DaWaK), 2019, pp. 179–188.
for multi-faceted analytics in the cultural heritage [22] E. Barbierato, M. Gribaudo, G. Serazzi, L. Tanca,
domain, Big Data and Cognitive Computing 4 (2020) Performance evaluation of a data lake architecture
7. via modeling techniques, in: Performance
Engi[
terprise data lake solutions, International Research [24] A. Agrahari, D. Rao, A review paper on big data: Journal of Engineering and Technology (IRJET) Vol- technologies, tools and trends, Int Res J Eng Techume 7 (2020). nol 4 (2017) 10. [12] M. Sha M, M. P. Rahamathulla, Cloud-based health- [25] A. S. Foundation, Hdfs architecture guide, 2006. care data management framework, KSII Transac- URL: https://hadoop.apache.org/docs/r1.2.1/hdfs_ tions on Internet and Information Systems (TIIS) 14 design.html.
(2020) 1014–1025. [26] A. S. Foundation, Apache nifi, 2006. URL: https: [13] N. C. Taher, I. Mallat, N. Agoulmine, N. El-Mawass, //nifi.apache.org/.
An iot-cloud based solution for real-time and batch [27] J. Alwidian, S. A. Rahman, M. Gnaim, F. Alprocessing of big data: Application in health- Taharwah, Big data ingestion and preparation tools, care, in: 2019 3rd international conference on bio- Modern Applied Science 14 (2020) 12–27. engineering for smart technologies (BioSMART), [28] A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, IEEE, 2019, pp. 1–8. M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. An