In recent years, so-called lakehouses have emerged as a new type of data platform that intends to combine characteristics of data warehouses and data lakes. Although companies started to employ the associated concepts and technologies as part of their analytics architectures, little is known about their practical medium- and long-term experiences as well as proven architectural decisions. Additionally, there is only limited knowledge about how lakehouses can be utilized efectively in an industrial context. Hence, it remains unclear under which circumstances lakehouses represent a viable alternative to conventional data platforms. To address this gap, we conducted a case study on a real-world industrial case, in which manufacturing data needs to be managed and analytically exploited. Within the scope of this case, a dedicated analytics department has been testing and leveraging a lakehouse approach for several months in a productive environment with high data volumes and various types of analytical workloads. The paper at hand presents the results of our within-case analyses and focuses on the industrial setting of the case as well as the architecture of the utilized lakehouse. This way, it provides preliminary insights on the application of lakehouses in industrial practice and refers to useful architectural decisions.
analyses and are typically less suited for many types of
advanced analytics [
With the growing range of capabilities for data acquisi- Data lakes attempt to close this gap by enabling the
tion and the current advances in the field of analytics [
Data platforms constitute the technical foundation for for use cases in which the analysis questions are already
all kinds of analytics applications within enterprises, as known in advance, while exploratory workloads with
they are capable of storing and managing huge amounts unknown analysis questions are more appropriately
supof data for analytical purposes and thus support data ported by data lakes. Consequently, enterprises often
collection, processing and analysis [
The most popular representatives of such frameworks 2. The Manufacturing Case are the open-source projects Delta Lake1, Apache Hudi2 and Apache Iceberg3. All of these frameworks provide libraries for various popular data query and processing engines, such as Apache Spark4 and Apache Flink5, and enhance them for additional data warehousing capabilities when working on data that resides on distributed ifle systems or object storages. This includes features for abstracting stored data as tables with relational characteristics, ensuring ACID properties and enabling eficient batch and stream processing [13, 14, 12]. Consequently, data can be flexibly stored in open formats and in a directly accessible manner, but still be conveniently processed and queried. This allows such lakehouses to cover large portions of the typical analytical workloads of data warehouses and data lakes.
In literature, several works can be found that propose and discuss diferent lakehouse implementations based on these frameworks, such as for the domains of healthcare [18, 19, 20], biomedical research [21], network management [22], IT security [23] and geospatial analytics [24]. However, these descriptions often lack details regarding the chosen architectural decisions and the extent to which these decisions have proven over time, especially in terms of aspects such as data organization, data modelling and data flow. Furthermore, to the best of our knowledge, there are yet no works available that present industrial real-world implementations of lakehouses for the manufacturing sector. As a consequence, it remains unclear under which circumstances and with which architectures lakehouses may represent a viable alternative to conventional data warehouses and data lakes for enterprises in this field. In an efort to address this gap, we conducted a case study on a real-world in- Table 1 dustrial case in which an analytics department developed, Characteristics of the investigated manufacturing case. tested and leveraged a lakehouse approach for the man- Source Data: agement and analytical exploitation of manufacturing Source Systems: data from the shop floor with a volume in the magnitude Data Types: of terabytes. This lakehouse has been in use for several Data Volumes: months in a productive environment with various analyt- Analytical ical workloads and hence represents a suitable candidate Workloads: for investigating architectural decisions. Analytics Types:
Section 2 explains the methodological approach of our Users: study and introduces the industrial setting of the case.
research method for complex topics and topics where the context needs to be taken into consideration [25]. Both applies to the research field of data platforms, as the construction of such platforms constitutes complex tasks that are subject to rapid developments and innovations, while architectural decisions and practical experiences of enterprises likely depend on contextual aspects, such as the domain and size of the enterprise, the available data volumes and the analytical use cases.
In the scope of our case study for the manufacturing case, we conducted interviews with one solution architect and one data engineer of the responsible analytics department, who both have been involved in the development of the corresponding lakehouse. The interviews lasted between 45 and 60 minutes and followed a semistructured approach, for which questions had already been prepared in advance, but were spontaneously supplemented by follow-up questions during the interviews. The asked questions were related to the context of the manufacturing case, including the available data sources and the requirements for the data analyses that are supposed to be performed, as well as various architectural aspects of the developed lakehouse (cf. Section 3). Afterwards, the transcribed answers of the interview participants were structured and analyzed with the help of qualitative coding techniques.
Table 1 summarizes important characteristics of the investigated case, including details about the source data and the intended analytical use cases.
Machine and Sensor Data Manufacturing Execution System Structured, Unstructured Terabytes Reporting, OLAP, Machine Learning, Near-realtime Reporting Descriptive, Diagnostic, Predictive Business Users, Data Analysts,
Data Scientists
2https://hudi.apache.org, accessed: 30.04.2024 3https://iceberg.apache.org, accessed: 30.04.2024 4https://spark.apache.org, accessed: 30.04.2024 5https://flink.apache.org, accessed: 30.04.2024
Event Hub Confluent Kafka
Data Stream Data
Stream Manufacturing
Data
Extract & Load
Harmonization
Delivery Serialized Data
Delta Lake
Tabulated Data
Delta Lake
Harmonized Data
Delta Lake Ingested Images Various Formats
Application Data
Delta Lake Machine Learning Prepared Images Various Formats
Queries Queries Data
Stream Data Science
Tools
Reporting OLAP Near-realtime Reporting Machine Learning
Cloud Object Storage
Azure Data Lake Storage Gen2 Batch Processing
Stream Processing
Relational Dataset (Table)
Data Files
Zone lect various types of data, reflecting the quality of the ifed with both the lakehouse approach in general, as well workpieces and the condition of the machines. This data as the architecture they have developed, since it allowed is primarily made available in a Manufacturing Execu- them to meet their requirements in terms of performance tion System (MES) and ingested as data stream into a and the required range of analytical workloads. We therelakehouse-like data platform. Besides structured mea- fore assume that their architectural decisions regarding surement values of the machines and sensors, this data the lakehouse have practically proven their suitability at also includes graphical images of particularly error-prone least in the medium term. parts of the produced technical components. A dedicated In the context of this work, lakehouses can largely be analytics department constisting out of multiple solution regarded as data lakes that consist of a distributed file architects and data engineers is involved in this case and system or an object storage and have been enhanced for responsible for the development and operation of the data additional data warehousing capabilities with the help platform. The goal of this data platform is the collection, of specific frameworks, such as Delta Lake, Apache Hudi management, preparation and analysis of the generated and Apache Iceberg [12]. Therefore, we utilized the Data manufacturing data in order to enable self-service ana- Lake Architecture Framework by Giebler et al. [26] as a lytics for business users and data analysts. This includes reference to guide and structure our within-case analysis traditional reporting and OLAP workloads, but also near- for the architecture of the data platform. However, we realtime reporting, since some analysis results are sup- limited ourselves to the aspects Data Organization, Data posed to be displayed on dashboards along the shop floor. Modeling, Data Flow, Data Storage and Infrastructure, as In addition, data scientists pursue to train machine learn- these were most strongly covered during the interviews. ing models from the available image data, which should Figure 1 illustrates the current architecture of the lakeenable the automatically detection of faulty workpieces house that is applied in the investigated manufacturing with the help of image classification techniques. case. Here, the lakehouse is depicted in the center, while the data sources and analytical workloads are shown on the left- and right-hand side. The lakehouse consists of 3. The Lakehouse Architecture multiple zones [27], which separate data of diferent granularity, quality and purpose. The individual aspects of As part of the case study, we examine the architecture of this architecture according to the Data Lake Architecture the lakehouse data platform that is employed in the man- Framework are discussed in the following sections. ufacturing case and, in particular, focus on interesting architectural decisions that were made during its development, for example with respect to aspects such as data organization, data modeling and data flow.
During the interviews, both participants confirmed to us that after several months of operation, they are satisthe Machine Learning Zone, where they can be used by preferred over continuously running stream processing data scientists for machine learning experiments. jobs for economic reasons, b) that ingested raw data is stored and managed as serialized JSON strings in tables 3.3. Data Modeling and not in raw text files, c) that data modelling is carried out only informally and that de-normalization techniques According to the participants of the interviews, data are applied in order to increase the query performance modeling is carried out only informally, meaning that at the expense of higher costs for storage space and d) data models are created ad-hoc depending on the struc- that zone models appear to be a suitable technique for ture of the data that is provided by the data sources data organization within lakehouses. and the requirements of the analytical use cases. Hence, In future work, we plan to compare this case with widely researched modelling approaches, such as normal several other real-world cases from diferent domains forms [28], multi-dimensional modeling [29] or the Data in terms of architectural similarities, the motivational Vault concept [30] are not explicitly applied. Moreover, factors for enterprises to utilize lakehouses, practical the data is intentionally de-normalized, as this enables a experiences and encountered challenges. This way, we higher query performance at the cost of increased storage want to further expand the findings of our work and space, which represents a reasonable trade-of for cloud become capable of generalizing them. environments due to the lower costs for storage space in comparison to computational resources. 3.4. Data Organization The lakehouse architecture of the manufacturing case applies a zone model for organizing the data of diferent granularity, quality and application-specificity within the lakehouse. In particular, this zone model defines four zones: A Raw Zone, which stores the raw and only slightly processed data, a Harmonization Zone, in which the relational, consolidated and unified data resides, a Delivery Zone for pre-aggregated, application-specific data and a Machine Learning Zone that holds data that is relevant for machine learning activities. These zones can be roughly mapped to the Raw Zone, Harmonized Zone, Delivery Zone and Explorative Zone of the Zone Reference Model (ZRM), which was originally proposed by Giebler at al. [27] for the data organization within data lakes. According to the participants of the interviews, both the use of a zone model in general, as well as the zones that were specifically selected for this case have proven their suitability. Therefore, it can be concluded that zone models for data lakes appear also to be relevant in the context of lakehouses and may be a suitable choice for organizing the data in these kind of data platforms.
This paper presented a real-world case in which a lakehouse has been developed and leveraged for the management and analysis of manufacturing data in industrial practice. In the scope of our study, we particularly focused on the architecture of the lakehouse, as well as the industrial setting and underlying goals. In this course, some interesting architectural decisions could be observed: Our study revealed that in the investigated case a) the periodic execution of stream processing jobs is