Optimizing Analytical Query Processing on Disaggregated Hardware Andreas Geyer 0 Supervised by: Wolfgang Lehner Dresden University of Technology (TU Dresden) , Dresden, 01069 , Germany

In a world of ever-growing amounts of data, hardware-scalability and energy-eficiency become more important with every year. Traditional scale-up and scale-out database management systems (DBMS) struggle to scale well with their growing analytical workloads. Due to this, the emerging technology of disaggregated hardware becomes more and more popular. However, there is no free ride and specific challenges arise. In my PhD topic, I want (i) to look into these challenges for analytical query workloads on disaggregated hardware and (ii) to provide appropriate solutions. First initial results concerning data movement are promising and show the potential of adapted solutions.

 eol>RDMA Disaggregated Memory Disaggregated Hardware 1. Introduction

theless and as we focus on analytical query processing (OLAP) as a prime representative for data-intensive workWith the ongoing shift to a data-driven world in almost loads, the eficient data exchange between MUs and CUs all application domains, the management and analyt- is a major challenge. ics of data gain importance. However, the demand for To minimize the amount of data to be transferred computing power as well as memory capacities is also between CUs and MUs, solution approaches such as growing to enable eficient data analytics over an ever- function-to-data (operator push-down) like in [ 3 ] or nearincreasing amount of data. To satisfy these ever-growing memory computing schemes [ 4 ] are heavily applied. hardware demands in a scalable and flexible way, the However, these approaches lead to the fact that they emerging technology of hardware disaggregation is con- cannot scale the computations, due to limited resources, sidered the "next big thing" [ 1 ]. Hardware disaggregation as we would be able to achieve it by the general decompois an approach that decomposes general-purpose mono- sition of CUs and MUs. Thus, they only provide limited lithic servers into separated, network-attached resource applicability. To overcome that shortcoming, we focus pools, each of which can be built, managed, and scaled on solutions regarding the data-to-function schema by independently. This hardware disaggregation ofers var- making the data transfer explicit as first-class citizen in ious valuable possibilities such as (i) fast, fine-grained such an architecture. With this explicit treatment, we are scalability depending on individual workloads, (ii) en- able to synchronize the assignments of computations to ergy proportionality, or (iii) resource sharing capabilities. CUs and the necessary data exchange in a scalable and However, there is no free ride and specific challenges lfexible way. This synchronisation includes several asarise. In my thesis, I want to focus on analytical query pects such as (i) diferent computations on the same data processing on disaggregated memory systems and solv- across diferent queries can be grouped at one CU, so that ing the specific challenges in that scope. the necessary data must be transferred only to this CU,

The foundation of our work is an appropriate system (ii) data transfer can be done in an asynchronous way to architecture, similar to the one from [ 2 ], with dedicated interleave it with the computation to hide latency, or (iii) units for computation and memory. These are named preparing data during transfer for subsequent processing Compute Units (CUs) and Memory Units (MUs). CUs and at the CU by e.g., adapting the data layout. MUs are explicitly decomposed and connected via a net- While still being in an early stage of my PhD thesis work. With modern network technologies like Com- (1st year), we argue that this research direction has high pute Express Link (CXL) or Remote Direct Memory Ac- potential. To show that, the remainder of this paper is cess (RDMA) over InfiniBand (IB) , there are already high- organized as follows: In Section 2, we discuss a selecthroughput, low-latency interconnects available. Never- tion of already existing solutions. Then, we give a more detailed view of our current approach including some preliminary results in Section 3. Finally, we conclude with a summary and outlook in Section 4.

Published in the Workshop Proceedings of the EDBT/ICDT 2023 Joint Conference (March 28-March 31, 2023, Ioannina, Greece) $ Andreas.Geyer@tu-dresden.de (A. Geyer)

© 2023 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org) Q1 Q2 Q3

Query Batcher Query Batch Query Optimizer Optimizer Goals Pipelines Pipeline Grouper

PU PU PU PU

 Memory PU PU Data Transfer Manager Scheduler Parameter Server Data Transfer Pipeline Group Executor

MUs PU PU PU

 Memory 2. Related Work

to the principle of scan sharing [ 11 ].

On the one hand, recent work also has just shown the Disaggregated hardware revolutionizes the design and ar- viability of CXL-attached main memory [ 12 ]. Our prochitecture of modern database systems and thus database totype implementation is currently based on one-sided researchers have just started to investigate the potential RDMA verbs, but our memory access layer is already implications of such a novel hardware model. For ex- prepared to also work with memory via CXL as soon as ample, [ 1, 3, 5, 6 ] discuss the general impact and among we have access to corresponding hardware. On the other other things infer a new architecture as well as database hand, DFI [ 13 ] is a framework to eficiently exploit highprimitives. We fully agree that disaggregation leads to speed networks, such as IB. They show that adding an an alteration of traditional query handling. abstraction layer on top of RDMA verbs does not impose

The authors of [ 2 ] introduce an approach for dis- a significant performance degradation. However, their tributed shared-memory databases (DSM-DB). Their sys- experiments are tailored towards tuple-based data protem architecture is similar to ours as we will describe cessing, whereas we focus on column- or batch-oriented in Section 3.1, but it focuses on OLTP workloads, while data transfer. we focus on OLAP. However, it will be interesting to compare this similar strategy to our own in the future.

There are already system prototypes like LegoOS [ 7 ], 3. Our Contributions PolarDB [ 8 ], Teleport [ 6 ], Farview [ 3 ], and more emerge.

LegoOS tackles the operating system side for steering The focus of my thesis lies on pipeline execution, as and controlling the actual hardware components, which state-of-the-art execution model for analytical query prois an extremely interesting feature for elasticity, but or- cessing (OLAP), introduced in [ 14 ], on disaggregated thogonal to our proposed processing model. PolarDB – hardware. The challenges coming with it arise from the very similar to our architectural blueprint – plans with main OLAP properties. First, ○ 1 it is necessary to access separate compute nodes but attributes the remainder of a lot of data for these workloads, which traditionally rethe resources to individual pools. Contrarily, we argue sults in scans of whole columns or even tables. Second, that dedicated units with individual compute resources ○ 2 a lot of queries are executed simultaneously and most as in our system architecture yield benefits, for example, probably access the same data multiple times. Therefore, the preservation of the opportunity for operator push- data transfer is a potential bottleneck and thus, an inteldown. In Teleport, the authors observe that the high ligent and optimized data transfer is crucial. The idea of network latency of ‘remote’ accesses is impacting data- making the data transfer explicit allows us to tune the intensive systems and thus opt for compute or operator pipeline execution in a way that the latency through netpush-down. Approaches with network-attached memory work communication is nearly negligible. Additionally, (NAM) [ 9, 10 ] are promising but lack the possibility of we argue that this allows us to utilize the given flexibiloperator push-down, similar to Teleport. Farview’s on- ity that disaggregated hardware ofers when it comes to demand provisioning of compute nodes paired with the resource management.

FPGA-controlled storage serves as a general inspiration for our work. However, Farview considers the execution 3.1. General System Architecture of individual pipelines, which is contrary to our processing model, which is based on shared data access similar As there is a wide variety of possibilities to structure a system based on disaggregated hardware, we start with 9 8 ]7 s [ e6 m iT5 ing4 s s 3 e c o r P2 1 0 0/3

Local NUMA 1/3

Overlap Chunked Pipeline Chunked Prefetch 2/3

3/3 Column Pipeline Column Prefetch

Local NUMA 1/3

Overlap Chunked Pipeline Chunked Prefetch 2/3

3/3 Column Pipeline Column Prefetch (a) 4 pipelines executed sequentially with 4 threads for each pipeline (b) 4 pipelines executed fully parallel with 1 thread for each pipeline our anticipated system design as introduced in our CIDR responsible for the sending process (one-sided IB verb 2023 publication [ 15 ]. Figure 1 depicts a related sketch. RDMA_WRITE) and writes to the RB of the remote sysAs already introduced, we separate our system into CUs tem. Since we make the bufers exclusive for each CU-toand MUs based on their respective task. While MUs fea- MU connection, we can prevent conflicts of concurrent ture high memory capacities (DRAM, NVRAM, ...) with a write processes to the same bufer. limited amount of compute resources, CUs provide a high Through extensive evaluation of diferent configuraamount of compute resources with a limited main mem- tions of our initial communication layer, we found a ory capacity. Thus, CUs are merely responsible for exe- multi-bufer approach as best fitting. This approach imcuting queries, managing the lifecycle of intermediates plies that there can be multiple SBs and RBs on each and feeding results to the clients. Apart from operator system. Thereby, we can hide the latency introduced by pushdown, base data has to be fetched from MUs. the consumption of the content from the RB and write

This architecture works with one-to-one connections continuously to other free bufers. With this approach, for CU and MU, as long as the MU holds all relevant we found that already a configuration with one SB and data. However, the idea is to have an N-to-M connection two remote RBs has big benefits in comparison to a singlebetween CUs and MUs. The connections are realised bufer approach. Using multiple threads on each node through a high-throughput, low-latency interconnect also improves the performance further. like IB or CXL, for instance. This communication layer is continuously developed

As this architecture is already realizable with commod- to further allow a multitude of other interconnect techity systems, we argue that it allows transferring knowl- nologies additional to RDMA over IB. edge from the well-known system architectures to the new one based on disaggregated hardware. In the ab- 3.3. Pipeline Group Concept sence of real disaggregated hardware, we emulate both CU and MU with standard monolithic server systems directly connected through IB. As soon as disaggregated hardware is available for us, we will apply our proposed architecture and components to it.

Based on the implementation of the communication layer, the pipeline-based processing model is re-evaluated on the given system architecture. As base tables are assumed to be in-memory on the MU, when answering the query, the data needs to be transferred from the MU to the CU. 3.2. Communication Layer The naïve implementation based on the state-of-the-art pipeline-execution model shows that the processing time Even with fast interconnects, the network is prone to for a pipeline is mainly dominated by the data transfer. be the bottleneck of the whole architecture, especially Following these results and tackling properties ○ 1 and in data heavy OLAP scenarios. Thus, we started our ○ 2 , the main target is to reduce the amount of transresearch by developing a flexible communication layer ferred data and interleave the communication with the based on RDMA over IB, which is well prepared to in- computation. Thus, we propose an approach of groupcorporate CXL as soon as we get access to the respec- ing pipelines with similar or the same data-needs into tive hardware. Following the communication scheme of pipeline-groups. This grouping allows us to prevent reRDMA there are reserved bufers on each system. We dundant data transfer. implemented this by a separation into Receive Bufer (RB) Figure 2 depicts two experiments to highlight the benand Send Bufer (SB) . As the name suggests, the SB is efits of our pipeline-group approach. Both graphs show the processing time for the execution of 4 simple pipelines depending on the amount of overlap within their dataneed. For these experiments Local means all data is located in-memory attached to the working CPU, NUMA that there is a NUMA-hop between the memory and the CPU, Chunked that the data is transferred over the network in smaller pieces and Column that whole data columns are transferred at once. In Figure 2a, the more traditional approach of executing each pipeline with full resources one after the other is displayed. Orthogonal to this, Figure 2b shows the fully parallel execution of the same four pipelines. It is visible, that a higher overlap in the data-need reduces the processing time of the four pipelines tremendously in both cases. However, with full parallelism, it is possible to nearly match the performance of the NUMA-curve, while the traditional approach does not perform that well. With these experiments, we argue that pipeline groups ofer the potential to nearly eliminate the latency introduced by network communication.

This work has been submitted to CIDR 2023 and was accepted for publication [ 15 ]. 3.4. Resource Adaption One of the key aspects of the pipeline group is also that it allows utilisation of the given flexibility of disaggregated hardware by scaling the resources individually with the workload. Several of the blue components of Figure 1 ofer dimensions to exploit this flexibility. The shown Pipeline Group Executor is capable of managing the resource allocation in the best fitting way. Thus, it can for example distribute the workload across multiple CUs to prevent network-bottlenecks or move the workload to a CU closer to the data-holding MU. With diferent scheduling strategies, it is possible to determine how many resources are needed. Hence, for example, scenarios, where there is a budget involved can greatly benefit from our approach as resources (CPU cores, RAM, etc.), are allocated just when they are needed and released after the work is done. Additionally, when we integrate some form of operator push-down, we can react nearly immediately on the side of the MU to the increased workload. More CPU power can be allocated to work on this operator, without impact on the other connections of the MU. However, even though we already have a proof of concept for our pipeline group approach, most of these components are still up for development.

4. Conclusion and Future Work

We outlined the advantages and challenges of DBMS on disaggregated hardware, gave a brief overview of the existing solutions and touched on why we think they are not suficient to solve the outlined challenges completely. Additionally, our described approach of pipeline group execution showed some results of our previous work, to prove that the concept is capable of solving the challenges of DBMS on disaggregated hardware as well as utilizing its flexibility and other advantages.

To research the topic of a DBMS processing model that makes the best use of the opportunities and finds solutions to the introduced challenges of disaggregated hardware will be one of the key topics of my PhD thesis. Thus, we will continue to develop our introduced pipeline group approach.

 [1] Q. Zhang , et al., Rethinking data management systems for disaggregated data centers , in: CIDR , 2020 . [2] R. Wang , et al., The case for distributed sharedmemory databases with rdma-enabled memory disaggregation , 2022 . [3] D. Korolija , et al., Farview: Disaggregated memory with operator of-loading for database engines , in: CIDR , 2022 . [4] G. Singh , et al., Near-memory computing: Past, present, and future , CoRR abs/ 1908 .02640 ( 2019 ). [5] Q. Zhang , et al., Understanding the efect of data center resource disaggregation on production dbmss , Proc. VLDB Endow . 13 ( 2020 ). [6] Q. Zhang , et al., Optimizing data-intensive systems in disaggregated data centers with teleport , in: SIGMOD , 2022 , p. 1345 - 1359 . [7] Y. Shan , et al., LegoOS: A disseminated, distributed OS for hardware resource disaggregation , in: USENIX ATC , 2019 . [8] W. Cao , et al., Polardb serverless: A cloud native database for disaggregated data centers , in: SIGMOD , 2021 , pp. 2477 - 2489 . [9] C. Binnig , et al., The end of slow networks: It's time for a redesign , CoRR abs/1504 .01048 ( 2015 ). [10] E. Zamanian , et al., The end of a myth: Distributed transactions can scale , Proc. VLDB Endow . 10 ( 2017 ) 685 - 696 . [11] L. Qiao , et al., Main-memory scan sharing for multicore cpus , PVLDB 1 ( 2008 ) 610 - 621 . [12] M. Ahn , et al., Enabling CXL memory expansion for in-memory database management systems , in: DaMoN, 2022 , pp. 8 : 1 - 8 : 5 . [13] L. Thostrup , et al., DFI: the data flow interface for high-speed networks , in: SIGMOD , 2021 , pp. 1825 - 1837 . [14] V. Leis , et al., Morsel-driven parallelism: a numaaware query evaluation framework for the manycore age , in: SIGMOD , 2014 , pp. 743 - 754 . [15] A. Geyer , et al., Pipeline group optimization on disaggregated systems , in: CIDR, 2023 .