Non-Uniform Memory Access (NUMA) architecture provides a multi-task run in parallel with di erent access latency between the nodes. The emergence of multi-core hardware o ers high processing power for multi-threaded Database Management Systems (DBMS). However, database threads run across multiple NUMA cores without exploring the hardware to its full potential. The impact of data movement between NUMA nodes is a major challenge for multi-threaded DBMS. The goal of our thesis is to nd out the e cient distribution of CPU-cores among the NUMA nodes in order to mitigate the data movement. We propose an abstract core allocation mechanism for query processing based on performance metrics. In our preliminary results, our mechanism is able to improve data tra c ratio between nodes in up to 3.87x with increased memory throughput in up to 27%.
0 250
The burgeoning ingestion of data requires new hardware to allow real-time data analysis. In our thesis we focus on Non-Uniform Memory Access (NUMA) hardware to boost throughput of multi-threaded Database Management Systems (DBMS) in Online Analytical Processing (OLAP). The NUMA architecture is formed by multi-core nodes with processors (or CPUs) attached to a memory bank. The memory access latency varies according to the distance between the node and the memory being accessed.
The varying memory access latency of NUMA impacts
the performance of multi-threaded query processing. We
observed the impact of NUMA in multi-threaded DBMS
executing the Volcano query parallelism model [
Threads Core 5T 6T 7T 8T 9T 10T 11T 12T 13T 1T4 T15 T16 17T 1T8 T19 20T Number 15 14 13 12 11 10 9 8 7 6 5 4 OS/MonetDB 32 1 0 threads to as many nodes and cores as possible to keep load balance, not considering the di erent access latencies in the NUMA architecture. The result is the constant migration of threads all over the nodes. Figure 1 shows the execution of the TPC-H query 6 (Q6) in MonetDB: a multi-threaded DBMS. Our NUMA architecture consists of 4-nodes with a Quad-Core AMD Opteron 8387 each. The OS tries to keep the load balancing between the cores causing many migration of threads with more data movement between the NUMA nodes. When running OLAP workloads, the current load balancing approach can cause many problems: resource contention, cache con icts, cache invalidation and ine cient data movement.
Many authors have investigated these problems by
trying to control the scheduling of threads and data movement
in NUMA nodes [
In this paper, we discuss our approach to tackle these three challenges, the contribution already published and current work in progress.
Over the last years, many di erent approaches were
presented to improve the performance of DBMS multi-core
parallelism in NUMA architecture. [
[
All of these contributions show important e orts into
making the DBMSs explore multi-core CPUs to their full
potential, but they di er from our goal as they allocate NUMA
cores statically to perform data and worker placement
strategies. Recently, [
In [
In contrast to the related work, we propose a dynamic multi-core allocation mechanism to mitigate the data movement providing to the OS the e cient sub-set of NUMA cores to perform database thread mapping.
Our proposal involves an elastic multi-core allocation mechanism to de ne the optimum number of cores to treat the time (a) Sparse Mode current workload. We de ne that the allocation of CPUcores should be performed dynamically to satisfy the demands of the workload and facilitate the OS thread scheduling with the least possible negative impact on performance. Our key idea is to consider data placement statistics from the OS side to de ne on which node the cores will be allocated. 3.1
The full description of our multi-core allocation
mechanism is presented in [
The challenge of the mechanism is to prevent both underutilization or overutilization of the system by nding out the local optimum number of cores (LONC) to accommodate a given workload. In this section, we de ne the LONC and also the allocation modes explored by the mechanism. 3.2.1
In the mechanism, we de ne resource usage thresholds (e.g., CPU or data tra c in the memory controller) that are used to decide when it is necessary to allocate or release cores. An adaptive algorithm decides on which node to allocate/release cores taking into account the accessed memory addresses kept in a priority queue data structure.
Figure 2 shows the allocation of closed (Dense) and far apart cores (Sparse). The adaptive allocation is a combination of both w.r.t. the e cient subset of cores. To nd the a nity between threads and data, our approach checks misses in the Translation Lookaside Bu er (TLB). When a TLB miss occurs, the OS maps the thread accessing the missed address to a node keeping this information in a data structure. Over time, new threads requesting the same address range are mapped to the same node where data is allocated. Therefore, we refer to as \the e cient subset of cores" the processing cores in the nodes with the requested address range. i m ( a )sm50 pm100 tse150 T200
0 250
In our adaptive mode, each entry of the priority queue keeps the PIDs of the active threads with their address spaces and the number of pages per NUMA node. The cores are allocated on nodes with more accessed pages and cores with the least number of accessed pages are released. In the implementation of the a nity between threads and data, the data structure stores the node IDs used to pin threads and allocate their address space. 3.2.2
The number of allocated cores takes into account the arithmetic CPU-load average of the active database threads. Formally, we de ne how to compute the number of cores, as follows: 8 w 9 nallocj(thmin < u < thmax)^p(nalloc) p(ntotal) (1)
To any OLAP workload w, there is a certain number of CPU cores nalloc such that the load of each core are between the minimum and maximum thresholds, in which the database performance p(nalloc) is equal or better than the performance p(ntotal) with all the CPU cores available in the hardware. The performance function p(x) relies on system counters provided by the OS and the database. 3.3
In our initial e orts in this thesis, we focused on OLAP
due to the pressure to transfer and compute large volumes
of data scattered across multiple NUMA nodes [
In a preliminary study, we ran the experiments with OLAP workload on a NUMA machine formed by 4-node with a QuadCore AMD Opteron 8387 each. Nodes are interconnected by Hyper-Transport (HT) link 3.x achieving 41.6 GB/s maximum aggregate bandwidth. We implemented our prototype in C language and we compare our mechanism to the Linux Debian 8 \Jessie" OS scheduling of threads spawn by the MonetDB (v11.25.5) DBMS. We let all the 16 cores available to the DBMS when running without the support of 4 3/)s B G 2( P T m e 1 M 0 OS/Mon Dense eD t B
S p as re
A d a ptive S0
S1
S2
S3 50 0.8 o i tra0.6 C I/M0.4 T H 0.2
used only the nodes S0 and S2.
Figure 6 shows the results of interconnection tra c in
relation with tra c HT/IMC ratio. In this experiment the
256 concurrent clients execute the 22 queries in random
order. The results show the reduction in the local/remote
per-query data tra c ratio of up 3:87x (2:47x on average).
Overall, the adaptive mode presented the best results. For
instance, we observed 2x smaller HT/IMC ratio than the
OS/MonetDB for queries such as Q9 that have the largest
number of joins operations. With less cores available, threads
locate data in the local node more often than the current
approach of letting all the cores available to the OS scheduler
(i.e., OS/MonetDB).
This paper describes the motivation and challenges to our
approach in the allocation of NUMA CPU-cores for
parallel execution of OLAP. In our rst evaluations, we discuss
the impact of data movement between NUMA nodes for
multi-threaded DBMS that hand over the mapping of query
threads to the OS. We show the di culty faced by the OS
scheduler to keep load balance, generating a vast amount of
data movement, interconnection tra c and cache
invalidation. In the initial contribution of this thesis [
As future directions, we plan to explore mixed workloads as we observed that OLAP not always need the entire setup of CPU-cores. Therefore, our agenda includes redesigning our abstract model to accommodate concurrent OLTP and OLAP. In particular, we plan to study the multi-core allocation in cloud computing environments that can particularly bene t from our model.
This work was partly funded by CAPES and CNPQ. 7.