Given the large amount of data that resides in relational database management systems (DBMS) and the fact that the DBMS often run on powerful servers, there exist considerable eforts for integrating machine learning (ML) support into the DBMS. This is particularly attractive in the explorative phase of data analytics that experiments with diferent ML algorithms on various subsets of the data. However, collocating ML and query processing on the same machine requires a scheduling mechanism that considers resource consumption, reasonable response times for interactive learning and the option to exploit both CPU and GPU. We propose DBMLSched, a scheduling mechanism that performs ML runtime prediction whenever possible and carefully monitors jobs and resources in the system, dynamically allocating ML jobs to their optimal device and pausing jobs if any performance interference occurs while at the same time avoiding starvation. Our preliminary results show a considerably improved response time compared to executing ML jobs in their order of arrival with less negative impact on concurrent queries.
time of random ML jobs on both CPU and GPU [17]. As workloads in their computational and resource require- a very general solution that is not restricted to specific ments making it necessary to rethink how to perform task scheduling.
In this paper, we focus on the exploratory phase of data science often performed in an interactive mode, and that experiments with a variety of ML learning jobs, exploring various subsets of the available data and smaller data sets, as a precursor to deciding which data sets and attributes are relevant to the problem. Thus, in our system, we expect many diferent variations of ML algorithms that run fairly short. We believe that it is beneficial to execute such jobs in the DBMS, as to exploit the DBMS query cessing is not significantly impacted by a learning task, fer. These ML jobs must then co-exist with traditional capacity to select data subsets and to avoid data trans- long. Fourth, concurrently running jobs might interfere Interference of ML jobs and SQL queries To illustrate how ML jobs afect concurrently executing SQL queries, we have the DBMS host the data set of the TPC-H benchmark and repetitively submit the medium-complex query 17 of the benchmark. In parallel, we submit twice a data science workflow also on the TPC-H data set. The pre-processing phase, among other things, submits SQL queries to retrieve the relevant learning data; learning uses K-means clustering. Pre-processing always executes on the CPU while the clustering job in the first submission executes on the CPU, and in the second submission on the GPU. in particular when it is executing on the CPU. Again, we a multi-process (or even multi-container) approach for propose the scheduler to be reactive and pause learning isolation purposes. However, DBMS often run in a single whenever database queries experience too much delay. process; thus in-database analytics will likely be based
We have integrated DBMLSched, an initial implemen- on multi-threading for concurrency. The right columns tation of these ideas, into AIDA [8], a Python-based in- of Figure 1 show the execution times for the various database analytic system residing inside a database en- combinations. The numbers above the bars indicate how gine. In our preliminary evaluation, DBMLSched was much slower the job runs compared to when it is run in able to provide response times that were up to 2.9x faster isolation. than if the jobs are executed in arrival order on the GPU We can see that the diferent combinations lead to while impacting SQL queries to an acceptable degree. quite diferent interference, with no interference between NN1 and NN3 as we originally anticipated, with significant interference between NN2 and NN3 for both multi2. Characteristics and Challenges threading and multi-processing, and with K-means and NN3 interfering little with multi-processing but signifiIn the interactive environment that we envision, rela- cantly with multi-threading. Running two NN3 jobs on tional queries and ML jobs can be submitted simultane- the CPU also causes interference. The underlying reaously by diferent clients. In this section, we investigate sons for these unexpected interferences can be related to several characteristics and challenges of executing them various causes such as bad concurrency within Python, concurrently. unexpected interactions between CPU and GPU, for instance for NN2, or suboptimal assignment of cores to diferent threads/processes.
Summary. While resource utilization can guide scheduling decisions, interferences can occur unexpectedly. A system that needs to schedule unknown ML programs and wants to support arbitrary ML libraries whose internals are not known, cannot simply concurrently run multiple jobs even if suficient resources appear to be available. Hence, monitoring the system state and adjusting scheduling dynamically will ofer better flexibility.
current ML jobs can impact each other. Our experiments use three neural networks NN1-NN3 with diferent input sizes and iterations, along with a K-Means algorithm.
The ideal device for NN1, NN2 and K-Means is the GPU due to the large amount of training data and complex computations. NN3 is a very small job and runs faster on the CPU.
Baseline Experiment. The first 4 columns in Figure 1 show the execution times for all four jobs when executed individually on their preferred device. NN1 and K-means use the GPU fully while NN2 only uses 78%. All three require 10% CPU utilization. NN3 uses only 50% of the CPU (half of the machine’s cores).
Concurrent Execution. We always run only one GPUbased job given the large GPU utilization of each, and then run NN3 concurrently on the CPU. In theory enough CPU capacity is available to not incur any interference.
We consider two scenarios, one where the ML jobs run within the same process using diferent threads, and the other where the ML jobs run in diferent processes. ML execution frameworks outside a DBMS will likely use
Figure 2 shows query response time (pink bars), and that work with termination conditions or for jobs that CPU utilization over time. The vertical lines mark when do not follow the paradigm that all iterations have the the pre-processing and clustering start and end. Response same length (that might be the case, e.g., with decision time for query 17 is 13 ms when it runs alone, increases trees), overall runtime is dificult to predict. We refer to slightly when the workflow starts pre-processing (con- them as implicit jobs. current SQL queries), and then increases heavily when learning starts on the CPU. In contrast, when we execute clustering on the GPU, there is little impact on query 3. DBMLSched response time. The result is directly related to the CPU utilization caused by the diferent phases of the data sci- We propose DBMLSched as a scheduling mechanism that ence workload, that reaches 80% only when learning controls the execution of ML jobs within a DBMS. As ML occurs on the CPU. jobs arrive DBMLSched assigns them to either GPU or
For interactive environments, query response times CPU. As the characteristics of the jobs are not known in are an important metric. For short queries, this might not advance and the DBMS might at the same time experience be an issue but for queries where the base response time various loads in regard to database queries DBMLSched is already close to a perceptible threshold, the impact due has an observer component that monitors the CPU/GPU to long-running ML jobs may be not acceptable, and it utilization overall and of the individual jobs. This inmight be desirable to hold back the execution of ML jobs. formation is used by the scheduler to decide where to However, real-time monitoring of the query response execute a job and also to halt jobs if there is interference. times requires tight integration with the database execution engine and will have to be custom designed for each database engine. Thus, we propose a more modular but still efective approach where the ML job scheduler dynamically monitors the CPU utilization of the system in order to make its scheduling decisions.
Integration into AIDA. AIDA [8] is a data science platform that facilitates the execution of analytical tasks within the database system. AIDA clients use a standardized set of Python APIs and interactive programming environments such as Jupyter to perform linear and relational algebra transformations or write complete ML functions. All calls/functions are shipped to the AIDA PrWedhicilteintgheErxeeicsustoimone TwiomrkestoofpMreLdiJcotbtshe execution server that resides inside the embedded Python intertimes of specific ML algorithms [ 18, 12], working with preter of the DBMS. It executes relational algebra opa wide variety of ad-hoc ML algorithms makes applying erations using the DBMS query engine while relational them challenging. However, it has already been observed algebra operations and ML functions are executed within that individual iterations of many popular learning al- the Python environment using existing ML libraries such gorithms often take similar amount of time. Indeed, we as NumPy and PyTorch. Data remains within the DBMS trained 6 models with diferent parameters ranging from space unless explicitly requested by the client. We have neural networks to clustering, both on CPU and GPU integrated DBMLSched into the AIDA server. We have with increasing number of iterations. The actual execu- also slightly adjusted the Pytorch-based API provided by tion times depend heavily on the ML algorithm, the input AIDA in order to facilitate easy extraction of the iterative data sizes, and whether execution is on CPU or GPU, but component of an ML job. On the server side, these ML all show linear behavior in regard to the number of itera- jobs are then intercepted and controlled by DBMLSched. tions.
ML programmers might indicate a certain condition that needs to be fulfilled to stop training (e.g., prediction error under a threshold). The number of iterations is thus determined during run-time. However, explicitly setting such termination conditions might not always be appropriate. Given the variety of algorithms, data scientists might not know in advance what a good termination condition is. Further, while exploring, they likely want to keep the execution times relatively short, in order to produce quick assessments for various algorithms. Thus, we believe that while not all, many ML programs will actually have a predefined number of iterations. We refer to them as explicit jobs. For these explicit jobs, once we know how long one iteration takes we can make a good estimation of the overall execution time. In contrast, jobs
is (1) to estimate the run-time of newly incoming jobs for both CPU and GPU, to (2) schedule them so that the overall service time is short and to (3) readjust when new jobs arrive or interference is observed. Given the complex interference we have observed in regard to concurrent ML jobs our current solution lets at most one job run on each CPU and GPU. We first outline our proposal assuming only explicit jobs and then extend the solution to also handle implicit jobs.
Runtime estimation When a new job arrives, DBMSched first checks whether it runs faster on CPU or GPU and estimates runtime. To do so, it pauses for each device the current job and executes the new job for a specific time. The current job is paused at the end of its current iteration because recent work [19] has shown that memory usage is the lowest at the end of an iteration, tors CPU and GPU utilization and responds to high utileading to less overhead when switching jobs. The length lization rates, which could be an indicator of resource of this estimation phase is set dynamically such that if contention. the new job is significantly shorter than the current job, The GPU utilization is recorded during the estimation it might actually finish during estimation. Otherwise, phase since we do not run any other ML job on the GPU, DBMLSched checks how many iterations have been com- while the diference between the CPU utilization during pleted during that time and confirms that they have the and before estimation is used as an estimation of the same length. From there it determines device preference actual CPU utilization of the job, in case CPU is also ocand run-time. Note that if the individual iterations of the cupied by queries or pre-processing of other ML jobs. At job do not have the same execution time or the job does any given time, the observer keeps track of the estimated not at least finish two iterations during the estimation CPU utilization of the jobs that are currently running on phase, then we cannot make any time prediction, and the CPU and GPU. job becomes an implicit job. Interference with queries If the actual CPU utilization
Scheduling jobs The service time for a job is the sum of is much higher and above a certain threshold, then this its execution time and all the execution times of jobs that indicates that there are concurrent SQL queries and / or execute before it in a given device. DBMLSched keeps pre-processing tasks which might be impacted by the two waiting queues, one for the GPU and one for the ML job running on the CPU. Thus, the job on the CPU is CPU. Whenever a new explicit job arrives with estimated paused. Once the CPU utilization falls again below this execution times on GPU and CPU, DBMSched reorders threshold, ML job execution on the CPU resumes. all jobs in both the GPU and CPU waiting lists accord- Interference with ML jobs As observed in Section 2, ing to the shortest remaining service time first (SRTF), a ML jobs running on the GPU might be impacted by jobs preemptive version of the shortest job, as it minimizes running on the CPU even if they only use a relatively the average amount of time each job has to wait until its small amount of CPU. In order to detect such interference, execution is complete when new jobs arrive on a contin- we observe GPU utilization. If it is significantly lower uous basis. Thus, we not only consider whether a job than the estimated GPU utilization and a job is running on runs faster on the GPU or the CPU but also consider its the CPU, we will pause the CPU job until the job running waiting time. Thus, a job might run faster on the GPU on the GPU has been completed. Once the GPU has than on the CPU but could be still scheduled on the CPU ifnished, rescheduling happens as described above. With because it would have to wait longer to even start on the this, jobs that were assigned during the last scheduling GPU. phase to the CPU might now be reassigned to the GPU.
Whenever a job completes on the GPU resp. CPU, We prioritize the GPU because we assume that most jobs the scheduler takes the first job from the correspond- run significantly faster on the GPU and the CPU might ing queue (if there is any) and usually executes it until mainly be a backup resource. Should there be many jobs completion. A new job, however, might preempt the that have true CPU afinity, other prioritization can be currently running job as described before. used.
Implicit jobs For implicit jobs, we can determine during the estimation phase whether they run better on the CPU or the GPU but we cannot estimate their run-time. 4. Preliminary Experiments Therefore, we split their execution into several tasks, each of which runs for a limited time, to avoid that they delay short-running jobs for too long. We guarantee that whenever they are scheduled at least one iteration is executed in order to guarantee progress. In order to ensure that our system is starvation free, we treat explicit jobs that have been staying in the system for a long time as implicit jobs.
We maintain separate queues for implicit jobs and schedule jobs for one device in a round-robin fashion, i.e., one explicit job, which will fully complete execution, and then one implicit job, which will execute for a quota of time and go to the end of the implicit queue. Service time calculations are accordingly adjusted considering in each round the time quota an implicit job will get.
ence with SQL queries. Figure 3 shows an execution over time where at the beginning a (blue) ML job is executed on the CPU and then a (red) ML job is scheduled on the GPU. Then first a lightweight SQL workload is activated causing a CPU utilization of roughly 20% if run alone in the system and then a heavier SQL workload with complex queries requiring roughly 40% of the CPU. If there is no observer (Fig. 3(a)) query response times are slightly higher for the light workload than if no ML job is run on the CPU (orange vs. add-on blue bars), while the more complex queries are heavily impacted by the ML job running on the CPU, having nearly 50% larger response times. In contrast, the observer (Fig. 3(b)) detects that the CPU threshold is passed and the ML job on the CPU is stopped, which prevents the severe delay.
In the next experiment, we look at how the scheduling policy takes efect on a hybrid workload of implicit and explicit jobs, and also how the choice of quota afects overall behavior. Our workload consists of 5 machine learning jobs: two implicit jobs LONG/SHORT submitted ifrst and last with three identical explicit neural network (NN) jobs in between, with a 3-second submission interval. Figure 4 shows that DBMLSched always performs better than sequential execution and short quotas for implicit jobs behave better than longer ones in 2 of the three situations. The shorter quotas allow the shorter jobs in the implicit and the explicit queues to be less delayed by the LONG job. Of course, one has to be careful not to set the quota too short as this could then delay the long job in case short jobs come into the system continuously. The values chosen for the time quota of implicit and long jobs clearly influence the prioritization of long/implicit jobs vs. short explicit jobs and are something that could be adjusted depending on the typical workload that might be submitted to the system and user expectations.
Lastly, we experimented during a 15-minute period to test the stable behavior of our scheduling system. The workload here consisted of a set of neural networks (both implicit and explicit), K-Means jobs and regression jobs of various lengths. Overall, service times using DBMLSched were 2.9x faster than when jobs are executed on the GPU sequentially in the order of arrival. Figure 5 shows the number of jobs that were in the system at any time during the experiment. We can see that relatively early in the experiment many jobs were submitted and our scheduler was able to work them of much quicker than sequential scheduling, keeping the waiting queues much shorter throughout the experiment. The reason is that our scheduler executes the shorter K-Means jobs first and also executes some jobs on the CPU, relieving the pressure on the GPU. Also as the short implicit jobs are executed in a round-robin fashion with the explicit jobs, they complete quickly.
arbitrary jobs and observes their behavior to perform scheduling decisions. In future work we would like to extend it to also consider memory aspects. Better control over which CPU cores are used by which tasks might also help in avoiding interference.
case for simpler data warehouses, in: ACM In[1] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Brad- ternational Conference on Management of Data bury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, (SIGMOD 2015), 2015, pp. 1917–1923. L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, [10] A. Yakovlev, H. F. Moghadam, A. Moharrer, J. Cai, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, N. Chavoshi, V. Varadarajan, S. R. Agrawal, T. KarB. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An nagel, S. Idicula, S. Jinturkar, N. Agarwal, Oracle auimperative style, high-performance deep learning toml: A fast and predictive automl pipeline, VLDB library, in: Advances in Neural Information Pro- Endow. 13 (2020) 3166–3180. cessing Systems 32: Annual Conference on Neural [11] Y. Peng, Y. Bao, Y. Chen, C. Wu, C. Guo, Optimus: Information Processing Systems (NeurIPS), 2019, An eficient dynamic resource scheduler for deep pp. 8024–8035. learning clusters, in: EuroSys Conf., 2018. [2] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, [12] H. Zhang, L. Stafman, A. Or, M. J. Freedman, SLAQ: Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, Quality-driven scheduling for distributed machine M. Devin, S. Ghemawat, I. J. Goodfellow, A. Harp, learning, in: Symp. on Cloud Computing (SoCC), G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, 2017, p. 390–404.
M. Kudlur, J. Levenberg, D. Mané, R. Monga, [13] D. Narayanan, K. Santhanam, F. Kazhamiaka, S. Moore, D. G. Murray, C. Olah, M. Schuster, A. Phanishayee, M. Zaharia, Heterogeneity-aware J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. cluster scheduling policies for deep learning workTucker, V. Vanhoucke, V. Vasudevan, F. B. Viégas, loads, in: USENIX Symp. on Operating Systems O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Design and Implementation (OSDI), 2020. Y. Yu, X. Zheng, Tensorflow: Large-scale ma- [14] W. Xiao, S. Ren, Y. Li, Y. Zhang, P. Hou, Z. Li, et al., chine learning on heterogeneous distributed sys- Dynamic scaling on GPU clusters for deep learning, tems, CoRR abs/1603.04467 (2016). in: USENIX Symp. on Operating Systems Design [3] L. Buitinck, G. Louppe, M. Blondel, F. Pedregosa, and Impl. (OSDI), 2020.
A. Mueller, O. Grisel, V. Niculae, P. Prettenhofer, [15] B. Wagner, A. Kohn, T. Neumann, Self-tuning query A. Gramfort, J. Grobler, R. Layton, J. VanderPlas, scheduling for analytical workloads, in: ACM Int. A. Joly, B. Holt, G. Varoquaux, API design for Conf. on Management of Data (SIGMOD), 2021, pp. machine learning software: experiences from the 1879–1891.
scikit-learn project, CoRR abs/1309.0238 (2013). [16] I. Sabek, T. S. Ukyab, T. Kraska, LSched: A [4] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gom- workload-aware learned query scheduler for anmers, P. Virtanen, D. Cournapeau, E. Wieser, J. Tay- alytical database systems, in: ACM Int. Conf. on lor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, Management of Data (SIGMOD), 2022. M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del [17] T. Lim, W. Loh, Y. Shih, A comparison of prediction Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, accuracy, complexity, and training time of thirtyK. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, three old and new classification algorithms, Mach. C. Gohlke, T. E. Oliphant, Array programming Learn. 40 (2000) 203–228.
with NumPy, Nature 585 (2020) 357–362. [18] M. Lattuada, E. Gianniti, D. Ardagna, L. Zhang, Per[5] J. M. Hellerstein, C. Ré, F. Schoppmann, D. Z. Wang, formance prediction of deep learning applications E. Fratkin, A. Gorajek, K. S. Ng, C. Welton, X. Feng, training in GPU as a service systems, Clust. ComK. Li, A. Kumar, The madlib analytics library put. 25 (2022) 1279–1302. or MAD skills, the SQL, VLDB Endow. 5 (2012) [19] W. Xiao, R. Bhardwaj, R. Ramjee, M. Sivathanu, 1700–1711. N. Kwatra, Z. Han, et al., Gandiva: Introspective [6] X. Li, B. Cui, Y. Chen, W. Wu, C. Zhang, Mlog: cluster scheduling for deep learning, in: USENIX Towards declarative in-database machine learning, Symp. on Operating Systems Design and ImplemenVLDB Endow. 10 (2017) 1933–1936. tation (OSDI), 2018, pp. 595–610. [7] M. E. Schüle, H. Lang, M. Springer, A. Kemper,
T. Neumann, S. Günnemann, In-database machine learning with SQL on GPUs, in: International Conference on Scientific and Statistical Database Management (SSDBM), ACM, 2021, pp. 25–36. [8] J. V. D’silva, F. De Moor, B. Kemme, AIDA - abstraction for advanced in-database analytics, VLDB
Endow. 11 (2018) 1400–1413. [9] A. Gupta, D. Agarwal, D. Tan, J. Kulesza, R. Pathak,