Empirical studies have repeatedly shown that in High-Performance Computing HPC users' resource estimations lack accuracy. Therefore, resource underestimation may remove the job at any step of computing and subsequently allocated resources will be wasted. Moreover, resource overestimation also will waste resources. In this work, to effectively utilize the overall HPC system, we proposed a new approach to predict the required resources such as; number of required CPUs, time slots etc. for newly submitted job. The study focused on predictive analytics tasks including regression and classification. A supervised machine learning system, comprising several models, was trained based on the collection of statistical data including per-job and per-user features collected from the reference queue systems. Results indicated that adding more features to the dataset improves the prediction accuracy. The possibility of designing a plugin to apply our machine learning system in practical applications was studied. A dynamically connected SLURM SPANK plugin was created that adds the “--predict-time” option and takes control on srun and sbatch commands while they are executed. It was found that the plugin enables practical use of our proposed machine learning system.
High-Performance Computing (HPC) applications are intrinsically computational and data
intensive. Because of the dynamicity of resources and on-demand user application requirements, job
scheduling in such environments is an NP-Complete and complex problem [
Machine learning plays an important role in predictive analytics as algorithms are capable to
be trained based on a historical dataset. Dagnino et al. have already discussed the applicability of
machine learning in data analytics [
Regression algorithms have continuous outputs and are used to predict the resources. They are as follows: Multilayer Perceptron (MLP), Random Forest Regression (RFR), Lasso Regression (LR), K-Nearest Neighbor Regression (KNN), Ordinary Least-Squares Regression (OLSR), Support Vector Regression (SVR), Ridge Regression (RR), Polynomial Regression (PR), and CART Regression (CARTR).
Classification algorithms are used when the output is a discrete label and are used to check the failure of jobs. They are as follows: Naive Bayes classifier, Kernel Support Vector Machines (SVM), CART classification.
To check the efficiency of our system we used collected statistics of Bluegene/P system installed at the Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University named after M.V Lomonosov. This statistic includes information on jobs run for almost 12 months in 2017. It has about 112000 instances. We used 80% of the dataset for training and 20% for testing. The per-job features, presented in Table 1, in first phase were used to train our machine learning. In second phase, 7 metrics known as per-user features, shown in Table 2, were added to our dataset to train our machine learning.
Our findings presented in [fig. 2a] show that in general, the predictions by MLP, RFR, KNN, PR, and CARTR are promising. After adding per-user features to the dataset, it was observed that most of the models had improved. RFR, KNN, and CARTR improved a little. However, the performance of MLP and PR drastically improved after adding per-user features to the dataset. Another promising finding was that after adding per-user features to the dataset all the models improved to predict the required time.
Feature time_limit num_cpus id name user group task_class
Type Numeric Numeric String String String String String
As [fig. 2b] shows, RFR is the best model and the highest increase in accuracy rate belongs to the SVR model. From [fig. 2c], we can see that CART classification is the best model for classification. Feature used_portion_of_time_limit avg_aborted_task average_congestion average_cpus duration wait_time / time_limit average_time_limit
To make our system applicable on the SLURM we need to create a plugin. Therefore, a component was designed to connect to the SLURM. It has the ability to collect statistics, analyze them and based on the analysis create a model. Using this model, the prediction could be done. Plugins may not only complement, but also modify the behavior of SLURM to optimize the system’s performance. SLURM has a centralized SLURMctld manager for monitoring resources and tasks. Every compute server (node) has a SLURMd daemon that could be compared with a remote shell: it waits for a job, runs that job, returns state and waits for more work. In addition, SLURM has several utilities for its users. In this work we are mainly interested in 2 utilities: srun - to start a job and sbatch - for placing jobs in the queue. More precisely, sending a batchscript (instructions to SLURM to perform the job) to the SLURM.
The standard SLURM package has a mechanism to predict only the starting time of a job.
However, this mechanism is very primitive and more often the time is overestimated. Moreover, there
is a software system for modeling the activity of computing cluster users based on the SLURM, which
uses the collection of statistics to simulate the load on a model of computing cluster under the control
of SLURM. This software lists several metrics used by the system administrators of clusters. This
approach has been tested on data from computing clusters of the Faculty of Computational
Mathematics and Cybernetics, Lomonosov Moscow State University and NIKIET JSC [
The task is done through design, develop and test the functionality of the component (or components) connected to the SLURM queuing system. The component collects statistical data and does analysis on the flow of computational tasks. The component is a system of two modules implemented as:
Add-ons to SLURM to add and handle the --predict-time option in the srun and sbatch commands to predict the job run time.
A Linux daemon that handles HTTP requests, trains the model, and etc. (hereinafter the main application). The application must be able to connect custom algorithms according to a given template.
In addition, special scripts are required to install these modules easily.
Our work has led us to conclude that adding new features to the dataset improves prediction accuracy. An innovative solution for the resource allocation problem was found. The possibility of writing a plugin to apply our machine learning system in practical applications was studied. It was found that designing a plugin allows the practical use of machine learning algorithms in decision making. However, it is required to improve the performance of this component. In future work, we will use this component to evaluate our algorithms on a real cluster to find the best method to do prediction.