Currently, a wide range of high-performance environments is available for a researcher to perform computations. It is a really difficult task to select an environment in which the computations will be completed as soon as possible. To solve this, you need to analyze the load of the environment computing resources, and also to predict their availability in the future. In this paper we describe a solution for prediction of computing resources load in a cluster environment using neural network models. We considered a process of configuring the neural network architecture: selection of activation functions, algorithms of initialization and updating of the weights of neurons. Training and testing was performed on a set of data for the load of the cluster "Sergey Korolev" for the period from November 2013 to December 2016.
Neural network prediction models are based on the use of neural networks that can be trained in regression problems and produce the output value, based on some input parameters, approximating the unknown functional dependencies of the output data on the input.
Neural network models were used in papers [
Neural network models were chosen for this study because of peculiarities of the task and historical data collected by us. We took into account the following aspects: time series of resources load are non-stationary, there are templates and periodic components in historical data, as well as segments with low and high load, corresponding to weekends, holidays and work days, time series have known minimal and maximum value.
To predict values of time series of this nature we can use neural networks, in fact solving the approximation problem of an
unknown function. Taking into account the papers [
MLP consist of neurons and connections between them (fig. 1). Neurons have a special transformation function – activation
function, each connection has characteristic called weight. Output signal of a neuron in a layer Z of MLP is determined using
equation 1[
N z j f ( wijui ) (1) i1 where ui – output signals of the layer Z, wij – weights of connections between i neuron of the previous layer and j neuron of the layer Z, f – activation function, zj – output signal of a neuron. In this paper, we used hyperbolic tangent function as an activation function for neurons of hidden layers.
Data Science / Y.S. Artamonov
The training of a neural network is a process of changing the weights of the neuron connections. The main goal of a learning algorithm is to find a configuration of the weights of all the links where the error function is minimized. In the task that we solve we use MSE (Mean Squared Error) as a criterion for model training using gradient descent method, as a final benchmark of a model we use MAE (Mean Average Error) since time series contains a lot of segments with zero value or sequential equal values, that is why we cannot use MASE (Mean Average Scaled Error) and MAPE (Mean Absolute Percentage Error).
Fig. 1. Structure of MLP with one hidden layer.
We use DeepLearning4j library for training and testing models based on MLP. This library provides battle proven tools and
algorithms for training and usage of various artificial neural networks, including the most popular neural network architectures,
learning and optimization algorithms. The library is written in Java and uses native extensions for computations on the CPU and
GPU to provide the required performance [
To train neural MLP networks the method of back propagation of the error is used with various modifications. The method is
an iterative gradient algorithm that is used to minimize the MLP error and to obtain the desired output values. The essence of the
method consists in propagation of error signals from the outputs of the network to its inputs, back to direct propagation of
signals in the usual mode of operation [
Primary parameters of the method and its modifications are: learning epochs count, learning rate, weights initialization algorithm, weights update algorithm, optimization algorithm, learning momentum.
In the task, we need to predict the number of occupied cluster nodes in several of the most intensively used groups of nodes. Target prediction interval – 12 hours, we need to predict 12 points of a time series, one mean value of cluster group load per one hour. We chose qdr_tmp and ddr_tmp cluster groups for training and prediction of a group load. Their load is of the greatest interest because of a large regular load.
To compare the learning methods with different modifications we chose the training parameters presented in Table 1. We compared the training of SL MLP and DL MLP models in the prediction task with 12 points of cluster load (each point – mean load of a cluster group for 1 hour). In training and forecasting, only time series data were considered and passed to neural network inputs. The optimal number of inputs, selected experimentally, is 6.
In the process of training, we fed to the input of the neural network various sets of consecutive 6 values of the series; we used random order of data sets. For each test set, 12 values were generated at the output of the neural network, which were compared with 12 values from the test set. Parameters i = 6, k = 12. The backward propagation of errors method is subject to the following problems: Hidden layer neurons count 15 1st: 20 2nd: 10
Modifications to the method of back propagation of errors with momentum and various updating algorithms of the link weights, such as Adadelta, enable us to fix or partially fix the above problems, accelerate training and reduce the error of MLP based prediction models.
In the study of neural network models we considered various configurations of training neural networks by the method of back propagation of errors. As parameters of the configuration in the training we used: the algorithm for initializing the weights of neurons, the algorithm for updating the weights of neurons and the optimization algorithm.
We tested 2 options for initializing the balance: uniform distribution (Uniform) and using the Xavier method. The following
algorithms for updating the weights of neurons were tested: Nesterov Accelerated Gradient (Nesterovs), adaptive gradient
descent (Adagrad), Adaptive learning rate (Adadelta), adaptive momentum estimation (Adam). Two optimization algorithms
were tested: linear gradient descent (LGD) and stochastic gradient descent (SGD). These optimizations and the parameters of
the gradient descent method and the back propagation of errors algorithm are described in paper [
The test used a training sample of length 6000 points and a test sample with a length of 1000 points, the sample data were obtained for the period from January 1, 2015 to January 1, 2016. The results of testing models with different learning parameters for solving the task of forecasting the cluster load are presented in Table 2, the RMSE (Root Mean Square Error) error values are given to estimate the dispersion of the forecast values. UNIFORM XAVIER UNIFORM XAVIER UNIFORM XAVIER UNIFORM XAVIER UNIFORM XAVIER UNIFORM XAVIER UNIFORM XAVIER UNIFORM XAVIER
NESTEROVS NESTEROVS NESTEROVS NESTEROVS ADADELTA ADADELTA ADADELTA ADADELTA ADAGRAD ADAGRAD ADAGRAD ADAGRAD
ADAM ADAM ADAM ADAM
LGD LGD SGD SGD LGD LGD SGD SGD LGD LGD SGD SGD LGD LGD SGD SGD
An example of forecasting cluster load data for a neural network with a single hidden layer is shown in fig. 2, a neural network with two hidden layers - in fig. 3. The dashed line shows the forecast values of the series. The graphs of the forecast values were obtained by calculating the forecast every 12 points.
As the final error metric, the mean absolute error (MAE) is selected, because the relative forecast error (MAPE) can not be used in series that include values close to or equal to zero. The distribution of MAE errors in the SL MLP and DL MLP models is shown in fig. 4, the distribution of errors of both models is close to normal.
Previously, the task of forecasting 12 cluster load points was solved by the time series prediction method using the maximum
resemblance sample (EMMSP) [
In addition to direct comparison of models, we tried to use all three models (EMMSP, SL MLP, DL MLP) together . In order to do this, we put forward a hypothesis: Each of the models is the best (shows the smallest MAE error) in a certain length of data L > M, where M is the number of prediction points. We tested this hypothesis for the data on which MLP models were tested. Each of the models retains its leadership at the average on a section of 24 to 36 points in length, which corresponds to a time interval of 1 to 1.5 days.
The error value for a simple adaptive selective model [
The prediction algorithms based on neural network models with one and two hidden layers are integrated into the Templet Web service, which enables users to estimate the task launch time. The forecast graphs and cluster load history are available to registered users of the system. In the future, we plan to provide users with an interactive hint about the number of available resources and the estimated time to start the task based on the task requirements (nodes, groups, software licenses) specified at the time of adding task to a batch queue.
The results of the cluster load forecasting can be applied to solve several types of tasks: increase the efficiency of cluster use (energy efficiency, load efficiency), selection of optimal environments and parameters for computations, planning of cluster growth and maintenance periods.
Methods of forecasting the loading of computing resources are most in demand now in cloud environments where they can enable commercial companies to reduce server maintenance costs or, on the contrary, to effectively adapt to the growing demands of customers.
This work is partially supported by the Russian Foundation for Basic Research (RFBR#15-08-05934-A), and by the Ministry of Education and Science of the Russian Federation within the framework of the State Assignments program (№ 9.1616.2017/ПЧ).