The use of artificial neural networks (ANN) requires solving structural and parametric identification problems corresponding to the choice of the optimal network topology and its training (parameter settings). In contrast to the problem of determining the structure, which is a discrete optimization (combinatorial), the search for optimal parameters is carried out in continuous space using some optimization methods. The most widely used optimization method in deep learning is the first-order algorithm that based on gradient descent (GD). In the given paper a comparative analysis of convolutional neural networks training algorithms that are used in tasks of image recognition is provided. Comparison of training algorithms was carried out on the Oxford17 category flower dataset with TensorFlow framework usage. Studies show that for this task a simple gradient descent algorithm is quite effective. At the same time, however, the problem of selecting the optimal values of the algorithms parameters that provide top speed of learning still remains open.
Deep Learning is a class of Artificial Neural Network (ANN) that has many
processing layers. There is huge number of ANN architectures variants in the literature.
ANNs can be used as a very effective technology to solving a wide class of problems
[
The use of ANN requires solving structural and parametric identification problems corresponding to the choice of the optimal network topology and its training (parameter settings). In contrast to the problem of determining the structure, which is a discrete optimization (combinatorial), the search for optimal parameters is carried out in continuous space using classical optimization methods. To train direct distribution networks with a teacher, algorithms are usually used that optimize some objective function. There are a lot of works that aim to improve ANN in different aspects (architecture design, choice and optimization of training algorithms and so on).
The most widely used optimization method in deep learning is the first-order
algorithm that based on gradient descent (GD). The BP algorithm is the standard training
method for ANN which uses GD. These methods can be split into the following
categories: batch gradient decent method, mini-batch gradient decent method, and
stochastic gradient decent method (SGD) [
The using of SGDs is the predominant methodology in training deep learning (CNN). 2
Initially, the convolution neural network structure was created taking into account the structural features of some parts of the human brain responsible for vision. The basis for the development of such networks is incorporated by three mechanisms: - local perception; - forming a set of layers in the shape of the characteristics maps (shared weights); - sub-sampling (sub-set).
Under the local perception it is understood that the input neuron receives not the whole picture but only some part of it. This helps to keep the image configuration during the transition from layer to layer.
The idea of shared weights means that a large number of connections used a small set of weights, i.e. each area of the image to which it is divided, will be processed by the same set of weights. Such artificial limitation of weights improves network’s generalization property.
CNN consists of the convolution layers, sub-sampling and fully connected neural network layers. 3
CNN got its name from the operator "convolution". The main purpose of convolution in the CNN case is to extract features from the input image.
Convolution keeps spatial relations among pixels, studying the features of the image, using the small batches of the input data.
Each neuron in the plane of the convolutional layer receives its inputs from the certain region of the previous layer (local receptive field).
Subsample layer zooms planes by local averaging of the neurons output values. Subsequent layers extract more common characteristics relied upon the picture distortion.
Each convolutional layer is followed by subsampling or computational layer which produces a reduction of the image dimension by local averaging the values of the neurons output.
The architecture of the convolution network is assumed that evidence of the feature’s existence is more important information than its exact location. Therefore, from a plurality of neighboring neurons in the map attributes one neuron with maximum value is chosen to map features of smaller dimension.
Difference between subsample layer and convolution layer is that in the convolution layer neighboring neurons overlap, which does not occur in the subsampling layer.
Thus, CNN is constructed by alternating of convolution and subsampling layers. At the output of the network several layers of fully connected neural network are usually installed. The input for these layers is the final feature’s map. Each neuron of the output layer is the perceptron, which has a non-linear activation function. 4
For convolutional neural network training a standard backpropagation algorithm and its various modifications can be used. The basis of this method is a stochastic gradient descent algorithm (Stochastic Gradient Descent).
4.1
Stochastic gradient descent (SGD) and its variants are the most widely-studied algorithms for optimization problems in machine learning and stochastic approximation.
The usual gradient descent is described by the following relation
(k 1) (k) J ( (k)), where – network parameter N 1 , J ( (k)) – the loss function; – training speed parameter (learning rate).
Algorithm (1) convergence is generally not guaranteed, but it is proved [
There are two main approaches to implementing gradient descent: Batch - when the training sample is viewed entirely at each iteration, and only after this is changed. This requires large computational cost. Stochastic (online) - where at each iteration of the algorithm from the training set some (random) object is selected. Thus, the vector is configurable for each newly selected object.
The following disadvantages are inherent to this algorithm: stuck in local minima and saddle points of the minimized functional. Slow convergence due to difficult terrain of the objective function when the plateau regions alternate with strong nonlinearity (the derivative of the plateau is almost zero, and sudden fall, on the contrary, can change the parameter estimation). Some of the parameters are updated less often than others, especially when in the data some informative but rare features are found. This has a bad effect on the nuances of the network rules generalization. On the other hand, giving too much importance to all rarely seen features can lead to overtraining. Too small value of parameter leads to slow convergence and stucking in local minima, while too large value of leads to "overshooting" the narrow global minima or no divergence at all.
Using the second-order methods discussed above requires calculating the Hessian derivative matrix for each pair of parameters, and, for the Newton’s method additionally its inverse matrix, i.e. implementation of these methods involves considerable computing effort.
Therefore, in practice, widespread methods are based on the stochastic gradient
method which has number of advantages [
Consider these methods in more detail. 4.2
The stochastic average gradient (SAG) algorithm [
The SAG iterations take the form where (k ) is the learning rate and a random index ik is selected at each iteration during which we set.
Essentially, SAG maintains in memory, the gradient with respect to each function in the sum of functions being optimized. The gradient value of only one such function is computed and updated in memory at each iteration. The SAG method estimates the overall gradient at by averaging the gradient values stored in memory.
However, the SAG technique can be utilized only with the smooth loss function and a convex objective function. The SAG has better convergence comparing to the SGD in tasks like convex linear prediction problems.
4.3
The SVRG algorithm [
N
i1 where %- interval update parameter. SVRG performs gradient updates by using following equation:
(k 1) (k) (gi (k)( (k) gi (k)(%) %).
The gradient can be calculated up to two times during each update. After w iterations, parameter % is updated and the next w iterations start. Through these update, (k 1) and the interval update parameter %will converge to the optimal , and then % 0 , and
gi (k)( (k) gi (k)(%) % gi (k)( (k) gi (k)( ) 0.
There are also many variants of such linear convergence stochastic optimization
algorithms, such as the SAGA algorithm [
4.4
Instead of depending on current gradient only for updating weights, the gradient
descent algorithm [
(k 1) (k) v(k 1); v(k 1) v(k) (1 ) J ( (k), where 0.9.
Later this pulse update becomes a standard for the gradient components upgrade. (4) (5) (6) (7) (8) 4.5
This algorithm [
v(k 1) v(k) (1 )x.
The gradient algorithm accumulates target network functions which is used during the parameters correction v(k 1) v(k) J ( (k)),
(k 1) (k) v(k).
More precise correlation algorithm for NAG has the form
(k 1) (k) v(k 1); v(k 1) v(k) J ( (k) v(k)).
(9)
(10)
(11)
(12)
(13)
(14)
(15)
(16)
Adagrad (adaptive gradient) [
M{g2 (k 1)} M{g2 (k)} (1 )g2 (k),
G(k) g(k), 4.6
In [
v(k 1) (k)v(k) (k)J ((k)); (k 1) (k ) (k )v(k ) (k 1)v(k 1) v(k 1) (k) (k 1) (k )v(k ) (1 (k 1)) (k ) J ((k )). where g(k) J ( (k); G(k) – the sum of the updates squares, - smoothing parameter that is required in order to avoid division by 0. The frequently updated last parameter G(k) is large (large denominator in (17)), i.e. the parameter will change slightly. Rarely changed parameters will change substantially. Parameter generally selected in range of 10-6 - 10-8.
4.8
Disadvantage of Adagrad is that G(k) in (17) can be increased without any
limitations. As the result, after short time an update becomes too small. This leads to
paralysis of the algorithm. RMSProp and Adadelta designed to solve this problem [
Adagrad modifies parameters in such a way that the frequently updated weights are adjusted less frequently. To do this, instead of the full sum of the updates averaged over history gradient square is used, i.e., moving average of the following form then instead of (17) we obtain
M{g2(k 1)} M{g2(k)} (1 )g2(k),
Adadelta is a continuation Adagrad [
Adadelta is different from RMSProp because we add to the numerator (17) the stabilizing member proportional to RMS from (k). In step k 1 value of RMS{ (k)}is not yet known, so the update of the parameters is implemented in three stages instead of two: at the first stage square of the gradient is accumulated, then is updated. And finally RMS{ (k)}, is updated (k 1) (k)
RMS{ (k)} RMS{g(k)} g(k); (18) (19) (20) (21) For RMSProp, Adadelta and Adagrad there is no need in very accurate choose of the learning curve - just its approximate value is needed. Usually it is advised to start snapping from 0,1-1, and leave 0.9. The closer to 1, the longer RMSProp and Adadelta with great RMS{ (k)} will much update rarely used weights. If 1 and RMS{ (k)} 0 , then Adadelta be longer "with a grain of salt" refers to a rarely used weights that can lead to paralysis of the algorithm, and intentionally cause "greedy" behavior, when the algorithm updates the first neurons that encode the best features.
4.10
Adam (Adaptive moment estimation) – another optimization algorithm. It combines
the idea of accumulation of the motion and the idea of a weaker weight updates for
typical features [
m(k 1) 1m(k) (1 1)g(k).
Nesterov is different from Adam because there is no need to accumulate , and the
gradient’s value. To obtain information about the gradient’s change in [
M{ (k 1) 2} M{ (k) 2} (1 ) (k) 2 ;
RMS{ (k )}
M { (k ) 2 } . (23) (24) (25) (26) v(k 1) 2v(k) (1 2 ) g(k 1) 2 ; (k 1) (k ) 1 2 (k 1) 1 1(k 1) m(k 1) v(k 1) where 1 , 2 [0,1) , g(k) – stochastic gradient off J at (k 1) , – step size, – a small constant.
Authors of Adam offered as defaults 1 0.9, 2 0.999, 108 and argue that an algorithm performs better or about the same as all previous algorithms on a broad set of datasets due to the initial calibration.
4.11
AdaMax algorithm [
Using the core update law of Adam [
k m(k 1) (1 1) 1i g(k i);
k v(k 1) (1 2 ) i g 2 (k).
2
i0
i0
It is easy to note that m(k 1) and v(k 1) are different because of the exponential
moving average utilization. In paper [
4.14
Nadam algorithm (Nesterov-accelerated Adaptive Moment Estimation) [
Gˆ (k 1) max Gˆ (k),G(k 1); g(k 1) 1g(k) (1 1) J ( (k); G(k 1) 2G(k) (1 2 ) J ( (k)2 ; G(k 1) 2G(k) (1 2 ) J ( (k)2 ; (35) (36) (37) (38) (39) (40) (41) (42) α= 0.001; β₁= 0.9; β₂ = 0.999; ε = 10⁻⁷.
4.16
In WNGrad algorithm (weight normalization Grad) [
Gˆ (k 1) max Gˆ (k), G(k 1) , where p (0,1 / 2] is the partially adaptive parameter (1/2 is the largest possible value for p and a larger p will result in non-convergence in the proof). When p 0, Padam reduces to SGD with momentum and when p 1/ 2 it is exactly Amsgrad.
It is empirically shown in [
The key difference between Adam and AdaShift (ADAptive learning rate method
with temporal SHIFTing) [
i 4.19
SWATS (Switching from Adam to SGD) [
While the focus of [
4.20
A strength of SGDs is that they are simple to implement and also fast for problems
that have many training examples. However, SGD methods have many disadvantages
(43)
(44)
(45)
[
A theoretical framework for the analysis of SGD parallelization performance has
been presented in [
We train and evaluate our CNN model on the Oxford17 category flower dataset [
Proposed CNN based application is implemented using TensorFlow [
A comparative analysis of gradient learning algorithms of convolutional neural
networks in solving visual recognition problem was held in this report. Studies show that
for this task quite effective is a simple gradient descent algorithm. Pulse usage in the
considered modifications led to some improvement in the recognition process, but it
also increased the computation cost. In the considered problems the most effective
algorithm is Adamax. In [
This project has been funded with support from the European Commission. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
Fig. 1. Example images from the Oxford flower dataset