In recent years, the popularity of deep neural networks used for various problem-solving tasks has increased dramatically. The main tasks include image classification and synthesis using convolutional and generative-adversarial neural networks. These types of networks need large amounts of training data to achieve the required accuracy and performance. In addition, these networks have a long training time. The authors of the paper analyzed and compared the gradient-based neural network learning algorithms. The biomedical image classification with the use of a convolutional neural network of a given architecture was carried out. A comparison of learning algorithms (SGD, Adadelta, RMSProp, Adam, Adamax, Adagrad, and Nadam) was made according to the following parameters: training time, training loss, training accuracy, test loss, and test accuracy. For the experiments, the authors used the Python programming language, the Keras machine learning library, and the Google Colaboratory development environment, which provides free use of the Nvidia Tesla K80 graphics processor. For the experiments tracking and logging the authors used the Weights & Biases service.
Learning algorithms are divided into first-order or second-order algorithms and evolutionary algorithms. First-order algorithms are based on the calculation of the first derivative of the error function. Therefore, these algorithms are also called gradient algorithms. Second-order algorithms use the second derivative to select the direction of error minimization. Evolutionary algorithms are built on the basis of genetic algorithms.
In [
The authors of article [
In the article [
In the article [
In the research study [
In these publications, researchers mostly paid attention to the analysis of existing algorithms. Only some of the authors compared learning algorithms.
Therefore, the limitations of these publications are that they only partially address the problem of learning algorithms comparison. Most of the publications are just about learning algorithms review when solving a bigger problem.
The main goal of any learning algorithm is to minimize the learning error and optimize the
network parameters. Modern classifiers [
Therefore, the actual task is the comparison of learning algorithms for biomedical image classification.
Modern algorithms for learning neural networks are based on error backpropagation and the gradient descent method. These algorithms are called gradient or first-order algorithms.
An important parameter of algorithms is the learning rate. This parameter controls how far to move in the direction opposite to the gradient of the function in one step. If the learning rate is low, the training time of the neural network can increase significantly. If the learning rate is high, the neural network may not reach the minimum error value [9]. Formally, it can be presented as follows: where θ refers to neural network parameters, ɑ is a learning rate,
is a gradient of the optimization function (loss).
The disadvantage of gradient descent is that the network parameters can be updated only after passing the full training dataset.
Among other gradient learning algorithms, stochastic gradient descent and mini-batch gradient descent are distinguished.
Stochastic gradient descent (SGD) differs from the usual one in that the network parameters are updated after each training iteration [10]. Therefore, when using this learning algorithm, the parameters of the neural network are updated much more often.
Mini-batch gradient descent uses data packets to update parameters [11]. The training dataset is divided into packets of the same size. Then each of the packets is sent to the network input, the gradient is calculated and the parameters are updated. Equation (1) can be represented in the following way: where
is a package of training examples.
Adagrad. The essence of this algorithm is that the learning rate adapts according to the network parameters [12]. The algorithm sets a lower learning rate for parameters that are associated with frequent features in the dataset. Then the equation with iterations will have the following form: where is a diagonal matrix, where each of the diagonal elements is the sum of the squares of the gradients with respect to the parameters θ at all previous iterations, including t, is a parameter with a small value that prevents division by 0 (usually ), is a gradient of the optimization function, .
The advantage of this algorithm is that a researcher does not need to set the learning rate manually. The authors use the default value for the learning rate, which is 1.0 [12].
The disadvantage of the algorithm is an accumulation of gradients from previous iterations. This leads to a decrease in the learning rate and a minor update of the network parameters.
Adadelta. This algorithm is an improved version of the previous algorithm. The Adadelta algorithm reduces the size of the matrix of accumulated gradients to a particular fixed value [13]. where RMS is a root mean square value, is a gradient of the optimization function,
The advantage of this algorithm is in no need for setting the initial value of the learning speed.
RMSProp. This algorithm is similar to the Adadelta algorithm. It was developed by Geoffrey Hinton [14]. The equations that describe the operation of the algorithm are as follows:
RMSprop was developed at around the same time as Adadelta. These algorithms solve the problem of monotonically decreasing learning rates in the Adagrad algorithm.
Adam та Adamax. Unlike the two previous algorithms, in addition to the squares of the previous gradients, the Adam algorithm also stores the previous gradients : where
are estimates of the mean and variance of the gradients, respectively [15].
The Adamax is a variant of the Adam algorithm.
Nadam. Nadam is a combination of RMSProp and Momentum algorithms. The first algorithm accumulates the squares of the gradient values, and the second algorithm accumulates the values of the previous gradients.
A training set of cytological images with a size of 64x64 pixels was used for the experiments. The initial dataset contains about 100 images. Therefore, the dataset was expanded to approximately 800 images using affine distortions. The Python programming language and the Rudi library [14] were used to expand the training data set.
Cytological images form a subset of biomedical images. Cytological images are images of cells of the organism. Examples of cytological images are shown in Figure 1.
Cytological image processing and analysis are reflected in works [
To compare the gradient descent-based training methods, a convolutional neural network model was built. As an input, the network accepts color cytological images with a size of 64x64 pixels and outputs a class label. The sequence of layers is given in Table 1.
Layer param kernel size = 5
slope = 0.2 kernel size = 5 slope = 0.2 rate = 0.5 kernel size = 3
slope = 0.2 kernel size = 3 slope = 0.2 rate = 0.5
As can be seen from Table 1, the network consists of several repeating blocks. Each block consists of a sequence of convolution layers, batch normalization, an activation layer, and a dropout layer. Each convolutional layer reduces by half the input volume.
The model is compiled using the categorical cross-entropy loss function. The number of learning epochs is 30.
The dataset is divided into learning and testing in the ratio of 80% to 20%. The batch size is set to 64.
The Tensorflow 2 library and the Python programming language were used to build the model and conduct experiments. The experiments were conducted in the Google Colaboratory environment on an Nvidia Tesla K80 graphics processor.
The results of the optimization of neural network parameters based on gradient descent are shown in Table 2.
All optimizer parameters are set to the default values specified in the Tensorflow library. The comparison of learning algorithms was made on the basis of the network training time, the value of the loss function, and the classification accuracy on the test dataset. The results of the experiments are shown in Table 3 and in the figures below.
Figure 2 shows the error graphs and accuracy graphs on the training and test datasets for the Adam optimizer. The accuracy curves on the training and test datasets show that the network is being retrained. This happens because there is a significant difference in accuracy between the training and test datasets. Accuracy on the training dataset was ~98%, and accuracy on the test dataset was
Figure 3 also shows a significant difference between the accuracy values on the training and test datasets: ~97% and ~75%, respectively.
The classification accuracy for the model with the RMSprop optimizer on the training and test datasets was ~97% and ~89%, respectively (fig.4).
The model with the Nadam optimizer also demonstrates significant overtraining. Accuracy on the training and test datasets is ~98% and 66%, respectively (fig.5).
As can be seen from Figure 6, the error and accuracy graphs are quite smooth. However, the classification accuracy on the training dataset was only ~73%. This is explained by another problem, namely underfitting. There are several options for its solution, such as an increase in the number of training epochs or an increase in the complexity of the used model.
The accuracy values on the training and test data sets are almost the same for the model with the Adagrad optimizer and equal to ~95% (fig.7).
Figure 8 shows that the error and accuracy curves on the test dataset for almost all models are not smooth, so the networks are retrained. This is evidenced by the significant difference in accuracy values on the training and test datasets. This problem can be solved by simplifying the neural network model or by increasing the number of images in the training dataset. On the other hand, the model with the Adadelta optimizer has an inverse problem called underfitting. To solve this problem, several techniques can be applied, such as increasing the complexity of the model or increasing the number of training epochs.
1. The authors of the research study conducted a comparative analysis of the gradient descentbased algorithms for optimizing neural network parameters (Adam, SGD, RMSprop, Nadam, Adadelta, Adagrad, and Adamax). The comparison was made according to the criteria of network training time, loss function values, and classification accuracy on the cytological image dataset.
2. Based on the cytological image dataset and the developed convolutional neural network model, the four best optimizers were selected according to the val_accuracy parameter: Adamax, Adadelta, Adagrad, and RMSprop.
3. The graphs of val_loss and val_accuracy on the test data set for optimizers (except Adadelta) are not smooth. Unlike other algorithms, Adadelta is an optimization algorithm with an adaptive learning rate. Therefore, parameters are updated with a smaller step, and during the training, there is no problem with retraining. As a result, the accuracy and error curves on the training and test datasets almost coincide and are smooth.
4. Since Adadelta is an adaptive algorithm, it is necessary to use a higher training rate at the beginning. This will significantly reduce the training time and ensure the convergence of the model.
In this work, only one neural network model was used. In future research, it is planned to apply discussed optimizers to more complex and huge models. The further research direction will also cover the application of optimizers for generative-adversarial networks.
[9] Amari, Shun-ichi. "Backpropagation and stochastic gradient descent method." Neurocomputing 5.4-5 (1993): 185-196. https://doi.org/10.1016/0925-2312(93)90006-O [10] Ketkar, Nikhil. "Stochastic gradient descent." Deep learning with Python. Apress, Berkeley, CA, 2017. 113-132.
[11] Hinton, Geoffrey, Nitish Srivastava, and Kevin Swersky. "Neural networks for machine learning lecture 6a overview of mini-batch gradient descent." Cited on 14.8 (2012): 2.
[12] Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12, 2121–2159. Retrieved from http://jmlr.org/papers/v12/duchi11a.html
[13] Zeiler, M. D. (2012). ADADELTA: An Adaptive Learning Rate Method. URL: http://arxiv.org/abs/1212.5701
[14] Hinton, G., Srivastava, N., Swersky, K. Overview of mini-batch gradient descent. URL: http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
[15] Kingma, D. P., & Ba, J. L. (2015). Adam: a Method for Stochastic Optimization.
International Conference on Learning Representations, 1–13. https://arxiv.org/abs/1412.6980
[
[17] Berezsky, O., Pitsun, O., Datsko, T., Derysh, B., Tsmots, I. & Tesluk, V. “Specified diagnosis of breast cancer on the basis of immunogistochemical images analysis”, IDDM’2020: 3rd International Conference on Informatics & Data-Driven Medicine, November 19–21, 2020, Växjö, Sweden. pp. 129-135. http://ceur-ws.org/Vol-2753/short5.pdf,
[18] Berezsky, O., Pitsun, O., Dubchak, L., Berezka, K., Dolynyuk, T. & Derish, B. Cytological Images Clustering. In: Shakhovska N., Medykovskyy M.O. (eds) Advances in Intelligent Systems and Computing V. CSIT 2020. Advances in Intelligent Systems and Computing. 2021; vol 1293. Springer, Cham. https://doi.org/10.1007/978-3-030-63270-0_12.