Justification of the selection of the architecture and hyperparameters of artificial neural networks (ANN), focused on solving various classes of applied problems, is a scientific and methodological problem. Optimizing the selection of ANN hyperparameters allows you to improve the quality and speed of ANN training. Various methods of optimizing the selection of ANN hyperparameters are known - the use of evolutionary calculations, genetic algorithms, etc., but they require the use of additional software. To optimize the process of selecting ANN hyperparameters, Google Research has developed the KerasTuner software tool. It is a platform for automated search of a set of optimal combinations of hyperparameters. In Kerastuner, you can use various methods - random search, Bayesian optimization, or Hyperband. In the numerical experiments conducted by the author, 14 hyperparameters were varied, including the number of blocks of convolutional layers and the filters forming them, the type of activation function, the parameters of the "dropout" layers, and others. The studied tools demonstrated high efficiency while simultaneously varying more than a dozen optimized parameters of the convolutional network. The calculation time on the Colaboratory platform for the various combined ANN architectures studied, including recurrent RNN networks, was several hours, even with the use of GPU graphics accelerators. For ANN, focused on the processing and recognition of retrospective information, an increase in the quality of recognition was achieved to 80 ... 95%.
Neural network technologies are successfully used in solving problems from various
areas of the economy, including industry, agriculture, and medicine [
The mentioned authors note the problems of justifying the choice of ins architecture and hyperparameters aimed at solving various classes of applied problems. There are known methods for optimizing ins hyperparameters, for example, using genetic algorithms, but this requires writing additional software.
Of particular interest is the publication of L. Li, K. Jamieson, G. DeSalvo, A.
Rostamizadeh, and A. Talwalkar, dedicated to the Keras Tune tool developed by Google
Research to optimize the process of selecting ins hyperparameters [
The issues of text data analysis, including in natural language (NLP), are considered in detail by such researchers as B. Bengforth, R. Bilbro, T. Ojeda, H. Palangi, A. Surkova, I. Chernobaev, who note additional difficulties in processing data in Russian [16-19]. 2
As a convenient software tool for creating software prototypes, the authors used the popular Python v. 3.7 language. To quickly create a software prototype, they used Google Colaboratory, a cloud platform from Google designed to distribute machine learning technologies and deep neural networks. The Colaboratory platform already has a lot of necessary libraries installed, as well as quite powerful Tesla K80 GPUs that significantly accelerate the learning process of neural networks.
Kerastuner was used as a tool for searching optimized hyperparameters. it allows creating custom instances of the Hyperband class, the parameters of which are shown in Table 1.
In the Kerastuner Toolkit, you can use Random search, Bayesian optimization, or
HyperBand methods [
To start the procedure for optimizing the ANN parameters, call the "tuner.search" method.
To test the functioning of the hyperparameter search module, you can use the wellknown Cifar-10 data set, which is built into TensorFlow.
In order to study the use of the Keras library's Kerastuner tool on the example of a convolutional ins, software modules for creating a network with hyperparameters that usually do not change during network training were adapted. You must specify a function that will provide variation of the necessary hyperparameters.
These parameters were the number of blocks of convolutional layers and their filters, the type of activation functions, parameters of regulatory layers “dropout”, types of Pooling, etc. (Figure 1).
It is possible to set the initial parameter values (default) from the range of variation.
After that, we create an instance of the tuner, which uses the “build_model(hp)”
function prepared above for building the model. In the fragment below, the “Hyperband”
class of the optimization algorithm will be used to search for ins hyperparameters.
Note that you can limit the number of ins launches with the max_trials parameter,
which is recommended to be set to the order of several hundred [
As output values, the module shows the dimension of the search hyperspace and the values of the variable ANN parameters at the current time, as well as the value of the "objective" value.
The iterative process of searching for combinations of parameters is quite lengthy and requires the use of a GPU. The optimization software module provides variation of parameters in space with dimension 14.
Visualization of the main results of optimization of ins parameters using tuner. search, performed on the Colaboratory platform using GPU graphics accelerators, is shown in Figure 4, a) ... d).
The diagrams in Figure 4 on the ordinate axis show the values of “validate accuracy” achieved by the ins during training on a test sample.
Diagram a) represents the influence of the number of convolutional network feature maps on the first layer, diagram b) on the second, diagram c) on the third, and diagram d) diagram - effect of the number of neurons in the first convolutional hidden layer. 4
The analysis of the diagram shows the influence of the set of basic hyperparameters of the optimized ANN on the value of its recognition accuracy. The value objective = 'val_accuracy', calculated from the test sample (Figure 3), was taken as an estimated indicator. Each of the variants of the influence of variation of individual hyperparameters presented in Figure 4 is characterized by multimodality, especially diagrams a) and b), so it is impossible to unambiguously recommend a priori a combination of preferred values of the studied hyperparameters.
After completing the procedure for selecting a combination of hyperparameters, you can get the best options from the models that were found in the search process, using the “get_best_models” function. It is also possible to view the numerical values of optimal hyperparameters that were found in the automated search process.
Note the significant calculation time, which is several hours even when using GPU graphics accelerators. Optimized neural networks are used to determine the authorship of natural language text corpora prepared for training.
It is experimentally established that among the key hyperparameters, the number of convolutional layers and neurons in them, as well as the parameters of convolutional layers and their combination, have the greatest influence. The study of the possibility of automated selection of ins hyperparameters using the “Kerastuner” tool showed the following.
The “Kerastuner” tool demonstrated high optimization efficiency while simultaneously varying one and a half dozen parameters of the convolutional network, but the counting time on the Colaboratory platform for the studied ANN architectures was several hours, even with the use of GPU graphics accelerators. For ins focused on processing and recognizing text information in natural language (NLP), the recognition quality has been improved to 80...95%.
Graphical analysis of the influence of variation of individual hyperparameters on the quality of ins operation revealed multimodality of diagrams for various combinations of hyperparameters, especially for the number of feature maps of convolutional layers, so it is impossible to recommend a priori a combination of preferred values of the studied hyperparameters. For this purpose, it is desirable to perform joint automated variation of hyperparameters to improve the quality of the ANN operation.
The reported study was funded by RFBR and EISR according to the research project No. 20-011-31648\20. 16. Puchkov, A., Dli, M., Kireyenkova, M.: Fuzzy classification on the base of convolutional neural networks. Advances in Intelligent Systems and Computing, 902, 379-91 (2020). 17. Suvajit, D. et al.: A comparative study of deep learning models for medical image classification. IOP Conf. Ser.: Mater. Sci. Eng., 263, 042097 (2017). 18. Lomakina, L., Rodionov, V. and Surkova, A.: Hierarchical clustering of text documents
Automation and Remote Control, 75(7), 1309-15 (2014). 19. Zhevnerchuk, D., Surkova, A., Lomakina, L. and Golubev, A.: Semantic modeling and structural synthesis of onboard electronics protection means as open information system Journal of Physics: Conference Series, 1015, 032157 (2018).