At present, the models and computer vision algorithms are increasingly used in various fields of activity. For example, in systems of sample analysis in medicine and pharmacology, in identification of individuals by a fingerprint, iris or face, in video surveillance security systems and in many other systems and applications. In connection with the growth of computing power and the emergence of big databases of images, it became possible to learn and use deep neural networks for solving the problems of classification and recognition. As to the image classification problem, the Convolutional Neural Networks showed themselves best of all; every year since 2012, they won the prestigious international contest - the ImageNet Large Scale Visual Classification Challenge (ILSVRC), in which such giants as Google and Microsoft participated. Thanks to the revealing of their capabilities, the convolutional neural networks are increasingly used for pattern recognition, image classification, object detection, semantic segmentation, and solving many other problems. The paper examines the evolution of the most efficient models and trends in development of architecture of convolutional neural networks, which are currently used for classification of images that have been included in the list of winners of this international competition, ILSVRC. More precisely, the key features of architecture and its annual variations are revealed on the background of increasing efficiency of practical application of these networks. The data of numerous experiments conducted over the past few years are summarized, classes of applied problems are analyzed, and estimates are given for an effectiveness of use of the considered convolutional neural networks. In fact, these performance estimates are based on evaluation of probability of adequate classification of images. On this basis, a generalized algorithm is formulated, and practical recommendations are proposed taking into account the problem features.
for identifying vehicles by their license plates and in many other systems and
applications [
A convolutional neural network is a neural network with a convolutional layer.
Usually in the convolutional neural networks there are also a sub-sampling layer
(pooling layer) and a fully connected layer. Convolutional neural networks are used
for pattern recognition, object detection, image classification, semantic segmentation,
and other tasks. In convolutional neural networks, layers of convolution and
subsampling consist of several “levels” of neurons, called feature maps, or channels. Each
neuron of this layer is connected to a small section of the previous layer, called a
receptive field. In the case of an image, a feature map is a two-dimensional array of
neurons, or simply a matrix. Other measurements can be used if another kind of data
is taken as input, for example, audio data (one-dimensional array) or volume data
(three-dimensional array) [
At the same time, although such networks are used quite successfully, the question of choosing the optimal architecture and setting the parameters of the neural network are remains unresolved. In this regard, the task of the work is to analyze the available experimental data using the most efficient convolutional neural networks used to classify images, in order to develop a general algorithm and practical recommendations on choosing the best architecture and setting the parameters of the neural network, according to the specifics of the problem. 2
This section presents the most efficient and widely used architectures of convolutional neural networks for classifying images that are arranged in chronological order. 2.1
The first neural network that won the ILSVRC image classification competition was AlexNet, in 2012, reaching a top-5 classification error of 15.31%. For comparison, the method that does not use convolutional neural networks received a classification error of 26.1%. AlexNet collected the latest technology at the time to improve the network. The architecture of this network is shown in Fig. 1.
Training network AlexNet due to the large number of network parameters occurred on two graphics processors (abbreviated GPU – Graphics Processing Unit), which reduced training time in comparison with learning based on the central processor (abbreviated CPU – Central Processing Unit). It also turned out that using the Rectified Linear Unit (ReLU) activation function instead of more traditional functions (sigmoids and hyperbolic tangent) made it possible to reduce the number of learning epochs by six times. This is due to the fact that the function of network activation Rectified Linear Unit allows you to overcome the problem of gradient attenuation inherent in other activation functions. Graphically, the activation function of the Rectified Linear Unit is shown in Fig. 2.
Also, a dropout technique (Dropout) was used in AlexNet, which randomly turns off each neuron on a given layer with a probability p at each epoch. Then, after learning the network, at the recognition stage, the weights of the layers to which the dropout was applied should be multiplied by 1/p. Technology Dropout acts as a regularizer, not allowing the network to retrain. To understand the effectiveness of this technique, there are several interpretations. First, this dropout causes neurons not to rely on neighboring neurons, but to learn to recognize more persistent signs. And the second, later, is that learning a network with a dropout is an approximation of learning a network of ensembles, each of which represents a network without some neurons. As a result, the probability of error is reduced, since the final decision is made not by one network, but by an ensemble, each network of which is trained differently. 2.2
The convolutional neural network ZF Net is the winner of ILSVRC 2013 with a top-5 classification error of 14.8%. The main achievement of this architecture is the creation of a filter visualization technique - a sweep network (deconvolutional network), consisting of operations, in a sense, reverse operations of the network. As a result, the network sweep displays a hidden layer of the network on the original image.
To study the behavior of the filter on a particular image using a trained neural network, you must first make a network output, then in the layer of the studied filter zero all weights, except the weights of the filter itself, and then apply the resulting activation to the network of the sweep network. The network sweeps consistently used operations Unpooling ReLU and filtering. The Unpooling operation partially restores the input of the corresponding sub-sampling layer by remembering the coordinates that the sub-sampling layer has selected. The ReLU operation is a regular layer that uses the ReLU function. The filtering layer performs the convolution operation with the weights of the corresponding convolution layer, but the weights of each filter are “inverted” vertically and horizontally. Thus, the initial activation of the filter moves in the opposite direction until it is displayed in the original image space. The architecture of the considered network is shown in Fig. 3. VGG Net is a convolutional neural network model that won the 2014 image classification competition. In this network, they refused to use filters larger than 3x3. Since the authors proved that the 7x7 filter layer is equivalent to three layers with 3x3 filters, and in this case 55% less parameters are used. Similarly, a 5x5 filter layer is equivalent to two layers with a 3x3 filter, which saves 22% of network parameters.
Features of the architecture and internal organization of this neural network are
shown in Fig. 4.
The Inveption-v1 convolution neural network is the winner of the ILSVRC 2014
competition with a top-5 error of 6.7%, also known as GoogleNet. The creators of this
network, led by Christian Szegedy, proceeded from the fact that after each layer of the
network it is necessary to make a choice whether the next layer will be a convolution
with a 3x3, 5x5, 1x1 filter or a subsampling layer. Each of these layers is useful – a
1x1 filter reveals a correlation between channels, while larger filters respond to more
global features, and a subsampling layer reduces dimensionality without large loss of
information. Instead of choosing which layer should be next, it is proposed to use all
layers at once, parallel to each other, and then merge the results into one. To avoid an
increase in the number of parameters, a 1x1 convolution is used in front of each
convolution layer, which reduces the number of feature maps. Such a block of layers was
called an Inception module. The architecture features of this neural network are
shown in Fig. 5.
40
Fig. 5. The architecture of the convolutional neural network GoogleNet [
Also, GoogLeNet abandoned the use of a fully connected layer at the end of the network, using the Average Pooling layer instead, which drastically reduced the number of parameters in the network. Thus, GoogLeNet, consisting of more than one hundred basic layers, has almost 12 times fewer parameters than AlexNet (about 7 million parameters against 138 million).
In the next iteration of the Inception module, called Inception-v2, the authors, as was done on the VGG network, decomposed the 5x5 layer into two 3x3 layers. Next, the Batch Normalization technique was used, which allows to multiply the learning speed by means of normalizing the distribution of layer outputs within the network.
In the same article [
The winner of the ILSVRC 2015 competition with a top-5 error of 3.57% was an ensemble of six networks of the ResNet (Residual Network) type, developed at Microsoft Research. The authors of ResNet have noticed that with the addition of new layers, the quality of the model grows to a certain limit (see VGG-19), and then begins to fall. This problem is called the degradation problem, a decrease in accuracy on the validation set.
The authors were able to find such a topology in which the quality of the model grows with the addition of new layers. A neural network can approximate almost any function, for example, some complex function H (x). Then it is true that such a network will easily learn the residual function: F (x) = H (x) – x. Obviously, that our initial objective function will be H (x) = F (x) + x. If we take a certain network, for example, VGG-19, and add twenty layers to it, we would like the deep network to behave at least as good as its shallow analogue.
The problem of degradation implies that a complex nonlinear function F (x), obtained by adding several layers, must learn the same transformation, if the previous layers had reached the quality limit. But this does not happen; it is possible that the optimizer simply cannot cope with adjusting the weights so that a complex non-linear hierarchical model does the same transformation. In order to "help" the network, it was proposed to introduce a missing connection (Shortcut Connections). The architecture features of this neural network is shown in Fig. 6. 2.6
After the success of applying the ResNet convolutional neural network, the following versions of the Inception network were introduced: Inception-v4 and InceptionResNet. In both cases, the Inception module was divided into modules A, B, and C for inputs with dimensions of 35x35, 17x17, and 8x8, respectively. Reduction blocks were also identified, in which the dimensionality decreases and the depth of the data inside the network increases. In Inception-v4, the main innovations are the replacement of Max Pooling with Average Pooling in the Inception modules themselves.
For Inception-ResNet, skipping connections have been added to the Inception modules. Two versions of the network were designed – Inception-ResNet-v1, which requires less computation, and Inception-ResNet-v2.
To estimate the convolutional neural network models in addition to the type of errors usually indicate the number of models in the ensemble and the number of notches images that were fed to the input of each model. For example, 10 notches means that four notches are made at the corners of the image, one notch in the center, and each notch is additionally horizontally inverted.
According to numerous experiments [
In the Tab. 1 shown the results of the considered neural networks with one model and one cutout based on ImageNet images (except ResNet-152, for which the result for 10 notches is indicated).
In Tab. 2 shown the results of using ensembles of models with many cutouts based on ImageNet images.
As can be seen from these tables, for five years, from 2012 to 2016, the Top-5 error on ImageNet for single models decreased almost four times (from 17% to 4.49%), and for the ensemble – almost five times ( from 15.40% to 3.10%).
Analyzing the experimental data (Tab. 1, Tab. 2), we can conclude that the choice of network architecture is made according to the following criteria: classification errors, performance, and the complexity of learning a neural network. For this, the following algorithm is usually used.
Initially, guided by certain requirements, they set a permissible classification error. For example, it is currently believed that a classification error when using human vision is in the range from 5% to 10%. If you look at the classification error of the latest convolutional neural networks, you can see that they are coping with this task as well as a human. This means that in the classification problems that a person solved classically, you can choose any network with an error not higher than the specified one. Based on the analysis of data in Tab. 2, we can conclude that the last five networks will suit us. But we need one. What should be done?
Next, the selection of an admissible network is made in order to satisfy the specified restrictions on labor intensity (estimates of labor intensity are given in Tab. 1), taking into account the available hardware capacities. This choice is also made taking into account the time constraints on the network learning process, since with the increase in the number of layers and network parameters, the training time will also increase.
The choice of complexity can also be ambiguous, since at the first stage five networks were chosen. In such a situation, the most important criterion is usually identified and the best network is selected by this criterion.
To improve the quality of the classification results, it is planned to use specialized
frame preprocessing models and algorithms [
In the course of considering the most effective models of convolutional neural networks used in our time for the purposes of image classification, an analysis of their architectural features was performed. According to numerous experiments, a generalization and analysis of the results of the efficiency of using neural networks for image classification (Tab. 1, Tab. 2) was made. On this basis, a generalized algorithm is formulated and practical recommendations are given regarding the choice of the best architecture of a neural network, respectively, the specifics of the problem.