Deep neural networks have become the state-ofart methods in many fields of machine learning recently. Still, there is no easy way how to choose a network architecture which can significantly influence the network performance. This work is a step towards an automatic architecture design. We propose an algorithm for an optimization of a network architecture based on evolution strategies. The algorithm is inspired by and designed directly for the Keras library [3] which is one of the most common implementations of deep neural networks. The proposed algorithm is tested on MNIST data set and the prediction of air pollution based on sensor measurements, and it is compared to several fixed architectures and support vector regression.
Deep neural networks (DNN) have become the
state-ofart methods in many fields of machine learning in recent
years. They have been applied to various problems,
including image recognition, speech recognition, and natural
language processing [
Deep neural networks are feed-forward neural networks with multiple hidden layers between the input and output layer. The layers typically have different units depending on the task at hand. Among the units, there are traditional perceptrons, where each unit (neuron) realizes a nonlinear function, such as the sigmoid function, or the rectified linear unit (ReLU).
While the learning of weights of the deep neural network is done by algorithms based on the stochastic gradient descent, the choice of architecture, including a number and sizes of layers, and a type of activation function, is done manually by the user. However, the choice of architecture has an important impact on the performance of the DNN. Some kind of expertise is needed, and usually a trial and error method is used in practice.
In this work we exploit a fully automatic design of
deep neural networks. We investigate the use of
evolution strategies for evolution of a DNN architecture. There
are not many studies on evolution of DNN since such
approach has very high computational requirements. To keep
the search space as small as possible, we simplify our
model focusing on implementation of DNN in the Keras
library [
The proposed algorithm is evaluated both on benchmark
and real-life data sets. As the benchmark data we use the
MNIST data set that is classification of handwritten digits.
The real data set is from the area of sensor networks for
air pollution monitoring. The data came from De Vito et
al [
The paper is organized as follows. Section 2 brings an overview of related work. Section 3 briefly describes the main ideas of our approach. In Section 4 our algorithm based on evolution strategies is described. Section 5 summarizes the results of our experiments. Finally, Section 6 brings conclusion. 2
Neuroevolution techniques have been applied successfully
for various machine learning problems [
There were quite many attempts on architecture
optimization via evolutionary process (e.g. [
On the other hand, studies dealing with evolution of deep neural networks and convolutional networks started to emerge only very recently. The training of one DNN usually requires hours or days of computing time, quite often utilizing GPU processors for speedup. Naturally, the evolutionary techniques requiring thousands of training trials were not considered a feasible choice. Nevertheless, there are several approaches to reduce the overall complexity of neuroevolution for DNN. Still due to limited computational resources, the studies usually focus only on parts of network design.
For example, in [
In [
In our approach we use evolution strategies to search for optimal architecture of DNN, while the weights are learned by gradient based technique.
The main idea of our approach is to keep the search space as small as possible, therefore the architecture specification is simplified. It directly follows the implementation of DNN in Keras library, where networks are defined layer by layer, each layer fully connected with the next layer. A layer is specified by number of neurons, type of an activation function (all neurons in one layer have the same type of an activation function), and type of regularization (such as dropout).
In this paper, we work only with fully connected feedforward neural networks, but the approach can be further modified to include also convolutional layers. Then the architecture specification would also contain type of layer (dense or convolutional) and in case of convolutional layer size of the filter. 4
Evolution strategies (ES) were proposed for work with
real-valued vectors representing parameters of complex
optimization problems [
Currently used evolution strategies may carry more
meta-parameters of the problem in the individual than just
a vector of mutation variances. A successful version of
evolution strategies, the so-called covariance matrix
adaptation ES (CMA-ES) [
In our implementation (n, m)-ES (see Alg. 1) is used. Offspring are generated using both mutation and crossover operators. Since our individuals are describing network topology, they are not vectors of real numbers. So our operators slightly differ from classical ES. The more detail description follows.
Algorithm 1 (n, m)-Evolution strategy optimizing realvalued vector and utilizing adaptive variance for each parameter procedure (n, m)-ES t ← 0
Initialize population Pt n by randomly generated vectors ~xt = (xt1, . . . , xtN,σ1t , . . . ,σ Nt )
Evaluate individuals in Pt while not terminating criterion do for i ← 1, . . . , m do choose randomly a parent ~xti, generate an offspring ~yt
i by Gaussian mutation: for j ← 1, . . . , N do σ ′j ← σ j · (1 + α · N(0, 1)) x′j ← x j + σ ′j · N(0, 1) end for insert ~yti to offspring candidate population Pt′ end for
Deterministically choose Pt+1 as n best individuals from P′
t Discard Pt and Pt′ t ← t + 1 end while end procedure
Individuals are coding feed-forward neural networks implemented as Keras model Sequential. The model implemented as Sequential is built layer by layer, similarly an individual consists of blocks representing individual layers.
I = ( [size1, drop1, act1,σ1size,σ1drop]1, . . . , [sizeH , dropH , actH ,σ Hsize,σHdrop]H ), where H is the number of hidden layers, sizei is the number of neurons in corresponding layer that is dense (fully connected) layer, dropi is the dropout rate (zero value represents no dropout), acti ∈ { relu, tanh, sigmoid, hardsigmoid, linear} stands for activation function, and σisize and σidrop are strategy coefficients corresponding to size and dropout.
So far, we work only with dense layers, but the individual can be further generalized to work with convolutional layers as well. Also other types of regularization can be considered, we are limited to dropout for the first experiments. 4.2
The operator crossover combines two parent individuals and produces two offspring individuals. It is implemented as one-point crossover, where the cross-point is on the border of a block.
Let two parents be
Ip1 = (B1p1, B2p1, . . . , Bkp1)
Ip2 = (B1p2, B2p2, . . . , Blp2), then the crossover produces offspring
Io1 = (B1p1, . . . , Bcp1, Bcpp22+1, . . . , Blp2)
p1 Io1 = (B1p2, . . . , Bcp2, Bcpp11+1, . . . , Bkp1),
p2 where cp1 ∈ { 1, . . . , k − 1} and cp2 ∈ { 1, . . . , l − 1} . 4.3
The operator mutation brings random changes to an individual. Each time an individual is mutated, one of the following mutation operators is randomly chosen: • mutateLayer - introduces random changes to one randomly selected layer. One of the following operators is randomly chosen: – changeLayerSize - the number of neurons is changed. Gaussian mutation is used, adapting strategy parameters σ size, the final number is rounded (since size has to be integer). – changeDropOut - the dropout rate is changed using Gaussian mutation adapting strategy parameters σ drop. – changeActivation - the activation function is changed, randomly chosen from the list of available activations. • addLayer - one randomly generated block is inserted at random position. • delLayer - one randomly selected block is deleted.
Note, that the ES like mutation comes in play only when size of layer or dropout parameter is changed. Otherwise the strategy parameters are ignored. 4.4
Fitness function should reflect a quality of the network represented by an individual. To assess the generalization ability of the network represented by the individual we use a crossvalidation error. The lower the crossvalidation error, the higher the fitness of the individual.
Classical k-fold crossvalidation is used, i.e. the training set is split into k-folds and each time one fold is used for testing and the rest for training. The mean error on the testing set over k run is evaluated.
The mean squared error is used as an error function: E = 100 1 N
∑( f (xt ) − yt )2,
N t=1 where T = (x1, y1), . . . , (xN , yN ) is the actual testing set and f is the function represented by the learned network. 4.5
The tournament selection is used, i.e. each turn of the tournament k individuals are selected at random and the one with the highest fitness, in our case the one with the lowest crossvalidation error, is selected.
Our implementation of the proposed algorithm is
available at [
5.1
For the first experiment we used real-world data from the
application area of sensor networks for air pollution
monitoring [
The sensor data contain tens of thousands measurements of gas multi-sensor MOX array devices recording concentrations of several gas pollutants collocated with a conventional air pollution monitoring station that provides labels for the data. The data are recorded in 1 hour intervals, and there is quite a large number of gaps due to sensor malfunctions. For our experiments we have chosen data from the interval of March 10, 2004 to April 4, 2005, taking into account each hour where records with missing values were omitted. There are altogether 5 sensors as inputs and 5 target output values representing concentrations of CO, NO2, NOx, C6H6, and NMHC.
The whole time period is divided into five intervals. Then, only one interval is used for training, the rest is utilized for testing. We considered five different choices of the training part selection. This task may be quite difficult, since the prediction is performed also in different parts of the year than the learning, e.g. the model trained on data obtained during winter may perform worse during summer (as was suggested by experts in the application area).
Table 1 brings overview of data sets sizes. All tasks have 8 input values (five sensors, temperature, absolute and relative humidity) and 1 output (predicted value). All values are normalized between h0, 1i.
The MNIST data set contains 70 000 images of hand written digits, 28 × 28 pixel each (see Fig. 1). 60 000 are used for training, 10 000 for testing. 0 0 0 0 5 5 5 5 10 10 10 10 15 15 15 15 20 20 20 20 250 5 10 15 20 25 250 5 10 15 20 25 250 5 10 15 20 25 250 5 10 15 20 25 0 0 0 0 5 5 5 5 10 10 10 10 15 15 15 15 20 20 20 20 250 5 10 15 20 25 250 5 10 15 20 25 250 5 10 15 20 25 250 5 10 15 20 25 For the sensor data the proposed algorithm was run for 100 generations for each data set, with n = 10 and m = 30. During fitness function evaluation the network weights are trained by RMSprop (one of the standard algorithms) for 500 epochs. Besides the ES classical GA was implemented and run on sensor data with same fitness function.
For the MNIST data set, the algorithm was run for 30 generations, with n = 5 and m = 10, for fitness evaluation the RMSprop was run for 20 epochs.
When the best individual is obtained, the corresponding network is built and trained on the whole training set and evaluated on the test set. 5.3
The resulting testing errors obtained by GA and ES in the first experiment are listed in Table 3. There are average, standard deviation, minimum and maximum errors over 10 computations. The performance of ES over GA is slightly better, the ES achieved lower errors in 15 cases, GA in 11 cases.
Table 4 compares ES testing errors to results obtained
by support vector regression (SVR) with linear, RBF,
polynomial, and sigmoid kernel function. SVR was trained
using Scikit-learn library [
The ES outperforms the SVR, it found best results in 17 cases.
Finally, Table 5 compares the testing error of evolved network to error of three fixed architectures (for example 30-10-1 stands for 2 hidden layers of 30 and 10 neurons, one neuron in output layers, ReLU activation is used and dropout 0.2). The evolved network achieved the most (10) best results.
Since this task does not have much training samples, also the networks evolved are quite small. The typical evolved network had one hidden layer of about 70 neurons, dropout rate 0.3 and ReLU activation function.
The second experiment was the classification of MNIST letters. As a baseline architecture was taken the one from Keras examples, i.e. network with two hidden layers of 512 ReLU units each, both with dropout 0.2. This network has a fairly good performance. It was trained 10 times and the results are listed in Table 2, together with results obtained by the evolved network.
The evolved network had also two hidden layers, first with 736 ReLU units and dropout parameter 0.09, the second with 471 hard sigmoid units and dropout 0.2. The ES found a competitive result, the evolved network achieved better accuracy than the baseline model. 6
We have proposed an algorithm for automatic design of DNNs based on evolution strategies. The algorithm was tested in experiments on the real-life sensor data set and MNIST dataset of handwritten digits. On sensor data set, the solutions found by our algorithm outperforms SVR and selected fixed architectures. The activation function dominating in solutions is the ReLU function. For the MNIST data set, the network with ReLU and hard sigmoid units was found, outperforming the baseline solution. We have shown that our algorithm is able to found competitive solutions.
The main limitation of the algorithm is the time complexity. One direction of our future work is to try to lower the number of fitness evaluations using surrogate modeling or to use asynchronous evolution.
Also we plan to extend the algorithm to work also with convolutional networks and to include more parameters, such as other types of regularization, the type of optimization algorithm, etc.
The gradient based optimization algorithm depends significantly on the random initialization of weights. One way to overcome this is to combine the evolution of weights and gradient based local search that is another possibility of future work.
This work was partially supported by the Czech Grant Agency grant 15-18108S and institutional support of the Institute of Computer Science RVO 67985807.
Access to computing and storage facilities owned by parties and projects contributing to the National Grid Infrastructure MetaCentrum provided under the programme "Projects of Large Research, Development, and Innovations Infrastructures" (CESNET LM2015042), is greatly appreciated.