J. Hlaváčová (Ed.): ITAT 2017 Proceedings, pp. 159–166 CEUR Workshop Proceedings Vol. 1885, ISSN 1613-0073, c 2017 P. Vidnerová, R. Neruda Evolution Strategies for Deep Neural Network Models Design Petra Vidnerová, Roman Neruda Institute of Computer Science, The Czech Academy of Sciences petra@cs.cas.cz Abstract: Deep neural networks have become the state-of- The proposed algorithm is evaluated both on benchmark art methods in many fields of machine learning recently. and real-life data sets. As the benchmark data we use the Still, there is no easy way how to choose a network ar- MNIST data set that is classification of handwritten digits. chitecture which can significantly influence the network The real data set is from the area of sensor networks for performance. air pollution monitoring. The data came from De Vito et This work is a step towards an automatic architecture al [21, 5] and are described in detail in Section 5.1. design. We propose an algorithm for an optimization of a The paper is organized as follows. Section 2 brings an network architecture based on evolution strategies. The al- overview of related work. Section 3 briefly describes the gorithm is inspired by and designed directly for the Keras main ideas of our approach. In Section 4 our algorithm library [3] which is one of the most common implementa- based on evolution strategies is described. Section 5 sum- tions of deep neural networks. marizes the results of our experiments. Finally, Section 6 The proposed algorithm is tested on MNIST data set brings conclusion. and the prediction of air pollution based on sensor mea- surements, and it is compared to several fixed architectures and support vector regression. 2 Related Work Neuroevolution techniques have been applied successfully 1 Introduction for various machine learning problems [6]. In classical neuroevolution, no gradient descent is involved, both ar- chitecture and weights undergo the evolutionary process. Deep neural networks (DNN) have become the state-of- However, because of large computational requirements the art methods in many fields of machine learning in recent applications are limited to small networks. years. They have been applied to various problems, in- There were quite many attempts on architecture opti- cluding image recognition, speech recognition, and natural mization via evolutionary process (e.g. [19, 1]) in previous language processing [8, 10]. decades. Successful evolutionary techniques evolving the Deep neural networks are feed-forward neural networks structure of feed-forward and recurrent neural networks with multiple hidden layers between the input and output include NEAT [18], HyperNEAT [17] and CoSyNE [7] al- layer. The layers typically have different units depending gorithms. on the task at hand. Among the units, there are traditional On the other hand, studies dealing with evolution of perceptrons, where each unit (neuron) realizes a nonlin- deep neural networks and convolutional networks started ear function, such as the sigmoid function, or the rectified to emerge only very recently. The training of one DNN linear unit (ReLU). usually requires hours or days of computing time, quite While the learning of weights of the deep neural net- often utilizing GPU processors for speedup. Naturally, work is done by algorithms based on the stochastic gradi- the evolutionary techniques requiring thousands of train- ent descent, the choice of architecture, including a number ing trials were not considered a feasible choice. Never- and sizes of layers, and a type of activation function, is theless, there are several approaches to reduce the overall done manually by the user. However, the choice of archi- complexity of neuroevolution for DNN. Still due to limited tecture has an important impact on the performance of the computational resources, the studies usually focus only on DNN. Some kind of expertise is needed, and usually a trial parts of network design. and error method is used in practice. For example, in [12] CMA-ES is used to optimize hy- In this work we exploit a fully automatic design of perparameters of DNNs. In [9] the unsupervised convo- deep neural networks. We investigate the use of evolu- lutional networks for vision-based reinforcement learning tion strategies for evolution of a DNN architecture. There are studied, the structure of CNN is held fixed and only a are not many studies on evolution of DNN since such ap- small recurrent controller is evolved. However, the recent proach has very high computational requirements. To keep paper [16] presents a simple distributed evolutionary strat- the search space as small as possible, we simplify our egy that is used to train relatively large recurrent network model focusing on implementation of DNN in the Keras with competitive results on reinforcement learning tasks. library [3] that is a widely used tool for practical applica- In [14] automated method for optimizing deep learning tions of DNNs. architectures through evolution is proposed, extending ex- 160 P. Vidnerová, R. Neruda isting neuroevolution methods. Authors of [4] sketch a Algorithm 1 (n, m)-Evolution strategy optimizing real- genetic approach for evolving a deep autoencoder network valued vector and utilizing adaptive variance for each pa- enhancing the sparsity of the synapses by means of special rameter operators. Finally, the paper [13] presents two version of procedure (n, m)-ES an evolutionary and co-evolutionary algorithm for design t ←0 of DNN with various transfer functions. Initialize population Pt n by randomly generated vectors ~xt = (xt1 , . . . , xtN , σ1t , . . . , σNt ) 3 Our Approach Evaluate individuals in Pt while not terminating criterion do In our approach we use evolution strategies to search for i ← 1, . . . , m do for optimal architecture of DNN, while the weights are choose randomly a parent ~xti , learned by gradient based technique. generate an offspring ~yti The main idea of our approach is to keep the search by Gaussian mutation: space as small as possible, therefore the architecture spec- for j ← 1, . . . , N do ification is simplified. It directly follows the implementa- σ ′j ← σ j · (1 + α · N(0, 1)) tion of DNN in Keras library, where networks are defined x′j ← x j + σ ′j · N(0, 1) layer by layer, each layer fully connected with the next end for layer. A layer is specified by number of neurons, type of insert ~yti to offspring candidate population Pt′ an activation function (all neurons in one layer have the end for same type of an activation function), and type of regular- Deterministically choose Pt+1 as n best individ- ization (such as dropout). uals from Pt′ In this paper, we work only with fully connected feed- Discard Pt and Pt′ forward neural networks, but the approach can be further t ← t +1 modified to include also convolutional layers. Then the end while architecture specification would also contain type of layer end procedure (dense or convolutional) and in case of convolutional layer size of the filter. 4.1 Individuals 4 Evolution Strategies for DNN Design Individuals are coding feed-forward neural networks im- Evolution strategies (ES) were proposed for work with plemented as Keras model Sequential. The model imple- real-valued vectors representing parameters of complex mented as Sequential is built layer by layer, similarly an optimization problems [2]. In the illustration algorithm individual consists of blocks representing individual lay- bellow we can see a simple ES working with n individuals ers. in a population and generating m offspring by means of Gaussian mutation. The environmental selection has two I=( [size1 , drop1 , act1 , σ1size , σ1drop ]1 , . . . , traditional forms for evolution strategies. The so called (n + m)-ES generates new generation by deterministically [sizeH , dropH , actH , σHsize , σHdrop ]H ), choosing n best individuals from the set of (n + m) par- ents and offspring. The so called (n, m)-ES generates new generation by selecting from m new offspring (typically, where H is the number of hidden layers, sizei m > n). The latter approach is considered more robust is the number of neurons in corresponding layer against local optima premature convergence. that is dense (fully connected) layer, dropi is the Currently used evolution strategies may carry more dropout rate (zero value represents no dropout), meta-parameters of the problem in the individual than just acti ∈ {relu, tanh, sigmoid, hardsigmoid, linear} a vector of mutation variances. A successful version of stands for activation function, and σisize and σidrop are evolution strategies, the so-called covariance matrix adap- strategy coefficients corresponding to size and dropout. tation ES (CMA-ES) [12] uses a clever strategy to approx- So far, we work only with dense layers, but the individ- imate the full N × N covariance matrix, thus representing ual can be further generalized to work with convolutional a general N-dimensional normal distribution. Crossover layers as well. Also other types of regularization can be operator is usually used within evolution strategies. considered, we are limited to dropout for the first experi- In our implementation (n, m)-ES (see Alg. 1) is used. ments. Offspring are generated using both mutation and crossover operators. Since our individuals are describing network 4.2 Crossover topology, they are not vectors of real numbers. So our operators slightly differ from classical ES. The more detail The operator crossover combines two parent individuals description follows. and produces two offspring individuals. It is implemented Evolution Strategies for Deep Neural Network Models Design 161 as one-point crossover, where the cross-point is on the bor- 4.5 Selection der of a block. Let two parents be The tournament selection is used, i.e. each turn of the tour- nament k individuals are selected at random and the one I p1 = (B1p1 , B2p1 , . . . , Bkp1 ) with the highest fitness, in our case the one with the low- I p2 = (B1p2 , B2p2 , . . . , Blp2 ), est crossvalidation error, is selected. then the crossover produces offspring Our implementation of the proposed algorithm is avail- able at [20]. Io1 = (B1p1 , . . . , Bcp1 p1 p2 , Bcp2+1 , . . . , Blp2 ) Io1 = (B1p2 , . . . , Bcp2 p2 p1 , Bcp1+1 , . . . , Bkp1 ), 5 Experiments where cp1 ∈ {1, . . . , k − 1} and cp2 ∈ {1, . . . , l − 1}. 5.1 Data Set 4.3 Mutation The operator mutation brings random changes to an in- For the first experiment we used real-world data from the dividual. Each time an individual is mutated, one of the application area of sensor networks for air pollution mon- following mutation operators is randomly chosen: itoring [21, 5], for the second experiment the well known • mutateLayer - introduces random changes to one ran- MNIST data set [11]. domly selected layer. One of the following operators The sensor data contain tens of thousands measure- is randomly chosen: ments of gas multi-sensor MOX array devices recording concentrations of several gas pollutants collocated with a – changeLayerSize - the number of neurons is conventional air pollution monitoring station that provides changed. Gaussian mutation is used, adapting labels for the data. The data are recorded in 1 hour in- strategy parameters σ size , the final number is tervals, and there is quite a large number of gaps due to rounded (since size has to be integer). sensor malfunctions. For our experiments we have chosen – changeDropOut - the dropout rate is changed data from the interval of March 10, 2004 to April 4, 2005, using Gaussian mutation adapting strategy pa- taking into account each hour where records with missing rameters σ drop. values were omitted. There are altogether 5 sensors as in- – changeActivation - the activation function is puts and 5 target output values representing concentrations changed, randomly chosen from the list of avail- of CO, NO2 , NOx, C6H6, and NMHC. able activations. The whole time period is divided into five intervals. • addLayer - one randomly generated block is inserted Then, only one interval is used for training, the rest is uti- at random position. lized for testing. We considered five different choices of the training part selection. This task may be quite difficult, • delLayer - one randomly selected block is deleted. since the prediction is performed also in different parts of Note, that the ES like mutation comes in play only when the year than the learning, e.g. the model trained on data size of layer or dropout parameter is changed. Otherwise obtained during winter may perform worse during summer the strategy parameters are ignored. (as was suggested by experts in the application area). Table 1 brings overview of data sets sizes. All tasks have 4.4 Fitness 8 input values (five sensors, temperature, absolute and rel- ative humidity) and 1 output (predicted value). All values Fitness function should reflect a quality of the network are normalized between h0, 1i. represented by an individual. To assess the generalization ability of the network represented by the individual we use a crossvalidation error. The lower the crossvalidation er- Table 1: Overview of data sets sizes. ror, the higher the fitness of the individual. Classical k-fold crossvalidation is used, i.e. the training Task train set test set set is split into k-folds and each time one fold is used for CO 1469 5875 testing and the rest for training. The mean error on the NO2 1479 5914 testing set over k run is evaluated. NOx 1480 5916 The mean squared error is used as an error function: C6H6 1799 7192 NMHC 178 709 1 N E = 100 ∑ ( f (xt ) − yt )2 , N t=1 The MNIST data set contains 70 000 images of hand where T = (x1 , y1 ), . . . , (xN , yN ) is the actual testing set and written digits, 28 × 28 pixel each (see Fig. 1). 60 000 are f is the function represented by the learned network. used for training, 10 000 for testing. 162 P. Vidnerová, R. Neruda 0 0 0 0 5 5 5 5 10 10 10 10 Table 2: Test accuracies on the MNIST data set. 15 15 15 15 20 20 20 20 25 25 25 25 model avg std min max 0 0 5 10 15 20 25 0 0 5 10 15 20 25 0 0 5 10 15 20 25 0 0 5 10 15 20 25 baseline 98.34 0.13 98.18 98.55 5 10 5 10 5 10 5 10 evolved by ES 98.64 0.05 98.55 98.73 15 15 15 15 20 20 20 20 25 25 25 25 0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25 and the results are listed in Table 2, together with results obtained by the evolved network. Figure 1: Example of MNIST data set samples. The evolved network had also two hidden layers, first with 736 ReLU units and dropout parameter 0.09, the sec- 5.2 Setup ond with 471 hard sigmoid units and dropout 0.2. The ES found a competitive result, the evolved network achieved For the sensor data the proposed algorithm was run for better accuracy than the baseline model. 100 generations for each data set, with n = 10 and m = 30. During fitness function evaluation the network weights are trained by RMSprop (one of the standard algorithms) 6 Conclusion for 500 epochs. Besides the ES classical GA was imple- mented and run on sensor data with same fitness function. We have proposed an algorithm for automatic design of For the MNIST data set, the algorithm was run for 30 DNNs based on evolution strategies. The algorithm was generations, with n = 5 and m = 10, for fitness evaluation tested in experiments on the real-life sensor data set and the RMSprop was run for 20 epochs. MNIST dataset of handwritten digits. On sensor data set, When the best individual is obtained, the corresponding the solutions found by our algorithm outperforms SVR and network is built and trained on the whole training set and selected fixed architectures. The activation function dom- evaluated on the test set. inating in solutions is the ReLU function. For the MNIST data set, the network with ReLU and hard sigmoid units 5.3 Results was found, outperforming the baseline solution. We have shown that our algorithm is able to found competitive so- The resulting testing errors obtained by GA and ES in lutions. the first experiment are listed in Table 3. There are av- The main limitation of the algorithm is the time com- erage, standard deviation, minimum and maximum errors plexity. One direction of our future work is to try to lower over 10 computations. The performance of ES over GA is the number of fitness evaluations using surrogate model- slightly better, the ES achieved lower errors in 15 cases, ing or to use asynchronous evolution. GA in 11 cases. Also we plan to extend the algorithm to work also with Table 4 compares ES testing errors to results obtained convolutional networks and to include more parameters, by support vector regression (SVR) with linear, RBF, poly- such as other types of regularization, the type of optimiza- nomial, and sigmoid kernel function. SVR was trained us- tion algorithm, etc. ing Scikit-learn library [15], hyperparameters were found The gradient based optimization algorithm depends sig- using grid search and crossvalidation. nificantly on the random initialization of weights. One The ES outperforms the SVR, it found best results in way to overcome this is to combine the evolution of 17 cases. weights and gradient based local search that is another Finally, Table 5 compares the testing error of evolved possibility of future work. network to error of three fixed architectures (for example 30-10-1 stands for 2 hidden layers of 30 and 10 neurons, one neuron in output layers, ReLU activation is used and Acknowledgment dropout 0.2). The evolved network achieved the most (10) best results. Since this task does not have much training samples, This work was partially supported by the Czech Grant also the networks evolved are quite small. The typical Agency grant 15-18108S and institutional support of the evolved network had one hidden layer of about 70 neu- Institute of Computer Science RVO 67985807. rons, dropout rate 0.3 and ReLU activation function. Access to computing and storage facilities owned by The second experiment was the classification of MNIST parties and projects contributing to the National Grid In- letters. As a baseline architecture was taken the one from frastructure MetaCentrum provided under the programme Keras examples, i.e. network with two hidden layers of "Projects of Large Research, Development, and Innova- 512 ReLU units each, both with dropout 0.2. This network tions Infrastructures" (CESNET LM2015042), is greatly has a fairly good performance. It was trained 10 times appreciated. Evolution Strategies for Deep Neural Network Models Design 163 Table 3: Errors on test set for networks found by GA and ES. The average, standard deviation, minimum and maximum of 10 evaluations of the learning algorithm are listed. GA ES avg std min max avg std min max CO part1 0.209 0.014 0.188 0.236 0.229 0.026 0.195 0.267 CO part2 0.801 0.135 0.600 1.048 0.657 0.024 0.631 0.694 CO part3 0.266 0.029 0.222 0.309 0.256 0.045 0.199 0.349 CO part4 0.404 0.226 0.186 0.865 0.526 0.108 0.308 0.701 CO part5 0.246 0.024 0.207 0.286 0.235 0.025 0.199 0.277 NOx part1 2.201 0.131 1.994 2.506 2.132 0.086 2.021 2.284 NOx part2 1.705 0.284 1.239 2.282 1.599 0.077 1.444 1.685 NOx part3 1.238 0.163 0.982 1.533 1.339 0.242 1.106 1.955 NOx part4 1.490 0.173 1.174 1.835 1.610 0.164 1.435 2.041 NOx part5 0.551 0.052 0.456 0.642 0.622 0.075 0.521 0.726 NO2 part1 1.697 0.266 1.202 2.210 1.506 0.217 1.132 1.823 NO2 part2 2.009 0.415 1.326 2.944 1.371 0.048 1.242 1.415 NO2 part3 0.593 0.082 0.532 0.815 0.660 0.078 0.599 0.863 NO2 part4 0.737 0.023 0.706 0.776 0.782 0.043 0.711 0.856 NO2 part5 1.265 0.158 1.054 1.580 0.730 0.111 0.520 0.905 C6H6 part1 0.013 0.005 0.006 0.024 0.013 0.004 0.007 0.018 C6H6 part2 0.039 0.015 0.025 0.079 0.034 0.010 0.020 0.050 C6H6 part3 0.019 0.011 0.009 0.041 0.048 0.015 0.016 0.075 C6H6 part4 0.030 0.015 0.014 0.061 0.020 0.010 0.010 0.042 C6H6 part5 0.017 0.015 0.004 0.051 0.027 0.011 0.014 0.051 NMHC part1 1.719 0.168 1.412 2.000 1.685 0.256 1.448 2.378 NMHC part2 0.623 0.164 0.446 1.047 0.713 0.097 0.566 0.865 NMHC part3 1.144 0.181 0.912 1.472 1.097 0.270 0.775 1.560 NMHC part4 1.220 0.206 0.994 1.563 1.099 0.166 0.898 1.443 NMHC part5 1.222 0.126 1.055 1.447 1.023 0.050 0.963 1.116 11 15 44% 60% 164 P. Vidnerová, R. Neruda Table 4: Test errors for evolved network and SVR with different kernel functions. For the evolved network the average, standard deviation, minimum and maximum of 10 evaluations of learning algorithm are listed. Task Evolved network SVR avg std min max linear RBF Poly. Sigmoid CO_part1 0.229 0.026 0.195 0.267 0.340 0.280 0.285 1.533 CO_part2 0.657 0.024 0.631 0.694 0.614 0.412 0.621 1.753 CO_part3 0.256 0.045 0.199 0.349 0.314 0.408 0.377 1.427 CO_part4 0.526 0.108 0.308 0.701 1.127 0.692 0.535 1.375 CO_part5 0.235 0.025 0.199 0.277 0.348 0.207 0.198 1.568 NOx_part1 2.132 0.086 2.021 2.284 1.062 1.447 1.202 2.537 NOx_part2 1.599 0.077 1.444 1.685 2.162 1.838 1.387 2.428 NOx_part3 1.339 0.242 1.106 1.955 0.594 0.674 0.665 2.705 NOx_part4 1.610 0.164 1.435 2.041 0.864 0.903 0.778 2.462 NOx_part5 0.622 0.075 0.521 0.726 1.632 0.730 1.446 2.761 NO2_part1 1.506 0.217 1.132 1.823 2.464 2.404 2.401 2.636 NO2_part2 1.371 0.048 1.242 1.415 2.118 2.250 2.409 2.648 NO2_part3 0.660 0.078 0.599 0.863 1.308 1.195 1.213 1.984 NO2_part4 0.782 0.043 0.711 0.856 1.978 2.565 1.912 2.531 NO2_part5 0.730 0.111 0.520 0.905 1.0773 1.047 0.967 2.129 C6H6_part1 0.013 0.004 0.007 0.018 0.300 0.511 0.219 1.398 C6H6_part2 0.034 0.010 0.020 0.050 0.378 0.489 0.369 1.478 C6H6_part3 0.048 0.015 0.016 0.075 0.520 0.663 0.538 1.317 C6H6_part4 0.020 0.010 0.010 0.042 0.217 0.459 0.123 1.279 C6H6_part5 0.027 0.011 0.014 0.051 0.215 0.297 0.188 1.526 NMHC_part1 1.685 0.256 1.448 2.378 1.718 1.666 1.621 3.861 NMHC_part2 0.713 0.097 0.566 0.865 0.934 0.978 0.839 3.651 NMHC_part3 1.097 0.270 0.775 1.560 1.580 1.280 1.438 2.830 NMHC_part4 1.099 0.166 0.898 1.443 1.720 1.565 1.917 2.715 NMHC_part5 1.023 0.050 0.963 1.116 1.238 0.944 1.407 2.960 17 2 2 4 68% 8% 8% 16% Evolution Strategies for Deep Neural Network Models Design 165 Table 5: Test errors for evolved network and three selected fixed architectures. Task Evolved network 50-1 30-10-1 30-10-30-1 avg std avg std avg std avg std CO_part1 0.229 0.026 0.230 0.032 0.250 0.023 0.377 0.103 CO_part2 0.657 0.024 0.861 0.136 0.744 0.142 0.858 0.173 CO_part3 0.256 0.045 0.261 0.040 0.305 0.043 0.302 0.046 CO_part4 0.526 0.108 0.621 0.279 0.638 0.213 0.454 0.158 CO_part5 0.235 0.025 0.283 0.072 0.270 0.032 0.309 0.032 NOx_part1 2.132 0.086 2.158 0.203 2.095 0.131 2.307 0.196 NOx_part2 1.599 0.077 1.799 0.313 1.891 0.199 2.083 0.172 NOx_part3 1.339 0.242 1.077 0.125 1.092 0.178 0.806 0.185 NOx_part4 1.610 0.164 1.303 0.208 1.797 0.461 1.600 0.643 NOx_part5 0.622 0.075 0.644 0.075 0.677 0.055 0.778 0.054 NO2_part1 1.506 0.217 1.659 0.250 1.368 0.135 1.677 0.233 NO2_part2 1.371 0.048 1.762 0.237 1.687 0.202 1.827 0.264 NO2_part3 0.660 0.078 0.682 0.148 0.576 0.044 0.603 0.069 NO2_part4 0.782 0.043 1.109 0.923 0.757 0.059 0.802 0.076 NO2_part5 0.730 0.111 0.646 0.064 0.734 0.107 0.748 0.123 C6H6_part1 0.013 0.004 0.012 0.006 0.081 0.030 0.190 0.060 C6H6_part2 0.034 0.010 0.039 0.012 0.101 0.015 0.211 0.071 C6H6_part3 0.048 0.015 0.024 0.007 0.091 0.047 0.115 0.031 C6H6_part4 0.020 0.010 0.026 0.010 0.051 0.026 0.096 0.020 C6H6_part5 0.027 0.011 0.025 0.008 0.113 0.025 0.176 0.058 NMHC_part1 1.685 0.256 1.738 0.144 1.889 0.119 2.378 0.208 NMHC_part2 0.713 0.097 0.553 0.045 0.650 0.078 0.799 0.096 NMHC_part3 1.097 0.270 1.128 0.089 0.901 0.124 0.789 0.184 NMHC_part4 1.099 0.166 1.116 0.119 0.918 0.119 0.751 0.096 NMHC_part5 1.023 0.050 0.970 0.094 0.889 0.085 0.856 0.074 10 6 4 5 40% 24% 16% 20% 166 P. Vidnerová, R. Neruda References [16] T. Salimans, J. Ho, X. Chen, and I. Sutskever. Evolu- tion Strategies as a Scalable Alternative to Reinforcement Learning. ArXiv e-prints, March 2017. [17] Kenneth O. Stanley, David B. D’Ambrosio, and Jason [1] Jasmina Arifovic and Ramazan Gençay. Using genetic al- Gauci. A hypercube-based encoding for evolving large- gorithms to select architecture of a feedforward artificial scale neural networks. Artif. Life, 15(2):185–212, April neural network. Physica A: Statistical Mechanics and its 2009. Applications, 289(3–4):574 – 594, 2001. [18] Kenneth O. Stanley and Risto Miikkulainen. Evolving neu- [2] H.-G. Beyer and H. P. Schwefel. Evolutionary strategies: ral networks through augmenting topologies. Evolutionary A comprehensive introduction. Natural Computing, pages Computation, 10(2):99–127, 2002. 3–52, 2002. [19] B. u. Islam, Z. Baharudin, M. Q. Raza, and P. Nallagown- [3] François Chollet. Keras. den. Optimization of neural network architecture using https://github.com/fchollet/keras, 2015. genetic algorithm for load forecasting. In 2014 5th Inter- [4] Omid E. David and Iddo Greental. Genetic algorithms for national Conference on Intelligent and Advanced Systems evolving deep neural networks. In Proceedings of the Com- (ICIAS), pages 1–6, June 2014. panion Publication of the 2014 Annual Conference on Ge- [20] Petra Vidnerová. GAKeras. netic and Evolutionary Computation, GECCO Comp ’14, github.com/PetraVidnerova/GAKeras, 2017. pages 1451–1452, New York, NY, USA, 2014. ACM. [21] S. De Vito, E. Massera, M. Piga, L. Martinotto, and G. Di [5] S. De Vito, G. Fattoruso, M. Pardo, F. Tortorella, and Francia. On field calibration of an electronic nose for G. Di Francia. Semi-supervised learning techniques in ar- benzene estimation in an urban pollution monitoring sce- tificial olfaction: A novel approach to classification prob- nario. Sensors and Actuators B: Chemical, 129(2):750 – lems and drift counteraction. Sensors Journal, IEEE, 757, 2008. 12(11):3215–3224, Nov 2012. [6] Dario Floreano, Peter Dürr, and Claudio Mattiussi. Neu- roevolution: from architectures to learning. Evolutionary Intelligence, 1(1):47–62, 2008. [7] Faustino Gomez, Juergen Schmidhuber, and Risto Miikku- lainen. Accelerated neural evolution through cooperatively coevolved synapses. Journal of Machine Learning Re- search, pages 937–965, 2008. [8] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org. [9] Jan Koutník, Juergen Schmidhuber, and Faustino Gomez. Evolving deep unsupervised convolutional networks for vision-based reinforcement learning. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, GECCO ’14, pages 541–548, New York, NY, USA, 2014. ACM. [10] Yann Lecun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 5 2015. [11] Yann LeCun and Corinna Cortes. The mnist database of handwritten digits, 2012. [12] Ilya Loshchilov and Frank Hutter. CMA-ES for hyper- parameter optimization of deep neural networks. CoRR, abs/1604.07269, 2016. [13] Tomas H. Maul, Andrzej Bargiela, Siang-Yew Chong, and Abdullahi S. Adamu. Towards evolutionary deep neural networks. In Flaminio Squazzoni, Fabio Baronio, Claudia Archetti, and Marco Castellani, editors, ECMS 2014 Pro- ceedings. European Council for Modeling and Simulation, 2014. [14] Risto Miikkulainen, Jason Zhi Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving deep neural networks. CoRR, abs/1703.00548, 2017. [15] F. Pedregosa et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825– 2830, 2011.