Due to many successful practical applications, deep neural networks and convolutional networks have become the state-of-art machine learning methods recently. The choice of network architecture for the task at hand is typically made by trial and error. This work deals with an automatic data-dependent architecture design. We propose an algorithm for optimization of architecture of convolutional network based on asynchronous evolution. The algorithm is inspired by and designed directly for the Keras library which is one of the most common implementations of deep neural networks. The proposed algorithm is successfully tested on MNIST and Fashion-MNIST data sets.
Various architectures of deep neural networks (DNN) have
become the state-of-art methods in many fields of machine
learning in recent years. They have been applied to
various problems, including image recognition, speech
recognition, and natural language processing [
Deep neural networks are feed-forward neural networks with multiple hidden layers between the input and output layer. The layers typically have different units depending on the task at hand. Among the units, there are traditional perceptrons, where each unit (neuron) realizes a nonlinear function, such as the sigmoid function, or the rectified linear unit (ReLU).
Convolutional networks (CNN) are family of DNN. They typically have three types of layers – convolutional layers, max-pooling layers, and dense (i.e. fullyconnected) layers. For the most common case of image processing, convolutional layers perform convolution of an input image to abstract high-level features. They are defined by a set of learnable filters. Max-pooling layers reduce the size of representation, their function is fixed and they are not learnable. Dense layers are usually used as the last layers of the network to perform the actual classification.
While the learning of weights (including filters) of the CNN is done by algorithms based on the stochastic gradient descent, the choice of architecture, including a number and sizes of layers, number and size of convolutional filters, size of pools in pooling-layers, and a type of activation function, is done manually by the user. However, the choice of architecture has an important impact on the performance of the CNN. Some kind of expertise is needed, and usually a trial and error method is used in practice.
In this work we exploit a fully automatic design of CNNs. We investigate the use of evolutionary algorithms for evolution of a CNN architecture. There are not many studies on evolution of CNN since such approach has very high computational requirements. To keep the search space as small as possible, we simplify our model focusing on implementation of CNN in the Keras library [3] that is a widely used tool for practical applications of DNNs and CNNs.
The approach described in this paper extends our previous results for evolving DNNs limited to networks with dense layers only [25, 24]. The proposed algorithm is evaluated on the MNIST and Fashion-MNIST data sets that are both classification tasks of small gray-scale images.
The paper is organized as follows. Next Section reviews related work. Section 3 describes the main ideas of our approach. Section 4 explains the main ideas of asynchronous evolution. Section 5 summarises the results of our experiments, and finally Section 6 brings conclusion. 2
Neuroevolution represents an attempt to train a neural network by means of evolutionary techniques [5]. In traditional neuroevolution, no gradient descent is usually involved, and both architecture and weights of the network undergo the evolutionary process. However, because of large computational requirements the applications are limited to small networks.
There were quite many attempts on architecture
optimization via evolutionary process (e.g. [
On the other hand, studies dealing with evolution of deep neural networks and convolutional networks started to emerge only very recently. The training of one DNN usually requires hours or days of computing time, quite often utilizing GPU processors for speedup. Naturally, the evolutionary techniques requiring thousands of training trials were not considered a feasible choice. Nevertheless, there are several approaches to reduce the overall complexity of neuroevolution for DNN. Still due to limited computational resources, the studies usually focus only on parts of network design.
For example, in [
In [
In our approach we use asynchronous evolution to search for optimal architecture of CNN, while the weights are learned by gradient based technique.
The main idea of our approach is to keep the search space as small as possible, therefore the architecture specification is simplified. It directly follows the implementation of CNN in Keras library, where networks are defined layer by layer. Layer is specified by its type – convolutional, max-pooling, dense. Dense layers are defined by a number of neurons, type of an activation function (all neurons in one layer have the same type of their activation function), and the type of regularization (such as dropout). Convolutional layers are defined by number of filters, size of the filter, and possibly the type of an activation function and type of regularization. Max-pooling layers are defined by the size of pool.
In this paper, we limit to networks that can be split into
two parts.The first part is a preprocessing part, it contains
only convolutional and max-pooling layers, and it is
responsible for the preprocessing of the input and
abstracting high-level features. The second part is a classifier and
it consists of dense layers. Such architecture corresponds
to the original proposal of the LeNet architecture [
In order to apply genetic algorithm (GA) to the search for an optimal CNN architecture, we have to be able to encode the architecture by an individual of the GA.
Our proposal of encoding closely follows the CNN description and implementation in the Keras [3] model Sequential. The model implemented as Sequential is built layer by layer, similarly the GA individual consists of blocks representing individual layers.
I = (I1, I2), I1 = ([type, params]1, . . . , [type, params]H1)
I2 = ([size, dropout, act]1, . . . , [size, dropout, act]H2) where I1 and I2 are the convolutional and dense part, respectively, H1, H2 is the number of layers in convolutional and dense part, respectively. The blocks in convolutional part encode type ∈ { convolutional, max − pooling} type of layer and params other parameters of the layer (for convolutional layer it is number of filters, size of filter, and activation function; for max-pooling layer it is only size of pool). The blocks in dense part code dense layers, so they consist of size the number of neurons, drop the dropout rate (zero value represents no dropout), act ∈ { relu, tanh, sigmoid, hardsigmoid, linear} activation function. 3.2
To produce new individuals in genetic algorithm we use recombination operators crossover and mutation. Crossover The crossover operator combines two parent individuals and produces two offspring individuals. It is implemented as one-point crossover, where the crossing point is determined at random, but on the border of a block only. The two parts of the individual are crossed over separately, so if parents are I = (I1, I2) and J = (J1, J2) we run crossover(I1, J1) and crossover(I2, J2).
Let the two parents be:
Ip1 = (B1p1, B2p1, . . . , Bkp1)
Ip2 = (B1p2, B2p2, . . . , Blp2), then, the crossover produces offspring:
Io1 = (B1p1, . . . , Bcp1, Bcpp22+1, . . . , Blp2)
p1 Io1 = (B1p2, . . . , Bcp2, Bcpp11+1, . . . , Bkp1),
p2 where cp1 ∈ { 1, . . . , k − 1} and cp2 ∈ { 1, . . . , l − 1} .
Thus, only the whole layers are interchanged between individuals.
Mutation The mutation operator brings random changes to an individual. Each time an individual is mutated, one of the following mutation operators is randomly chosen (each of mutation operators has its own probability): • mutateLayer - introduces random changes to one randomly selected layer. • addLayer - one randomly generated block is inserted at random position. If it is inserted to the first part of the individual, its either convolutional layer or maxpooling layer; otherwise it is dense layer. • delLayer - one randomly selected block is deleted.
When mutateLayer is performed, again one of the available operators is chosen. For dense layers they are: • changeLayerSize - the number of neurons is changed.
The Gaussian mutation is used, the final number is rounded (since size has to be an integer). • changeDropOut - the dropout rate is changed using the Gaussian mutation. • changeActivation - the activation function is changed, randomly chosen from the list of available activations.
For convolutional layers: • changePoolSize - the size of pooling is changed. • changeNumberOfFilters - the number of filters is changed. The Gaussian mutation is used, the final number is rounded. • changeFilterSize - the size of the filter is changed. • changeActivation - the activation function is changed, randomly chosen from the list of available activations. 3.3
Fitness function should reflect a quality of the network represented by an individual. To assess the generalization ability of the network represented by the individual we use a crossvalidation error. The lower the crossvalidation error, the higher the fitness of the individual.
Classical k-fold crossvalidation is used, i.e. the trainin g set is split into k-folds and each time one fold is used for testing and the rest for training. The mean loss function on the testing set over k run is evaluated.
For the classification tasks, categorical crossentropy is used as the loss function. As a parental selection operator, the tournament selection is used in our algorithm. It works as follows, in each turn of the tournament, k individuals are selected at random, and the one with the highest fitness—in our case the one with the lowest crossvalidation error—is selected. 4
In classical genetic algorithm the individuals are evaluated in generations. In each generation, new individuals are produced based on operators selection, mutation, and crossover, their fitness is evaluated and they replace the old generation. The fitness evaluations are independent and can be done in parallel.
There are many approaches to parallelization of
genetic algorithms. In general, there are three classes
of parallel GA approaches – single population
masterslave model, single population fine-grained, and
multipopulation coarse grained model (see [
In our work we use the master-slave parallelization method (Fig. 2). It works with single population of individuals, and the evaluations of individuals are performed in parallel – the master stores the population and the slaves evaluate the fitness. The algorithm itself is same as for serial GA, each individual may compete and mate with any other.
The fitness evaluation is parallelized so that a fraction of the population is assigned to each of the processors. Communication occurs only when a slave receives an individual to evaluate and when a slave returns a fitness value.
The algorithm is normally defined as synchronous, i.e. the master waits for the slaves to receive the fitness values for all the population and only then it proceeds to the next generation. Such synchronous master-slave algorithm has exactly the same properties as a simple sequential algorithm.
However, the disadvantage of such synchronous parallel approach is that in general, not all fitness evaluations require same time. In our particular case, some individuals represent large networks and require long time to evaluate, on the other hand there might be very small networks that are evaluated much faster. At the end of each generation, there are processors that are already finished and waiting for the ones evaluating large networks.
As a solution to this problem we have chosen to use
asynchronous evolution [
One of the features of asynchronous approach is that it is naturally biased towards solution with faster fitness evaluation. In our case we consider this feature to be an advantage, because we are more interested in smaller networks with shorter learning time. As our practical experiments in the next section imply, this feature is not harmless, it seems in fact not sufficient and maybe a further discrimination of larger networks in fitness function can be beneficial.
The (synchronous and asynchronous) master-slave approach can be easily and efficiently implemented both on shared-memory and distributed-memory parallel computers. On a shared-memory multiprocessor, the population is stored in memory and each slave process can access the individuals assigned to itself. On a distributed-memory computer, the master process is responsible for storing the population, sending the individuals to other processes (slaves), collecting the results and producing new generation by genetic operators.
procedure ASYNCEA(MINPOPSIZE,POPSIZE) P ← 0/ while | P| < minPopSize do if not node available then
wait() end if while node available do ind ← RandomIndividual() evaluate(ind) end while evaluatedInd ← getEvaluatedIndividual() P ← P ∪ { evaluatedInd} end while ind ← produceIndividual() evaluate(ind) while termination criterion not met do evaluatedInd ← getEvaluatedIndividual() P ← P ∪ { evaluatedInd} if | P| > popSize then
discard the worst individual from P end if ind ← produceIndividual() evaluate(ind) end while end procedure 5
For our experiments we have chosen the well known
MNIST data set [
Each data set contains 70 000 images of 28 × 28 pixel. 60 000 are used for training, 10 000 for testing. In MNIST, there are images of handwritten digits (see Fig. 3), while in Fashion-MNIST are images of fashion objects (Fig. 4).
Our implementation of the proposed algorithm is available at [23].
On the MNIST data set, the evolved architecture has even less number of weights than the baseline model. However, on the Fashion-MNIST the evolution was not so successful and the evolved architecture is quite bloated. The architecture complexity is not reflected in the fitness function directly, but the asynchronous evolution should tend to prefer smaller networks.
We have also compared the synchronous parallelization with asynchronous one in terms of time requirements. We 0 0 0 0 5 5 5 5 10 10 10 10 15 15 15 15 20 20 20 20 250 5 10 15 20 25 250 5 10 15 20 25 250 5 10 15 20 25 250 5 10 15 20 25 0 0 0 0 5 5 5 5 10 10 10 10 15 15 15 15 20 20 20 20 250 5 10 15 20 25 250 5 10 15 20 25 250 5 10 15 20 25 250 5 10 15 20 25 We have proposed an algorithm for automatic design of convolutional networks based on asynchronous evolution. The algorithm was tested in experiments on two image classification tasks, the MNIST and Fashion-MNIST data sets. The evolved networks were compared to baseline models, and they achieved competitive results (in terms of slightly better classification accuracies). We have shown that it is possible to automatically find solutions comparable to those designed by human expert.
The main limitation of the presented algorithm is its time complexity. The possibility to trade human expert knowledge for computational resources should be seen as an advantage in several scenarios. Our main motivation is to develop an autonomous system capable of creating machine learning models without human intervention. This may be useful for the cases where no expert is available or a new task without prior experience is encountered. Also, in critical cases where even a small performance gain is necessary, our approach has demonstrated its usability.
This work was partially supported by the Czech Grant Agency grant 18-23827S and institutional support of the Institute of Computer Science RVO 67985807.
Access to computing and storage facilities owned by parties and projects contributing to the National Grid Infrastructure MetaCentrum provided under the programme "Projects of Large Research, Development, and Innovations Infrastructures" (CESNET LM2015042), is greatly appreciated.
One direction of our future work is to try to lower
the number of fitness evaluations using surrogate
modelling [
ing deep neural networks. In Proceedings of the Compan- [21] M. Suganuma, S. Shirakawa, and T. Nagao. A genetic proion Publication of the 2014 Annual Conference on Genetic gramming approach to designing convolutional neural netand Evolutionary Computation, GECCO Comp ’14, pages work architectures. CoRR, abs/1704.00764, 2017.
1451–1452, New York, NY, USA, 2014. ACM. [22] B. u. Islam, Z. Baharudin, M. Q. Raza, and P. Nallagown[5] D. Floreano, P. Dürr, and C. Mattiussi. Neuroevolution: den. Optimization of neural network architecture using from architectures to learning. Evolutionary Intelligence, genetic algorithm for load forecasting. In 2014 5th Inter1(1):47–62, 2008. national Conference on Intelligent and Advanced Systems [6] F. Gomez, J. Schmidhuber, and R. Miikkulainen. Accel- (ICIAS), pages 1–6, June 2014.
erated neural evolution through cooperatively coevolved [23] P. Vidnerová. GAKeras. synapses. Journal of Machine Learning Research, pages 937–965, 2008. [7] I. Goodfellow, Y. Bengio, and A. Courville.
Deep Learning. MIT Press, 2016. github. om/PetraVidnerova/GAKeras, 2017. [24] P. Vidnerová and R. Neruda. Evolution strategies for deep neural network models design. In Proceedings ITAT 2017: Information Technologies - Applications and Theory.
Aachen & Charleston: Technical University & CreateSpace Independent Publishing Platform, 2017 - (Hlavácˇová,
J.), CEUR Workshop Proceedings, V-1885 , pages 159–166. [25] P. Vidnerova and R. Neruda. Evolving keras architectures for sensor data analysis. In 2017 Federated Conference on Computer Science and Information Systems (FedCSIS), pages 109–112, Sept 2017. [26] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.