Neural network models have a number of hyperparameters that must be chosen along with their architecture. This can be a heavy burden on a novice user, choosing which architecture and what values to assign to parameters. In most cases, default hyperparameters and architectures are used. Signi cant improvements to model accuracy can be achieved through the evaluation of multiple architectures. A process known as Neural Architecture Search (NAS) may be applied to automatically evaluate a large number of such architectures. A system integrating open source tools for Neural Architecture Search (OpenNAS), in the classi cation of images, has been developed as part of this research. OpenNAS takes any dataset of grayscale, or RBG images, and generates Convolutional Neural Network (CNN) architectures based on a range of metaheuristics using either an AutoKeras, a transfer learning or a Swarm Intelligence (SI) approach. Particle Swarm Optimization (PSO) and Ant Colony Optimization (ACO) are used as the SI algorithms. Furthermore, models developed through such metaheuristics may be combined using stacking ensembles. In the context of this paper, we focus on training and optimizing CNNs using the Swarm Intelligence (SI) components of OpenNAS. Two major types of SI algorithms, namely PSO and ACO, are compared to see which is more e ective in generating higher model accuracies. It is shown, with our experimental design, that the PSO algorithm performs better than ACO. The performance improvement of PSO is most notable with a more complex dataset. As a baseline, the performance of ne-tuned pre-trained models is also evaluated.
generating more complex architectures, such as CNNs, are at early stages of
development. Consequently they are poorly documented and often unreliable
[
The development of OpenNAS integrates several metaheuristic approaches in a single application used for the neural architecture search of more complex neural architectures such as convolutional neural networks. Furthermore, the e ectiveness of NAS in generating good neural architectures for image classi cation is evaluated. Standard approaches to NAS, using the AutoKeras framework, are also incorporated into the system design.
A key aspect of the study is to contrast Swarm Intelligence (SI) algorithms
for NAS. Consequently, Particle Swarm Optimization (PSO) [
2.1
CNNs are feed-forward Deep Neural Networks (DNNs) used for image
recognition. The original CNN architecture was proposed by LeCun [
AutoML involves the automation of the entire machine learning pipeline
including data augmentation, feature engineering, model selection, choice of hyper
parameters and nally neural architecture selection and creation. By constrast,
NAS has a more narrow focus in that it concentrates on neural architecture
selection and creation [
Tree-based Pipeline Optimization Tool (TPOT) is an open source python
package that uses genetic programming in optimizing the machine learning
pipeline [
AutoKeras [
Neural architecture search is the process of automatically nding and tuning
DNNs. It has been shown that DNNs have made remarkable progress in solving
many real world problems such as image recognition, speech recognition and
machine translation [
A basic approach to NAS is the brute force training and evaluation of all possible model combinations. On completion, the best performing model is selected. However, this is impractical due to the combinatorics of the problem. Using metaheuristics, such as swarm intelligence, is an alternative which seeks the best model within reasonable time constraints. 2.4
Swarm Intelligence, a category of Evolutionary Computing, has been used for
classi cation problems in the following forms: Particle Swarm Optimization
(PSO) [
Particle Swarm Optimization PSO belongs to the class of swarm
intelligence techniques and is a population-based stochastic technique for solving
optimization problems developed in 1995. An open source python library, for CNN
optimization using the PSO algorithm was developed by Fernandes et al [
Using ACO, a system known as DeepSwarm was developed by Byla and Pang
[
With arti cial neural networks, there are many parameters to choose from such as the number of hidden network layers, number of neurons per layer, type of activation function, choice of optimizer and so on. The nal network design often depends on the problem domain and is typically achieved in a time consuming trial and error fashion.
Similar problems exist with CNNs but these problems are exacerbated by the length of time, and amount of computational resources required to train such networks. Clearly, a core objective of NAS is to nd good network performance within acceptable time limits through the reduction of both the number of networks tested and the length of time required for their evaluation. The implementation of NAS can be achieved through a variety of approaches including transfer learning using pre-trained networks, network morphism or swarm intelligence. Using these approaches as its pillars, a NAS system (OpenNAS) has been built which tackles such problems 3. OpenNAS does not enforce a particular architecture but rather it allows novel and interesting architectures to be discovered.
In this work we focus on the swarm intelligence component of the OpenNAS system. The swarm optimization techniques currently used are Particle Swarm Optimization and Ant Colony Optimization.
The PSO algorithm determines how the principal CNN layer types, and their associated hyperparameters, are connected together. The generated models consist of architectures using a mix of convolutional, average pooling, max pooling and fully connected layers. In addition, dropout layers and batch normalization layers are also added to alleviate over tting. The hyperparameters associated with each layer type are indicated in Table 1. 3 https://github.com/seamusl/OpenNAS-v1
Particle architectures, i.e. model architectures, are compiled for a number of epochs and evaluation is carried out using the standard loss function of crossentropy loss. Particle architectures with the smallest loss are selected by the algorithm. The number of epochs parameter for pBest must be carefully chosen since it is the main driver of both run time and model accuracy.
Using an ACO approach, the parameters used for model training in the exploration process are highlighted in Table 2. Two test con gurations are considered. In the rst case, 8 ants are used with 30 epochs and in the second case, 16 ants are used with 15 epochs. The depth parameter was xed at 20.
Fine tuning was implemented by initially removing the fully connected layers from the top of the model. Two blocks are then added, each of which has a fully connected layer, a batch normalization layer and a dropout layer. The hybrid structure is then trained with the new dataset. Fine tuning of a VGG16 network is illustrated in Figure 1. The high level view of the system architecture is presented in Figure 2. The system is organized into the following python modules: OpenNAS, pre-processor, trainer, ensemble, super stacker, syscon g and loader. The pre-train function uses transfer learning as either a feature extractor or to ne tune the pre-trained networks of VGG16, VGG19, MobileNet or ResNet50.
With the swarm function, PSO or ACO can be used to search for the best
neural architecture. Existing open source python libraries were customized for
both PSO and ACO functionality. Particle swarms were implemented using a
psoCNN library [
Existing NAS tools, such as AutoKeras, were also integrated into the OpenNAS system. AutoKeras is a powerful open source library which provides functions to automatically search for optimal architectures for deep learning models. However, this library is still in beta development and the associated documentation is quite poor.
With the ensemble module, there are options to build stacked ensembles using either homogeneous or heterogeneous base learners. These learner outputs are subsequently passed to a suite of meta learner algorithms. The system generates the optimal neural architecture model using the chosen heuristic. 5
Two datasets were chosen for the experimental design, namely CIFAR10 [
In order to test variance and reproducibility, each con guration was run 5 times on both CIFAR10 and Fashion Minst which resulted in the evaluation of 4000 CNN architectures for this phase of the study.
was used to evaluate the performance of both PSO con gurations. It is clear from Table 3 that the PSO model trained on swarm settings of a lower population and higher number of iterations (population of 10 and 20 iterations) performed signi cantly better. In terms of accuracy, the mean performance was 3.5% better.
MAacxc MAecacn StADcecv (Tmi mine) Layers Population 10 Iterations 20 0.900 0.853 Population 20 Iterations 10 0.883 0.818
Both con gurations have a very low standard deviation for model accuracy indicating a high level of reproducibility between test runs. At a mean run time of 21.9 hours for the rst con guration and 18.6 hours for the second con guration, the PSO search for CNN architectures is a slow process considering high performance workstations, with NVIDIA GeForce GTX 1080 Ti graphic cards, were used.
trained on the Fashion Mnist dataset (Table 4), achieved much higher accuracy compared with models developed using CIFAR10 data. Similar to CIFAR10, the low standard deviation associated with both implementations of Fashion Mnist models indicate the PSO approach produces consistent results between di erent test runs.
The stochastic nature of metaheuristics impacts the run times associated with PSO for both Fashion Mnist and CIFAR10. In all tests, no clear pattern emerged with regard to run times: CIFAR10 was faster using a population of 10 with 20 iterations whereas Fashion Mnist was faster with a population of 20 with 10 iterations. Therefore, in terms of run time, no clear conclusion could be drawn by doubling the population and halving the iterations.
With regard to the impact of swarm settings on model accuracy for Fashion Mnist, again there is little to separate the con gurations. With a mean accuracy of 93.5% for a population of 20 with 10 iterations and a corresponding mean model accuracy of 93.2% using a population of 10 with 20 iterations, no clear conclusion can be drawn.
Therefore, unlike CIFAR10, changing the swarm settings by doubling
population and halving iterations does not impact model accuracy in the case of
Fasion Mnist. Both con gurations for the PSO algorithm perform well on this
dataset.
Similar to other metaheuristics, there are several parameters which can be tuned
for optimal neural architecture search using Ant Colony Optimization [
the results from Table 5 indicate that a greater number of ants leads to higher model accuracy. The improvement in max model accuracy achieved, through doubling the number of ants and halving the number of epochs, was modest at just 1.2%. The impact on run time for a small increase in accuracy was severe. Doubling the number of ants e ectively doubled the run time (even though the number of epochs was halved). The standard deviation for accuracy is very low indicating good reproducibility between the various test runs.
of ACO models using Fashion Mnist data is highlighted Table 6. It can be seen that both con gurations perform well resulting in accuracies greater than 93%. The di erence in mean model accuracy between con guration A (8 ants and 30 epochs) is trivial when compared to con guration B (8 ants and 30 epochs). However, similar to ACO on CIFAR10, the di erence in run time is very significant for con guration B. E ectively it took over 7 hours longer to achieve an accuracy improvement of 0.1%.
Clearly in the case of a simpler dataset such as Fashion Mnist, using a number
of ants in excess of 8 is not worth doing. This nding is similar to that seen with
the more complex CIFAR10 dataset, above. Therefore choosing the number of
ants, used for this ACO implementation, is an important consideration impacting
run time performance. As anticipated, the standard deviation for accuracy is also
very low indicating good reproducibility between the various test runs.
The OpenNAS performance of all models across both datasets is illustrated in
Figure 3. The results demonstrate performance comparable to that achieved by
the pso-CNN [
The highest accuracy of OpenNAS in CIFAR-10 classi cation was 90.0%. This was achieved using using a PSO-derived model. By comparison, DeepSwarm achieved a top accuracy of 88.7%.
With Fashion Mnist data, the highest performing model for OpenNAS is again a PSO derived model with an accuracy of 94.3%. This result compares very favourably with the SOA accuracy of 94.6%. The highest performing model for DeepSwarm achieved an accuracy of 93.56%.
In the case of pso-CNN, experiments were conducted on Fashion Mnist but not on CIFAR-10. The best performing pso-CNN model on Fashion Mnist was 91.9% without dropout and 94.5% with dropout.
The ndings clearly show that a PSO approach leads to higher model accuracies given that DeepSwarm is exclusively based on an ACO approach.
The pre-trained networks of MobileNet and RestNet50 delivered the poorest performance with CIFAR10. The other pre-trained networks, using VGG architectures, performed very well on the same dataset.
With a more complex dataset, such as CIFAR10, the mean performance improvement of the PSO algorithm is signi cant when compared with ACO. With con gurations used in this study, PSO achieved a mean accuracy of 85.3% on CIFAR10 compared with an ACO mean accuracy of 82.2%.
The approach taken by ACO, in determining the best architecture is very di erent to the PSO approach. With ACO, simpler models are initially evaluated at lower depths with progressively more complex models being evaluated at deeper search levels. Therefore at search depth 1, there is essentially just a single hidden layer being evaluated. The number of ants speci ed creates new architectures which simply vary the hyper parameters used for that layer. With each new depth being explored, an additional layer is added to the architecture being explored.
Furthermore, the ACO approach enables the targeting of hyper parameter optimization within a given layer type rather than optimizing at the overall architecture level. Specifying a large number of ants, with a reduced depth, ensures the search space is restricting to studying the e ects of layer hyper parameters rather than model depth and the constituent layers. By comparison, the number of layers in PSO generated models is entirely stochastic. 7
The OpenNAS approach identi es the hyperparameters within each layer of networks used for image classi cation of grayscale and color datasets. In addition the number and type of layers for the neural architecture are also identi ed. This combined approach generates model architectures which achieve competitve accuracies when classifying the CIFAR10 and Fashion Mnist datasets.
The results of swarm intelligence algorithms, in the context of this study, have generated impressive performances. However, in many cases, their performance is only marginally better than ne tuned pre-trained VGG models. The accuracies of PSO derived models have been shown to exceed those of ACO derived models in the image classi cation of grayscale and color datasets.
In addition, the OpenNAS integrated approach, using both PSO and ACO algorithms, yields higher accuracies when compared with DeepSwarm which relies on a single metaheuristic.