We propose an incremental training method that partitions the original network into sub-networks, which are then gradually incorporated in the running network during the training process. To allow for a smooth dynamic growth of the network, we introduce a look-ahead initialization that outperforms the random initialization. We demonstrate that our incremental approach reaches the reference network baseline accuracy. Additionally, it allows to identify smaller partitions of the original state-of-the-art network, that deliver the same nal accuracy, by using only a fraction of the global number of parameters. This allows for a potential speedup of the training time of several factors. We report training results on CIFAR-10 for ResNet and VGGNet.
When dealing with a classi cation task on a new dataset of images, a widely used strategy is to start by training a few state-of-the-art networks that were developed for similar datasets. The main disadvantage of this approach is that only too late in the process we learn whether the network is not well suited for the dataset. At that moment, the training is stopped, the network is adapted and the process is restarted from scratch, discarding all the previously collected information. Thus, in the presence of large, complex datasets, the global timeto-solution tends to reach the order of several weeks or even months.
A great deal of research is now focused in optimizing the depth of a neural network (NN) and most of the proposed methods involve growing the network by gradually adding one or more layers [1{3]. Although this is a logical step towards evolving network architectures, to the best of our knowledge there is no work that compares the accuracy and performance of the networks obtained by dynamically adapting their structure during training, with the original one trained from scratch.
In this paper we present our contribution towards gradually training deep state-of-the-art convolutional neural networks (CNNs) with no loss in accuracy. During incremental training we start the learning process with a shallow network. When the network performance stops improving because of its limited capacity, we transfer the knowledge of the shallow network to a deeper one and continue the training. In this way the time and resources spent in training the initial shallow network are not wasted and the deeper network has a better initialization point. Additionally, since the incremental training method easily quanti es (by construction) the value added by each network extension, it is simple to de ne custom trade-o functions (e.g., time vs accuracy) to stop the network expansion when desired criteria are met. The long-term goal is to develop a fully-automated framework that optimizes neural network structures for speci c tasks. However, this requires knowing what type of changes need to be performed to the network, which is outside the scope the current work.
The rest of the paper is organized as follows. Section 2 brie y summaries related works, Section 3 provides an overview of our methodology, while Section 4 presents the results of our experiments. Finally, concluding remarks and future works are summarized in Section 5. 2
There is a clear tendency in the literature towards automatizing the design of
NN. In AdaNet [
Auto-Net [
On the same track, a framework for large scale image classi ers [
Other works [7{9] that discover good combination of hyper-parameters do not employ neuro-evolution, therefore the discovery requirements in terms of time and resources are less expensive, but have other drawbacks such as limited search space, considerable loss in accuracy, limited usability, and need of retraining from scratch for a considerable amount of time after each network alteration. 3 http://automl.chalearn.org/
Other steps towards incremental training are presented in [
Let us consider a generic CNN N composed of n layers. The incremental training method proposed in this work begins by partitioning the original network N into K sub-networks Sk, with k = 1; :::; K and K n. In our current implementation, the partitioning is done a priori and does not change throughout the training process, although this will be subject of future studies. A sub-network can be composed of one or several layers, with the only constraint that at least one of the layers must contain trainable parameters; in other words, pooling and dropout cannot constitute a sub-network by themselves. The classi er block at the end of the network, which might include fully-connected layers or global pooling layers, is not considered in the K partitions.
Following the scheme illustrated in Fig. 1, the training process starts with sub-network S1 attached to the classi er block. To determine when is the optimal time to add the second sub-network S2 between S1 and the classi er, we compute every window size (ws) epochs the improvement in the validation accuracy. When the improvement observed is below a threshold we stop the training and we increase the network depth by adding the next sub-network. This process, which is illustrated in Fig. 2, repeats until all K sub-networks are incorporated or until another custom criteria is met.
Each time a new sub-network Sk+1 is inserted in the current architecture, its weights need to be initialized. This step is delicate, as our experiments show empirically that a non-optimal initialization (e.g., random) might prevent the network from reaching the same accuracy as the original one, or might slow down signi cantly the incremental training process (Section 4). To overcome this issue, we propose a more e cient initialization technique, that we refer to as look-ahead. Given a current network constituted by k sub-networks, our look-ahead method consists of training Sk+1 for a few epochs based on the input generated by the already trained uppermost part of the network. This strategy provides a more informed starting point for the new sub-network, since it looks ahead towards more complex features that can be obtained based on the previously learned ones. We remark that the weights of the original sub-network S1 + S2 + ::: + Sk are freezed during the look-ahead process that initializes Sk+1; this implies that no time is spent in back-propagating in that region of the network. Moreover, the depth of the look-ahead tends to be comparably smaller than the depth (Step 1) (Step 2) (Step 3) (Step 4) of the nal network, therefore the training of the look-ahead is not considered expensive. In this section, we train several state-of-the-art CNNs to compare the performance of our incremental training method, with the one of the classical algorithm. All runs involve single-precision arithmetic and are performed on IBM4 POWER8 Minsky compute nodes, equipped with four Nvidia P100 GPUs.
In all experiments we use the CIFAR-10 dataset. CIFAR-10 is a collection
of 60,000 RGB images of 32x32x3 pixels each, evenly distributed across 10
mutually exclusive classes. On top of it, we also use the simple data augmentation
described in [
The aim of our experiments is to demonstrate the advantages of the incremental training method compared to the regular one. Therefore, in our runs we do not employ all the advanced tricks (e.g., learning rate schedule) needed to reach the highest state-of-the-art accuracy. The values of the most relevant hyper-parameters are summarized in Table 1, Panel B. The partitioning of the networks for the incremental training is mainly based on simplicity. We group layers with the same number of lters (in the case of convolutions) and the same number of output nodes (in the case of dense layers). We tested di erent types of network partitioning observing no major impact in the overall performance.
In Fig. 3 we show a comparison between the regular and incremental training methods: for the latter, we plot results using both random and look-ahead initialization. The validation accuracy ( rst row) shows that the look-ahead outperforms the random initialization of the sub-networks. This behavior is better illustrated in the case of ResNet-56 network, in contrast with the behavior of VGGNet, as the over-parametrized network mitigates the outperformance. On the second row we estimate the cost per inference of each sub-network in terms of FLOPs. The plots highlight the bene t of our incremental training, that performs a much lower amount of calculations for the majority of the training time. 4 IBM, the IBM logo, ibm.com, OpenPOWER are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Other product and service names might be trademarks of IBM or other companies.
Net++ Random Lookahead
Baseline 20 40 60 80 100 120 140
Epochs Net++ Random Lookahead
Baseline 20 40 60 80 100 120 140
Epochs Random Lookahead
Baseline 20 40 60 80 100 120 140
Epochs Random Lookahead
Baseline 20 40 60 80 100 120 140
Epochs 1 2 3 4 Cumulative FLOPs ResNet-56
Random Lookahead Baseline 5 1e146
Random Lookahead
Baseline 0.40.0 0.2 0.4
Cum0u.l6ative0F.8LOPs1.0 1.2 1e115.4 VGGNet 0.4 0
On the third and last row we observe that, while the regular version of the VGGNet reached 90% accuracy using 1.4 PFLOP, the incremental version needed 0.8 PFLOP and only 40% of the total number of parameters. In both networks the look-ahead training converged faster.
On top of these advantages, the incremental training allows to trace the importance of each sub-network during the training. For instance, in the case of VGGNet, the last two sub-networks bring a marginal improvement (less than 1% accuracy), while increasing by 16% the inference cost and by 80% the number of parameters. In Fig. 4 we show that only 5% of the VGGNet paramaters are enough to reach 85% accuracy, while with 42% we arrive to the baseline accuracy. Therefore, the incremental training can be of great bene t for applications in which the last 1-2% of accuracy are not crucial and a slightly lower performance is within accepted limits. Indeed, it can dynamically determine to stop the network expansion early, saving a lot of computational resources without compromising accuracy.
1.0 We proposed an incremental training method for CNNs. We demonstrated that our method reaches same or slightly better accuracy than regular training methods on state-of-the-art networks. This was achieved by using a more informed initialization of the network extension, which we call the look-ahead. Clear bene ts of the incremental training are the faster convergence, the intuitive understanding of the importance of each sub-network in the overall performance, and the smooth synergy between training and optimal network depth discovery. In future works, we plan to generalize the present approach to other types of networks, such as Recurrent Neural Networks and Long Short Term Memory.