Recent works on network pruning [16, 12] showed that randomly weighted neural networks contain supermasks (subnetworks) the performance of which is comparable with similarly sized, trained networks. These results underpin the Lottery Ticket Hypothesis [3]: the efectiveness of deep neural networks rely on lucky initialization. While these works define sparse supermasks, we demonstrate that dense supermasks can also be found by applying structured pruning. We remove components from randomly weighted neural networks - neurons from fully connected layers - such that the loss of the networks decreases continuously. This results in smaller, dense networks whose accuracy is higher than their initial version.
The trend to improve the accuracy of modern deep neural networks is to increase
the number of parameters and the number of layers [
Since there is an increasing need to apply neural networks on resource
limited devices (autonomous drones, cars, mobile phones or other embedded
devices), pruning methods were developed that aim is to eliminate redundant
parameters [
Pruning results show that smaller networks are able to represent complex distributions but current optimization methods fail to tune their weights adequately. However, optimizing in larger dimensional space and eliminating the parameters afterwards can solve the problem.
Contrary to this, a recent work pointed out that smaller, sparse networks can
be trained as good as their dense counterparts, if their weights are initialized
properly [
Following the work of [
Our work is inspired by the results of [
While this work studies fully connected networks on a small dataset we would like to extend these experiments to convolutional neural networks and larger datasets as well.
To prune neurons from a network, we compare the magnitude based pruning with the linear filter ensembles (LFE) method. As a control case, we also apply random pruning where neurons are removed randomly from the network. In the following we briefly present the magnitude and LFE pruning methods.
Magnitude pruning uses the parameter magnitude as importance value: parameters with small magnitude have less importance and their elimination chance is higher. In case of structured pruning, the importance of the ℎ neuron is estimated by calculating its L2 norm: = ︁[ ∑︁ | w, | where w, denotes the ℎ parameter of the ℎ neuron and is the estimated importance.
Linear filter ensembles
[
optimal by evaluating the pruned network on a validation set, we simply set this to a constant value. Therefore, in each pruning iteration % of the neurons are removed from the layer.
LFE estimates the importance of the filters (or neurons) by evaluating the network loss using filter ensembles. A filter ensemble is a group of filters (in case of fully connected layers a group of neurons) from a given layer. Each ensemble is represented by a binary vector, that defines which filters should be used during the network evaluation. For each filter ensemble an score is assigned that is calculated from the network loss. Ensembles with small loss get high score while solved by minimizing the Euclidean 2-norm: large loss produces smaller scores. matrix and = [ 1, . . . , ]
Given the { , } pairs, vector is constructed and the ·
= [ 1, . . . , ] = equation is ℒ = ‖ − · ‖ .
= [ 1, . . . , ] is used as importance indicators, where large means more importance value.
We experiment with LFE, magnitude and random based pruning on the LeNet300-100 [8] architecture and the MNIST dataset [9]. MNIST is a dataset that contains grayscale images of handwritten digits ranging from 0 to 9. The dataset contains 60 000 samples in total: 50 000 images belongs to the training set and 10 000 to the test set. Each image has a size of 28 × 28 pixels and the digits are centered and self-normalized on them.
We experiment with a 3-layer fully connected network, called LeNet-300-100 [8] on the MNIST dataset. This network has an input layer that contains 784 units – one unit for each pixel from the MNIST image. The input layer is followed by two fully connected layers, containing 300 and 100 units respectively and use the ReLU activation function. The output layer contains 10 units and uses softmax activation function. Since we are measuring the performance of randomly initialized, untrained networks, we do not apply any training on them (we do not modify the parameters, only remove them if required). The weights values correspond to the initial values sampled from normal distribution with 0 mean, 0.1 standard deviation. All biases are set to 0.
Using the pruning methods (without training the network), we iteratively remove 6 and 2 units from the first and second layers. Therefore, the network size shrinks from 100% to 1.77%. During each iteration, we measure the network loss and accuracy on the test set. We also experiment with one-shot pruning using LFE: instead of removing the neurons iteratively, the network size is decreased to the target value simultaneously. In each pruning method we repeat the process 5 times and report the average loss and accuracy, as well as the min and max of these values. To apply LFE pruning, we insert mask layers after the hidden layers.
These mask layers define the active and inactive units (neurons) from the previous layer: units with 0 masks are turned of, therefore, the weights of that unit could not contribute to the network output. We use the cross entropy loss function to calculate the score of diferent filter ensembles. 3.1. LeNet-300-100 results First, we experiment with the LeNet-300-100 fully connected network. Results of the experiment is presented in Figure 1 and Figure 2. As Figure 1 shows, the network accuracy with random pruning remains around the initial 10%. This value is expected in case of a classification problem with 10 categories. This corresponds to the intuition that a randomly initialized network after random pruning became a smaller but still random network.
Surprisingly, the magnitude based pruning could not increase the network
accuracy at all. While [
In contrast to this, LFE pruning (iterative and one-shot as well) significantly increases the network accuracy, as more and more units are removed from the network. Applying one-shot pruning, the network accuracy peaks around 29%, where 80% of the parameters are removed from the network. This increase is more significant in case of the iterative LFE: the network accuracy reaches 37% ± 4%, where the network size is only 35.34% of the initial network (108 and 36 neurons in the first and second layers respectively). After this value, the accuracy quickly decreases and stops at the initial 11.9% with 1.77% of the initial network size.
Figure 2 shows the loss of the networks during parameter pruning. With LFE
pruning (iterative and one-shot as well) the network loss decreases as expected: it
reaches a minimum value and starts to increase afterwards. However, random and
magnitude pruning produces a continuously decreasing loss which is in contrast
with the accuracy curve. One would expect that randomly removing parameters
from a random network will not influence the network loss. However, as more and
more parameters are eliminated, the network output (probabilities) becomes more
and more uniform (the network assigns near 0.1 probability for all the 10 digits).
This results decreasing cross entropy loss that asymptotically reaches the value
ℒ = − log(0.1) ∼ 2.3025.
3.2. Wide-LeNet results
Motivated by the results of [
These results validates the hypothesis that randomly initialized networks contain supermasks – with accuracy far from random – not only on sparse form but in dense form as well.
In this work we experimentally showed that randomly initialized, untrained neural networks contain dense subnetworks with an accuracy far better than chance. Using magnitude and LFE pruning, we iteratively pruned the randomly initialized LeNet300-100 architecture and measured the network loss and accuracy. With LFE pruning, we found dense subnetworks with an accuracy of 37% on the MNIST dataset. Moreover, applying the same pruning method on Wide-LeNet, we achieved 63.5% accuracy, without training the parameters at all.
In the future we would like to expand these experiments to bigger networks and real world datasets as well.