In the last decade, deep learning models competed for performance at the price of tremendous computational costs. Such a critical aspect recently attracted more attention for both the training and inference phases. The latter is obviously orders of magnitude lower than the training complexity, but on the other hand, it contributes many times, which impacts eficiency on edge or embedded devices. Inference can be made eficient through neural network pruning, which consists of parameters and neurons' removal from the model's topology while maintaining the model's accuracy. This results in reduced resource and energy requirements for the models. This paper describes two pruning procedures for lowering the operations required during the inference phase and a method to exploit the resulting sparsity. The same cannot be applied at training time: we show it is possible to borrow similar ideas to reduce the cost of gradient backpropagation by disabling the computation for selected neurons.
1. Introduction works are widely used in various tasks, such as speech recognition and computer vision. However, modern architectures require many parameters to generalize well, resulting in large model sizes, high computational and memory resources, and significant energy consumption during training and inference.
In this paper, we present our research on neural network pruning, which involves removing the less essential elements of the network to reduce the model resource requirements. Specifically, we explore the design of pruning procedures (Sec. 2 and Sec. 3), the efect of pruning on network features (Sec. 4), and the practical application of pruned networks to reduce energy consumption (Sec. 5).
We present two pruning techniques capable of
squeezing the model size incrementally during training:
LOBSTER [
Ital-IA 2023: 3rd National Conference on Artificial Intelligence, organEvelop-O ∗Corresponding author.
In this section, we present LOBSTER [
LOBSTER uses a sensitivity-based regularization to promote sparsity in the network topology. Specifically, we define the sensitivity of a network parameter as the derivative of the loss function with respect to that parameter. Parameters with low sensitivity have little impact on the loss function when perturbed and can be pruned without compromising performance. LOBSTER achieves sparsity by gradually shrinking parameters with low sensitivity using a regularize-and-prune approach.
The sensitivity is defined as !!+’& "+",(’&#"", ) ’ ) ) ( ’ )+"$#’&!!+$ ’,#-’*!+$ ()*! &!+$ ℓ2(),&#&" !!! * ( * * ) ( *-’"* $" "’."$’",% ℓ2)*-’$’# ),&! )%!+!)* (a) *-’"!*&","*+ (b)
SeReNe [
LOBSTER allows training a network from scratch, neurons from the network. To achieve this, we leverage
thanks to its loss-based sensitivity formulation. More- the concept of neuron’s sensitivity, defined as the
variaover, it avoids additional derivative computations or tion of the network output with respect to the neuron’s
second-order derivatives, unlike other sensitivity-based activity:
appErxopaecrhimese.nts on multiple architectures and datasets , ( , , ) = 1 ∑ | , | , (2)
demonstrate that LOBSTER outperforms several com- =1 ,
petitors in multiple tasks. It achieves competitive com- where represents the network’s output and , the
pression ratios with minimal computational overhead -th neuron of the -th layer activity.
and without compromising performance. The results of During training, all the parameters of low-sensitivity
the pruning procedure for LeNet-5 trained on the MNIST neurons are shrunk, making it possible to remove them
dataset and ResNet-18 trained on ImageNet are shown from the network. When the ℓ2 norm of a neuron’s
pain Figure 1. LOBSTER achieves state-of-the-art sparsi- rameters approach zero, the neuron no longer emits
sigifcation and classification errors for both architectures. nals (except for the bias), and can be pruned. We propose
Sparse VD [
Finally, we show that structured sparsity provides benefits when storing the neural network topology and pa- rons from the architecture. The resulting models do not rameters. Table 1 shows the results obtained applying require any particular software or hardware to speed up SeReNe on the LeNet-5 architecture trained on MNIST. their inference.
SeReNe achieves a high compression ratio and pruned We were able to perform benchmarks for both mobile
neurons, outperforming the considered references. The devices [
Overall, these results demonstrate the feasibility of deploying pruned neural networks on various resource4. Structured pruning for constrained devices, opening up new opportunities for low-power devices bringing deep learning to the edge.
In this section, we present some empirical results that
demonstrate how pruning (especially structured prun- 5. Neurons at Equilibrium (NEq)
ing) can produce a network model that requires fewer
resources to perform inference. To achieve this, we built All the works presented up to this point focused on
reducthe s i m p l i f y library [
−1 ilarity between all the outputs of the -th neuron at time and at time − 1 for the whole validation set Ξ . We can say that the -th neuron is at equilibrium when it can is the cosine simin FLOPs, and the network’s generalization capability at the end of training. Our results show that NEq consistently reduces the number of FLOPs with minimal or no performance drop. While the amount of saved computation is similar for the stochastic approach with fixed probabilities in all the considered scenarios, the loss in performance varies depending on the architecture and dataset. In contrast, NEq adapts to the particular setup and saves the largest FLOPs for a given performance, with a lower performance loss even when the stochastic approach is tested with the same FLOPs saving.
In this paper, we shared the research experiences we developed in the context of compressing large neural models. Our story has begun with classical unstructured pruning of model parameters, e.g. connections between neurons, where the target is the highest sparsification with the lowest performance impairment. This approach, while very sound from a theoretical point of view, does not guarantee significant eficiencing of the inference phase, when the model is deployed on actual devices. Therefore, we described the structured pruning alternatives that aim at removing whole neurons, thus uncovering the real pruning potential in saving memory and reducing the latency. Finally, we show that pruning can also be exploited at training time to cut the cost of backward propagation. In particular, we introduced NEq, a technique to disable the computation of gradients of neurons that have reached equilibrium: this amounts to pruning the backpropagation graph, and decreasing the number of operations during training. This technique can reduce the cost of training modern neural networks. for neural network compression, in: 2021 IEEE International Conference on Image Processing (ICIP), IEEE, 2021, pp. 3527–3531. [11] J. Flich, L. Medina, I. Catalán, C. Hernández, A. Bragagnolo, F. Auzanneau, D. Briand, Eficient inference of image-based neural network models in reconfigurable systems with pruning and quantization, in: 2022 IEEE International Conference on Image Processing (ICIP), 2022, pp. 2491–2495. doi:10.1109/ICIP46576.2022.9897752.