Elite BackProp: Training Sparse Interpretable Neurons

Abstract

In this paper we present a method called Elite BackProp (EBP) to train more interpretable convolutional neural networks (CNNs) by introducing class-wise activation sparsity; after training, each class will be associated with a small set of elite filters that fire rarely but highly activate on visual primitives from images of that class. Our method is broadly applicable as it does not require additional object part annotations during training. We demonstrate experimentally that EBP realizes high degrees of activation sparsity with no accuracy loss and enhances the performance of a rule extraction algorithm that distils the knowledge from a CNN, by inducing more compact rules that use fewer atoms to describe the decisions of a CNN while maintaining high fidelity compared with other solutions. This happens because EBP induces sparse compositional representations that reuse and combine primitive filters. Such representations can assist in understanding the reasoning behind a CNN and build trust into their decisions.

Keywords

• eol>neural-symbolic integration
• training interpretable CNNs
• activation sparsity
• rule extraction

1. Introduction

Training neural networks to be interpretable [ 1, 2 ] or interpreting their decisions [ 3, 4, 5, 6 ], has received a great deal of attention in recent years. Multiple rule extraction algorithms have been proposed that aim to distil the knowledge from a CNN and intepret its decisions [ 7, 8, 9, 10, 11 ], many of which rely on thresholding filter activations to determine whether a filter can be considered active (a process known as quantisation [ 9, 12 ]), detecting a specific pattern across images. After combining active filters, rules are formed to explain the classification decision of the CNN, where each atom used in explanations corresponds to an active filter.

One can regard compact explanations as more interpretable as there is less information for the reader to assimilate. Thus, ideal explanations are composed of smaller sets of rules which are in turn composed of smaller sets of atoms that can be reused across the rule set and across diferent classes in diferent combinations. Such compactness is more dificult to achieve if the original CNN itself encodes a large number of redundant representations, i.e. in which diferent convolutional filters co-activate on the same concepts.

We aim to tackle the aforementioned ineficiencies by introducing an algorithm called Elite BackProp (EBP) that enforces class-wise activation sparsity. That is, EBP trains CNNs to associate each class with a handful of elite filters that fire rarely but highly activate on images from that class. By filter we mean the set of weights that make a single channel in the convolutional layer. Every filter is assigned a probability of being active on each class based on its frequency and magnitude of activation on images from that class. Filters are ranked for each class according to their activation probability and the top-K filters for a class form its elite. If a filter activates on an image from a class that has low probability of activating that filter, then a penalty inversely proportional to filter’s ranking is applied. Since only the elite filters will have strong activations for each class, the model is incentivised to learn more compositional representations by reusing and combining primitive filters to describe more complex concepts rather than learning redundant multiple filters for each class separately.

Activation sparsity has been used extensively in the literature in many diferent forms [ 13, 14, 15, 16, 17, 18, 19, 20, 21, 22 ] and it has also been hypothesized that sparsely activated neurons are more interpretable than neurons that activate frequently [ 23 ]. To the best of our knowledge, EBP is the first to induce class-wise activation sparsity as we define it above.

Our experiments show that models trained with EBP maintain their original accuracy while realizing high degrees of sparsity. Furthermore, we demonstrate that the group activation sparsity induced by EBP benefits a rule extraction algorithm that distils the knowledge from a trained CNN and explains the output classification in an interpretable logical language over quantized filter activations represented as logical atoms. We show that EBP yields extracted programs with high fidelity (accuracy in approximating the original model) that use much fewer atoms compared with other activation sparsity methods. Moreover, we present qualitative examples of how the receptive field of filters trained with EBP is much more interpretable compared to training without as it consistently detects a specific pattern across diferent images.

This paper is organized as follows: in Section 2 we review prior work in activation sparsity and interpretability. In Section 3, we present EBP and Section in 4 we conduct experimental evaluation of EBP. Section 5 concludes by summarizing our results and discussing future work.

2. Related Work

Work in the field of neural-symbolic integration [ 3, 8, 12, 24, 25 ] concerns both post-hoc and selfinterpretable models of explainability for neural networks, and employ symbolic representations such as decision trees or logic programs to explain a network’s behavior. ProtoPNet [ 1 ] trains interpretable CNNs by partitioning the input image into sub-regions and associating each region with a prototypical part of some class that “looks like” the extracted region. Another idea introduced in [ 2 ] uses a “filter loss”that pushes a filter to represent an object part of a particular class, but not others. Both lines of work difer from EBP in that they both add a new structural layer whereas EBP only adds a penalty to the loss that enforces class-wise activation sparsity.

Another line of research focuses on sparsifying connections or activations in neural networks in order to reduce overfitting and redundancy in representations while optimizing the model in terms of accuracy and speed. The literature in weight sparsification and compression is vast, see for example [ 26 ] and references therein. However, our work is only concerned with activity sparsity, and not weight sparsity.

Sparse activity was initially inspired by sparse coding [ 23 ] and has been used in the literature in many diferent forms. The commonly used ReLU activation function [ 13 ] induces activation sparsity in a fraction of neurons and recently authors in [ 18 ] proposed three activation functions to induce spike-like sparsity. A k-Winners-Take-All activation (k-WTA) function which retains only the k highest activations from a layer and sets all the rest to zero was used in [ 17 ] to improve adversarial robustness. This is a natural generalization of the boolean K-Winners-TakeAll network [ 27 ] which was motivated by biological neural circuits and had boolean outputs. Authors in [ 28 ] also utilized k-WTA in k-Sparse Autoencoders. Another winner-take-all method is Duty Cycle [ 19 ] which sparsifies activations in a layer like k-WTA but in addition it sparsifies connections preceding that layer by initializing it from sparse random distribution and also it introduces a boosting term to favor units that have not been recently active in order to encourage every unit to be equally active and hence maintain the representational power of the model. Our method difers from these works in that it associates images of each class to a common group of winners/elites without modifying the forward pass or pruning connections. Instead it penalizes filters that activate on images from a class that has low probability of activating those iflters.

Dropout [ 15 ] has been widely used to prevent co-adaptation of neurons by randomly dropping them during training with uniform probability. Sparseout [ 16 ] extended Dropout by imposing an penalty on the activations, allowing one to choose the level of activation sparsity. Recently DASNet [ 20 ] introduced a dynamic activation sparsity method, utilizing a winners-take-all dropout technique. DASNet behaves as a mask between layers and prunes low-ranking neurons in terms of their activation magnitude at run-time for computational speedups. In contrast EBP is not concerned with pruning and the activations are not masked at runtime.

Other solutions induce activation sparsity by applying regularization penalties; [ 29 ] penalizes the deviation of the expected activation of the hidden units from a low fixed level to achieve a sparsity level layer-wise and [ 14 ] added a cross entropy term between the average probability of unit activation and the desired sparsity level that encourages the activation probability of a neuron to be close to . [ 30 ] used a clustering based regularization approach to obtain sparse representations. Recently [ 21 ] exploited an 1 activation regularizer for computational benefits and [ 22 ] improved the results using a variant of ReLu in conjuction with a regularizer based on Hoyer sparsity metric [ 31, 32 ]. Both [ 21, 22 ] induce activation sparsity layer-wise, i.e., a certain percent of neuron activations in each layer will be retained and [ 22 ] introduces a novel approach for activation pruning for computational speedups. In contrast, EBP induces class-wise sparsity, meaning that images from a specific class will be associated with the same group of elite filters and no pruning is performed.

3. Method

We denote by the number of classes in a dataset and by the ground truth class of an image ∈ . Given a conv-layer of a CNN with filters {1(), . . . , ( ) } and a batch of images {, . . . , }, let () = ((1), (2), . . . , () ) be the feature map output of at the -th layer, where each (), = 1, . . . , , is a 2D activation matrix that is the output of convolving the feature map of layer − 1 with the -th filter for -th image, followed by ReLU and maxpooling (if present). (0) = denotes the input image. The activation () of filter () for the -th image in the batch at layer is defined as the spatial average of activations: () = 1 ()

∑︁ ∑︁ |( )|, () where , denotes the height and width of the feature map at layer respectively for the image and (· ) stands for the (, ) spatial coordinates. A filter () is said to be active for an image if its activation () > , where is a specified threshold.

EBP associates each class with a handful of elite filters that activate rarely and have strong activation magnitude on images from that class. This is accomplished by assigning each filter a probability of being active for each class based on the frequency and magnitude of activations on images from that class during training. Then, for each class filters are ranked and the top- form its elite. controls the degree of sparsity and representation power of the model (more sparse for lower ). Filters that activate on images from a class that do not belongs to its elite are penalized, with penalties inversely proportional to their ranking.

On each iteration and for each image of class in the batch, EBP stores the -dimensional vector (1 , . . . , () ) of average filter activations (Eq. 1) from a layer accumulatively in a () vector as follows: ←

+ ((1), . . . , () ).

Afterwards, for each class , filters are ranked based on their history of accumulated activations: ≻ +1 if > +1 , for ∈ {1, . . . , − 1}, where = (1 , . . . , ) and = ∑︀ , = 1, . . . , denotes the accumulated activations of the -th filter for class (layer () is removed from the superscript for notational convenience). The highest activated filters for each class form its elite and are stored in a set = {1 , . . . , }. For each filter we define a probability of being active for class as in Equation 4 and we penalize filters which activate on an image of class , with penalties (1:) proportional to (1 − ) (i.e., the probability of filter being inactive): (1:) = ∑︁ ∑︁(1 − ) ,

() =1 =1 where = 1 − , where 1:, denotes the set of weights from layer 1 up to , = 1 if ̸∈ and 0 otherwise, is the -th index of and the -th sorted accumulated activation (Eq. 3) for class . The total loss we optimize in each batch is a combination of cross entropy loss and : (, ˆ) = − ∑︁ ∑︁ log ˆ + (1:), =1 =1 where = 1 if the -th observation is of class and 0 otherwise, ˆ is the predicted probability that is of class and ∈ R controls the regularization strength. In our experiments we have chosen for simplicity the number of elites per class to be equal to for all classes, and we leave experimenting with diferent for each class for future work. We refer to the Appendix 6 for further discussion and visualization of the class probabilities of images before and after applying EBP. (1) (2) (3) (4) (5)

4. Experiments

We conduct experiments on 2 image classification datasets; a 6-class animal subset [ 2 ] of Pascal VOC [ 33 ] and a toy 3-class subset of places365 dataset [ 34 ] consisting of the classes forest road, highway and street which we will refer to as the road dataset. Regarding (train, valid, test) splits, we further split the original train-val to get a test set because annotations were not available for the original test set. We use (3341, 594, 1399) for the 6-class Pascal subset and (10444, 1500, 3054) for the 3-class road dataset. We quantitatively compare EBP in terms of accuracy, activation sparsity and rule extraction (Section 4.1) with several activation sparsity methods from the literature. We show that EBP results in higher sparsity without sacrificing accuracy and also that rules extracted from a CNN trained with EBP use fewer atoms and have higher fidelity. Qualitative examples of extracted rules in the road dataset using EBP are shown in Fig. 3. From k-winner-take-all methods we compare with k-WTA and Duty Cycle and from regularization methods we compare with [ 29 ] which we refer to as EASR (Expected Activation Sparsity Regularization). We use Sparseout to investigate the efect of dropout-like methods. Since our method is concerned with sparsifying existing layer activations, we do not compare with [ 1, 2 ] because they require additional, specialised neural layers to enforce interpretability.

For each method, we initialized VGG-16 with weights pretrained on ImageNet and followed the preprocessing and augmentations in [ 35 ]. We finetuned the last dense layer for 50 epochs with Adam optimizer [ 36 ], ridge regularization parameter 0.005 and learning rate 5 × 10− 5 using an NVIDIA GPU 1080 Ti 12GB and Tensorflow 1. Afterwards, we applied each method after the conv13 layer and resumed training all layers for 100 epochs with learning rate 10− 6. We select the conv13 layer because filters in deeper layers in CNNs tend to represent object parts and more semantic concepts than earlier layers [ 37, 38 ]. Regarding the hyperparameters, for EBP we trained with diferent ∈ {10, 25, 50, 100, 200, 300} (same for all classes) and regularization values2 ∈ Λ = {0.1, 0.01, 0.001, 0.0001}. For EASR we used ∈ {0.0005, 0.0001, 0.00005, 0.00002} and for k-WTA we trained with density ratio ∈ {0.05, 0.1, 0.2, 0.5, 0.8, 0.9, 0.95}. For Duty Cycle we tune the density coeficient ˆ that controls the percentage of neurons that are expected to be active in {0.1, 0.2, 0.5, 0.7, 0.9} and the boosting coeficient from 1 up to 512, incrementing it in powers of 2. For Sparseout we experimented with values for ∈ {0.3, 0.5} and ∈ {0.5, 1, 1.5, 2, 2.5, 3}.

4.1. Sparsity and Rule Extraction

We measure the activity sparsity after training with each method with various metrics such as Hoyer measure [ 32 ], Lifetime and Population kurtosis, Treves-Rolls lifetime (T-R Life.), TrevesRolls population (T-R Popul.) and activity sparseness [ 39 ] on images from road and 6-class subset of Pascal dataset. The rationale behind choosing this toy road dataset to assess the sparsity and benefits in a simple rule extraction algorithm (Section 4.1) is that scenes contain topics that are shared between diferent classes and we don’t want to learn separate filters for each of them. For example, trees appear in all classes, hence it is preferable to learn one filter 1https://www.tensorflow.org/ 2for smaller values and = 0.01 the convergence was slower. Training for 100 epochs was necessary to determine the best model but with higher values and lower the models converged in 30 − 50 epochs. that fires in response to trees (in all classes) rather than having a filter per class. Results in Tables 1 and 2 indicate that EBP achieves high activity sparsity without sacrificing accuracy. Fig. 2 depicts that filters trained with EBP consistently detect a specific pattern across images and that they fire in response to smaller and more compact semantic regions which are more interpretable and have activation patterns that are more clear.

Though there is no consensus in the literature regarding a metric for interpretability, the size of an explanation has been proposed as an option [ 8, 12 ]. To assess the extent to which EBP benefits rule extraction by inducing parsimonious compositional representations that re-use and combine primitive filters to form new concepts, we show how a binary decision tree trained as a symbolic approximation of a CNN’s behavior is more compact when that CNN is trained with EBP than without. To perform rule extraction we first quantize each filter activation. A iflter is considered active if its activation (Eq. 1) is above ( + ) where and denote the mean and standard deviation of filter activations after conv13 layer across all images. The tree takes the thresholded filter activations of the conv13 layer as the input and the CNN’s output classification as the target so that for a given input instance the induced decision path from the root to the leaf node provides a symbolic explanation of the original CNN’s classification of that instance. Each such path can be regarded as a separate rule over a set of atoms, e.g. ∧ ¬ ∧ → highway states (Fig. 3 top-left) that if filters A and C are active and filter B is inactive, then the input is a highway.

We approximated VGG-16 on the road and 6-class subset of Pascal dataset by generating decision trees using the sklearn3 library’s DecisionTreeClassifier class using entropy criterion and cost complexity pruning to choose the optimal depth. Fig. 1 (1a-c, 2a-c) shows the tradeof between sparsity, fidelity and accuracy after training using diferent hyperparameter values for each method. Results in Table 1 and Fig. 1 demonstate that programs extracted from a CNN trained with EBP have higher fidelity and use consistently fewer unique atoms and rules in explanations compared to other approaches. This means that EBP induces more compositional representations that re-use and combine filters instead of using redudant multiple filters for each class separately. Fig. 3 shows some rules extracted from a CNN trained with EBP and the receptive field of filters activation. For visualization purposes we used a tree of depth 3.

5. Conclusion and future work

In this paper we proposed a method that promotes group activation sparsity to encourage more parsimonious and interpretable representations in a CNN. We demonstrated that models trained with EBP realize high degrees of activation sparsity without sacrificing accuracy and benefit a rule extraction algorithm that distils the knowledge from a CNN; fewer atoms are used in explanations while maintaining high fidelity. In future work we aim to conduct experiments with diferent values for each class , further analysis of diferent layers and rule extraction and possibly an application with emphasis on studying the benefits of compositionality. Another future direction is to embed rules into neural networks, in order to enhance their interpretability and generalization capabilities. Rule embedding can assist in neural-symbolic integration, which is an important step towards bridging the gap between neural networks and symbolic representations.

6. Appendix

6.1. Best hyperparameters for each method

For each method in Tables 1 and 2 we select hyperparameter that yield the best accuracy on valiation set. If diferent hyparameters yielded similar accuracy (less than 0.5% accuracy deviation) we choose the parameters that yielded much higher Hoyer sparsity. The best parameters for each dataset are shown in Table 3. Fig. 1 shows the results on diferent hyperparameter runs.

For EBP higher values for resulted in lower sparsity, higher number of atoms used in explanations and lower fidelity. This is intuitive because most of the kernels in conv13 layer are redundant for classification on road dataset or on the 6-class subset of Pascal. Moreover from Fig. 1 is evident that as increases EBP performs similarly with other methods from the literature, however, EBP performs better for lower .

Road dataset Method parameters EBP = 25, = 0.001 Sparseout = 0.3, = 2.5 DutyCycle ^ = 0.7, = 8 -WTA = 0.8 EASR = 0.00005 6-class subset of Pascal dataset

parameters = 25, = 0.001 = 0.3, = 2 ^ = 0.7, = 8

= 0.2 = 0.00001

6.2. Probabilities of filter activations for each class

In Section 3 we defined the probability of a filter being active for a particular class as in Equation 4. This probability is dynamically computed during training from the history of activations. Fig. 4 (top) shows the evolution of a filter activation for each class before and after applying EBP. Before applying EBP the filter fired in responce to a collection of patterns (trees, road, cyclists), hence the probability of filter activation was spread across diferent classes (blue histogram on Fig. 4). Moreover it was dificult to assess the cause of its activation. After training with EBP the filter fires in responce to a specific pattern (trees) which is shared between classes but mostly present in forest road. Fig. 4 (bottom) shows the evolution of another filter that is shared between classes. Before applying EBP the filter fired in responce to trafic signs and trees. After training with EBP the filter rfies in responce to trafic signs only which are present mostly in highway and street classes (hence the probability for those classes has been boosted in orange histogram). Observe that before applying EBP the probability of activation for forest road was higher (since the presence of trees and trafic signs combined is higher in forest road that in street and highway class).

6.3. Discussion of proposed method and regularization

In the proposed algorithm in Section 3 elite filters were not penalized during training, since the assigned probability from Equation 4 is 1. This can be altered and introduce penalties for all iflters (elite included) by defining = 1 − 1 . Penalizing all filters may induce even more activity sparsity. In our experiments we used the setup described in Section 3.

Furthermore, we recommend applying ridge weight regularization on the ( + 1) layer following the EBP -th layer to constrain weights in a small Euclidean ball. The reason is that EBP penalizes activations on layer according to a probability distribution and if we do not impose any constraints on the weights following that layer then the model has the freedom to learn arbitrarily large weights on layer + 1 and possibly negate our penalization. In our experiments we did not use additional regularization after conv13 (where EBP was applied) in order to make our method directly comparable with other approaches from the literature. However our preliminary experiments show no significant diference after adding this additional regularization layer in the road and 6-class subset of Pascal dataset.

References

[1] C. Chen , O. Li , D. Tao , A. Barnett , C. Rudin , J. K. Su , This looks like that: Deep learning for interpretable image recognition , in: Advances in Neural Information Processing Systems , volume 32 , Curran Associates , Inc., 2019 .

[2] Q. Zhang , Y. N. Wu , S.-C. Zhu, Interpretable convolutional neural networks , in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018 , pp. 8827 - 8836 .

[3] J. Townsend , T. Kasioumis , H. Inakoshi , ERIC: extracting relations inferred from convolutions , in: 15th Asian Conference on Computer Vision , Kyoto, Japan, Revised Selected Papers, Part III , volume 12624 of Lecture Notes in Computer Science, Springer, Nov. 30 - Dec. 4, 2020 , pp. 206 - 222 .

[4] R. Guidotti , A. Monreale , S. Ruggieri , F. Turini , F. Giannotti , D. Pedreschi , A survey of methods for explaining black box models , ACM Comput. Surv . 51 ( 2018 ).

[5] S. M. Lundberg , S.-I. Lee , A unified approach to interpreting model predictions , vol. 30 , in: Advances in Neural Information Processing Systems , Curran Associates, Inc., 2017 .

[6] M. T. Ribeiro , S. Singh , C. Guestrin , "why should i trust you?": Explaining the predictions of any classifier, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , San Diego California, USA, June 12-17, 2016 , pp. 97 - 101 .

[7] N. Frosst , G. E. Hinton , Distilling a neural network into a soft decision tree , ArXiv abs/1711 .09784 ( 2017 ).

[8] R. Andrews , J. Diederich , A. B. Tickle , Survey and critique of techniques for extracting rules from trained artificial neural networks , Knowledge-Based Systems 8 ( 1995 ) 373 - 389 .

[9] H. Jacobsson , Rule extraction from recurrent neural networks: Ataxonomy and review , Neural Computation 17 ( 2005 ) 1223 - 1263 .

[10] Q. Zhang , R. Cao , F. Shi , Y. N. Wu , S.-C. Zhu, Interpreting cnn knowledge via an explanatory graph , Proceedings of the AAAI Conference on Artificial Intelligence 32 ( 2018 ).

[11] Q. Zhang , Y. Yang , H. Ma, Y. N. Wu , Interpreting cnns via decision trees , in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019 .

[12] J. Townsend , T. Chaton , J. M. Monteiro , Extracting relational explanations from deep neural networks: A survey from a neural-symbolic perspective , IEEE Transactions on Neural Networks and Learning Systems 31 ( 2020 ) 3456 - 3470 .

[13] X. Glorot , A. Bordes , Y. Bengio , Deep sparse rectifier neural networks , in: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , Fort Lauderdale, USA, 2011 , pp. 315 - 323 .

[14] V. Nair , G. E. Hinton , 3d object recognition with deep belief nets , in: Advances in Neural Information Processing Systems , volume 22 , Curran Associates , Inc., 2009 .

[15] N. Srivastava , G. Hinton , A. Krizhevsky , I. Sutskever , R. Salakhutdinov , Dropout: A simple way to prevent neural networks from overfitting , J. Mach. Learn. Res . 15 ( 2014 ) 1929 - 1958 .

[16] N. Khan , I. Stavness , Sparseout: Controlling sparsity in deep networks , in: Advances in Artificial Intelligence - 32nd Canadian Conference on Artificial Intelligence , Canadian AI 2019 , Kingston , ON , Canada, May 28 -31, 2019 , Proceedings, volume 11489 of Lecture Notes in Computer Science, Springer, 2019 , pp. 296 - 307 .

[17] C. Xiao , P. Zhong , C. Zheng , Enhancing adversarial defense by k-winners-take- all , International Conference on Learning Representations (ICLR) ( 2020 ).

[18] P. Bizopoulos , D. Koutsouris , Sparsely activated networks , IEEE Transactions on Neural Networks and Learning Systems 32 ( 2021 ) 1304 - 1313 .

[19] S. Ahmad , L. Scheinkman, How can we be so dense? the benefits of using highly sparse representations, ICMLWorkshop on Uncertainty and Robustness in Deep Learning ( 2019 ).

[20] Q. Yang , J. Mao , Z. Wang , H. Li , Dasnet: Dynamic activation sparsity for neural network eficiency improvement , in: 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI) , 2019 , pp. 1401 - 1405 .

[21] G. Georgiadis , Accelerating convolutional neural networks via activation map compression , in: Conference on Computer Vision and Pattern Recognition (CVPR) , 2019 , pp. 7078 - 7088 .

[22] M. Kurtz , J. Kopinsky , R. Gelashvili , A. Matveev , J. Carr , M. Goin , W. Leiserson , S. Moore , N. Shavit , D. Alistarh , Inducing and exploiting activation sparsity for fast inference on deep neural networks , in: Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , 2020 , pp. 5533 - 5543 .

[23] F. D. J. Olshausen B. A. , Sparse coding with an overcomplete basis set: A strategy employed by v1? , Vision research 37 ( 2007 ) 3311 - 3325 .

[24] I. Donadello , L. Serafini , A. D. Garcez , Logic tensor networks for semantic image interpretation , IJCAI'17 , AAAI Press, 2017 , p. 1596 - 1602 .

[25] A. S. d. Garcez , D. M. Gabbay , K. B. Broda , Neural-Symbolic Learning System: Foundations and Applications , Springer-Verlag, Berlin, Heidelberg, 2002 .

[26] R. Ma , L. Niu , A survey of sparse-learning methods for deep neural networks , in: 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI) , 2018 , pp. 647 - 650 .

[27] E. Majani , R. Erlanson , Y. Abu-Mostafa , On the k-winners-take-all network , in: Advances in Neural Information Processing Systems , volume 1 , Morgan-Kaufmann, 1989 .

[28] A. Makhzani , B. Frey , K-sparse autoencoders , arXiv preprint arXiv:1312.5663 ( 2013 ).

[29] H. Lee , C. Ekanadham , A. Y. Ng , Sparse deep belief net model for visual area v2 , in: Proceedings of the 20th International Conference on Neural Information Processing Systems , NIPS'07, Curran Associates Inc., Red Hook , NY , USA, 2007 , p. 873 - 880 .

[30] R. Liao , A. Schwing , R. S. Zemel , R. Urtasun , Learning deep parsimonious representations , in: Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, Curran Associates Inc., Red Hook , NY , USA, 2016 , p. 5083 - 5091 .

[31] K. Kimura , T. Yoshida, Non-negative matrix factorization with sparse features , in: 2011 IEEE International Conference on Granular Computing , 2011 , pp. 324 - 329 .

[32] P. Hoyer , Non-negative matrix factorization with sparseness constraints , Journal of machine learning research ( 2004 ) 1457 - 1459 .

[33] M. Everingham , L. Van Gool , C. K. I. Williams , J. Winn , A. Zisserman , The pascal visual object classes challenge 2012 (voc2012) results , 2012 . URL: http://www.pascal-network. org/challenges/VOC/voc2012/workshop/index.html.

[34] B. Zhou , A. Lapedriza , A. Khosla , A. Oliva , A. Torralba , Places: A 10 million image database for scene recognition , IEEE Transactions on Pattern Analysis and Machine Intelligence 40 ( 2018 ) 1452 - 1464 .

[35] A. Krizhevsky , I. Sutskever, G. E. Hinton , Imagenet classification with deep convolutional neural networks , in: Advances in Neural Information Processing Systems , volume 25 , Curran Associates , Inc., 2012 .

[36] D. P. Kingma , J. Ba , Adam: A method for stochastic optimization , in: 3rd International Conference on Learning Representations, ICLR 2015 , San Diego, CA, USA, May 7- 9 , 2015 .

[37] D. Bau , B. Zhou , A. Khosla , A. Oliva , A. Torralba , Network dissection: Quantifying interpretability of deep visual representations , 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ( 2017 ) 3319 - 3327 .

[38] S. Odense , A. Garcez , Layerwise knowledge extraction from deep convolutional networks , ArXiv abs/ 2003 .09000 ( 2020 ).

[39] B. Willmore , D. Tolhurst , Characterizing the sparseness of neural codes, Network (Bristol , England) 12 ( 2001 ) 255 - 270 .