We present a novel neurosymbolic framework called NeSyFOLD to extract logic rules from a CNN and create a NeSyFOLD model to classify images. NeSyFOLD's learning pipeline is as follows: (i) We first pre-train a CNN on the input image dataset and extract activations of the last layer kernels as binary values; (ii) Next, we use the FOLD-SE-M rule-based machine learning algorithm to generate a logic program that can classify an image-represented as a vector of binary activations corresponding to each kernel-while producing a logical explanation. The rules generated by the FOLD-SE-M algorithm have kernel numbers as predicates. We have devised a novel algorithm for automatically mapping the CNN kernels to semantic concepts in the images. This mapping is used to replace predicate names (kernel numbers) in the rule-set with corresponding semantic concept labels. The resulting rule-set is interpretable, and can be intuitively understood by humans. We compare our NeSyFOLD framework with the ERIC system that uses a decision-tree like algorithm to obtain the rules. Our framework has the following advantages over ERIC: (i) In most cases, NeSyFOLD generates smaller rule-sets without compromising on the accuracy and fidelity; (ii) NeSyFOLD generates the mapping of filter numbers to semantic labels automatically.
Explainability in AI is an important issue that has resurfaced in recent years as deep learning models have become larger and are applied to an increasing number of tasks. Some applications such as autonomous vehicles, disease diagnosis, and natural disaster prevention are very sensitive areas where a wrong prediction could be the diference between life and death. Motivated by the need for explainability we introduce a neurosymbolic framework NeSyFOLD, that gives a global explanation for the predictions made by a CNN.
The NeSyFOLD framework uses a Rule Based Machine Learning (RBML) algorithm called
FOLD-SE-M [
We compare our NeSyFOLD framework with the ERIC system [
We developed an algorithm for mapping the significant kernels extracted from the CNN to the semantic concept(s) that they represent. We show in our experiments that the NeSyFOLD model is able to either outperform or meet ERIC’s accuracy and fidelity while generating a significantly smaller-sized rule-set. The NeSyFOLD-Group model is able to outperform the NeSyFOLD model and generate an even smaller rule-set.
We use rule-set size as a metric of interpretability. We also propose a novel algorithm for automatic semantic labeling of the predicates in the rule-set extracted by NeSyFOLD while ERIC’s rule set has to be labeled manually. In addition to the accuracy, we also compare NeSyFOLD’s fidelity with ERIC’s. There is always a trade-of between explainability and fidelity due to some information loss caused by the discretization of continuous values which is seen in both systems.
Default Rules: The FOLD-SE-M algorithm [
can be formalized as the normal logic programming rule:
:- 1, 2, . . . , , not 1, not 2, . . . , not . where ’s and ’s are positive predicates and not represents negation-as-failure. We call such rules default rules. Thus, the default () : ¬()
() will be represented as the following default rule in normal logic programming: flies(X) :- bird(X), not penguin(X).
We call bird(X), the condition that allows us to jump to the default conclusion that X flies, the default part of the rule, and not penguin(X) the exception part of the rule.
FOLD-SE-M [
There is a lot of past work which focuses on visualizing the outputs of the layers of the
CNN. These methods try to map the relationship between the input pixels and the output of the
neurons. Zeiler et al. [
Qi et al. [11] propose an Explanation Neural Network (XNN) which learns an embedding in high-dimension space and maps it to a low-dimension explanation space to explain the predictions of the network. A sentence-like explanation including the features is then generated manually. No rules are generated and manual efort is needed. Chen et al. [ 12] introduce a prototype layer in the network that learns to classify images in terms of various parts of the image. They assume that there is a one to one mapping between the concepts and the filters. We do not make such an assumption. Zhang et al. [13, 14] learn disentangled concepts from the CNN and represent them in a hierarchical graph so that there is no assumption of a one to one filter-concept mapping. However, no logical explanation is generated. Bologna et al. Some popular methods for generating explanations are LIME [15] and its extended rule-based version [16] which generate local explanations based on features. Our system generates global explanations.
We now proceed to explain the methodology behind the learning, inference, and semantic
labeling pipeline used in our NeSyFOLD framework. Our learning and inference pipelines are
similar to and inspired by the algorithm used by Townsend et al. for ERIC [
We start by training the CNN on the input images for the given classification dataset in the conventional manner, namely, by using a loss function to get the loss and then propagating the gradient w.r.t the loss through the CNN via backpropagation. Any optimization technique can be used for updating the weights.
Quantization: Quantization is the process of binarization of the kernel outputs. The quantization function is defined in eq. (3). is the threshold for a specific kernel . Each kernel in the last convolution layer produces a feature map , for a given image. We collect all such feature maps from all kernels for all images. For each kernel , an activation threshold is calculated by using eq. (4) and eq. (5). and are hyperparameters. Once the is calculated for each iflter then using eq. (3) and eq. (2) we get the binarization , for each image and each kernel. Iterating through all images we obtain a table of size × where is the number of images in the training data and is the number of kernels. Each row represents the binarized output of the CNN corresponding to each image. If the kernel grouping is true then for each kernel the top 10 images that activate it are selected. Then the most similar kernels are identified by calculating cosine similarity. Finally, the kernel groups are decided by using a hyperparameter which we call the “similarity threshold”. Each column in the table represents groups of similar kernels in this case.
Rule Extraction: The FOLD-SE-M algorithm is used on the binarization table. It extracts the rules that explain a target class in terms of the kernels/kernel-groups. Each predicate in the body of the rules is a kernel in the CNN. An example rule could be: target(X,‘2’) :- not 3(X), 54(X), not ab1(X).
This rule can be interpreted as “Image X belongs to class ‘2’ if kernel 3 is not activated and kernel 54 is activated and the abnormal condition (exception) ab1 does not apply. Semantic labeling : After the rules are generated we find the semantic labels corresponding to the predicates/kernels in the rule-set for better interpretability. We use a dataset that provides the corresponding semantic segmentation mask along with the training images. We take an intersection of the semantic segmentation masks with the feature maps produced by each kernel for each image and label that kernel as the majority concept which it is activated by in most images. Note that we currently support labeling the rule-set used by the NeSyFOLD model, where each predicate is a single kernel. The semantic labeling algorithm for the NeSyFOLDGroup model’s rule-set is part of future work. The rules with filters replaced by semantic labels are generated. The same example rule from above may now look like: target(X,‘bathroom’) :- not bed(X), bathtub(X), not ab1(X). , =(,, ) (2) (3) (4) (5) (,, ) = {︃1, if , >
0, otherwise , =||,||2
= · + √︂ 1 ∑︁(, − )2
The inference pipeline of NeSyFOLD is relatively straightforward. To classify a new image, we feed it to the CNN and the kernel feature maps for each kernel in the last layer are obtained. Then, as in the learning pipeline using Equations (2) and (3), we get the binarizations for each kernel output and generate the binarization table as in the learning pipeline. Next, we run the FOLD-SE-M toolkit’s built-in rule interpreter on the labeled/unlabelled rule-set.
First, we compare our NeSyFOLD and NeSyFOLD-Group models with the model generated by the ERIC system and report the accuracy, fidelity, number of unique predicates/atoms in the rule-set, number of rules generated and the size of the rule-set. For ERIC, the rules generated are in the form of CNF formula where each atom is a binarized kernel output or its negation. Size is calculated as the total number of antecedents for ERIC and the total number of predicates in the bodies of the rules that constitute the logic program generated by NeSyFOLD.
We used a VGG16 CNN with pre-trained weights on the Imagenet dataset [17]. We trained for
100 epochs with a batch size of 32. We used the Adam [18] optimizer and applied class weights
for imbalanced data. We also used 2 regularization of 0.005 on all layers and a learning rate
of 5 × 10− 7. We used a decay factor of 0.5 and patience of 10 epochs. Also, we resized all
images to 224 × 224. We used an of 0.6 and of 0.7. We used almost the same parameters
as used by Townsend et al. in [19] for training so that we could do a fair comparison. For
this experiment, we used the German Trafic Sign Recognition Benchmark (GTSRB) [20], MNIST
[21] and the Places [22] dataset. We selected 5 classes from the Places dataset namely, {desert
road(de), driveway(dr), forest road(f), highway(h), street(s)}. In Table 4 and Table 2 we have
shown the “dedrf", “dedrh", “dedrs" and “defs" class combinations. We cite the performance
metrics of ERIC on these 4 class combinations from the paper [
NeSyFOLD-Group equals the NeSyFOLD model w.r.t accuracy and w.r.t fidelity in all but one dataset where they are comparable. The grouping of filters helps in generating better features for the binarization table.
Second, we examine the efect of semantic labeling of the rule-set generated for the NeSyFOLD model. We use the ADE20k dataset for mapping the predicates in the rule-set obtained by NeSyFOLD trained on 3 diferent classes of the Places dataset separately. These classes are {bathroom, bedroom, kitchen} and {desert road, driveway, forest road}.
Below we show the labeled rule-set and the interpretation of the first rule for each class from both rule-sets in English. The interpretation of the other rules can be done similarly. We used a of 0.5 and of 1− 2. {Rule-set 1: } target(X,‘kitchen’) :- cabinet1(X), not ab1(X), not ab2(X), not ab3(X). target(X,‘bedroom’) :- bed1(X). target(X,‘bathroom’) :- countertop1_toilet1_sink1(X). target(X,‘bathroom’) :- not cabinet3(X), not countertop1_toilet1_sink1(X), bathtub1(X). target(X,‘bedroom’) :- bed2(X), not mirror1_countertop2(X). target(X,‘kitchen’) :- cabinet7(X). ab1(X) :- not cabinet4(X), not cabinet8(X). ab2(X) :- not cabinet2(X), bed1(X). ab3(X) :- not cabinet6(X), countertop1_toilet1_sink1(X),not cabinet5(X).
The interpretation of the first rule of the ‘kitchen’ class after expanding the ab1(X), ab2(X) and ab3(X) predicates is the following: “Image X is a kitchen if it has cabinets and no countertoptoilet-sink combination and no bed.” The same interpretations for the ‘bedroom’ and ‘bathroom’ classes are “Image X is a bedroom if there is a bed” and “Image X is a bathroom if there is a combination of countertop, toilet, and sink”, respectively. This rule-set has 9 rules, 13 Unique predicates and the total size is 19. It achieves 80% accuracy on the test set. {Rule-set 2:} target(X,‘forest_road’) :- trees2(X), not house4(X), not ab1(X). target(X,‘driveway’) :- house5_building2(X). target(X,‘desert_road’) :- not trees1(X), not ab2(X). target(X,‘forest_road’) :- not house2(X), not trees2(X), trees3(X). target(X,‘driveway’) :- not house5_building2(X), not road2(X). target(X,‘forest_road’) :- not road2(X). ab1(X) :- not trees3(X), house3_building1(X). ab2(X) :- not ground1_road1(X), house1(X).
The interpretation of the first rule for the ‘forest road’ class after expanding the ab1(X) predicate is the following: “Image X is a forest road if it has trees and no houses or buildings”. Likewise, the first rule for ‘desert road’ and ‘driveway’ classes represents: “Image X is a desert road if there are no trees and no houses” and “Image X is a driveway if there is a house/building”, respectively. This rule-set has 8 rules, 10 Unique predicates and the total size is 16. It achieves 83% accuracy on the test set. For an example image classified as a “forest road”, the justification can be obtained from s(CASP) querying target(img, ‘forest_road’) to Rule-set 2 JUSTIFICATION_TREE: ‘target’ holds (for img, and forest_road),because ‘trees2’ holds (for img), and there is no evidence that ‘house4’ holds (for img), andthere is no evidence that ‘ab1’ holds (for img), because‘trees3’ holds (for img).
MODEL: {target(img,forest_road), trees2(img), not ab1(img), trees3(img)}
From the justification, it can be clearly seen that the first rule was invoked. Notice that the FOLD-SE-M algorithm orders the rules according to the number of images covered by the rules in the form of a decision list. That is, the topmost rule will classify the most images, and the bottommost the least. Hence the topmost rules make more sense as they capture all the concepts that can make the sharpest distinction among the 3 classes. The bottom rules capture the outliers that were not classified by the above rules. For example, the 6th rule in Rule-set 2 will be invoked only if all the other rules above it were not applicable. This intuitively suggests that the image is a ‘forest_road’ image if there is not a particular type of road (not road2(X)), but only if it does not fit in the more frequently seen bin of images of ‘forest_road’ which have trees and no houses or buildings (as captured by rule 1). Note, the heavy biases in the data that the CNN learns would get captured as defaults in the top rules. Also, both s(CASP) and FOLD-SE-M rule interpreters can automatically generate a justification for each example that is classified.
Third, since each predicate represents a kernel’s binarized output and for each predicate its corresponding semantic concept(s) is known, there should be a way to utilize the kernels present in the rules to form new rules. We ran an experiment where we trained a CNN on the {desert, highway, lake} classes and generated the labeled rules using NeSyFOLD. We then wrote a new rule target(X, ‘beach’):- sand(X), water(X). using the predicates associated with water and sand from the existing rule-set. We used this rule only to classify images of the ‘beach’ class of the Places dataset. Out of the 1000 images, 690 (69%) were correctly classified as ‘beach’ by the rule. This is an interesting insight as it shows that the CNN filters can be individually used as concept/object detectors.
In this paper we have shown that using FOLD-SE-M algorithm leads to a significant reduction in the rule-set size without compromising on the accuracy in most cases. The kernel grouping intuitively leads to better features in the binarization table and hence the FOLD-SE-M algorithm is able to represent the information in fewer rules. The semantic labeling algorithm we propose makes the NeSyFOLD model’s rule-set easier to understand. A natural direction for future research is modifying the semantic labeling algorithm to accomodate the predicates that represent groups of kernels generated by the grouping algorithm. We believe this will be an important step for writing new rules to classify images of unseen classes during training. [9] M. Denil, A. Demiraj, N. de Freitas, Extraction of salient sentences from labelled documents, 2014. URL: https://arxiv.org/abs/1412.6815. doi:10.48550/ARXIV.1412.6815. [10] K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: Visualising image classification models and saliency maps, 2013. URL: https://arxiv.org/abs/1312.6034. doi:10.48550/ARXIV.1312.6034. [11] Z. Qi, S. Khorram, L. Fuxin, Embedding deep networks into visual explanations, Artificial
Intelligence 292 (2021) 103435. doi:10.1016/j.artint.2020.103435. [12] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, J. K. Su, This looks like that: deep learning for interpretable image recognition, Advances in neural information processing systems 32 (2019). [13] Q. Zhang, R. Cao, Y. N. Wu, S.-C. Zhu, Growing interpretable part graphs on convnets via multi-shot learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017. [14] Q. Zhang, R. Cao, F. Shi, Y. N. Wu, S.-C. Zhu, Interpreting CNN knowledge via an explanatory graph, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. [15] M. T. Ribeiro, S. Singh, C. Guestrin, "why should i trust you?": Explaining the predictions of any classifier, 2016. URL: https://arxiv.org/abs/1602.04938. doi: 10.48550/ARXIV.1602. 04938. [16] M. T. Ribeiro, S. Singh, C. Guestrin, Anchors: High-precision model-agnostic explanations, in: Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [17] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255. [18] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, 2014. URL: https://arxiv.
org/abs/1412.6980. doi:10.48550/ARXIV.1412.6980. [19] J. Townsend, M. Kudla, A. Raszkowska, T. Kasiousmis, On the explainability of convolutional layers for multi-class problems, in: Combining Learning and Reasoning: Programming Languages, Formalisms, and Representations, 2022. URL: https://openreview.net/ forum?id=jgVpiERy8Q8. [20] J. Stallkamp, M. Schlipsing, J. Salmen, C. Igel, Man vs. computer: Benchmarking machine learning algorithms for trafic sign recognition, Neural Networks 32 (2012) 323–332. URL: https://www.sciencedirect.com/science/article/pii/S0893608012000457. doi:https: //doi.org/10.1016/j.neunet.2012.02.016, selected Papers from IJCNN 2011. [21] Y. LeCun, L. Bottou, Y. Bengio, P. Hafner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (1998) 2278–2324. [22] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, A. Torralba, Places: A 10 million image database for scene recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence (2017).