Interpretability of Convolutional Neural Networks (CNNs) is often crucial for their application in real world scenarios. We aim to provide such interpretations in terms of the semantic content and conceptual structure CNNs acquire from their training data. Recent advances in Explainable AI have shown that CNNs are capable of learning hierarchical relationships between semantic categories in the form of taxonomic classifications. However, accurate evaluation of this ability is an open challenge, of which two aspects are non-trivial: constructing symbolic representations of semantic content after training, and quantification of its adequacy with respect to the semantics of the application domain. Existing evaluation methods are typically restricted to standard CNN performance metrics and do not take into account the underlying decision-making process in terms of explicit semantic structure in the domain. In this paper we propose a taxonomy extraction method for supervised CNN classifiers to capture how symbolic class concepts and their hypernyms from a given domain are hierarchically organised in the model's subsymbolic representation. In addition, we propose a taxonomy ground truth comparison method to evaluate the “semantic adequacy” of the extracted hierarchy of class concepts. Our approach is tested using VGG-16, ResNet-18, ResNet-152 and trained on CIFAR-100 and ImageNet. Results show the influence of the dataset quality and architecture depth on semantic adequacy, as suggested by recent literature [1]. We also observe that existing techniques for injecting external knowledge to the models during the training phase may lead to better taxonomies. This suggests that the hierarchical-aware models may have a semantic advantage over their respective original architectures. Finally, we provide a fine grained approach for analysing CNN interpretability in terms of its semantic content.
Since their inception in 1989 [
The adoption of Convolutional Neural Networks (CNNs) in critical domains such as in healthcare
[
In the field of Explainable AI (XAI), the issue of evaluating and comparing deep learning models
beyond their accuracy over a test set has been tackled by methods that provide global interpretations.
Examples include detection of concept importance [
In this work, we provide methods for semantic interpretability of CNNs, based on how hierarchical
relationships between semantic concepts are captured by the model’s internal representation. To this
aim, we propose a taxonomy extraction method to derive a domain taxonomy from a trained CNN.
For example, for a model trained over the ImageNet dataset [
By comparing such taxonomic axioms with a relevant ground truth class hierarchy (e.g., WordNet for ImageNet) we can evaluate quantitatively how faithful the extracted taxonomy is to the selected groundtruth. In the current context this metric is referred to as semantic adequacy. Note that, independent of model accuracy, a higher semantic adequacy score implies better transparency by allowing interpretation of model behaviour in terms of user-accepted concepts. Moreover, it makes objective semantic comparison between models possible, provided that their training data is drawn from classes in similar or related taxonomies.
In order to achieve these goals, we build upon the idea of co-activation graph [
Experimental results indicate that the CNNs used can learn direct and transitive subclass relationships reasonably well, especially considering that they were not explicitly trained for that and did not have any a priori information about semantic hierarchies. An encouraging finding from our results is that hierarchy-aware ResNets achieved the best results for both datasets. This support the intuition that if we inject external knowledge into a neural network during the training process, we can make the model more interpretable by making such knowledge explicit in relation to the network’s decision process. Our experiments also show that the taxonomy extraction and the idea of semantic adequacy can enhance global interpretability of deep models in terms of the semantics pertaining to class hierarchy encoded in their internal representations.
The rest of this paper is organised as follows. Section 2 presents related work for the topics of global interpretation of CNNs and taxonomy extraction. Section 3 elaborates on the methodology for extracting taxonomies from co-activation graphs built for CNNs. In Section 4 an experiment using CNNs trained for CIFAR-100 and ImageNet is conducted. In Section 5 we present our conclusions and discuss the obtained results.
In XAI, global interpretations aim to provide insights into the behaviour of a model as a whole, instead
of explaining individual inference events. E.g., TCAV [
Using a diferent approach, [
In complex domains, constructing taxonomies by hand is an expensive task. Some approaches have looked into taxonomy extraction from non-textual, structured and semi-structured data, such as knowledge graphs. Authors in [12, 15] propose an unsupervised method to find hypernym axioms. Vector embeddings for each node in the knowledge graph are calculated, resulting in a class centroid and radius in the latent space. Then, based on the distance between the centroids, they form axioms and construct their transitive closures to build a taxonomy.
The main drawback of the approach in [15] is that concepts need to be directly represented as nodes in the knowledge graph. The method proposed in [12] can help overcome this issue. Based on node embeddings, their approach uses hierarchical clustering over the latent space to find a hierarchical structure. By mapping concepts (or types) to cluster-nodes in this structure, a typed tree is formed from which to construct a taxonomy. The advantage is that the clustering phase does not use any information regarding the types. This makes it more suitable in our setup, where these types are typically not represented by nodes in the graph. In this work, we use co-activation graphs as a graph representation for CNNs and adapt the method in [12] to this graph for the taxonomy extraction. In the next section, the methodology to combine the two is presented in detail.
Recent works have shown that CNNs can learn the underlying hierarchical structure between classes
during the training phase even though they were not explicitly trained for this specific task. Motivated
by this observation, authors in [
In their work, the authors in [
On other hand, the method proposed by [16] adds extra classification layers after the final
classification layer of the original architecture. We consider the method from [
Following [
We assume class concepts are organised in a taxonomy, and that a neural network was optimised only to discriminate between leaf node classes. Our goal is to reconstruct, into symbolic form, the full taxonomy from the internal sub-symbolic structure of the model. To this end, we introduce a novel extraction method, designed such that the extracted taxonomy reflects how the model organises output classes and their hypernyms hierarchically in its internal representation.
We use co-activation graphs [
Given a trained neural network model, the nodes of its co-activation graph stand in one-to-one correspondence with its neurons. Each pair of nodes in the graph is connected by a weighted edge determined by the Spearman correlation coeficients between neuron activation values on a test data set. For neurons in dense layers this process is straightforward: there is a single activation value per data sample. For neurons in convolutional layers, a single activation value is obtained by applying average pooling on the feature map.
The resulting graph provides relevant information on how dependencies between internal representations impact the global behaviour of a classifier. This makes co-activation graphs a suitable intermediate representation to analyse how a model transforms sub-symbolic information from pixels, via its internal representations, into an assignment to a semantic class. Next, we will discuss how to transform the statistical information in the co-activation graph into a semantic structure.
We modify the extraction method in [12] such that it applies to co-activation graphs. Three phases can be distinguished: embedding of graph nodes into vector representation, followed by agglomerative clustering, and finally assignment of semantic types to clusters. We now discuss further details of the method. For the embedding of nodes from the co-activation graph to vector representations we make use of existing embedding functions [18, 19]. The results from the first phase is a data set containing a vector for each node in the co-activation graph.
Agglomerative clustering starts by creating a leaf cluster for every vector in . At each iteration, the two closest clusters are merged according to some chosen distance metric. Clustering terminates when there is a single cluster containing every vector in , resulting in a tree structure over the vectors. Assigning semantic types to the tree will turn it into a taxonomy.
In order to assign types to clusters, the method calculates the F-score (, ) for each cluster and type , which indicates how well represents the entities in . The F-score can be calculated as shown in Equation 1, where , is the number of entities with type in cluster , is the number of entities in and is the number of entities with type . A high F-score(C,t) indicates that cluster contains mostly entities of type and the highest number of entities of type is contained in . (, ) = 2 ⋅
, + (1)
We remove clusters that are not associated to a type and evaluate the extracted taxonomy by precision, recall and F-score over edges of the taxonomy (“direct” evaluation) and its transitive closure (“transitive” evaluation).
The experimental evaluation has two distinct goals. First, to measure correlations of class similarity in the embedding space with semantic similarity in the ground truth taxonomy. This indicates the ability of learned representations to separate symbolic concepts in the taxonomy. Second, to evaluate the semantic adequacy of the extracted taxonomy with respect to its ground truth. This indicates the degree of transparency allowed by the particular models as expressed in semantic terms, and the suitability of our method for semantic interpretation in general.
Experiments were conducted using CIFAR-100 and ImageNet datasets. For CIFAR-100 we trained VGG-16, ResNet-18 and hierarchy-aware ResNet-18 (noted HA-ResNet-18). For ImageNet we trained VGG-16, ResNet-152 and hierarchy-aware ResNet-152 (noted HA-ResNet-152).
For each dataset and model, a co-activation graph containing only those connections with correlation higher than 0.3 was built. We extracted the taxonomy using the pipeline in Figure 1. Node embeddings were generated using two diferent embedding methods in order to check if the results are consistent across diferent strategies: Node2Vec [ 19] and Fast Random Projection (FastRP) [18]. For both Node2Vec and FastRP we used the algorithm implemented by Neo4j.
After calculating the node embeddings using the two methods above, the next phase is to apply the agglomerative clustering algorithm. For this phase we have tested diferent distance criteria and metrics, which influence the merging strategy of the agglomerative clustering. The metrics were euclidean and cosine (when applicable) while the distance criteria used were: average (UPGMA), weighted (WPGMA), complete, centroid and ward.
At this point, we end up with a hierarchical clustering tree, and the next phase is to assign types to the clusters that best represent the entities from each type. Because we want to compare the extracted taxonomy with the WordNet hierarchy, we extracted the types directly from WordNet using the nltk python package. Then, for each type and each cluster , we have calculated the (, ) using Equation 1 and assigned types to clusters by solving the corresponding Linear Sum Assignment problem. Finally, the clusters and types that are not associated are removed, and the resulting tree hierarchy represents the extracted taxonomy.
The goal of this analysis is to check whether the produced node embeddings can encode semantic similarity between classes represented in that space. For this analysis, we have first calculated the semantic similarity for every pair of classes in the dataset. This process was done by using the path similarity metric available from the nltk package, which calculates how similar two concepts are based on the shortest path that connects them on the WordNet hierarchy. We have then calculated the spearman correlation between semantic similarity among classes and cosine similarity between the corresponding classes in the embedding space.
In Table 1 it is possible to observe that there is a positive correlation between semantic similarity and cosine similarity among classes in the embedding space. This is a first indication that the CNNs have learned semantic relationships between classes, which supports the idea that it may be possible to reconstruct the taxonomical relationships between them. The HAResNet152 architecture trained on ImageNet achieved the highest correlation, which gives a first evidence that the additional information injected during the training phase helped this model to better capture the semantic relationships between classes. It can also be noted that the results from Node2Vec are higher than FastRP for CIFAR-100 while FastRP performs better for ImageNet, which indicates that the choice of the embedding method may be incluenced by the density of the graph. However, the correlation only is not a strong metric to compare the semantic adequacy between each architecture since they do not expose which semantic relationships were learned by the model. The semantic adequacy comparison between models is done in a more detailed way in the second analysis.
In this second analysis we evaluate the semantic adequacy of the taxonomies extracted using our method by comparing them with the ground truth hierarchy extracted from WordNet. This evaluation is conducted following the same principles used in [12], which uses both a direct and a transitive evaluation. For example, an axiom such as ℎℎ ⊂ is going to cause a negative efect in the direct evaluation, because, according to WordNet, the direct hypernym for a ℎℎ is ℎℎ . In the transitive evaluation, the goal is to evaluate high level axioms. Using the previous example, ℎℎ ⊂ causes a positive efect in the transitive evaluation because there is a transitive relationship between types ℎℎ and , also according to WordNet.
For this analysis, we also evaluated the semantic adequacy of an untrained model using the VGG-16 architecture initialised using random parameters, with the purpose of providing a lower bound baseline. The precision, recall and F-score are calculated using the edges of the graphs representing the extracted taxonomies and the edges of the graph representing the ground truth, as described in Equation 2. For the direct evaluation we calculate the evaluation metrics based directly on the respective graphs whereas for the transitive evaluation we consider the transitive closure.
Ground truth : = ( , ), Experimental : = ( , ) ⎧ = ⎪ ⎪
= ⎨ ⎪ ⎪ - = 2 ⋅ ⎩ | ∩ |
| | | ∩ | | | | ∩ | | | + | | (2) models we tested if there were a diference in their F-score distribution using the student t-test. The null-hypothesis is that there is no diference between two distributions and p-value ≤ 0.05 rejects the null hypothesis and indicates there is a statistical diference between the two distributions.
From Table 4 it is possible to observe that, for the transitive evaluation, all HA-ResNet variations are statistically more semantic adequate than pure ResNets and VGG-16. This is an encouraging evidence that the injection of external knowledge during the training phase may help achieving more interpretable models. We can also see that VGG-16 does not perform better than any other model, which may indicate that the architecture depth can have an efect on semantic adequacy.
Overall, our evaluation shows that the taxonomies extracted from CNNs using our method can
achieve reasonable direct and transitive F-scores, even when the models were not trained explicitly
for that. The best results in our analysis were generated by the hierarchical-aware architectures, as
proposed by [
We proposed a method for semantic interpretability of deep representations by extracting taxonomies from the internal structure of trained CNNs. Our approach represents a CNN as a co-activation graph and adapts the taxonomy extraction method in [12] to such graph. We then introduce the concept of semantic adequacy to measure how well a model captures the hierarchical relationship between classes from a given domain by comparing its extracted taxonomy to a ground truth such as WordNet.
The proposed taxonomy extraction method and semantic adequacy together can help in comparing and choosing among diferent CNNs by exposing how well each model learned the semantic relationships from a given dataset instead of relying purely on performance metrics. The next steps include adapting the semantic adequacy in order to provide more fine-grained information, since the value is currently associated to the extracted taxonomy as a whole. In this case we expect the metric to inform which specific parts of the taxonomy are more or less adequate according to the ground truth. For example, the extracted taxonomy may be more adequate for a specific subtree (e.g. dogs) but less adequate for another (e.g. primates). This information can be useful not only for deciding when to trust a given model but also for transfer learning, where a model may not be suitable for a given task even though it may provide high accuracy.
This publication has emanated from research supported by Science Foundation Ireland (SFI) Grant Number SFI/12/RC/2289_P2, co-funded by the European Regional Development Fund. [12] F. Martel, A. Zouaq, Taxonomy extraction using knowledge graph embeddings and hierarchical clustering, in: Proceedings of the 36th Annual ACM Symposium on Applied Computing, SAC ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 836–844. URL: https://doi.org/10.1145/3412841.3441959. doi:1 0 . 1 1 4 5 / 3 4 1 2 8 4 1 . 3 4 4 1 9 5 9 . [13] Y. Goyal, A. Feder, U. Shalit, B. Kim, Explaining classifiers with causal concept efect (cace), 2019.
URL: https://arxiv.org/abs/1907.07165. doi:1 0 . 4 8 5 5 0 / A R X I V . 1 9 0 7 . 0 7 1 6 5 . [14] C.-K. Yeh, B. Kim, S. O. Arik, C.-L. Li, T. Pfister, P. Ravikumar, On completeness-aware conceptbased explanations in deep neural networks, 2019. URL: https://arxiv.org/abs/1910.07969. doi:1 0 . 4 8 5 5 0 / A R X I V . 1 9 1 0 . 0 7 9 6 9 . [15] P. Ristoski, S. Faralli, S. P. Ponzetto, H. Paulheim, Large-scale taxonomy induction using entity and word embeddings, in: Proceedings of the International Conference on Web Intelligence, WI ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 81–87. URL: https://doi.org/10.1145/3106426.3106465. doi:1 0 . 1 1 4 5 / 3 1 0 6 4 2 6 . 3 1 0 6 4 6 5 . [16] R. L. Grassa, I. Gallo, N. Landro, Learn class hierarchy using convolutional neural networks,
CoRR abs/2005.08622 (2020). URL: https://arxiv.org/abs/2005.08622. a r X i v : 2 0 0 5 . 0 8 6 2 2 . [17] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, Curran Associates Inc., Red Hook, NY, USA, 2012, p. 1097–1105. [18] H. Chen, S. F. Sultan, Y. Tian, M. Chen, S. Skiena, Fast and accurate network embeddings via very sparse random projection, 2019. URL: https://arxiv.org/abs/1908.11512. doi:1 0 . 4 8 5 5 0 / A R X I V . 1 9 0 8 . 1 1 5 1 2 . [19] A. Grover, J. Leskovec, node2vec: Scalable feature learning for networks, 2016. URL: https: //arxiv.org/abs/1607.00653. doi:1 0 . 4 8 5 5 0 / A R X I V . 1 6 0 7 . 0 0 6 5 3 .