Deep Neural Networks (DNNs) have achieved remarkable performance on a range of tasks. A key step to further empowering DNN-based approaches is improving their explainability. In this work we present CME: a concept-based model extraction framework, used for analysing DNN models via concept-based extracted models. Using two case studies (dSprites, and Caltech UCSD Birds), we demonstrate how CME can be used to (i) analyse the concept information learned by a DNN model (ii) analyse how a DNN uses this concept information when predicting output labels (iii) identify key concept information that can further improve DNN predictive performance (for one of the case studies, we showed how model accuracy can be improved by over 14%, using only 30% of the available concepts).
(a) (b) extracting concept information. DNNs have been shown to perform hierarchical feature extraction, with layers closer to the output utilising higher-level data representations, compared to layers closer to the input [16, 17]. This implies that choosing a single layer imposes an unnecessary trade-of between low- and high-level concepts. On the other hand, CME is capable of eficiently combining latent space information from multiple layers, thereby avoiding this constraint.
Finally, existing methods typically represent concept explanations as a list of concepts, with their relative importance with respect to the classification task. In contrast, our approach describes the functional relationship between concepts and outputs, thereby showing in more detail how the model utilises concept information when making predictions. cision trees [23, 24], or other more readily explainable models [25] to approximate complex models, in order to study their behaviour. Provided the approximation quality (referred to as fidelity ) is high enough, an extracted model can preserve many statistical properties of the original model, while remaining open to interpretation.
However, extracted models generated by existing methods represent their decision-making using the same input representation as the original model, which is typically dificult for the user to understand directly. via human-understandable concepts, making them easier to interpret.
Instead, our extracted models represent decision-making remainder of this section. ing how it can be used to analyse DNN models using concept-based extracted models.
In this section we present our CME approach, describ- sume partial access to ⋆, such that a small set of output function, mapping data-points in their concept representation to output space . Thus, when processing an input , a DNN
can be seen as converting this input into an interpretable concept representation using
, and using to predict the output from this representation. The significance of this decomposition is further discussed in Appendix A.
decomposable, by attempting to approximate with an extracted model ̂ ∶
→ . In this case, ̂ is defined as ̂ ( ) = ̂ ( ̂ ( )), using input-to-concept ̂ and outputto-concept ̂ extracted by CME from the original DNN.
3.3. Input-to-Concept ( ̂ ) When extracting ̂ from a pre-trained DNN, we assume we have access to the DNN training data and labels {( (0), (0)), ..., ( ( ), ( ))}. Furthermore, we as training points { (0), ..., ( −1)} have concept labels { (0), ..., ( −1)} associated with them, while the remaining points { ( ), ...,
( + )} do not (in this case
= − ). labelled dataset and concept unlabelled dataset. Using 3.1. Formulation We consider a pre-trained DNN classifier ∶ ( ⊂ ℝ , ⊂ ℝ ), where ( ) = is mapping an input ∈ to an output class ∈ . For every DNN layer , we denote the function ∶ → , ( ⊂ ℝ ) as a mapping from the input space to the hidden representation space , where of hidden units, and can be diferent for each layer.
Similarly to [18, 19], we assume the existence of a denotes the number ( ), obtaining ( (0), ..., ( + )). Using this data and the pute the layer’s representation of the input data = concept labels, we construct a semi-supervised dataset, consisting of labelled data {( (0), (0)), ..., ( ( −1) , ( −1))}, → , these datasets, we generate ̂ by aggregating concept label predictions across multiple layers of the given
hidden units, we comcepts associated with the input data. is defined such concept representation ⊂ ℝ , defining distinct con- and unlabelled data { ( ), ..., ( + )}. Next, we rely on Semi-Supervised Multi-Task Learnthat every basis vector in spans the space of possi- ing (SSMTL) [26], in order to extract a function ∶ ble values for one particular concept. We further assume the existence of a function ⋆ ∶ → , where → , which predicts concept labels from layer ’s hidden space. In this work, we treat each concept as ⋆( ) = is mapping an input to its concept represen- a separate, independent task. Hence, ( ) is decom . Thus, ⋆ defines the concepts and their values (referred to as the ground truth concepts) for every input defined as posed into separate tasks (one per concept), and is ( ) = ( 1( ), ..., ( )) where each ( ) position of functions and , such that ( ) = ( ( )).
→ is an inputto-concept function, mapping data-points from their tation ∈ . The function ∶ → is a concept-toinput representation ∈ to their concept represen- computing the predictive loss of function with re
= arg min ( , ) ∈ tation point. 3.2. CME
decomposable, if it can be well-approximated by a com- for predicting that concept as shown in (1): obtain a set of functions ( ∈ {1.. }
) predicts the value of concept from Repeating this process for all model layers
. = {
| ∈ {1.. } {1.. }}. For every concept , we define the “best” layer , we ∧ ∈ (1) (2): spect to a concept . Finally, we define ̂ as shown in ̂ ( ) = ( 1 ◦ 1 1 ( ), ..., ◦ ( ))
(2)
for every concept ∈ {1.. } is equal to the value computed by from that input’s representation in layer . Overall, ̂ encapsulates concept information contained in a given DNN model, and can be used to analyse how this information is represented, as well as to predict concept values for new inputs. • Task 1: This task consists of determining the shape concept value from an input image. For every image sample, we define its task label as the shape concept label of that sample. • Task 2: This task consists of discriminating between all possible shape and scale concept value combinations. We assign a distinct identifier to each possible combination of the shape and scale concept labels. For every image sample, we deifne its task label as the identifier corresponding to this sample’s shape and scale concept values.
has to learn to recognise a specific concept from an input image. Task 2 explores a relatively more complex scenario, in which a DNN has to learn to recognise combinations of concepts from an input image. 3.4. Concept-to-Label ( ̂ ) We setup extraction of ̂ as a classification problem, in which we train ̂ to predict output labels from concept labels predicted by ̂ . We use ̂ to generate concept labels for all training data points, obtaining a set of con- 4.1.2. Model cept labels { (0), ...,
( + )}. Next, we produce a labelled dataset, consisting of concept labels and corresponding
use it to train ̂ in a supervised mented with using Decision Trees (DTs), and Logistic Regression (LR) models for representing ̂ , as will be discussed in Section 5. Overall, ̂ can be used to analyse how a DNN uses concept information when making , (0)), ..., ( ( + ) , ( + ))}, and predictions.
white shape images, procedurally generated from all possible combinations of 6 ground truth independent oficial dSprites repository. 4
4.1.1. Classification Tasks We define framework:
3https://github.com/dmitrykazhdan/CME 4https://github.com/deepmind/dsprites-dataset/ licly available at3. 4.1. dSprites We evaluated CME using two datasets: dSprites [27], in this section imply that we know precisely which and Caltech-UCSD birds [28]. All relevant code is pub- concepts the models had to learn, in order to achieve dSprites is a well-established dataset used for evaluating unsupervised latent factor disentanglement approaches. dSprites consists of 2D 64×64 pixel black-and- 4.2. Caltech-UCSD Birds (CUB) concepts (color, shape, scale, rotation, x and y position). ages of 200 bird species with every image annotated manner. We experi- layers with ReLUs, 50% dropout [30] and a softmax out
model [29] for each task. Both models had the same architecture, consisting of 3 convolutional layers, 2 dense put layer. The models were trained using categorical cross-entropy loss, and achieved 100.0 ± 0.0% classification accuracies on their respective held-out test sets.
4.1.3. Ground-truth Concept Information
100.0 ± 0.0% task performances (shape for Task 1, and shape and scale for Task 2). We refer to this as the ground truth concept information learned by these models.
For our second dataset, we used Caltech-UCSD Birds 200 2011 (CUB). This dataset consists of 11,788 imlabels for every image sample. using 312 binary concept labels (e.g. beak and wing colour, shape, and pattern). We relied on concept preprocessing steps defined in [ 18] (used for de-noising concept annotations, and filtering out outlier concepts), which produces a refined set of = 112 binary concept 4.3.2. CBM We relied on the standard CUB classification task, which As discussed in Section 4.2.3, we do not have access to consists of predicting the bird species from an input ground truth concept information learned by the CUB image. model. Instead, we rely on the pre-trained sequential bottleneck model defined in [ 18] (referred to as CBM 4.2.2. Model in the rest of this work). CBM is a bottleneck model, obtained by resizing one of the layers of the CUB model We used the Inception-v3 architecture [31], pretrained to match the number of concepts provided (we refer on ImageNet [32] (except for the fully-connected lay- to this as the bottleneck layer), and training the model ers) and fine-tuned end-to-end on the CUB dataset, in two steps. First, the sub-model consisting of the following the preprocessing practices described in [33]. layers between the input layer and the bottleneck layer The model achieved 82.7 ± 0.4% classification accuracy (inclusive) is trained to predict concept values from on a held-out test set. We refer to this model as the input data. Next, the submodel consisting of the layCUB model in the rest of this work. ers between the layer following the bottleneck layer and the output layer is trained to predict task labels 4.2.3. Ground-truth Concept Information from the concept values predicted by the first submodel. Unlike dSprites, the CUB dataset does not explicitly Hence, this bottleneck model is guaranteed to solely define how the available concepts relate to the output rely on concept information that is learnable from the task. Thus, we do not have access to the ground truth data, when making task label predictions. Thus, this concept information learned by the CUB model. benchmark serves as an upper bound for the concept information learnable from the dataset, and for the task performance achievable using this information. Impor4.3. Benchmarks tantly, CBM does not attempt to approximate/analyse We compare performance of our CME approach to two the CUB model, but instead attempts to solve the same other benchmarks, described in the remainder of this classification task using concept information only. section. We use the first CBM submodel as a ̂ benchmark, representing the upper bound of concept information 4.3.1. Net2Vec learnable from the data. We use the second submodel as a ̂ benchmark, representing the upper bound of We rely on work in [34] for defining benchmark ̂ func- task performance achievable from predicted concept tions for the three tasks. Work in [34] attempts to information only. Finally, we use the entire model as predict presence/absence of concepts from spatially- an ̂ benchmark. We make use of the saved trained averaged hidden layer activations of convolutional lay- model from [18], available in their oficial repository 5. ers of a CNN model. Given a binary concept , this approach trains a logistic regressor, predicting the presence/absence of this concept in an input image from 5. Results the latent representation of a given CNN layer. In case of multi-valued concepts, the concept space has to be We present the results obtained by evaluating our apbinarised, as discussed in Section 2.2. In this case, the proach using the two case studies described above. binarised concept value with the highest likelihood is We obtain the concept labelled dataset by returning returned. the ground-truth concept values for a random set of
Unlike CME, [34] does not provide a way of selecting samples in the model training data. For dSprites, we the convolutional layer to use for concept extraction. found that a concept labelled dataset of a 100 samples or We consider the best-case scenario by selecting, for all more worked well in practice for both tasks. Thus, we tasks, the convolutional layers yielding the best concept fix the size of the concept labelled dataset to 100 in all extraction performance. For all tasks, these layers were of the dSprites experiments. For CUB, we found that a convolutional layers closest to the output (the 3rd conv. concept labelled dataset containing 15 or more samples layer in case of dSprites tasks, and the final inception per class worked well in practice. Thus, we fix the size block output layer in case of the CUB task). of the concept labelled dataset to 15 samples per class in all of the CUB experiments. In the future, we intend to explore the variation of model extraction performance (a) Task 1
(b) Task 2 tions for all concepts with the size of the concept labelled dataset in more detail. 5.1. Concept Prediction Performance First, we evaluate the quality of ̂ functions produced by CME, Net2Vec, and CBM. For both dSprites tasks, we relied on the Label Spreading semi-supervised model [35], provided in scikit-learn [36], when learning the functions for CME. For CUB, we used logistic regression functions instead, as they gave better performance. 5.1.1. dSprites aged over 5 runs). As discussed in Section 4.1.1, we have access to the ground truth concept information learned by these models (shape concept information for Task 1, and shape and scale concept information for Task 2). For both tasks, ̂ functions extracted by
concepts relevant to the tasks, whilst achieving a low performance on concepts irrelevant to the tasks. Thus,
mation contained in the task models. For both tasks, ̂ functions extracted by Net2Vec achieved a much lower performance on the relevant concepts. 5.1.2. CUB
not explicitly define how the concepts relate to the output task labels. Thus, we do not know how relevant/important diferent concepts are, with respect to task label prediction. In this section, we make the conservative assumption that all concepts are relevant, Figure 2 shows predictive performance of the ̂ func- tions on all concepts for the two dSprites tasks (aver- put samples with corresponding concept labels , and cept importance in more detail in Section 5.3. when evaluating ̂ functions, and explore relative con- can be evaluated by observing their diferent values of . A larger
scores for score implies a
Firstly, we relied on the ‘average-per-concept’ met- bigger proportion of samples had at least relevant rics introduced in [18] when evaluating the ̂ function concept predicted incorrectly.
( , ̂ ) = ∑ ( , ≠ ̂ , )
(4)
Using a held-out test set, we plot the ues for
∈ [0, ..., 112], as shown in Figure 3 (averaged over 5 runs). Importantly, ̂ function performances
metric valperformances, by computing their 1 predictive scores for each concept, and then averaging over all concepts. respectively (averaged over 5 runs).
We obtained 1 scores of 92 ± 0.5%, 86.3 ± 2.0%, and 85.9 ± 2.3% for CBM, CME, and Net2Vec ̂ functions,
ber of concepts, it is crucial to measure how concept mispredictions are distributed accross the test samples. For instance, consider a dSprites Task 2 ̂ function that achieves 90% predictive accuracy on both shape and scale concepts. The average predictive accuracy on relevant concepts achieved by this ̂ will therefore be 90%. However, if the two concepts are mis-predicted for strictly diferent samples (i.e. none of the samples have both shape and scale predicted incorrectly at the same time), this means that 20% of the test samples will have one relevant concept predicted incorrectly.
when using them for task label prediction, this implies that consequent task label prediction will not be able to achieve over 80% task label accuracy. This efect becomes even more pronounced in case of a larger number of relevant concepts.
Consequently, we defined a novel cumulative misprediction error metric, which we refer to as the ‘misprediction-overlap’ (MPO) metric. Given a test set = {( (0), (0) ), ..., ( ( )
, ( ))} consisting of + 1 ina prediction set = {( ̂ (0) ), ..., ̂ ( ) , } fraction of samples in the test set, that have at least relevant concepts predicted incorrectly, as shown in Equation 3 (where (.) denotes the indicator function): computes the ( , , ) = ( , ̂ ) >= ) (3) Here,
can be used to specify which concepts to measure the mis-prediction error on (i.e. in case some of the provided concepts are irrelevant). Under our assumption of all concepts being relevant, we defined as shown in Equation 4: 1 =0 ∑ ( =0 ground truth concepts (i.e. the ⋆ performance), in which case = 1 for = 0, and = 0 otherwise. Net2Vec simplicity obtained values within 1% deviation from the corresponding CME values for all , and is therefore omitted here for
and worse than according to the
metric. (ii) the Inception-v3 model having a relatively large number of convolutional layers, implying that the final relevant to concept prediction.
Importantly,
showed that both CBM and CME ̂ functions had a significant proportion of test samples with incorrectly-predicted relevant concepts (e.g. CME had an MPO score of 0.25 at
= 4, implying that 25% of all test samples have at least 4 concepts predicted incorrectly). In practice, these mispredictions can have a significant impact on consequent task label predictive performance, as will be further explored in the next section. 5.2. Task Performance In this section, we evaluate the fidelity and performance of the extracted ̂ models. For all CME and Net2Vec ̂ functions evaluated in the previous section, we trained output-to-concept functions ̂ , predicting class labels from the ̂ concept predictions. Next, for every ̂ , we defined its corresponding ̂ as discussed in Section 3, ̂ , we evaluated its fidelity and its task performance, using a held-out sample test set. Table 1 shows the ifdelity of extracted models, and Table 2 shows the task via a composition of ̂ and its associated ̂ . For every In the previous section, we demonstrated how CME can performance for these models (averaged over 5 runs).
respectively, as described in Section 4. task performances of 100±0%, 100±0%, and 82.7±0.4%,
ing that CME successfully approximated the original dSprites models. Furthermore, these scores were con ̂ models.
For the CUB task, both CME and Net2Vec ̂ models achieved relatively lower fidelity and task performance scores (in this case, performance of CME was very similar to that of Net2Vec). Crucially, the CBM model also achieved relatively low fidelity and accuracy scores (as anticipated from our implies that concept information learnable from the data is insuficient for achieving high task accuracy.
Hence the relatively high CUB model accuracy has to be caused by the CUB model relying on other non-concept information. Thus, the low fidelity of CME and Net2Vec is a consequence of the CUB model being non-conceptdecomposable, implying that it’s behaviour cannot be explained by the desired concepts. The next section discusses possible approaches to fixing this issue.
metric analysis). This 5.3. Intervening be used to identify whether a model relies on desired concepts during decision-making. In this section, we demonstrate how CME can be used to suggest model (i) concepts being binary (requiring no binarisation) (99%+) fidelity and task performance scores, indicatconvolutional layer likely learned higher-level features, siderably higher than those produced by the Net2Vec Manifold Types Using ground-truth concept information and hidden space visualisation, it is possible to inspect the nature of latent space manifolds, with respect to specific concepts. Firstly, this inspection allows to build an intuition of how concept information is represented in a particular latent space. Secondly, it is possible to use this information when selecting the types of ̂ functions to use during concept extraction.
For instance, some manifolds consist of “blobs” encoding distinct concept values (e.g. row shape, columns dense, dense_1), suggesting that the latent space is clustered with respect to a concept’s values.
Variation Across Layers cept information and hidden space visualisation, it is also possible to inspect how concept information representation varies across layers of a DNN model. behaviour is consistent with user expectations (model Firstly, this inspection allows to build an intuition of verification ), (ii) identifying specific concepts or conhow concept-related information is transformed by the cept interactions (if any) causing incorrect behaviour DNN. Secondly, it is possible to use this information to (model debugging), (iii) extracting new knowledge about identify the ‘best’ layers to extract concept information how concept information can be used for solving a parfrom. For instance, both rows shape and scale illus- ticular task (knowledge extraction). Further examples trate that the manifolds of higher layers become more and analysis of extracted ̂ functions can be found in unimodal (separating concept values) with respect to Appendix D. the relevant concepts. Importantly, this analysis, together with the definition of ̂ allows using diferent layers for extracting diferent concepts. 6. Conclusions
Overall, we argue that CME concept extraction can be well-integrated with existing latent space analysis We present CME: a concept-based model extraction approaches, in order to study which concept informa- framework, used for analysing DNN models via concepttion is learned by a DNN, and how this information is based extracted models. Using two case-studies, we represented across DNN layers. This type of inspec- demonstrate how CME can be used to (i) analyse contion can have numerous applications, including: (i) cept information learned by DNN models (ii) analyse inspecting which concepts a model has learned, and how DNNs use concept information when making preverifying whether it has learned the desired concepts dictions (iii) identifying key concept information that (useful for model explanations and model verification ), can further improve DNN predictive performance. CME (ii) inspecting how concept information is represented is a model-agnostic, general-purpose framework, which across diferent layers (useful for fine-grained model can be combined with a wide variety of diferent DNN analysis), (iii) extracting concept predictions from a models and corresponding tasks.
DNN (useful for knowledge extraction). Further exam- In this work, we assume a fixed set of concept laples and analysis of extracted ̂ functions can be found bels available to CME before model extraction begins in Appendix C. (i.e. the concept-labelled dataset). In the future, we intend to explore active-learning based approaches to obtaining maximally-informative concept labels in an 5.4.2. Concept-to-Output ( ̂ ) interactive fashion. Consequently, these approaches ̂ functions encapsulate how a DNN uses concept infor- will improve extracted model fidelity by retrieving the mation when making predictions. Hence, these func- most informative concept labels, and reduce manual tions can be inspected directly, in order to analyse concept labelling efort. model behaviour represented in terms of concepts. An Given the rapidly-increasing interest in concept-based example is given in Figure 6, in which we plot the deci- explanations of DNN models, we believe our approach sion tree ̂ function extracted by CME from the Task 1 can play an important role in providing granular conceptmodel. Further examples are given in Appendix D. based analyses of DNN models.
Overall, inspection of ̂ functions can be used for (i) verifying that a DNN uses concept information correctly during decision-making, and that it’s high-level
man, J. W. Vaughan, H. Wallach, Manipulating and measuring model interpretability, arXiv AW acknowledges support from the David MacKay preprint arXiv:1802.07810 (2018). Newton research fellowship at Darwin College, The [10] B. Kim, M. Wattenberg, J. Gilmer, C. J. Cai, Alan Turing Institute under EPSRC grant EP/N510129/1 J. Wexler, F. B. Viégas, R. Sayres, Interpretabil& TU/B/000074, and the Leverhulme Trust via the Lev- ity beyond feature attribution: Quantitative testerhulme Centre for the Future of Intelligence (CFI). BD ing with concept activation vectors (TCAV), in: acknowledges support from EPSRC Award #1778323. J. G. Dy, A. Krause (Eds.), Proceedings of the DK acknowledges support from EPSRC ICASE scholar- 35th International Conference on Machine Learnship and GSK. DK and BD acknowledge the experience ing, ICML 2018, Stockholmsmässan, Stockholm, at Tenyks as fundamental to developing this research Sweden, July 10-15, 2018, volume 80 of Proceedidea. ings of Machine Learning Research, PMLR, 2018, pp. 2673–2682. URL: http://proceedings.mlr.press/ References v80/kim18d.html.
[11] B. Zhou, Y. Sun, D. Bau, A. Torralba, Interpretable [1] B. Goodman, S. Flaxman, European union regula- basis decomposition for visual explanation, in: tions on algorithmic decision-making and a “right Proceedings of the European Conference on Comto explanation”, AI magazine 38 (2017) 50–57. puter Vision (ECCV), 2018, pp. 119–134. [2] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Ben- [12] A. Ghorbani, J. Wexler, J. Y. Zou, B. Kim, Towards netot, S. Tabik, A. Barbado, S. García, S. Gil-López, automatic concept-based explanations, in: AdD. Molina, R. Benjamins, et al., Explainable ar- vances in Neural Information Processing Systems, tificial intelligence (xai): Concepts, taxonomies, 2019. opportunities and challenges toward responsible [13] C.-K. Yeh, B. Kim, S. O. Arik, C.-L. Li, P. Ravikumar, ai, Information Fusion 58 (2020). T. Pfister, On concept-based explanations in deep [3] A. Adadi, M. Berrada, Peeking inside the black- neural networks, arXiv preprint arXiv:1910.07969 box: A survey on explainable artificial intelligence (2019).
(xai), IEEE Access 6 (2018). [14] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler,
[
Accountability, and Transparency, 2020, pp. 648– [15] Y. Goyal, U. Shalit, B. Kim, Explaining classifiers 657. with causal concept efect (cace), arXiv preprint [5] P.-J. Kindermans, S. Hooker, J. Adebayo, M. Alber, arXiv:1907.07165 (2019).
K. T. Schütt, S. Dähne, D. Erhan, B. Kim, The (un) [16] G. E. Hinton, Learning multiple layers of reprereliability of saliency methods, in: Explainable sentation, Trends in cognitive sciences 11 (2007) AI: Interpreting, Explaining and Visualizing Deep 428–434.
Learning, Springer, 2019, pp. 267–280. [17] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Tor[6] D. A. Melis, T. Jaakkola, Towards robust inter- ralba, Object detectors emerge in deep scene cnns, pretability with self-explaining neural networks, arXiv preprint arXiv:1412.6856 (2014). in: Advances in Neural Information Processing [18] P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, Systems, 2018, pp. 7775–7784. E. Pierson, B. Kim, P. Liang, Concept bottleneck [7] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, models, in: Proceedings of Machine Learning M. Hardt, B. Kim, Sanity checks for saliency maps, and Systems 2020, International Conference on in: Advances in Neural Information Processing Machine Learning, 2020, pp. 11313–11323.
Systems, 2018, pp. 9505–9515. [19] F. D.-V. Isaac Lage, Human-in-the-loop learning [8] B. Dimanov, U. Bhatt, M. Jamnik, A. Weller, You of interpretable and intuitive representations, in: shouldn’t trust me: Learning models which con- ICML Workshop on Human Interpretability, 2020. ceal unfairness from multiple explanation meth- URL: http://whi2020.online/static/pdfs/paper_31. ods, in: European Conference on Artificial Intelli- pdf .
gence, 2020. [20] R. Andrews, J. Diederich, A. B. Tickle, Survey and [9] F. Poursabzi-Sangdeh, D. G. Goldstein, J. M. Hof- critique of techniques for extracting rules from trained artificial neural networks, Knowledgebased systems 8 (1995) 373–389. ceedings of the IEEE conference on computer vi[21] J. R. Zilke, E. L. Mencía, F. Janssen, Deepred– sion and pattern recognition, 2018, pp. 4109–4118. rule extraction from deep neural networks, in: [34] R. Fong, A. Vedaldi, Net2vec: Quantifying and International Conference on Discovery Science, explaining how concepts are encoded by filters Springer, 2016, pp. 457–473. in deep neural networks, in: Proceedings of the [22] D. Chen, S. P. Fraiberger, R. Moakler, F. Provost, IEEE conference on computer vision and pattern Enhancing transparency and control when draw- recognition, 2018, pp. 8730–8738. ing data-driven inferences about individuals, Big [35] D. Zhou, O. Bousquet, T. N. Lal, J. Weston, data 5 (2017) 197–212. B. Schölkopf, Learning with local and global con[23] R. Krishnan, G. Sivakumar, P. Bhattacharya, Ex- sistency, in: Advances in Neural Information tracting decision trees from trained neural net- Processing Systems 16, 2004.
works, Pattern recognition 32 (1999). [36] F. Pedregosa, G. Varoquaux, A. Gramfort, [24] M. Sato, H. Tsukimoto, Rule extraction from V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretneural networks via decision tree induction, in: tenhofer, R. Weiss, V. Dubourg, J. Vanderplas, IJCNN’01. International Joint Conference on Neu- A. Passos, D. Cournapeau, M. Brucher, M. Perral Networks. Proceedings (Cat. No. 01CH37222), rot, E. Duchesnay, Scikit-learn: Machine learning volume 3, IEEE, 2001, pp. 1870–1875. in Python, Journal of Machine Learning Research [25] D. Kazhdan, Z. Shams, P. Liò, Marleme: A multi- 12 (2011).
agent reinforcement learning model extraction [37] L. v. d. Maaten, G. Hinton, Visualizing data using library, arXiv preprint arXiv:2004.07928 (2020). t-sne, Journal of Machine Learning Research 9 [26] Q. Liu, X. Liao, L. Carin, Semi-supervised multi- (2008) 2579–2605.
task learning, in: Advances in Neural Information
Processing Systems, 2008. [27] L. Matthey, I. Higgins, D. Hassabis, A. Lerch- A. Concept Decomposition ner, dsprites: Disentanglement testing sprites dataset, https://github.com/deepmind/dsprites- The results and findings presented in existing work on dataset/, 2017. concept-based explanations suggests that users often [28] C. Wah, S. Branson, P. Welinder, P. Perona, S. Be- think of tasks in terms of concepts and concept interaclongie, The caltech-ucsd birds-200-2011 dataset tions (see Section 2.1 for further details). For instance, (2011). consider the task of determining the species of a bird [29] Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, from an image. A user will typically perform this task R. E. Howard, W. E. Hubbard, L. D. Jackel, Hand- by first identifying relevant concepts (e.g. wing color, written digit recognition with a back-propagation head color, and beak length) present in a given image, network, in: Advances in neural information pro- and then using the values of these concepts to infer the cessing systems, 1990, pp. 396–404. bird species, in a bottom-up fashion. [30] N. Srivastava, G. Hinton, A. Krizhevsky, On the other hand, Machine Learning (ML) modI. Sutskever, R. Salakhutdinov, Dropout: els usually rely on high-dimensional data representaA simple way to prevent neural networks tions, and infer task labels directly from these highfrom overfitting, Journal of Machine Learn- dimensional inputs (e.g. a CNN produces a class label ing Research 15 (2014) 1929–1958. URL: from raw input pixels of an image).
http://jmlr.org/papers/v15/srivastava14a.html. Consequently, Concept Decomposition (CD) approaches [31] C. Szegedy, V. Vanhoucke, S. Iofe, J. Shlens, Z. Wo- attempt to explain the behaviour of such ML models by jna, Rethinking the inception architecture for decomposing their processing into two distinct steps: computer vision, in: Proceedings of the IEEE concept extraction, and label prediction. In concept conference on computer vision and pattern recog- extraction, concept information is extracted from the nition, 2016, pp. 2818–2826. high-dimensional input data. In label prediction, con[32] A. Krizhevsky, I. Sutskever, G. E. Hinton, Ima- cept information is used to produce the output label. genet classification with deep convolutional neu- Hence, CD approaches attempt to explain ML model ral networks, in: Advances in neural information behaviour in terms of human-understandable concepts processing systems, 2012, pp. 1097–1105. and their interactions in a bottom-up fashion, parallel[33] Y. Cui, Y. Song, C. Sun, A. Howard, S. Be- ing human-like reasoning more closely. longie, Large scale fine-grained categorization Importantly, whilst this work focuses on CNN modand domain-specific transfer learning, in: Pro- els and tasks, the notion of CD can in principle be applied to any ML model and task. concept-related knowledge stored in these models. Consequently, we believe that CME will be invaluable in A.1. CBMs situations where concept-related information is expensive/dificult to obtain, or is only partially-known. In CBMs can be seen as a special case of models perform- these cases, a user may interact with existing DNN moding CD, in which CD behaviour is enforced by design. els via CME, in order to refine existing concept-related Hence, these models explicitly consist of two submod- knowledge. els, with the first submodel extracting concept infor- It should be noted that a CBM can trivially be apmation, and the second submodel using this concept proximated using CME, by defining ̂ as the output of information for producing task labels. Importantly, a CBM’s concept bottleneck layer, and defining ̂ as non-CBM models can still demonstrate CD behaviour. the CBM’s submodel producing task labels from the For instance, the dSprites Task 2 model was shown to bottleneck layer output. have CD behaviour, with relevant concept information extracted in the dense layers, and used for classification A.3. Further Discussion decisions.
A.2. CBMs & CME DNN is concept-decomposable, by attempting to approximate it with an extracted model that is conceptThe utility of CBMs is that they produce models explic- decomposable by design (i.e. explicitly consists of two itly encouraged to use CD. Consequently, these models separate stages). Intuitively, if a given DNN learns and are much more likely to rely on the desired concepts relies on concept information of the specified concepts during decision-making, and be more aligned with a during label prediction, this concept information will user’s mental model of the corresponding task. be contained in the DNN latent space. Hence, the DNN
However, a given DNN model can already exhibit decision process could be separated into two steps: conCD behaviour, and use the desired concept information cept information extraction, and consequent task label (e.g. as was the case with both dSprites task models). prediction.
In this case, costly modifications and model re-training Importantly, existing CD-based approaches (such as are unnecessary. As discussed in Section 3, CME can those discussed in Section 2.2) require the set of conextract concept information from pre-trained DNNs by cepts and their values to be (i) suficient to solve the training ∗ concept predictors (where denotes the corresponding classification task (i.e. the class labels number of DNN layers used in concept extraction, and can be predicted from concept information with high denotes the number of concepts). As demonstrated accuracy) (ii) learnable from the data (i.e. the DNN in Section 5, these concept predictors can consist of model will be able to learn concept information from simpler models (e.g. LRs), trained on only a fraction the given dataset), in order to achieve high task perforof the DNN training data. Thus, the computational mance. cost of training these concept predictors is significantly However, these works do not discuss how to handle smaller, compared to training a bottleneck model on cases where these assumptions do not hold (e.g. as was all the training data, as done in the case of CBMs. the case with the CUB task). Thus, exploring ways
More importantly, CBM models require knowledge of eficiently discovering relevant concepts suficient of existing concepts and available concept annotations. for solving a given task, as well as ways of ensuring In practice, these annotations are often expensive to whether this concept information is learnable from the produce, especially for large datasets and/or a large data are both important research directions for future number of concepts. Furthermore, information about work. which concepts are relevant and/or suficient for solving a given task is often not fully available either. Instead, CME is capable of using existing DNN models B. dSprites Dataset to extract this information automatically in a semisupervised fashion, making concept discovery (identi- B.1. Description fying the relevant concepts), and concept annotation both faster and cheaper.
Overall, CME permits eficient interaction with pretrained DNN models, which can be used to leverage dSprites is a dataset of 2D shapes, procedurally generated from 6 ground truth independent concepts (color, shape, scale, rotation, x and y position). Table 3 lists the concepts, and corresponding values. dSprites consists of 64 × 64 pixel black-and-white images, generated from in [18]. Further details regarding layer naming and/or all possible combinations of these concepts, for a total concept naming can be found in 6. For all concepts, conof 1 × 3 × 6 × 40 × 32 × 32 = 737280 total images. cept values become significantly better-separated after the Mixed_7c layer. However, the figure shows that Table 3 concept values are still quite mixed together for some dSprites concepts and values of the points, even for later layers. This low separability indicates that concept values will still be mis-predicted for some of the points, and that concept extraction for the CUB task will likely perform suboptimally.
Name Color Shape
Scale Rotation Position X Position Y
Values white square, ellipse, heart 6 values linearly spaced in [0.5, 1] 40 values in [0, 2 ] 32 values in [0, 1] 32 values in [0, 1] B.2. Pre-processing
tion Y (keeping every other value only), and select 8 of the 40 values for Rotation (retaining every 5th value). This step makes the dataset size more manageable (reducing it from 737280 to 3 ∗ 6 ∗ 8 ∗ 16 ∗ 16 = 36864 samples), whilst preserving its characteristics and properties, such as concept value ranges and diversity.
The CUB model has a considerably larger number of layers, and a considerably larger number of task concepts. Hence, for the sake of space, we demonstrate an example here using only 6 diferent model layers of the CUB model, and showing only the top 5 important concepts identified in Section 5.3. In this Figure, the concepts are named using their indices, and the layers are named following the naming convention used
Figure 9 shows the decision tree extracted for dSprites Task 2. Overall, this model has correctly learned to diferentiate between classes based on the shape and scale concepts (note: there are 3 × 6 shape and scale concept values, for a total of 18 output classes). 6https://github.com/yewsiang/ConceptBottleneck/tree/master/CUB