Interpretable deep learning aims at developing neural architectures whose decision-making processes could be understood by their users. Among these techniqes, Concept Bottleneck Models enhance the interpretability of neural networks by integrating a layer of human-understandable concepts. These models, however, necessitate training a new model from the beginning, consuming significant resources and failing to utilize already trained large models. To address this issue, we introduce “AnyCBM”, a method that transforms any existing trained model into a Concept Bottleneck Model with minimal impact on computational resources. We provide both theoretical and experimental insights showing the efectiveness of AnyCBMs in terms of classification performances and efectivenss of concept-based interventions on downstream tasks.
Numerous national and international regulatory frameworks underscore the transformative
potential of artificial intelligence (AI). However, they also warn of the inherent risks associated
with such powerful technology, emphasizing the importance of careful monitoring and strict
protections. For instance, the recent AI Act [
Concept Bottleneck Models (CBMs) [
Concept-based models : → learn a map from a concept space to a task space [
AnyCBM (Figure 1) is a method designed to convert any opaque neural network architecture into a Concept Bottleneck Model (CBM) that is interpretable. The fundamental innovation of AnyCBMs involves the use of an external model that processes embeddings from a trained black box model. These embeddings, denoted as ℎ() ∈ () ⊆ R, are encoded into a set of supervised concepts ∈ . Subsequently, these concepts are mapped back into embeddings ℎ() ∈ () ⊆ R. This process allows for the embedding space of the black box model to be translated into a more understandable and interpretable form, where each concept represents a meaningful feature or characteristic that explains the decision-making process of the neural network. The following definition formalizes AnyCBMs.
Definition 3.1 (AnyCBM). Given a black box model : → and a set of concepts , a AnyCBM is a tuple of models ( , ) such that, the following diagram commutes: ()
()
More specifically, the concept predictor : () → encodes black box embeddings into concepts, and the task encoder : → () maps concepts back into black box embeddings. In practice, the commutative diagram describes how the interpretable mapping through via and should be consistent with the direct transformation of the black box . Also properties and capabilities of AnyCBMs can directly be derived from the commutative diagram as it constraints the relationships among the transformations , , and .
In the following we present two practical case studies.
Case 1: is the identity function on When is the identity function, (ℎ()) = ℎ() for all ℎ() ∈ (), and () = (). The diagram simplifies, and we have:
∘ = id Theorem 3.2. If is the identity function on , then is injective: = id =⇒ : ˓→ () (1) Proof. Assume (1) = (2). Since is surjective, there exist ℎ1, ℎ2 ∈ () such that (ℎ1) = 1 and (ℎ2) = 2. Then,
ℎ1 = ( (ℎ1)) = (1) = (2) = ( (ℎ2)) = ℎ2.
Thus, 1 = 2, proving that is injective.
Significance: This property implies that can uniquely reconstruct elements of () from
, despite not being injective. For example, if represents a lossy compression, then
could be an error-correcting decoding where no information is lost despite compression.
Case 2: independent training In many practical cases, concept predictors and task encoders
are independently trained to reduce concept leakage [
Theorem 3.3. If and are independently trained and is a multi-layer neural network, then cannot be surjective.
Proof. Assume for contradiction that is surjective. The surjectivity of would require that every point in () is the image of some point in . Given the independent training, the domain of is finite, specifically 2. Since () ⊆ R, the mapping : → () must pull from a set with finite cardinality 2 to cover R which is a contradiction. Hence, cannot be surjective.
Significance: This theorem indicates that the surjectivity of depends on the way we train the concept bottleneck. This means that, under independent training, AnyCBMs are not invertible, even when represents an invertible transformation.
Our experiments aim to answer the following questions: • How is AnyCBMs classification performance on concepts and downstream tasks compared to standard CBMs and black boxes? • How efective are concept interventions in AnyCBM compared to concept interventions in CBM? • Is it possible to train AnyCBM with a dataset slightly diferent from the one used to train the black-box model? This section describes essential information about the experiments.
In our experiments, we use two diferent datasets commonly used to evaluate CBMs: MNIST
even/odd [
In our analysis, we use ROC-AUC scores to measure classification performance in concepts and
downstream tasks and to measure the efectiveness of concept-based interventions in improving
classification performance in downstream tasks. To measure the efectiveness of interventions,
we follow a similar approach to the one described by Espinosa Zarlenga et al. [
In our experiments, we compare AnyCBMs with standard CBMs and with an end-to-end blackbox model in terms of generalisation performance. We compare AnyCBMs’ interventions with the efectiveness of interventions in standard CBMs.
AnyCBMs match black box and CBM performances in terms of classification accuracy on concepts and downstream tasks (Table 1), AnyCBMs perform just as well as the original black-box models on which they are based when it comes to accurately completing tasks. Additionally, the accuracy with which these models handle concepts is equal to that of other similar Concept Bottleneck Model architectures. This suggests that AnyCBMs could be a valuable tool for making existing black-box models easier to understand. Using AnyCBMs, we might be able to explain how these complex models work and, in particular, which encoded information is inside the layers of the models, making them more transparent and accessible for further analysis and improvement.
AnyCBM interventions are as efective as in Concept Bottleneck Models (Figure 2) AnyCBMs are as responsive to concept-based interventions as standard CBMs. This means that when concepts predicted by AnyCBMs are manually changed by human experts at test time, they efectively impact the downstream task accuracy. This finding underlines the ability of AnyCBMs to interact with domain experts as it would have been expected by CBMs. In
MNIST even/odd Task ROC AUC Concept ROC AUC 99.8 ± 0.0 99.8 ± 0.0 99.8 ± 0.0 99.6 ± 0.0 98.8 ± 0.3 addition, this represents a successful method to steer the behaviour of the model modifying human-understandable concepts.
AnyCBM can be trained with a diferent dataset from the one used to train the blackbox model (Table 2) One can initially train a black box model with a dataset, which could be larger or more beneficial for addressing the downstream task. Subsequently, the AnyCBM module can be trained on a slightly diferent dataset that includes concept annotations. As demonstrated in Table 2, this approach does not compromise the model’s performance in terms of task accuracy when both the black-box model and AnyCBM are utilised in the original dataset. It also partially accurately predicts concepts in the original dataset, even when there is a distribution shift. This indicates that AnyCBM can alleviate a significant constraint of CBMs, which is the requirement for concept annotations in the dataset used to train the entire model. In addition, the dataset used to train the AnyCBM module could contain only input and concept annotations, without the need for label annotations.
Advantages In the age of Large Models with billions of parameters, the development of solutions that do not require retraining to enhance their capabilities is crucial. AnyCBMs successfully meet this need, as they do not require the alteration of the weights of a pre-trained black-box model. This enables any black-box model to acquire the extra features of CBMs, such as the interpretability of the latent space and the capacity to change the model’s behaviour through concept interventions. Furthermore, we believe that AnyCBM can be trained using a dataset that is smaller than the one used to train the original black-box because it has a consistently smaller number of parameters. Interestingly, the dataset can even be distinct (for instance, we might train the model with a dataset without concepts while training AnyCBM with a slightly diferent dataset that has only concept annotations), mitigating the CBMs’ constraint of needing concept annotations for the training set used to train the model. Under these circumstances, it might be intriguing to determine whether certain concepts can be accurately predicted from the latent embeddings of black-box models. If some concepts are unpredictable, this could suggest that the black-box models did not grasp that particular concept in the prior training, either due to the dataset employed or its irrelevant role in task prediction. Limitations Although the model gains the benefits of CBMs, it also takes on some of their drawbacks. The primary constraint is the necessity for concept data to train the AnyCBM component, although this is somewhat alleviated by the reduced need for concept annotations and the option to utilise an alternate dataset for their extraction.
Future work We underscore the importance of delving deeper into AnyCBM and its benefits,
while also trying to mitigate its drawbacks. For example, it would be intriguing to examine its
application in multimodal contexts, where automatic concept extraction could be feasible, as
suggested in [
This paper introduces Any Concept Bottleneck Models (AnyCBMs), a method for transforming opaque neural networks into interpretable Concept Bottleneck Models (CBMs), allowing for insights into the decision-making process in terms of concept-based explanations and interventions. This paper analyses practical case studies that demonstrate the properties and limitations of AnyCBMs in enhancing interpretability while maintaining high classification performances from both a theoretical and an experimental perspective. These results suggest how AnyCBMs could represent a computationally efective solution to enhance the interpretability of existing trained or fine-tuned black-box neural networks, allowing also for concept-based interventions in the black-box latent space. This study was funded by TRUST-ME (project 205121L_214991), SmartCHANGE (GA No. 101080965) and XAI-PAC (PZ00P2_216405) projects.