In the field of explainable AI, a vibrant efort is dedicated to the design of self-explainable models, as a more principled alternative to post-hoc methods that attempt to explain the decisions after a model opaquely makes them. However, this productive line of research sufers from common downsides: lack of reproducibility, unfeasible comparison, diverging standards. In this paper, we propose CaBRNet, an open-source, modular, backward-compatible framework for Case-Based Reasoning Networks: https://github.com/aiser-team/cabrnet.
architectures. Such models are based on the principle of case-based reasoning (CBR), in which
new instances of a problem (the classification task) are solved using comparisons with an
existing body of knowledge that takes the form of a database of prototypical representations of
class instances. In other words, the classification of an object is explained by the fact that it
looks like [
In this context, we propose CaBRNet (Case-Based Reasoning Networks), an open-source PyTorch library that proposes a generic approach to CBR models. CaBRNet focuses on modularity, backward compatibility, and reproducibility, allowing us to easily implement existing works from the state of the art, propose new and innovative architectures, and compare them within a unified framework using multiple dedicated evaluation metrics. 2. CaBRNet: One framework to rule them all...
In a field of research as wide as XAI, where the proliferation of tools and methods is increasing, seeking a common framework is a natural temptation that often leads to yet another tool that evolves besides the rest. To minimize this risk, CaBRNet is designed with three main objectives, essential for a lasting acceptance: supporting past state-of-the-art methods through backward compatibility, facilitating present developments by striving for modularity, and ensuring their reusability in future works through reproducibility.
While each model has its own specific properties ( e.g., decision tree in ProtoTree, scoring sheet in
PIP-Net, linear layer in ProtoPNet), CBR image classifiers from the state of the art [
More precisely, as shown in Figure 1 and Table 1, the architecture of a CBR image classifier starts with a backbone – usually the feature extractor of an existing convolutional neural network (CNN) such as ResNet50 – with optional additional layers to reduce dimensionality (e.g., the fourth layer of a ResNet50 followed by a convolutional layer with sigmoid activation).
The feature extractor outputs a set of vectors, that form the latent space, and is followed by a similarity layer whose role is to compute the similarity between the latent vectors of the input image and a series of reference vectors, called prototypes, each obtained from an image from the training set. From the similarity scores, the process of locating the prototypes inside images (attribution) and visualizing the relevant pixels (e.g., bounding box, cropping, heatmaps) is also parametrized by a configuration file, as described in Table 1. Finally, a decision <model_arch.yml> top_arch:
name: ProtoTree extractor: backbone: arch: resnet50 layer: layer4 weights: IMAGENET1K_V1 add_on: conv1: type: Conv2d params: koeurtn_cehl_asniznee:ls1: 256 bias: False sigmoid1:
type: Sigmoid classifier: name: ProtoTreeClassifier params: num_classes: 200 depth: 9 ...
resnet50 (torchvision.models)
lryae1 lryae2 lryae3 lryae4 l_vooagp lfrscyae create_feature_extractor (torchvision.models.feature_extraction) backbone add_on
layers lryae1 lryae2 lryae3 lryae4 cvon1 iisodg1m class ConvExtractor class Prototree class PrototreeClassifiersimilarities similarity layer prototypes decision tree decision layer, parametrized by an architecture-dependent python class (e.g., decision tree, linear layer), assigns a score to each class based on the similarity scores, the highest being the decision (object classification).
In addition to these architectural choices, CaBRNet also allows specifying the parameters necessary for the training of the models (e.g., objective functions, number of epochs, optimizer to use, which parts of the neural network can be updated and when, training dataset, preprocessing). For more information regarding the training and data configuration, we refer the reader to the CaBRNet documentation.
In its current version (v0.2), and as shown in Table 1, CaBRNet implements two architectures
(ProtoPNet and ProtoTree) and five attribution methods:
• Upsampling of the similarity map with cubic interpolation, as in [
In the coming months, we plan to add support for more architectures (see Sec. 4).
Significant work has already been carried out on CBR classifiers. Therefore, ensuring that any result obtained with previously existing codebases (e.g., model training) can be reused in the CaBRNet framework is paramount and has been one of our main and earliest priorities, in an efort to ensure backward compatibility.
First, any previously trained model can be loaded and imported within the CaBRNet framework, i.e., models trained using the original code proposed by the authors of ProtoPNet and ProtoTree
Parameter extractor/backbone/arch extractor/backbone/layer extractor/add_on classifier attribution/type view/type
can be used by our framework for other purposes (e.g., benchmarks) without having to retrain the model from scratch.
Second, our implementation of existing architectures (currently, ProtoPNet and ProtoTree) supports two modes: i) default mode, where all operations have been reimplemented so that they are as up-to-date as possible with the latest PyTorch versions; ii) compatibility mode, as a sanity check (i.e., to make sure that our implementation does not deviate from previous implementations), whose purpose is to be accurate with previous works at the operation level1.
Reproducibility is key for the transparency of research and good software engineering alike. However, replicating machine learning (ML) results can be particularly tricky2. One reason for this is the inherent randomness at the core of ML programs (and the lack of documentation of settings related to that randomness). The rapid update pace of common ML frameworks 1Backward compatibility was rigorously tested using unit tests covering all aspects of the process, from data loading to model training, to pruning and prototype projection. 2See https://reproducible.cs.princeton.edu/ leads to the regular deprecation of APIs and features, which may make a code impossible to run without directly changing its source, or going through the rabbit hole of dependency hell. As an example, the mixed-precision training setting, available from PyTorch v1.10, uses diferent types for CPU and GPU computations, which may lead to diferent results. One last part is the under-specification of the dataset constitution (curation process, classes imbalance, etc.) and preprocessing pipelines (test/train splits, data transformations).
Reproducibility can cover many notions. In [
To address the randomness variation, we initialize the various random number generators (RNGs) using a fixed seed that is stored along the training parameters. Thus, while loading a model’s parameters, the seed used for its training will be loaded as well, guaranteeing that given the same hardware and hyperparameter configuration, the variability induced by the pseudo-randomness is identical. We also support the possibility to save checkpoints at various steps of the process, and we include the current state of all RNGs to restore the training process exactly as it was.
As we believe that reproducibility is improved by a good documentation, we also provide detailed installation instructions, a tutorial, the API reference and the CaBRNet backend reference. The major part of this documentation is automatically generated from the source code, ensuring consistency between the code and the documentation at all times.
In this section, we describe our design choices for the CaBRNet benchmark, with the purpose of proposing a framework for the systematic evaluation of CBR models.
Current CBR models rely on two assumptions: i) proximity in the latent space (w.r.t. a given
distance metric) is equivalent to similarity in the visual space; ii) there exists a simple mapping
between a latent vector and a localized region in the original image, due to the architecture of
the feature extractor (CNN). These assumptions have recently been put to the test by various
metrics4, that are already implemented or that we aim to implement in CaBRNet to streamline
the evaluation of CBR models. In particular, CaBRNet currently integrates:
• the perturbation-based explanation of [
In the coming months, we plan to add support for more evaluation metrics (see Sec. 4).
3.2. Stop wasting time retraining state-of-the-art models
In our opinion, reproducing results from the state of the art (i.e., re-training models) – to serve as
points of comparison with a new approach – can waste valuable time and computing resources
4Regarding the relevance and/or efectiveness of these metrics, we refer the reader to the original papers.
from AI researchers, when that time could be dedicated to improving that new approach. This
issue mostly arises when proposing new evaluation metrics5 that must be applied to a wide
range of state-of-the-art models. Thus, one of the objectives for CaBRNet is to publish pre-trained
state-of-the-art CBR models, so that they can be used readily by the XAI research community.
For the sake of transparency, and as stated in Sec. 2.3, each published model is also associated
with information ensuring a level of reproducibility, should a researcher wish to re-train these
models. Moreover, in an efort to improve the statistical significance of all related experiments,
we plan to publish at least three models per model configuration and dataset . As an example, we
have already published 6 models6 trained on the Caltech-UCSD Birds 200 (CUB200 [
CaBRNet is open-source and welcomes any external contribution. We also strongly
encourage the research community to publish trained models that can be reused within our
evaluation framework. On our side, future developments include: i) support for ProtoPool[
Ideally, the modularity of CaBRNet design will appeal to both AI researchers wanting to experiment with CBR approaches, and industrial actors interested in deploying those approaches in a principled and traceable way. Our goal is for CaBRNet to become the development framework for new and innovative CBR approaches by the community.
The authors wish to thank Bartek Jura (Poznań Institute of Technology) and Meike Nauta (Datacation) for their kind feedback on this paper, and Jules Soria (CEA-List) for carrying out some of the training experiments that have been published.
This work was supported by French government grants managed by the Agence Nationale de la Recherche under the France 2030 program with the references ”ANR-23-DEGR-0001” and ”ANR-23-PEIA-0006”, as well as a Research and Innovation Action under the Horizon Europe Framework with grant agreement Nr.101070038. Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them. 5When using common metrics (e.g., model accuracy), it is possible to avoid retraining models by simply referring to the results provided by the original authors. 6Available at https://zenodo.org/records/10894996.