Introduction

Deep learning (DL) is an area of machine learning (ML) that has been rapidly growing in recent years. Employed for training of deep artificial neural network architectures, its use has been extensively explored, due to its impressive results in various areas, such as bioinformatics [1], natural language processing [2,3], and image processing [4,5]. Some of the main applications in image processing are in medicine, where AI-based Computer-aided diagnosis (AI-CAD) solutions often use DL. In these cases, DL models support medical diagnoses based on images such as magnetic resonance imaging (MRI), X-rays, computed tomography and conventional photographs.

Despite considerable community interest, practical application of AI-CAD solutions encounters obstacles, including the lack of interpretability features in models. These models are often perceived as black boxes, making it challenging for humans to understand their internal reasoning, which raises trust issues among experts and regulatory concerns [6,7,8]. Current AI-CAD solutions often lack robustness, being susceptible to biases during training and failing to provide experts with confidence estimations or limitations regarding the results. This poses challenges, especially in healthcare settings, where less experienced professionals may overly rely on computational models [6,9,10].

ProtoPNet, introduced by Chen et al. [11], is a deep neural network (DNN) architecture aiming to enhance interpretability features within DNN models. During inference, ProtoPNet showcases prototypes that share similar features with the input image in a "bag-of-features" format. Several studies have applied the ProtoPNet architecture in medical image analysis. For instance, Mohammadjafari et al. [12] used ProtoPNet to classify MRI brain scans as either healthy or indicative of Alzheimer's disease. Vaseli et al. [13] introduced ProtoASNet, a modified version of ProtoPNet tailored for handling spatio-temporal data and integrating aleatory uncertainty estimation into prototypes in the context of aortic stenosis. Wei et al. [14] presented the MProtoNet model, specifically designed for tumor brain classification tasks in multi-parametric MRI (mpMRI) analysis.

The use of DL is also affected by limited availability of large datasets, especially in supervised learning. Besides data being abundant, labeling requires expert input, resulting in high demands in costs and time. To address this issues, deep active learning (DAL) emerges as a feasible approach, extending active l earning (AL) [15] concepts to work with DNNs. DAL assumes that a model with comparable results can be achieved using less but carefully selected training instances. Several works have investigated DAL in medicine, with satisfactory outcomes with less data compared with models trained on the entire dataset [16,17,18,19,20]. Smailagic et al. [21] proposed the O-medal method, which employs online training and eliminates the necessity of the complete model re-training at each DAL cycle. This method presents a more feasible scheme within the context of DNNs.

Limited research exists on the intersection of DAL and interpretable models. Phillips et al. [22] used LIME interpretability to aid experts in understanding query batches and tracking uncertainty bias. Das et al. [23] employed AL for anomaly detection, providing experts with explanations of model decisions using tree-based ensembles. They introduced a novel formalism for compact descriptions to enhance diversity and generate interpretable rule sets. Liu et al. [24] proposed a DAL approach based on model interpretability, aiming to maximize representativeness in unlabeled data by selecting examples from different linear separable regions of an interpretable DNN. Furthermore, Mondal and Ganguly [25] trained an explainer model alongside a classifier to select new instances based on dissimilarity of explanations compared to already labeled instances. Except for Das et al. [23], none of these works applied the approaches with medical data. Das et al. [23] only explored the use in tabular data.

The main contribution of our work is ProtoAL 1 , a novel model capable of integrating an interpretable DNN into the DAL framework, specifically designed for medical image analysis. ProtoAL achieves interpretability through prototypes, providing explanations based on image patches from the training dataset, aligning closely with clinical practices in a visually intuitive manner. Additionally, we investigated how the DAL framework impacts the objective of reducing the required training instances to achieve results comparable to conventionally trained models. To our knowledge, ProtoAL is the first model to integrate an intrinsically interpretable DNN through visual prototypes into the DAL framework. To assess its effectiveness, ProtoAL was compared against baselines using robust predictive performance measures, such as the area under the precision-recall curve (AUPRC). Our model, ProtoAL, aims to integrate interpretable DL into the DAL framework, with focus on the AI-CAD to medical imaging. The DAL framework allows train a learning model, 𝑀 , using a training set, ℒ, consisting of selected instances based on a search strategy 𝑄, which criteria typically target uncertainty instances or aim to enhance diversity. These selected instances are part of a large, unlabeled dataset 𝒰, and are labeled by an oracle before being added to ℒ. The oracle can operate online, where domain experts label the selected instances; or in a "simulated" manner, where ground-truth labels are hidden and only revealed for the selected instances. The DAL framework assumes that selected instances offer pertinent information about the problem, which enables a model achieve comparable performance to one trained on a fully random instance dataset while needing fewer examples. This is particularly beneficial in medical context, where large datasets of unlabeled data exists, but labeled datasets suitable for DL training are scarce, allowing reduce the expenses associated with expert labeling. The DAL cycle is repeated until a stop condition is reached, as a satisfactory performance, budget constraints or depletion of the 𝒰 dataset.

Methods

Labeled

Figure 1 illustrates our workflow, divided in two cycles: the outer DAL cycle, referring to the process described above; and the training of the interpretable model itself, with its inner workings (detailed in the Subsection 2.2). The proposed ProtoAL model uses the ProtoPNet as the base model ℳ, integrating an intrinsically-oriented DNN into the DAL framework. This allows the interpretable capabilities of the ProtoPNet to be incorporated into the DAL framework, requiring fewer training instances and taking advantage of the more informative and significant ones. With ProtoAL, the adoption of models like ProtoPNet in the medical context becomes more feasable, where large labeled datasets are scarce.

Deep Active Learning

We use the O-medal method [21] to define our DAL framework. The O-medal operate similarly to the framework described above, but differs by avoiding the need to retrain the model from scratch at each DAL iteration. It offers a more reliable, better fitting, and computationally efficient approach. Also, ℒ does not include all previously labeled instances. Instead, it consists of newly labeled instances and a partition 𝑝 of the previously labeled data.

The 𝒬 strategy was based on uncertainty estimation. A common approach involves using Monte Carlo Dropout (MC Dropout) [26], with the Dropout technique [27] working as a Bayesian approximation. During the inference of the 𝒰 instances, 𝑇 forward steps are performed, and averaging is applied across each instance. The 𝑛 most uncertain instances are labeled and added to the ℒ set [26]. Throughout this, all layers except the dropout layers remain frozen. As oracle, we adopted the simulated strategy, only revealing the instances labels when selected to labeling.

Prototypical Neural Network

The ProtoPNet [11] is an interpretability-oriented DNN that uses prototypes to explain the reasoning of the learning model. Unlike explainability in posthoc approaches, the ProtoPNet proposes an intrinsically oriented method for achieving interpretable DNN, trough imposed constraints on the learning process, which considers the explanations during training.

In summary, the ProtoPNet model compromises a backbone neural architecture 𝑓 , such as VGGNet [28], ResNet [29], or any other selected architecture, extended by a prototypical layer 𝑔 p and a fully connected layer ℎ. The input feature extraction is realized by 𝑓 , while 𝑔 p learns 𝑚 prototypes P = {p 𝑗 } 𝑚 𝑗=1 of 𝐻 1 × 𝑊 1 × 𝐷 shape, with 𝐻, 𝑊 , and 𝐷 representing height, width and dimensions of the prototypes [11]. The prototype activation pattern acts as a patch representing some prototypical image patch in the original pixel space, where 𝑝 can be understood as a latent representation of some prototypical part. The squared 𝐿 2 distances between the input patches and the prototypes of 𝑝 are inverted into similarity scores, forming an activation map where the values indicate how strongly a prototypical part is present in the image. The activation map is reduced using global max-pooling to a single similarity score, indicating how much a prototypical part is present in some input image patche. The 𝑚 similarity scores produced by 𝑔 p are multiplied by the outputs of ℎ and normalized using softmax, yielding the predicted probabilities of the image [11].

The main advantage of ProtoPNet is provide visual explanations of the predictions, which it optimizes and learns alongside the classifier. These prototypes are related to instances from the training set, representing real cases and attributes.

Experiments

Dataset

The experiments were carried out using the Messidor dataset [30] 2 of diabetic retinopathy (DR). It compromises 1200 color images of the eyes fundus, obtained from three different ophthalmologic departments. Experts evaluated each image and classified it based on the retinopathy grade and risk of macular edema. Retinopathy grade ranges from 0 (normal) to 3, considering the number of microaneurysms, hemorrhages and the presence of neovascularization. Corrections were made by correcting mislabeling and removing duplicated images files according to the instructions in the dataset download page 3 .

Following the preprocessing outlined in Smailagic et al. [21], we grouped the retinopathy grades as healthy (DR = 0) or diseased (DR ≥ 1). The risk of macular edema feature was not used in the experiments. The images were resized to 512 × 512, and data were augmented by randomly applying rotations up to 15 degrees, horizontal flips and scaling in the range [0.9, 1]. The train, validation and test sets were composed of 759, 190 and 238 instances, respectively.

Baselines

We compared the ProtoAL model in two perspectives, to observe both the interpretability and DAL framework factors. For such, we adopted three baselines, targeting distinct contexts: (i) A vanilla ResNet-18 model, trained conventionally and without interpretability features, with access to the entire training dataset from beginning; (ii) the ProtoPNet standalone baseline, with the aim of evaluate the performance of the interpretability model used in ProtoAL without incorporating it into the DAL framework; and (iii) the ProtoAL with random search strategy, utilizing as search strategy the selection of random instances from 𝒰. This baseline allows to observe the impact of the MC Dropout as search strategy in the performance of the ProtoAL model. The ResNet-18 and ProtoPNet were pretrained on the ImageNet dataset [31], both when used as baselines and backbone in the ProtoAL.

Implementation details

As mentioned earlier, the ProtoAL model follows the structure of the O-medal method, while employing a ProtoPNet as DNN model. The model hyper-parameters were optimized using grid search [32]. This was deemed the most appropriate approach given the computational constraints associated with exhaustive search. The runs were executed by varying seed values (0, 1,2,5,10,12,42,123,1234,12345), and hyper-parameters for both the DAL method and ProtoPNet model, as MC Dropout steps (10,30,50), instances to label per DAL iteration (10,20,30), batch size (32,64) and epochs per DAL iteration (5,10,20).

We conducted runs using both random (ProtoAL-Random) and MC Dropout (ProtoAL-MC) as search strategy. The stop condition for the DAL cycle was determined as no remaining instances left to be labeled in 𝒰. The number of runs was dynamically adjusted based on the labeled instances per DAL iteration. The fixed hyper-parameters follows the Smailagic et al. [21] work.

The ℒ set initially consisted of 100 randomly selected instances. The percentage of previously labeled examples forming ℒ was set to 0.875.

A ResNet-18 was used as the backbone DNN of the ProtoAL, with prototype layers featuring 256 channels. We settled 12 prototypes, with 6 prototypes allocated for each class (healthy and diseased). During training, ProtoPNet undergoes warm-up for 5 epochs during the first DAL iteration. After warm-up, joint training is performed for 𝑒 epochs (epochs per DAL iteration hyper-parameter), succeeded by a projection (push) step and last layer optimization for 15 steps. Excepting for warm-up, this cycle repeats for all DAL iterations. We used the Adam optimizer, with learning rates as outlined in Chen et al. [11]. Learning rate was decreased exponentially per epoch. All experiments were run on Nvidia Tesla V100 GPU, with 32GB VRAM. The DNN models were implemented with PyTorch 2.0.1 framework and Python 3.10.

Experimental Results

During the grid search, we trained a total of 540 ProtoAL models. We selected the best model configuration using AUPRC, based on validation set results. The best run was achieved by a model trained with a batch size of 32, employing 10 iterations of MC Dropout per instance, and selecting 30 new examples at each DAL iteration. For the joint optimization phase, the ProtoPNet model underwent 10 training epochs. Table 1 presents a comparison of results between ProtoAL with MC dropout (ProtoAL-MC) and the baselines evaluated on the test set.

The ProtoAL-MC achieved an AUPRC of 0.79, along with an F1-Score and accuracy of 0.81. The ProtoAL-Random, utilizing random search strategy, achieved 0.77 in AUPRC and 0.79 in both F1-Score and accuracy. These results demonstrate that the MC Dropout search strategy yielded superior results, particularly evident in the AUPRC metric.

ProtoPNet and ResNet-18 obtained comparable or superior results to ProtoAL in specific metrics. It is worth noting that both baselines had access to the entire training dataset from the start, being more exposed to the instances. It can be observed that ProtoPNet exhibits a decline Through prototypes, a domain expert can observe the most similar instances for both disease and healthy cases, along with their respective relevant regions in the input image. As interpretability criteria, experts can rely on visual inspection, and prototype similarity scores and weights, assessing whether the model's inferences are meaningful and relevant. For instance, a expert can verify whether a highly relevant prototype corresponds to a meaningful region of the input image, assess if it contains features consistent with the predicted output, and observe the similarity score it achieves. With such information, an expert has the tools to more accurately calibrate their confidence and trust in the model's prediction.

Discussion and conclusion

Our model, ProtoAL, integrates an interpretable DNN model with prototypes into a DAL framework. Specifically tailored for the AI-CAD context of medical imaging, the aim was to enhance the reliability of AI-CAD solutions in practical medical contexts while exploiting the capacity to comprehend the model decisions and training in limited labeled datasets. Quantitative results presented in Table 1 demonstrate the success of providing a interpretable model while utilizing a reduced amount of training data, addressing two key challenges in the adoption of AI in medical contexts: lack of interpretability and scarcity of labeled datasets. The qualitative results in Figure 3 illustrate prototypes relevant to the network's inference reasoning for a diseased input image, providing visual explanations that domain experts can interpret. ProtoAL offers interpretability features lacking in the ResNet-18 baseline, which enhance the practical usability of ProtoAL as an AI-CAD solution while maintaining a performance level similar to that of the ProtoPNet model, albeit with reduced training data demands. However, preliminary experiments with ProtoPNet using a greater number of prototypes resulted in the repetition of similar or identical prototypes, which is undesirable as it does not promote prototype diversity. Furthermore, our work is limited to quantitative evaluation using standard metrics such as AUPRC and F1-Score, which do not assess the interpretability characteristics of the model.

Future works could explore how to enhance integration of interpretability features within the DAL framework, including leveraging information from prototypes to refine search strategies. Moreover, investigate how to promote prototype diversity and integrate domain experts in evaluation of the results would provide valuable insights and a important qualitative analysis.