Introduction

There is a significant research interest in supporting Alzheimer's Disease (AD) diagnosis with Deep Learning (DL) models [1]. Existing diagnostic guidelines often integrate the clinical evaluation of the patient with structural Magnetic Resonance Imaging (sMRI), to detect pathological brain patterns like gray matter atrophy.

Brain alterations in sMRI might support the early and differential diagnosis and the prediction of disease's progression. There are sets of common practices used for analysing sMRI acquisition, but there are still no universally accepted methods [2,3,4]. In addition, information collected from sMRI should be interpreted together with the patient's age, as there are anatomical brain changes due to the physiological ageing process [5,6].

DL architectures can facilitate the analysis of neuroimaging data, and might be able to identify unconventional AD subtypes and extract yet unknown image-based biomarkers [7,8]. Prototypical-Part (PP) networks combine the advantages of DL models in an interpretable-bydesign architecture, and are collecting interesting results in medical imaging applications where the black-box nature of standard DL models poses controversy [9].

There are currently different variants of PP networks, including PIPNet [10], originally applied to 2D images and then extended to handle 3D scans [11]. PIPNet showed appealing properties in the medical imaging domain [12], including a reduced number of part-prototypes, semantic significance of learned prototypes, and ability to cope with Out-of-Distribution data (which might be particularly useful in dementia diagnosis, where unusual neurodegeneration pattern are reported [4]). However, sMRI data should be interpreted together with patients' demographics to discern age-related image alteration from pathological alteration, and existing PP models cannot be directly applied to perform this task. Adding non-image prototypes to the standard PP architecture is a non-trivial task, and there are no unique strategies available. There are some works which integrate the concept of learning prototypes from multiple modalities which are based on the concatenation (deterministic prototypes) or on the multimodal feature extraction (shifted prototypes). However, available models cannot be applied to our task, as specifically designed for images and textual data [13].

We present Patch-based Intuitive Multimodal Prototypes Network (PIMPNet), the first multimodal prototype classifier which learns 3D image part-prototypes and prototypical values from structured data, to predict patient's cognitive level in AD from sMRI and age values.

Method

This section introduces the architecture (cf. Sect. 2.1 and Fig. 1) and the training process (Sect. 2.2), of PIMPNet.

Proposed Model: PIMPNet

We propose an age-prototypes layer integrated into the original PIPNet 3D model [11] to create our multimodal architecture. In contrast to "ordinary" age-binning for the inclusion of age information, the age-prototypes layer has the advantages of: (i) being able to learn important age values for the diagnostic task performed (which might not be equally distributed, and might not be easily identifiable in priors); (ii) not to assign different age bins to two patients of similar age close to the bin boundary.

Our PIMPNet has an input layer which takes the 3D image x 𝑖𝑚𝑔 ∈ R 𝑐ℎ×𝑆×𝑅×𝐶 and the age x 𝑎𝑔𝑒 ∈ R 1 as input, where 𝑐ℎ, 𝑆, 𝑅, 𝐶 respectively represents the number of channels, slices, rows and columns of the input image volume. Image x 𝑖𝑚𝑔 and age x 𝑎𝑔𝑒 are processed in parallel. A CNN backbone processes x 𝑖𝑚𝑔 , z = 𝑓 (x 𝑖𝑚𝑔 ; w 𝑓 ) extracting 𝑀 3-dimensional In parallel, we have the age-prototypes layer, constituted by 𝑁 trainable 1-dimensional tensors t 𝑎𝑔𝑒,𝑛 ∈ R 1×𝑁 , which aims to learn prototypical age values for the classification task. This layer computes age prototype presence scores p 𝑎𝑔𝑒 ∈ [0., 1.] 𝑁 , a similarity measurement between the input age to every age prototype, defining a smooth age binning1 :

p 𝑎𝑔𝑒,𝑛 = 1 √︂ 1 + (︁ x𝑎𝑔𝑒−t𝑎𝑔𝑒,𝑛 𝑡 )︁ 2𝑠 ,(1)

where t 𝑎𝑔𝑒 are trainable parameters and 𝑡 and 𝑠 are hyper-parameters which regulate the band and the slope of the similarity function. We then have a prototypes layer which concatenates the image and age prototype presence scores obtaining a layer of

𝐿 = 𝑀 + 𝑁 prototypes p ∈ [0., 1.] 𝐿 p = 𝑐𝑜𝑛𝑐𝑎𝑡(p 𝑖𝑚𝑔 , p 𝑎𝑔𝑒 ).

PIMPNet Training

We optimize PIMPNet's parameters by integrating the training of age prototypes into the original PIPNet training process [10]. This includes two main stages: (1) Self-Supervised pre-training of Image-Prototypes, and (2) PIMPNet training.

As the original PIPNet [10], the 1st stage generates positive pairs x ′ 𝑖𝑚𝑔 , x ′′ 𝑖𝑚𝑔 by applying data augmentation transformation to x 𝑖𝑚𝑔 selected so that humans consider the two views similar. These are used to minimize the loss function 𝜆 𝐴 ℒ 𝐴 + 𝜆 𝑇 ℒ 𝑇 by updating w 𝑓 , where

ℒ 𝐴 = − 1 𝐷𝐻𝑊 ∑︀ (𝑑,ℎ,𝑤)∈𝐷×𝐻×𝑊 𝑙𝑜𝑔(z ′ :,𝑑,ℎ,𝑤 • z ′′ :,𝑑,ℎ,𝑤

) is an Alignment Loss which optimizes positive pairs to activate the same prototype. Together with a softmax over z :,𝑑,ℎ,𝑤 , the alignment results in near-binary encodings where an image patch corresponds to exactly one prototype.

ℒ 𝑇 = − 1 𝑀 ∑︀ 𝑙𝑜𝑔(𝑡𝑎𝑛ℎ( ∑︀ p 𝑖𝑚𝑔,𝑏 ) + 𝜖

) is a Tanh-Loss used to prevent the trivial solution that one prototype node is activated on all image patches in each image in the dataset and instead activates multiple distinct prototypes per batch 𝑏. Only during training, output scores are calculated as o = log((pw 𝑐 )2 + 1), acting as regularization for sparsity.

The 2nd training stage includes the training of age prototypes, optimization of classification performance and image-prototypes fine-tuning for the downstream classification task. The optimization minimizes 𝜆 𝐴 ℒ 𝐴 + 𝜆 𝑇 ℒ 𝑇 + 𝜆 𝐶 ℒ 𝐶 by updating w 𝑓 , t 𝑎𝑔𝑒 , w 𝑐 , where ℒ 𝐶 is the Log-likelihood classification loss

Evaluation

We used the multimodal dataset from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database 3 . We selected "ADNI1 Standardized Screening Data Collection for 1.5T scans" processed with Gradwarp, B1 non-uniformity, and N3 correction, obtaining 307 CN and 243 AD sMRI brain scans and the corresponding patients' age. We report statistics on patients' demographic of the selected ADNI cohort in Table 1. We preprocessed sMRI data inspired by the pre-processing pipeline applied in previous works [15]. We tranformed all the images to the common ICBM152 Non-Linear Symmetric 2009c standard space [16] with affine registration. We selected the grey matter structures by applying the ICBM152 Non-Linear Symmetric 2009c brain and kept a margin of 3 from its first and last non-empty slices. We applied an image downsampling of 2 and we scaled all image intensities to the range [0-1] with a min-max normalization. We implemented PIMPNet using PyTorch and MONAI 4 , training our models on an Intel Core i7 5.1 MHz PC, 32 GB RAM, equipped with an NVIDIA RTX3090 GPU with 24 GB of embedded RAM. As CNN backbones we used ResNet-18 3D pretrained on Kinetics400 [17] and ConvNeXt-Tiny 3D pretrained on the STOIC medical dataset (Study of Thoracic CT in COVID-19) [18]. We finetuned PIMPNet with Adam optimizer using the same hyperparameter settings of the original PIPNet [10]. We only reduced the batch size to 12 to adapt it to our computational capabilities and we set the learning rate of age prototypes to 0.1 5 . We arbitrarily set the number of age prototypes = 5 evenly spaced between 40 and 90 to cover the patients' age range of our dataset. For the age similarity function, we respectively set 𝑡 = 4 and 𝑠 = 8 6 . We performed 5-fold cross-validation with patient-wise splits. 20% of training images are used for validation.

We evaluated the models in terms of classification performance and with functionally grounded metrics of explainability. Results are reported in Tables 2 and 3. We compared PIMPNet performance (sMRI + age) with PIPNet-3D (sMRI only) [11], to evaluate if including age information improves diagnostic performance. We measured performance using Accuracy (Acc), Balanced Accuracy (Bal Acc), Sensitivity (SENS, Acc of Cognitive Normal class), Specificity (SPEC, Acc of Alzheimer's Disease class), and F1 score (F1). We measured the Global size (GS) of the model as the total number of prototypes, and the Local size (LS) of explanations as the number of detected prototypes in a single 3D sMRI, averaged over all the images in the test set. Additionally, we report the Sparsity (Sp) of the decision layer as the percentage of zero-weights in the linear classification layer [10] to assess the compactness of the prototypes-classes layer. We further assessed whether prototypes are consistently located in the same brain region, and the purity of the prototypes in terms of the anatomical regions included based on the CerabrA atlas annotation [19]. More specifically, the Prototypes Localization Consistency (LC 𝑝 ) evaluates the differences in the coordinate centre of the prototypical part in the input image, while the Prototype Brain Entropy (H 𝑝 ) as a measure of purity computes the Shannon Entropy of the brain regions included in the prototypical part [11]. We show the learned age prototypes t 𝑎𝑔𝑒 from five different folds (denoted as Mx where x indicates the current fold) in Table 4.

Discussion and Conclusion

Both PIPNet and PIMPNet with the ResNet-18 3D backbone achieve higher classification performance than with the ConvNext-Tiny backbone. Our preliminary results also show that the proposed Age-Prototype layer can learn prototypical age values; however, these do not improve classification performances compared to the baseline model. Our functionally-grounded evaluation of prototypes shows that all models learn prototypes consistently located in the same anatomical brain regions (low LC 𝑝 values). We also observe that the models trained with the ConvNeXt-Tiny 3D backbone have higher compactness. This might partially explain the lower performance scores (the number of prototypes learned is not enough for performing the diagnosis), but is an interesting observation for future research as such a highly compact model can be considered more interpretable than larger ones and can be easily evaluated by domain experts. We also observe that the image prototypes of the ConvNeXt-Tiny 3D backbone are generally purer 7 . Despite purity being a desirable property for prototypes [20], because of the design of the purity metric, also a prototype which only includes the background, i.e., a clinically irrelevant prototype, will have high purity 8 . In summary, we proposed PIMPNet, an interpretable multimodal prototype-based classifier. The proposed architecture is the first prototypes-based network which performs an interpretable classification based on the detection of prototypes learned from different data modalities (3D images and age information). We applied PIMPNet to the binary classification of Alzheimer's Disease from 3D sMRI images together with the patient's age. Despite the usage of age prototypes do not improve predictive performance compared to the model trained with only images, we identified different potential reasons which define the future directions of our work. First, as the original PIPNet training paradigm includes a pretraining stage [10] of image prototypes, we plan to include an age prototypes pretraining step w.r.t. the log-likelihood classification loss. Second, we also plan to work on the model's design. As the simple concatenation of the prototype presence score might not be able to properly represent the relationship between age and image prototypes for the downstream task, we plan to combine image and age prototypes using a different (but still interpretable) classifier than a scoring-sheet system.