Zero-shot anomaly detection (ZSAD) ofers potential for identifying anomalies in medical imaging without task-specific training. In this paper, we evaluate CLIP-based models, originally developed for industrial tasks, on brain tumor detection using the BraTS-MET dataset. Our analysis examines their ability to detect medical-specific anomalies with no or minimal supervision, addressing the challenges posed by limited data annotation. While these models show promise in transferring general knowledge to medical tasks, their performance falls short of the precision required for clinical use. Our findings highlight the need for further adaptation before CLIP-based models can be reliably applied to medical anomaly detection.
Anomaly detection (AD) in medical imaging plays a critical role in identifying and diagnosing diseases,
often detecting rare or subtle anomalies that may go unnoticed by human observers [
In this paper, we explore the potential of CLIP-based models for medical anomaly detection by
focusing on a brain tumor detection task using the BraTS dataset [
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
We tested selected CLIP-based abnormality detection models to detect and segment brain metastases.
This task was chosen due to the clinical significance of brain metastases, which are common secondary
tumors that pose significant challenges in diagnosis and treatment. They are indeed characterized by
an extreme variability in terms of lesion size, shape and localisation within the brain, thus representing
an optimal case study for AD. To this aim, we used the the BraTS 2023 Brain Metastases [
For this work, we used only the BraTS 2023 training set (165 patients), splitting 70% for training and 30% for testing. The whole tumor ground truth mask (WC = Nonenhancing tumor core + Surrounding non-enhancing FLAIR hyperintensity + Enhancing Tumor) was used as the target mask. As a case study, only the axial view of t2w images was considered. CLIP’s preprocessing standardization was applied using OpenAI’s ImageNet mean and standard deviation on each slice separately. Data augmentation techniques were employed to reduce overfitting during training.
CLIP (Contrastive Language-Image Pretraining) is a large-scale vision-language model that has shown
remarkable success in zero-shot image classification tasks [
ZSAD aims to identify anomalous patterns in images from categories that are not present during
training. Given an image ∈ R× × 3, the objective is to compute an image-level anomaly score
∈ [
exp(cos(F(pa,tc)h, Fanomalous)) exp(cos(F(pa,tc)h, Fnormal)) + exp(cos(F(pa,tc)h, Fanomalous)) where cos(· , · ) denotes the cosine similarity, and , refer to the spatial location of the patch. The image-level anomaly score is then computed by aggregating the anomaly map to provide an overall score for the image.
Several variations of the above described approach have been developed, introducing features such as object-agnostic and learnable text prompts or value-wise attention mechanisms for fine-grained localization. In these cases, a set of auxiliary data train = {(, )}=1, where represents the images and ∈ {0, 1}× are the corresponding ground-truth masks for anomalies, is typically available to train the adaptation layers. In this paper, we evaluate four AD methods that adapt CLIP for zero-shot and few-shot anomaly detection in medical imaging.
• AnomalyCLIP [
In addition to using the standard CLIP backbone, we also leverage a specialized version of CLIP
for medical imaging, called PMC-CLIP [
In this section, we describe the benchmark designed to evaluate four CLIP-based anomaly detection (AD)
methods—VAND, AnomalyCLIP, AnoVL and AdaCLIP — on the brain tumor segmentation task using the
BraTS dataset. The benchmark is designed to assess the capability of these methods, originally developed
for industrial AD tasks, in detecting medical anomalies such as brain tumors. We explore diferent
training strategies to test the generalization of these models across domains, including industrial,
medical, and brain tumor datasets. More in detail, we designed four experimental setups:
• (Industrial) Pretrained on Industrial AD Dataset: In the first setup, we evaluate each method
in its original form, using models pretrained on an industrial anomaly detection dataset. This
setup allows us to examine the generalization capabilities of these models in detecting medical
anomalies such as brain tumors without any additional medical-specific training.
• (Finetune) Finetuned on BraTS: Next, we finetune the models using the BraTS dataset, which
contains labeled examples of brain tumors. This setup assesses the improvement in performance
when the models are adapted to a specific medical domain through supervised learning. In
this setting, the CLIP model weights are frozen. Only the adaptive layers introduced by each
modification are finetuned on BraTS.
• (Brats) Training from scratch: In this experiment, we train the model in its original form, with
a randomly initialized adapter layer, on the BraTS dataset. As in the previous setting, the CLIP
model weights are frozen.
• (PMC) Using PMC-CLIP [
These methods are designed for 2D image processing. Since brain MRIs are 3D volumes, we treat
each slice (axial view) as a 2D image and reconstruct the 3D anomaly map by stacking the 2D anomaly
maps. For all four setups, we measure the performance of each method using Dice score (equivalent to
the F1-score) as well as sensitivity, specificity, and positive predictive value (PPV). These metrics are
computed for each 3D volume and averaged across all test subjects. Additionally, we compute the Area
Under the Receiver Operating Characteristic curve (AUROC) and the maximum F1-score (F1-max) [
The 3D Dice scores across subjects and models are summarized in Fig. 1. This plot compares the performance of four anomaly detection models (AdaCLIP, AnomalyCLIP, VAND, and AnoVL) for each of the four experimental setups (pretrained on industrial datasets, finetuned on BraTS, training on BraTS from scratch, and training on BraTS using the PMC-CLIP backbone).
AdaCLIP demonstrates the highest variance in performance across experiments, with its 3D Dice Scores ranging from approximately 0.1 to 0.7, particularly better when training the adapters on the BraTS dataset. VAND also shows relatively high performance, especially when using the PMC-CLIP backbone, where its scores vary between 0.1 and 0.6. In contrast, AnomalyCLIP shows consistently lower Dice Scores across all experiments, particularly struggling with scores close to 0 when using the industrial and industrial-finetuned weights. AnoVL shows the weakest performance overall, with very low scores across all experiments, rarely exceeding 0.1, reflecting its poor capacity for anomaly detection in this setting.
For further comparison, we analyze additional metrics as reported in Table 1. Notably, the trend
observed in the AUROC values contrasts sharply with the Dice scores. We attribute this discrepancy to
the AUROC’s sensitivity to data imbalance [
AnoVL AnomalyCLIP AnomalyCLIP AnomalyCLIP AnomalyCLIP
VAND VAND VAND VAND
Industrial Finetune
Brats
PMC Industrial Industrial Finetune
Brats
PMC Industrial Finetune
Brats PMC industrial , finetune , brats , and pmc . lesions within large 3D volumes. Consequently, AUROC values may provide an inflated representation of performance that does not accurately reflect segmentation quality.
To further investigate model performance, we analyzed Dice Scores per subject across the experiments, as shown in Fig. 2. Due to its low performance we exclude AnoVL from this analysis. Each plot presents 2D Dice Scores across individual slices (blue bars) and the corresponding 3D Dice Scores (red crosses). This comparison illustrates the extent of variability between 2D slice-level performance and 3D volumetric segmentation accuracy.
• VAND exhibits relatively stable 2D segmentation accuracy across subjects, especially when trained on BraTS from scratch and when using PMC-CLIP backbone, where Dice Scores typically range from 0.4 to 0.8. However, the 3D Dice Scores are notably lower for most subjects, with values between 0.2 and 0.5, highlighting challenges in volumetric segmentation. To note, while showing similar variability of Dice score on 2D slices, the experiment on PMC-CLIP backbone reports higher 3D Dice scores compared to industrial-CLIP with training on BraTS, suggesting that, for this architecture, medical-specific CLIP embeddings are more suitable for the task of anomaly detection. • AdaCLIP also shows a wide range of 2D Dice Scores, particularly excelling in the training on BraTS, where it achieves values as high as 0.8 in certain subjects. However, like VAND, the corresponding 3D Dice Scores are consistently lower, ranging between 0.2 and 0.4 for many subjects. • AnomalyCLIP, on the other hand, underperforms across all datasets. Similarly to VAND, the greatest improvement is obtained with the use of the PMC-CLIP and training the adaptive layers on BraTS. However, its 2D Dice Scores are generally below 0.3 for most subjects, and the 3D Dice Scores are even lower, barely exceeding 0.1 for the majority of cases. This suggests poor performance for anomaly detection in both 2D and 3D segmentations.
To understand how segmentation performance changes spatially within the brain, we plotted the mean 2D Dice Score for each discretized distance from the brain center (normalized from -1 to 1) in Fig. 3. The number of slices containing lesions at each distance is also shown (yellow bars), providing insight into how lesion location afects segmentation accuracy.
For both VAND and AdaCLIP, segmentation accuracy is highest near the upper part of the brain (normalized distance around 0.25), where the number of lesion-containing slices is also highest. Dice Scores reach values between 0.4 and 0.7 at these upper-central regions, while they drop to around 0.2 towards the periphery (distances near -1 and 1). This suggests that these models struggle more with peripheral brain regions, where lesion frequency is lower or more variable. AnomalyCLIP, however, despite showing a similar trend as the other methods, achieves lower performance across all distances overall, with mean Dice Scores consistently below 0.3. This suggests that the model is unable to efectively capture lesion characteristics regardless of their location in the brain.
In summary, there is a trend in achieving reasonable 2D segmentation performance near the center of the brain but struggle towards the periphery. Their 3D performance remains lower than their 2D scores, suggesting challenges in volumetric segmentation.
The results from our experiments show consistently low performance for all models on the BraTS brain tumor segmentation task, with Dice scores below 50%, even when using PMC-CLIP, a visuallanguage model pretrained on a medical dataset. Although a comprehensive quantitative comparison is challenging due to variations in medical applications and datasets, we observe that the AUROC values obtained in our experiments (Table 1) are generally lower than those reported in the related studies. Indeed, despite pixel-level Dice scores not being reported for Brain MRI, AdaCLIP presents AUROC values ranging from 0.772 to 0.904, while AnomalyCLIP reports values between 0.789 and 0.897 for ZSAD across other (2D) medical imaging applications (e.g. skin, colon, thyroid). Concerning Brain anomaly detection, their reported image level AUROC values exceed 90%. However, such high values do not align with pixel-level metrics observed in our experiments, also due to the fact that AUROC might not be sensitive enough to capture anomaly segmentation quality. Therefore, the Dice score should be preferred as a more reliable metric for performance evaluation in this context, particularly given the sparsity of anomalous voxels.
Nevertheless, the observed discrepancy may be attributable to the nature of the BraTS dataset, which
presents unique challenges such as greater variability in tumor appearances and difuse boundaries
compared to the possibly simpler brain MRI datasets used in related work experiments, e.g.
BrainMRI [
Finally, several other CLIP-based models have already been designed for medical imaging, including
MedicalCLIP [
This work was partially supported by the “Bando 5× mille Ministero della Salute 2024" assigned from the IRCCS Humanitas.