Fingerprint Identification of Generative Models Using a MultiFormer Ensemble Approach Notebook for ImageCLEF Lab at CLEF 2024 Md. Ismail Siddiqi Emon1,† , Mahmudul Hoque1 , Md Rakibul Hasan1 , Fahmi Khalifa2,† and Md Mahmudur Rahman1,* 1 Department of Computer Science, SCMNS School, Morgan State University, Baltimore, Maryland 21251, USA 2 Electrical & Computer Engineering Dept., School of Engineering, Morgan State University, Baltimore, Maryland 21251, USA Abstract In the ever-changing realm of medical image processing, ImageCLEF brought a new dimension with the Identifying GAN Fingerprint task, catering to the advancement of visual media analysis. This year, the author presented the task of detecting training image fingerprints to control the quality of synthetic images for the second time (as task 1) and introduced the task of detecting generative model fingerprints for the first time (as task 2). Both tasks are aimed at discerning these fingerprints from images, on both real training images and the generative models. The dataset utilized encompassed 3D CT images of lung tuberculosis patients, with the development dataset featuring a mix of real and generated images, and the test dataset. Our team ’CSMorgan’ contributed several approaches, leveraging multiformer (combined feature extracted using BLIP2 and DINOv2) networks, additive and mode thresholding techniques, and late fusion methodologies, bolstered by morphological operations. In Task 1, our optimal performance was attained through a late fusion-based reranking strategy, achieving an F1 score of 0.51, while the additive average thresholding approach closely followed with a score of 0.504. In Task 2, our multiformer model garnered an impressive Adjusted Rand Index (ARI) score of 0.90, and a fine-tuned variant of the multiformer yielded a score of 0.8137. These outcomes underscore the efficacy of the multiformer-based approach in accurately discerning both real image and generative model fingerprints. Keywords GAN Fingerprint, Multiformer, BLIP2, Generative Model Fingerprint, Training Data Fingerprint, DINOv2, Late Fusion, Thresholding, Reranking, ARI Scoring, CT image denoising 1. Introduction In 2004 ImageCLEF [1] was established for medical imaging which has pioneered advancements in this field. After it advanced to its second edition [2], it marked a significant evolution by the inclusion of medical Generative Adversarial Networks (GANs) tasks. And, the first iteration of its kind came in 2023. This task aimed to explore a specific hypothesis which was that the GANs might embed "fingerprints" of real images within the synthetic medical images they generate. Confirming this hypothesis could have significant implications. It might lead to a reconsideration of the copyright status of synthetic images. This would challenge the conventional view that synthetic images are entirely artificial. In recent years, there has been a substantial surge in the application of GANs and diffusion models within the medical domain [3, 4, 5]. These sophisticated architectures are capable of generating synthetic images and facilitating their translation across different modalities. This burgeoning utilization underscores the transformative potential of GANs and diffusion models in enhancing medical imaging and diagnostics. Medical imaging professionals have been exploring various applications of GANs in medical image CLEF 2024: Conference and Labs of the Evaluation Forum, September 09–12, 2024, Grenoble, France * Corresponding author. † These authors contributed equally. $ mdemo1@morgan.edu (Md. I. S. Emon); mahoq1@morgan.edu (M. Hoque); mdhas1@morgan.edu (M. R. Hasan); fahmi.khalifa@morgan.edu (F. Khalifa); md.rahman@morgan.edu (M. M. Rahman) € https://github.com/ismailEmonFu (Md. I. S. Emon); https://github.com/HoqueMahmudul (M. Hoque); https://github.com/Hasan-MdRakibul (M. R. Hasan); https://mdrahmanlab.com/ (M. M. Rahman)  0000-0003-0595-229X (Md. I. S. Emon); 0009-0006-5532-4135 (M. Hoque); 0000-0002-6179-2238 (M. R. Hasan); https://orcid.org/0000-0003-3318-2851 (F. Khalifa) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings analysis, such as creating artificial medical images and distinguishing between real and fake images. They have developed effective architectures like "Attention GAN" [6] and "ABC GAN" [7] to produce lifelike medical images, which assist with various tasks, including training AI models and protecting patient privacy. However, despite these advancements [8, 9], a major challenge remains: differentiating between real and synthetic medical images. This is an area where scientists continue to focus their efforts. Generative models [10, 11, 12, 13], a recent AI innovation, have driven significant advancements across multiple domains [14, 15, 16], including generative medical imaging [17, 18, 19], such as authors from [17] shows that synthesizing high-resolution images of skin lesions with Generative Adversarial Networks (GANs) can address the lack of labeled data and skewed class distributions in skin image analysis. Using progressive growing, they produce realistic dermoscopic images that are difficult for expert dermatologists to distinguish from real ones, outperforming other GAN architectures like DCGAN and LAPGAN. Synthetic biomedical images serve vital roles in research, healthcare professional training [20], and patient care enhancement from anisotropic diffusion [21] to AnoGAN, an unsupervised deep convolutional GAN to identify anomalies in imaging data for disease markers, demonstrated by accurately detecting anomalies in retinal OCT images [22]. Additionally recent advancement demonstrates that [23] discriminating between malignant and benign lung nodules remains challenging, necessitating CAD systems to assist radiologists. Using unsupervised learning with Deep Convolutional- Generative Adversarial Networks (DC-GANs), they aim to generate realistic lung nodule samples, hypothesizing that difficult-to-differentiate imaging features will be highly discriminative, thereby improving diagnostic accuracy, training radiologists, and generating realistic samples for deep network training.They address challenges such as data scarcity, cost, and ethical considerations associated with real patient data acquisition. The 2024 ImageCLEFmedical GANs Task provides a forum to investigate the influence of GANs on the generation of artificial biomedical images, facilitating the examination of the potential advantages and ethical concerns of their application. This includes two basic objectives: (1) how to identify fingerprints of training data in synthetic biomedical images (inspection), and (2) how to find fingerprints of different generative models on images that they are designed to generate (differentiation). This essentially allows researchers to compare models and highlight the characteristics, patterns, or features present in synthetic images that would separate the models other than that they might not be alike. This dataset includes axial slices of 3D CT images of around 8000 lung tuberculosis patients and acts as a great resource for research. Task 1 aims to identify the real images that GANs’ images were generated from, so as to address privacy and security concerns associated with the use of artificial images [18]. The datasets were provided by the organizers, with a development set divided into marked artificial and real images with training while mentioning no information on the percentage of the used and unused images from this set is disclosed. This extensive dataset is useful to rigorously test the hypotheses and push the community forward with discoveries in biomedical image synthesis. Task 2 is to detect the fingerprints of generative models in GANs-generated images. The authors of ImageCLEFmed GANs argue that one way to understand this behavior is to envision that each AI model imparts its own distinct “signature” on the images it generates. So, here, our aim is to reveal these hidden signatures to identify what makes each model unique. It’s like reading differently styled handwriting from different authors but in the AI area. We do not simply aim to differentiate models but rather to gain a deeper understanding of them, by examining the hidden patterns and subtleties contained within the synthetic images. 2. Datasets In the second edition of the ImageCLEFmed GANs challenge, the organizers presented two tasks: "Identify training data fingerprints" and "Detect generative models’ fingerprints." For the first task, the hypothesis is that when images are generated using diffusion models, the original real images leave specific fingerprints in the generated models. If this hypothesis is true, it could impose additional restrictions on publishing or sharing such images publicly, as they would be as sensitive as the original images. Conversely, if the hypothesis is false, it could lead to a vast dataset of artificially generated images using diffusion models, potentially revolutionizing the medical imaging field. Figure 1: Examples of CT images: (a) from the dataset for identifying training data fingerprints; (b) not from the dataset for identifying training data fingerprints; (c) generated from the dataset for identifying training data fingerprints, and (d) generated from the dataset for detecting generative model fingerprints. The second task, although different in its subjective nature, shares a similar objective: identifying generative model fingerprints in images produced by various diffusion models. The challenge is to determine whether these models imprint unique fingerprints in the generated images. The organizers did not disclose the number of diffusion models used, but the approach remains consistent regardless of the quantity. The author of the ImageCLEFmed GANs task provided both a development set and a training set. For Task 1, there were two datasets consisting of axial slices of 3D CT images from approximately 8,000 lung tuberculosis patients. The artificial slice images, sized at 256×256 pixels, were generated using various undisclosed generative adversarial networks and diffusion neural networks. Over 12,000 generative images were included, and the test set contained a total of 8,000 images. For Task 2, a dataset comprising 3,000 generated image files was provided. Figure 1 depicts examples of images from the original dataset. 3. Proposed Methodology Here we are going to explain the details of the approaches utilized in our submission for both tasks: “Identify training data fingerprints” and “Detect generative models’ fingerprints”. For the task of identifying training data fingerprints, we first performed morphological operations to reduce noise in the CT images. This preprocessing step was crucial for improving the quality of synthetic medical images generated by GANs. Subsequently, we implemented BLIP and DINOv2 as image signature generators. As illustrated in Figure 2, these morphological operations helped control the quality of the images. After preprocessing, we conducted individual feature rankings for each model and then concatenated the feature rankings from both models. We also performed dimensionality reduction on the concatenated features to enhance the ranking process. Finally, we applied late fusion to combine the results from the previous steps, optimizing the identification of training data fingerprints. Figure 2: A block diagram of the sequential processing steps (a) and detailed schematic of the the developed system for identifying training data and detecting generative models’ fingerprints (b). The process in (b) begins with morphological operations to reduce noise in CT images, followed by using BLIP and DINOv2 for image signature generation. After concatenating and reducing the dimensionality of features using an encoder and PCA, late fusion is applied to optimize the identification of training data. 3.1. Morphological operations During image processing image opening is widely utilized for effective morphological operation. It primarily aimed to remove small objects from 3D axial CT images. But at the same time preserves the size and shape of larger structures. Which is an effective noise-releasing mechanism. As illustrated in Figure 3, electronic noise in CT images often originates from the combination of the detector system and the reconstruction kernel, with sharper kernels typically resulting in noisier images. According to [24], this noise is a consequence of efforts to enhance image quality without increasing the radiation dose. Our hypothesis is that applying image opening will eliminate small, noisy, and irrelevant details, thereby potentially enhancing the efficiency of fingerprint detection in medical images. By preserving key features, image opening maintains the integrity of essential image details, ensuring that the important structures remain intact while the noise is reduced. Image opening consists of two main steps erosion (Eq. 1) followed by dilation (Eq. 2). Image opening (3) can be represented as: 𝐴 ⊖ 𝐵 = {𝑧 ∈ 𝐸|𝐵𝑧 ⊆ 𝐴} (1) ⋃︁ 𝐴⊕𝐵 = 𝐴𝑏 (2) 𝑏∈𝐵 𝐴 ∘ 𝐵 = (𝐴 ⊖ 𝐵) ⊕ 𝐵 (3) Here in equation (1) above image erosion has been performed, where 𝐴 and 𝐵 are input image and structuring elements respectively. And 𝐵𝑧 is the translation of 𝐵 by 𝑧. Then in equation (2), image Figure 3: (a) Noisy CT images sampled from the ImageCLEF GAN dataset for task training image fingerprint identification; (b) Processed images after applying the image opening operation, which involves an erosion step followed by dilation. (c) Noisy CT images sampled from the ImageCLEF GAN dataset for the task of generative model fingerprint identification; (d) Processed images after applying the image opening operation for task 2, which also involves an erosion step followed by dilation. dilation is performed where 𝐴𝑧 is the translation of 𝐴 by 𝑧. In the equation (3), # is the image opening operation. 3.2. Multiformer To build a multiformer we have chosen two foundation models which are BLIP and DINOv2 as backbone architecture, see Fig. 4, for which details are given below. 3.2.1. BLIP Architecture Bootstrapping Language-Image Pre-training 2 (BLIP2) [25, 26] utilize a Visual Transformer (ViT) [27] for image encoding, dividing images into patches and encoding them into a sequence of embeddings with a [CLS] token representing the global image feature. This method is computationally efficient compared to traditional object detectors. BLIP is a multimodal Mixture of Encoder-Decoder (MED), operates in three modes: unimodal encoder [28] (similar to BERT for text), image-grounded text encoder (incorporates visual information via cross-attention layers), and image-grounded text decoder (uses causal self-attention layers for text generation). During pre-training, we jointly optimize three objectives: two for understanding and one for generation. Each image-text pair undergoes one forward pass through the ViT and three through the text transformer for different tasks. The Image-Text Contrastive Loss (ITC) inspired from [29, 30] to align visual and textual feature spaces, improving vision-language understanding by encouraging positive image-text pairs to have similar representations. Figure 4: A thematic block diagram of multiformer leveraging the powerful architecture of BLIP2 and DINOv2. The Image-Text Matching Loss (ITM) [30] learns fine-grained multimodal representations through a binary classification task to determine whether image-text pairs are matched, utilizing a hard negative mining strategy. The Language Modeling Loss (LM) trains the model to generate textual descriptions from images using cross-entropy loss, enhancing the model’s capability to convert visual information into coherent captions. To maximize efficiency and leverage multi-task learning, the text encoder and decoder share parameters except for the self-attention layers, which capture the differences between encoding and decoding tasks. This shared architecture benefits from improved training efficiency and effective multi-task learning. The BLIP2 feature extractor is a multi-modal model designed to extract and integrate features from both images and text. It begins with patch embedding for images, converting each image 𝑋 into patches 𝑃 using a convolutional layer, 𝑃 = Conv(𝑋). These patches are then processed through a transformer encoder, 𝐹𝑣 = Transformer𝑣 (𝑃 ), to capture visual relationships. Image data 𝐼 is tokenized and embedded into high-dimensional space, 𝐸𝑡 = Embedding(𝐼), and passed through another transformer encoder, 𝐹𝑡 = Transformer𝑡 (𝐸𝑡 ). The cross-modal attention mechanisms enhance its performance to state of the art, 𝐹𝑣𝑡 = CrossAttention(𝐹𝑣 , 𝐹𝑡 ), are used to align and integrate visual features. The model is pre-trained on large datasets with paired images and text to learn these representations. The pre-trained model can be fine-tuned for downstream tasks to improve performance on specific datasets. The BLIP2 feature extractor thus provides robust, high-level features suitable for tasks such as image classification, object detection, and text-image matching. 3.2.2. DINOv2 Architecture DINOv2 [31] is an enhanced version of DINO [32], integrating various improvements and using a larger, more diverse dataset to accelerate and stabilize training at scale. It utilizes the LVD-142M curated dataset, which includes data from sources like ImageNet [33], Google Landmarks [34], Mapillary SLS [35], and Food-101 [36]. The dataset is de-duplicated using FAISS [37] batch searches and embeddings. Training combines DINO and iBOT losses with SwAV centering, involving a learnable student and an EMA teacher. Key techniques include multi-crop cross-entropy for global image representation, patch-level masking, and separate weights for image and patch objectives. The teacher’s softmax- centering is replaced by Sinkhorn-Knopp batch normalization, and the KoLeo regularizer ensures batch uniformity. High-resolution images are used towards the end of pre-training. DINOv2’s implementation features FlashAttention [38, 39] for efficiency, nested tensors from xFormers, stochastic depth, and mixed-precision PyTorch FSDP. Distillation [40, 41] into smaller models from a larger teacher model is also included. Ablation studies cover model selection, data curation strategies, model scaling, and loss objectives. DINOv2 achieves results comparable to weakly supervised text models like EVA- CLIP on ImageNet and performs better than SSL methods like Mugs, EsViT, and iBOT. It shows strong performance in domain generalization, image and video classification, instance recognition, and semantic segmentation. For depth estimation, DINOv2 uses a linear layer on frozen tokens and experiments with concatenation of ViT [27] layers/blocks and regression over a DPT decoder, achieving superior results on datasets like NYUd, KITTI, and SUN-RGBd. Qualitative results demonstrate effective semantic matching and foreground extraction using PCA on patch features. We utilized the DINO Vision Transformer because it is a sophisticated image processing model capable of handling and analyzing images through a systematic series of steps. Below is a description of how its architecture operates. Initially, the input image is divided into smaller patches, which are then embedded into a higher-dimensional space using a convolutional layer. This embedding captures the local features of each patch. These embedded patches pass through multiple nested tensor blocks, each consisting of several components: layer normalization to standardize inputs, memory-efficient attention mechanisms to focus on different parts of the image, and multi-layer perceptrons for feature refinement. Each block also includes layer scaling and dropout layers to improve training stability and prevent overfitting. The model incorporates 12 of these blocks, creating a deep network capable of learning complex image representations. After processing through all blocks, a final normalization layer is applied, followed by an identity layer that prepares the features for subsequent tasks. This architecture allows the DINO ViT to effectively learn and process detailed image features, making it suitable for various image processing applications. 3.3. Autoencoder for feature reduction Initially, we employed raw features for clustering to assign labels to the test set. Subsequently, we integrated robust features from both the BLIP and DINOv2 models. To enhance the clustering results, we implemented an Autoencoder to encode this extensive feature set into a more meaningful and reduced representation. This dimensionality reduction facilitated improved clustering performance. Autoencoders (AEs) have been quite popularised in AI research over recent years and consequently, there have been many studies and advancements made in this subject, There are mainly two types, they are Fundamental AEs And Variants. The simplest ones, as used in our working example, one can find, are auto-associative neural networks, connected to a multi-layer perceptron, where input is being reconstructed. The encoders consist of an encoder that compresses an input vector with recognition weights into a code vector and a decoder that decompresses the input vector from a code vector with generative weights. This structure allows each layer of a deep network to be trained separately using the basic AE as a building block. The encoder activation function (sf) may be, for example, sigmoid or hyperbolic tangent, a weight matrix (W) is a bias vector (b), and x Wine are used in the computation of the hidden representation of an input vector x with a 𝑦 = 𝑓Θ (𝑥) = 𝑠𝑓 (𝑊 𝑥 + 𝑏), where 𝑊 is a weight matrix, 𝑏 is a bias vector, and 𝑠𝑓 is the encoder activation function (e.g., sigmoid or hyperbolic tangent). The hidden representation 𝑦 is then decoded back to a reconstruction vector 𝑧 using 𝑧 = 𝑔Θ (𝑦) = 𝑠𝑔(𝑊 ′ 𝑦 + 𝑏′ ), where sg is the activation function of the decoder. The aim is to make the reconstruction( z ) as close to the input( x ) as possible. To simplify training, the weight matrix 𝑊 ′ is often constrained to be the transpose of 𝑊 (tied weights), reducing the number of free parameters. This mapping process ensures that each input is transformed into a hidden representation and then reconstructed, enabling effective learning and data representation. 4. Experimental Result Analysis 4.1. System Specification and Parameter Settings The proposed model is implemented on a Google Cloud Vertex AI instance, utilizing a NVIDIA V100 Tensor Core GPU. This GPU boasts 5120 CUDA cores, 640 Tensor cores, up to 32 GB of HBM2 memory, and a memory bandwidth reaching up to 900 GB/s. These specifications allow for highly efficient processing and accelerated computation, which are essential for handling the complex tasks and large datasets involved in our model’s execution. For the BLIP architecture, the parameters used make it better in feature extraction. The input image size was normalized at 256×256 pixels, followed by data augmentations of random cropping, horizontal flipping with probability 𝑝 = 0.5, rotation within ±20∘ , and color jittering with brightness adjustment factors [0.75, 1.25]. Feature Extraction used the pre-trained ViT large model as the backbone. The value chosen for the learning rate 𝜂 was 0.0005 and a batch size 𝐵 of 16 was employed. The AdamW optimizer where weight decay factor was added was used for loss function minimization 𝐿(𝜃) Normalization layers were used for scaling the features within the same scale of any particular signal for the dataset. For Dinov2 architecture, a number of key parameters were defined to maximally optimize their performance. The model was pre-trained on a large companys own well curated set of images, making it a great base for later fine-tuning. The size of the input image was kept uniform during training, 224 × 224 pixels, where several data augmentations like center cropping, random affine transformations, resizing, and normalizing, were performed. We used a learning rate 𝜂 of 0.001 and batch size (B) of 32. Adam optimizer with 𝛽1 = 0.9 and 𝛽2 = 0.999 was used to minimize the loss function 𝐿(𝜃) where 𝜃 denotes the model parameters. We applied a dropout with a probability of p = 0.5 and layer normalization to prevent overfitting and maintain training stability. For the Dinov2, BLIP we then tried to tune these parameters carefully to get as high performance as possible to identify generative model fingerprintfs and test image fingerprints. I executed these procedures while training the item co-occurrence model with data augmentation, learning rates, and optimization technique and achieved high quality and stable item embeddings. 4.2. Identify Training Data Fingerprints Experiments For identifying training data fingerprints, we first reduced noise in the CT images through morphological operations. We then used BLIP and DINOv2 as image signature generators. As shown in Figure 2, these steps improved the quality of synthetic medical images generated by GANs. We ranked features individually and after concatenation, performed dimensionality reduction, and used late fusion to refine our fingerprint identification results. For submission 1, we implemented the "additive mode thresholding" technique, which considers local variations in image intensity to enhance image processing. First, we reduced the dimension of the feature vector using Principal Component Analysis (PCA). We then combined all the features and weighted them by the total. The mode of this final weighted result was used as the threshold. For the test images, we applied a similar weighting approach: if the weighted value was less than the mode, the image was tagged as not used; otherwise, it was tagged as used. This method allowed us to account for local intensity variations, improving the accuracy of our thresholding. In submission 2, which we titled "additive average thresholding," we took a different approach. We calculated the final result for each subject and then averaged these results across all subjects. This average became the threshold value for classification. By using the average, we aimed to create a more generalized threshold that could effectively classify the images based on the overall distribution of the data. For submission 3, we used an encoder model to handle the extensive feature set generated by the backbone models. The encoder compressed this concatenated feature set, reducing its dimensionality. With the reduced feature set, we applied both mode and mean thresholding techniques. This dual approach allowed us to leverage the strengths of both thresholding methods, providing a robust classification mechanism. In submission 5, we employed a late fusion strategy to combine the decisions from the previous four methods. Late fusion involves aggregating the results at the decision level rather than at the feature level. We used majority voting to finalize the classification, ensuring that the combined decisions of the different methods provided a more accurate and reliable result. This ensemble approach helped to miti- gate the weaknesses of individual methods and improved the overall performance of our classification system. For the final submission 6, we performed reranking using the Agglomerative Clustering algorithm. This algorithm conducts hierarchical clustering with a bottom-up approach, allowing us to specify parameters such as the number of clusters, distance metric, and linkage criterion. The reranking was based on decisions from the previous submissions. 4.3. Identify Generative Model Fingerprints Experiemnts To identify generative models’ fingerprints, we initially reduced noise in the CT images using mor- phological operations. We then employed the pre-trained BLIP2 and DINOv2 architectures for feature extraction. As illustrated in Figure 2, our objective was to accurately label each subject with the corresponding model number, determining which generative or diffusion model produced each image. For submissions 1 and 2, we utilized a combination of feature sets from BLIP and DINOv2 titled ’multiformer’ architecture with different augmentation techniques. In submission 1, we applied center cropping and random affine transformations, along with resizing, normalizing, and other standard setups. In submission 2, we used random cropping, random horizontal flipping, random rotation, and color jittering. These augmentations introduced variations in size, orientation, and brightness to the training dataset, enhancing the model’s robustness and accuracy. We then used k-means and agglomerative clustering to assign labels to each subject. Submissions 3 and 4 followed a similar feature extraction method, but we applied PCA and autoencoder was applied for combined feature dimensionality reduction, respectively. The same clustering algorithms were then used for label assignment. In submissions 5 and 6, we leveraged solely the BLIP architecture. Submission 5 used a BLIP base model for feature extraction, while submission 6 utilized the BLIP pre-trained ViT large model. The normalized feature sets were subsequently fed into clustering algorithms for labeling. For submissions 7 and 8, we performed ensemble voting and reranking based on the decisions from previous submissions. Ensemble voting combined results at the decision level rather than the feature level, employing majority voting to determine the final classification, ensuring a more accurate and reliable outcome. For reranking, we applied Density-Based Spatial Clustering (DBSCAN). This algorithm identifies clusters by ensuring each point within a cluster has a neighborhood defined by a specified radius, containing at least a minimum number of points, thereby separating dense regions from areas with fewer points. 4.4. Results and Discussions The results presented in Table 1 show the metrics for datasets 1 and 2, referred to as db1 and db2, for the task of identifying training data fingerprints. Initially, we applied morphological opening operations, which first erode the image and then dilate the eroded image using the same structuring element. After performing these operations, we passed the images to our vision transformer models for further processing. Table 1 Submission Results for the Task 1: Identifying Training Data Fingerprints. Dataset 1 (db1) Dataset 2 (db2) Submission ID Approach Accuracy Precision Recall F1-score Accuracy Precision Recall F1-score 1 Additive Mode Thresholding 0.502 0.502 0.489 0.495 0.491 0.491 0.487 0.489 2 Additive Mean Thresholding 0.504 0.504 0.515 0.509 0.499 0.499 0.483 0.491 3 Autoencoder Mode 0.492 0.492 0.486 0.489 0.5 0.5 0.509 0.504 4 Autoencoder Mean 0.495 0.495 0.512 0.503 0.497 0.497 0.496 0.496 5 Reranking 0.51 0.511 0.449 0.478 0.495 0.494 0.436 0.463 6 Reranking v2 0.495 0.495 0.485 0.49 0.506 0.506 0.45 0.476 Submission 1 utilized the Dinov2 model with additive mode thresholding. This model employs task-agnostic and cognitive approaches through self-supervised learning and features a multipurpose backbone derived from a pre-trained, extensive, and well-curated image set, powered by a visual transformer model. Additive mode thresholding was then used to assign labels to each image. This method achieved (shown in Table 1) an accuracy and precision of 0.5025 for dataset 1, and an accuracy and F1 score of 0.491 for dataset 2. As shown in Table 2 the overall accuracy was 0.492. Submission 2 used the Blip architecture with additive average thresholding. This model employs an unimodal decoder to extract pattern signatures from images. Subsequently, additive average thresholding was applied to assign labels to each image. This approach resulted in an accuracy and precision of 0.50425 for dataset 1, and an accuracy and F1 score of 0.4995 for dataset 2. As shown in 2.As shown in Table 2 the overall accuracy for submission 2 was 0.501875. Submissions 3 and 4 are based on the concatenated multiformer feature fusion. We subsequently applied PCA and autoencoder for dimensionality reduction to both feature sets derived by the Dinov2 and Blip models. This approach yielded our best results, with submission 4 achieving an accuracy and precision of 0.49575 for dataset 1. For dataset 2, the highest accuracy and F1 score of 0.5005 were attained, making it the best among these submissions. As shown in Table 2 the overall accuracy for submissions 3 and 4 was 0.496357 and 0.4957 respectively. Table 2 Overall Submission Results for the Identifying Training Data Fingerprints task Submission Identify Identify Identify Identify training data training data training data training data "fingerprints"- "fingerprints"- "fingerprints"- "fingerprints"- Accuracy Precision Recall F1-score 1 0.492 0.497 0.497 0.497 2 0.5 0.501875 0.501875 0.501875 3 0.496 0.496375 0.496375 0.496375 4 0.5 0.4957 0.4957 0.4957 5 0.47 0.500875 0.500875 0.500875 6 0.483 0.502625 0.502625 0.502625 In the final two submissions, 5 and 6, we employed a reranking technique. Submission 6 provided the highest accuracy for dataset 1, reaching 0.51. Meanwhile, for dataset 2, submission 5 achieved the best accuracy at 0.506. As shown in Table 2 the overall accuracy for submissions 5 and 6 was 0.500875 and 0.502625 respectively. Moreover for Task 2, we investigated the intriguing notion that generative models might leave distinct marks on the images they create. Our goal was to determine whether different models have unique "fingerprints" within the synthetic images they produce. By closely examining these images, we aimed to uncover the specific characteristics that define each model’s output. This time the author of the ImageCLEFmed GANs [8] provided results are represented by the Adjusted Rand Index (ARI), which measures the similarity between two clusters. The ARI improves upon the Rand Index by considering the likelihood of chance agreements between clusters, thus enhancing its reliability. ∑︀ (︀𝑎𝑖𝑗 )︀ [︁∑︀ (︀𝑎𝑖. )︀ ∑︀ (︀𝑎.𝑗 )︀]︁ (︀𝑛)︀ 𝑖𝑗 2 − 𝑖 2 𝑗 2 / 2 ARI = [︁∑︀ (︀ )︀ ∑︀ (︀ )︀]︁ [︁∑︀ (︀ )︀ ∑︀ (︀ )︀]︁ (︀ )︀ (4) 1 𝑎𝑖. 𝑎.𝑗 𝑎𝑖. 𝑎.𝑗 𝑛 2 𝑖 2 + 𝑗 2 − 𝑖 2 𝑗 2 / 2 Index−Expected Index As shown in Eqn (4) we can see that the ARI is calculated with the formula: ARI = Max Index−Expected Index . Here, the Index represents the raw agreement index, which counts pairs of elements that are either in the same or different clusters in both the true and predicted clusterings. The Expected Index accounts for the expected value of the raw index if the cluster assignments were random, while the Max Index is the maximum value of the raw index, indicating perfect clustering. The ARI ranges from -1 to 1, where 1 signifies perfect agreement, 0 indicates a random clustering, and -1 suggests no agreement. The formula leverages a contingency table and binomial coefficients to adjust the Rand Index for change, providing a more accurate measure of clustering similarity. Table 3 Overall Submission Results for the Generative Model Fingerprint Detection Task Submission ARI Score 1 0.8137499357777883 2 0.9000159097044281 3 0.26753081555895303 4 0.36560477207139175 5 0.0013132463035679 6 0.0017768435 7 0.1785452554 8 0.2323909988 Among our total of eight submissions shown in Table 3 Submission 2 stood out with the highest ARI score of 0.900, demonstrating a strong agreement between the predicted clustering and the ground truth, and showcasing its exceptional performance in data clustering. In contrast, Sub5 and Sub6 recorded very low ARI scores of 0.001 and 0.002, respectively, indicating poor alignment between the predicted clusters and the actual data. Our analysis of the results revealed a spectrum of scores for each submission, illustrating how well they matched the real data. While submissions like Sub2 performed exceptionally well, others such as Sub5 and Sub6 fell short of expectations. Overall, our findings highlight the distinctive marks generative models leave on the images they produce, which could aid in recognizing and attributing these images to specific models in the future. 5. Conclusions In the dynamic field of medical image processing, ImageCLEFmed GANs has launched a pioneering initiative with the Identifying GAN Fingerprint task, alongside the task of detecting generative model fingerprints. In this paper, we tackled the first task by proposing six approaches, utilizing additive thresholding, autoencoder, and reranking techniques to classify images as used or not used for generating synthetic images. To address the task of detecting generative models’ fingerprints, we implemented eight approaches involving Dinov2, Blip, and ensemble feature fusion. These findings highlight the significance of our efforts in advancing medical image analysis techniques. Moving forward, we plan to apply advanced noise-removing techniques to leverage pixel-level connectivity. Additionally, we aim to develop a unified framework that integrates classical and transformer-based architectures to enhance our ability to detect imprint signatures. 6. Acknowledgments This work was supported by the National Science Foundation (NSF) grant (ID. 2131307) “CISE-MSI: DP: IIS: III: Deep Learning-Based Automated Concept and Caption Generation of Medical Images Towards Developing an Effective Decision Support”. References [1] B. Ionescu, H. Müller, A. Drăgulinescu, J. Rückert, A. Ben Abacha, A. Garcıa Seco de Herrera, L. Bloch, R. Brüngel, A. Idrissi-Yaghir, H. Schäfer, C. S. Schmidt, T. M. Pakull, H. Damm, B. Bracke, C. M. Friedrich, A. Andrei, Y. Prokopchuk, D. Karpenka, A. Radzhabov, V. Kovalev, C. Macaire, D. Schwab, B. Lecouteux, E. Esperança-Rodier, W. Yim, Y. Fu, Z. Sun, M. Yetisgen, F. Xia, S. A. Hicks, M. A. Riegler, V. Thambawita, A. Storås, P. Halvorsen, M. Heinrich, J. Kiesel, M. Potthast, B. Stein, Overview of ImageCLEF 2024: Multimedia retrieval in medical applications, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction, Proceedings of the 15th International Conference of the CLEF Association (CLEF 2024), Springer Lecture Notes in Computer Science LNCS, Grenoble, France, 2024. [2] A. Andrei, A. Radzhabov, D. Karpenka, Y. Prokopchuk, V. Kovalev, B. Ionescu, H. Müller, Overview of 2024 ImageCLEFmedical GANs Task – Investigating Generative Models’ Impact on Biomedical Synthetic Images, in: CLEF2024 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Grenoble, France, 2024. [3] S. Xun, D. Li, H. Zhu, M. Chen, J. Wang, J. Li, M. Chen, B. Wu, H. Zhang, X. Chai, Z. Jiang, Y. Zhang, P. Huang, Generative adversarial networks in medical image segmentation: A review, Computers in Biology and Medicine 140 (2022) 105063. URL: https://www.sciencedirect.com/science/article/ pii/S001048252100857X. doi:https://doi.org/10.1016/j.compbiomed.2021.105063. [4] A. Anantatamukala, K. M. Krishna, N. B. Dahotre, Generative adversarial networks assisted machine learning based automated quantification of grain size from scanning electron micro- scope back scatter images, Materials Characterization 206 (2023) 113396. URL: https://www. sciencedirect.com/science/article/pii/S1044580323007556. doi:https://doi.org/10.1016/j. matchar.2023.113396. [5] A. Makhlouf, M. Maayah, N. Abughanam, C. Catal, The use of generative adversarial networks in medical image augmentation, Neural Comput. Appl. 35 (2023) 24055–24068. [6] H. Tang, H. Liu, D. Xu, P. H. S. Torr, N. Sebe, Attentiongan: Unpaired image-to-image translation using attention-guided generative adversarial networks, CoRR abs/1911.11897 (2019). URL: http: //arxiv.org/abs/1911.11897. arXiv:1911.11897. [7] D. Mindlin, M. Schilling, P. Cimiano, Abc-gan: Spatially constrained counterfactual generation for image classification explanations, in: L. Longo (Ed.), Explainable Artificial Intelligence, Springer Nature Switzerland, Cham, 2023, pp. 260–282. [8] A. Andrei, A. Radzhabov, I. Coman, V. Kovalev, B. Ionescu, H. Müller, Overview of ImageCLEFmed- ical GANs 2023 task – Identifying Training Data "Fingerprints" in Synthetic Biomedical Images Generated by GANs for Medical Image Security, in: CLEF2023 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece, 2023. [9] B. Ionescu, H. Müller, A. Drăgulinescu, W. Yim, A. Ben Abacha, N. Snider, G. Adams, M. Yetisgen, J. Rückert, A. Garcıa Seco de Herrera, C. M. Friedrich, L. Bloch, R. Brüngel, A. Idrissi-Yaghir, H. Schäfer, S. A. Hicks, M. A. Riegler, V. Thambawita, A. Storås, P. Halvorsen, D. J. A. A. A. R. I. C. V. K. A. S. G. I. Nikolaos Papachrysos, Johanna Schöler, H. Manguinhas, L. Ştefan, M. G. Constantin, M. Dogariu, J. Deshayes, A. Popescu, Overview of ImageCLEF 2023: Multimedia retrieval in medical, socialmedia and recommender systems applications, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction, Proceedings of the 14th International Conference of the CLEF Association (CLEF 2023), Springer Lecture Notes in Computer Science LNCS, Thessaloniki, Greece, 2023. [10] C. Jang, Y. Lee, Y.-K. Noh, F. C. Park, Geometrically regularized autoencoders for non-euclidean data, in: The Eleventh International Conference on Learning Representations, 2023. URL: https: //openreview.net/forum?id=_q7A0m3vXH0. [11] T. Rhodes, T. Bhattacharjee, D. D. Lee, Learning from demonstration using a curvature regularized variational auto-encoder (curvvae), in: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022, pp. 10795–10800. doi:10.1109/IROS47612.2022.9981930. [12] J. Z. Kim, N. Perrin-Gilbert, E. Narmanli, P. Klein, C. R. Myers, I. Cohen, J. J. Waterfall, J. P. Sethna, 𝛾-vae: Curvature regularized variational autoencoders for uncovering emergent low dimensional geometric structure in high dimensional data, 2024. arXiv:2403.01078. [13] D. Ahn, H. Cho, J. Min, W. Jang, J. Kim, S. Kim, H. H. Park, K. H. Jin, S. Kim, Self-rectifying diffusion sampling with perturbed-attention guidance, 2024. arXiv:2403.17377. [14] Y. Skandarani, P.-M. Jodoin, A. Lalande, Gans for medical image synthesis: An empirical study, 2021. arXiv:2105.05318. [15] W. Ahmad, H. Ali, Z. Shah, S. Azmat, A new generative adversarial network for medical images super resolution, Scientific Reports 12 (2022). URL: http://dx.doi.org/10.1038/s41598-022-13658-4. doi:10.1038/s41598-022-13658-4. [16] W. Zhang, W.-K. Cham, Hallucinating face in the dct domain, IEEE Transactions on Image Processing 20 (2011) 2769–2779. doi:10.1109/TIP.2011.2142001. [17] C. Baur, S. Albarqouni, N. Navab, Generating highly realistic images of skin lesions with gans, in: OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis: First International Workshop, OR 2.0 2018, 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, Third International Workshop, ISIC 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16 and 20, 2018, Proceedings 5, Springer, 2018, pp. 260–267. [18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, Advances in neural information processing systems 27 (2014). [19] A. F. Frangi, S. A. Tsaftaris, J. L. Prince, Simulation and synthesis in medical imaging, IEEE transactions on medical imaging 37 (2018) 673–679. [20] C. Wang, G. Yang, G. Papanastasiou, S. Tsaftaris, D. Newby, C. Gray, G. Macnaught, T. MacGillivray, Dicyc: Gan-based deformation invariant cross-domain information fusion for medical image synthesis, Information Fusion 67 (2021) 147–160. doi:10.1016/j.inffus.2020.10.015. [21] P. Perona, J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Transactions on pattern analysis and machine intelligence 12 (1990) 629–639. [22] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, G. Langs, Unsupervised anomaly detection with generative adversarial networks to guide marker discovery, in: International conference on information processing in medical imaging, Springer, 2017, pp. 146–157. [23] M. J. Chuquicusma, S. Hussein, J. Burt, U. Bagci, How to fool radiologists with generative adversarial networks? a visual turing test for lung cancer diagnosis, in: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), IEEE, 2018, pp. 240–244. [24] M. Diwakar, M. Kumar, Ct image noise reduction based on adaptive wiener filtering with wavelet packet thresholding, 2014 International Conference on Parallel, Distributed and Grid Computing (2014) 94–98. URL: https://api.semanticscholar.org/CorpusID:15070114. [25] J. Li, D. Li, C. Xiong, S. C. H. Hoi, Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, in: International Conference on Machine Learning, 2022. URL: https://api.semanticscholar.org/CorpusID:246411402. [26] J. Li, D. Li, S. Savarese, S. Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. arXiv:2301.12597. [27] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, 2021. arXiv:2010.11929. [28] J. Devlin, M. Chang, K. Lee, K. Toutanova, Pre-training of deep bidirectional transformers for language understanding, Minneapolis, MN: Association for Computational Linguistics (2019) 4171–4186. [29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from nat- ural language supervision, in: International Conference on Machine Learning, 2021. URL: https://api.semanticscholar.org/CorpusID:231591445. [30] J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, S. C. H. Hoi, Align before fuse: Vision and language representation learning with momentum distillation, in: Neural Information Processing Systems, 2021. URL: https://api.semanticscholar.org/CorpusID:236034189. [31] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, P. Bojanowski, Dinov2: Learning robust visual features without supervision, 2024. arXiv:2304.07193. [32] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, A. Joulin, Emerging properties in self-supervised vision transformers, 2021. arXiv:2104.14294. [33] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, IEEE, 2009, pp. 248–255. URL: https://ieeexplore.ieee.org/abstract/document/5206848/. [34] T. Weyand, A. Araujo, B. Cao, J. Sim, Google landmarks dataset v2 – a large-scale benchmark for instance-level recognition and retrieval, 2020. arXiv:2004.01804. [35] F. Warburg, S. Hauberg, M. Lopez-Antequera, P. Gargallo, Y. Kuang, J. Civera, Mapillary street-level sequences: A dataset for lifelong place recognition, in: Proceedings of IEEE Con- ference on Computer Vision and Pattern Recognition 2020, IEEE, United States, 2020. URL: http://cvpr2020.thecvf.com/,https://ieeexplore.ieee.org/xpl/conhome/9142308/proceeding, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020 ; Conference date: 14-06-2020 Through 19-06-2020. [36] L. Bossard, M. Guillaumin, L. Van Gool, Food-101 – mining discriminative components with random forests, in: European Conference on Computer Vision, 2014. [37] M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, H. Jégou, The faiss library, 2024. arXiv:2401.08281. [38] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré, Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. arXiv:2205.14135. [39] T. Dao, Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. arXiv:2307.08691. [40] N. Reimers, I. Gurevych, Making monolingual sentence embeddings multilingual using knowledge distillation, 2020. arXiv:2004.09813. [41] N. Brown, A. Williamson, T. Anderson, L. Lawrence, Efficient transformer knowledge distillation: A performance review, 2023. arXiv:2311.13657.