Our team's primary contribution to the ImageCLEFmedical GANs 2024 task is as follows. This task aims to assess whether synthetic medical images generated by Generative Adversarial Networks (GANs) contain identifiable features from the training data. We employed a similarity-based classification method, categorizing real images based on their similarity to generated images. In this work, we utilized various similarity calculation methods to evaluate the similarity between real and generated images. We calculated the similarity for original images, noisy images, and features extracted through a feature network. On the validation dataset, our similarity-based approach achieved an F1-score of 0.732 and an accuracy of 0.760. In the submitted results, our best F1-score was 0.598. Our experimental results demonstrated that our method could distinguish between images that were “used” and those that were “not used”.
Deep learning models have significant potential in supporting medical diagnosis and treatment,
achieving remarkable results in various medical image analysis tasks. However, training these models requires
vast amounts of data, which is often challenging to obtain. Deep generative models, capable of
generating highly realistic medical images, have been used to create large synthetic datasets to facilitate model
training[
Nevertheless, since generative models model the probability distribution of the data, synthetic images produced by these models may threaten the privacy of patient images used in training. Recent studies have shown that medical images, such as chest X-rays and MRI scans, can be used to re-identify patients, exacerbating concerns about privacy breaches.
To identify potential privacy threats associated with the use and sharing of synthetic medical data
in real-world scenarios, a new challenge has been introduced as part of the ImageCLEFmedical CLEF
challenge 2024[
The generated images are produced by learning the data distribution of real images, meaning that
the synthetic images are statistically similar to the real ones. The closer the data distribution of the
generated images is to the real images, the higher the quality of the generated images, making them
visually closer to real images. In this work, our task is to perform binary classification to categorize
images as used or not used. To achieve this goal, we calculate similarity scores to determine the category
of the images, classifying images with high similarity scores[
We employed three diferent methods to calculate similarity. First, we directly computed the similarity between the original generated images and real images by comparing their pixel values. Second, we applied noise to the original images and then calculated the similarity between the noisy images and the real images, which enhanced the robustness of the images. Finally, we extracted features from the images using advanced deep learning models to obtain high-dimensional features and then calculated the similarity between these features. These features contain deep information about the images, providing a more accurate reflection of the similarity between images.
By using these three methods, we were able to comprehensively evaluate the similarity between generated and real images, efectively accomplishing the binary classification task. This multi-method approach not only improved the accuracy of the classification but also provided richer information and stronger guarantees for image processing and analysis.
Synthetic images[
Nataraj et al. proposed a novel method combining co-occurrence matrices and deep learning techniques to detect GAN-generated fake images. The authors extracted co-occurrence matrices from the three color channels in the pixel domain and trained a deep convolutional neural network (CNN) model. The method demonstrated good generalization capability when trained on one dataset and tested on another.
GANs[
In conclusion, previous studies have demonstrated the potential of GANs in generating synthetic medical images and performing image-to-image translation tasks. However, distinguishing between synthetic and real medical images remains an active area of research, requiring robust methods to ensure the reliability and integrity of generated data.
To conduct a statistical analysis of the images, we categorized them into three groups: generated images, used images, and not used images. It was observed that the similarity between the images was significantly high. To validate the similarity between the data, we employed the Three-Component Weighted Structural Similarity Index (3-SSIM) to calculate the similarity. 3-SSIM is an improved version of SSIM (Structural Similarity Index). Unlike SSIM, which compares the entire image as a whole, 3-SSIM evaluates the similarity in edge, texture, and smooth regions separately, assigning diferent weights to these components to obtain the final assessment result.
The Three-Component Weighted Structural Similarity Index calculates the similarity between images by separately evaluating the edge information, texture areas, and smooth regions, and then assigns diferent weights to these components. The final similarity score is obtained by summing these weighted values. We performed statistical analysis on three sets of data: the similarity between real images (real-real), the similarity between generated and real images (generated-real), and the similarity between generated images (generated-generated).
The experimental results presented in Table 1 reveal that the similarity between generated images is higher than the similarity between generated and real images, which in turn is higher than the similarity between real images. This is because the generated images are produced by the same model, resulting in relatively higher similarity. The similarity between generated and real images is slightly lower due to the inclusion of two types of images; the used images likely have a higher similarity to the generated images, while the not used images have a lower similarity. The similarity values for real-real are slightly lower, but the diference is not significant. When calculating the data in the table, the self-comparison similarity was excluded as it is identical and has no practical significance. Therefore, the maximum similarity in real-real and gen-gen is not 30.
Similarity calculation methods play a crucial role in image processing, machine learning, and information retrieval. This paper provides a detailed introduction to several common similarity calculation methods, including Euclidean distance, cosine similarity, and Structural Similarity Index (SSIM). Similarity calculation is used to measure the degree of similarity between two objects, such as images, feature vectors, or strings. These methods are widely applied in image classification, clustering, retrieval, and recommendation systems. Each method has its own applicable scenarios and advantages, making the choice of an appropriate similarity calculation method critical to the performance and efectiveness of algorithms. As shown in Figure 1, this paper presents an illustration of diferent image similarity calculations.
When calculating image similarity, Euclidean distance is a commonly used method that measures the diference between two images at the pixel level. This method assumes that images can be represented as points in a high-dimensional space, where each pixel’s color value (typically RGB values) is considered a dimension in this space. Euclidean distance calculates the straight-line distance between these two points (i.e., the two images), with a smaller distance indicating higher similarity between the images. First, it is essential to ensure that the two images have the same dimensions; in this task, the generated images and the real images are of identical sizes. The formula for calculating the Euclidean distance is as follows.
⎯⎸ (A, B) = ⎷⎸∑︁ ( − )2
=1 Where A and B are two n-dimensional vectors.
Cosine similarity is another commonly used method for measuring image similarity, particularly in content-based image retrieval (CBIR) systems. Cosine similarity assesses the similarity between two vectors by measuring the cosine of the angle between them. The core idea is that the closer the directions of the two vectors, the more similar they are, regardless of their magnitudes. Calculating cosine similarity involves converting each image into a vector form. This typically entails flattening the pixel values of the image or features extracted from the image (such as color histograms, texture descriptors, shape features, etc.) into a one-dimensional vector. The cosine similarity value ranges from -1 to 1, where 1 indicates identical directions (very similar), 0 indicates orthogonality (no similarity), and -1 indicates completely opposite directions. Cosine similarity focuses on directional similarity, ignoring magnitude. In some cases, two images might be very similar in terms of certain feature ratios, but the absolute diferences in actual pixel values could be significant.
cos( ) =
A · B ‖A‖B‖ = √︁∑︀=1 2 √︁∑︀=1 2 ∑︀=1
Where A and B are two vectors. A · B represents the dot product of vectors A and B. ||A|| and ||B|| represent the magnitudes of vectors A and B.
Structural Similarity (SSIM) is a more intuitive and efective method for calculating image similarity. SSIM considers the luminance, contrast, and structural information of images, which allows it to more accurately reflect the human visual system’s perception of image quality. SSIM first calculates the luminance diference between two images. The luminance comparison is achieved by calculating the mean values of the images, which reflects the overall brightness levels of the images. Next, SSIM calculates the contrast diference. The contrast comparison is achieved by calculating the standard deviation of the images; the greater the standard deviation, the higher the image contrast. Finally, SSIM compares the structural information of the two images. This step is achieved by calculating the covariance of the images. Covariance reflects the linear relationship between the pixels of the images, capturing the structural characteristics of the images.
SSIM(, ) =
(2 + 1) (2 + 2) ︀( 2 + 2 + 1)︀ (︀ 2 + 2 + 2)︀
Where x and y are corresponding blocks of the two images. and are the mean values of image blocks x and y. and are the standard deviations of image blocks x and y. is the covariance of image blocks x and y.
Noise can significantly afect the quality of images, thereby impacting the results of similarity calculations. Common types of noise include Gaussian white noise and salt-and-pepper noise.
Gaussian White Noise: This type of noise follows a normal distribution, typically with a mean of zero, and the standard deviation can be set according to the actual situation. Gaussian noise adds a random value to each pixel of the image, resulting in an overall blurring efect.
Salt-and-Pepper Noise: This type of noise randomly changes image pixels to either white (255) or black (0) with a certain probability, commonly occurring during image transmission. Salt-and-pepper noise creates random white or black spots in the image, making it appear speckled and unclear. As shown in Figure 2, similarity calculation is performed on the two images with added noise
Experiments show that noise significantly impacts image similarity values. After adding noise, the similarity between images decreases notably, efectively expanding the diferences between images. Gaussian white noise and salt-and-pepper noise degrade image quality in diferent ways, thus afecting the results of similarity calculations. As a method for measuring image similarity, it is crucial to consider the impact of noise on the results and to apply appropriate noise reduction measures in practice to improve the accuracy of similarity calculations.
We carried out assessments across two distinct experiments. Initially, we split the validation set into two equal portions, designating one half as the test set to streamline our experimental analysis. Subsequently, we submitted our findings to the IMAGECLEFMED GANS 2024: IDENTIFY TRAINING DATA FINGERPRINTS competition. This challenge was addressed as a binary classification issue, and its evaluation criteria encompassed several critical performance metrics: F1-score, accuracy, precision, recall, and specificity. Notably, the F1-score has been chosen as the principal metric for this year’s evaluation. The definitions of these metrics are as follows:
Precision =
Recall =
+
+ Specificity =
+ (1) (2) (3) Accuracy =
+ + + + F1-score =
Precision · Recall Precision + Recall (4) (5)
In this experiment, we conducted tests on both the original images and images with added noise. We used Euclidean distance, cosine similarity, and Structural Similarity Index (SSIM) to calculate the similarity between images. Based on the similarity scores, we performed classification to obtain accuracy, precision, recall, and F1-score.
To highlight the diferences between images, we introduced varying degrees of noise into the images for the experiments. Specifically, we added diferent levels of noise, including Gaussian noise and salt-and-pepper noise, to assess their impact on similarity scores and classification performance. This approach helped us understand how noise degrades image quality and afects similarity calculations.
By systematically introducing diferent noise levels, we were able to evaluate the efectiveness and stability of diferent similarity calculation methods under various noise conditions. This comprehensive experimental design allowed us to more accurately measure the impact of noise on image similarity calculations, providing valuable insights for future image processing and analysis improvements. The experimental results, summarized in the table 2 illustrate the performance metrics of each similarity calculation method under diferent noise levels.
We made predictions for all 4,000 images generated by each model and submitted these prediction results. To evaluate the performance of the models, we used the F1-score as the primary evaluation metric, as it comprehensively considers both precision and recall, providing a more holistic assessment of performance. Additionally, we used accuracy as a secondary metric to measure the correctness of the model across all predictions. This evaluation process ensured that we could fully understand the performance of the models under diferent conditions. We submitted a total of eight diferent results. The detailed scores are summarized in the table 3, showing the specific performance and corresponding evaluation scores for each submission. These results help us to further analyze and improve the models, enhancing their efectiveness in practical applications.
As shown in Table 3, our three best experimental results are presented. Through our experiments, we found that adding noise significantly increases the diferences between images. We performed similarity calculations on the images with added noise and classified them based on these similarity scores. Notably, using the SSIM similarity calculation method yielded the best results on the test set.
In this study, we employed various similarity calculation methods to classify images. By setting similarity thresholds, we classified the images based on the similarity between real images and generated images. Diferent similarity calculation methods were used to evaluate the similarity between real and generated images. We performed similarity calculations on both the original images and the images with added noise. Next, we will investigate calculating similarity based on extracted features. When calculating feature similarity, it is crucial to ensure that the features contain more detailed information to capture the subtle diferences between diferent images. latent difusion models, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10684–10695. [12] J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.