ImageCLEF, a platform dedicated to the evaluation and advancement of visual media analysis, has conducted a novel challenge in its medical track titled Controlling the Quality of Synthetic Medical Images created via GANs. This task investigates the hypothesis whether Generative Adversarial Networks (GANs) generated synthetic images retain distinct characteristics indicative of the real images used for training. This research aims to detect these characteristics and analyze the relationship between real and artificial biomedical image datasets. The task focuses on analyzing test image datasets to identify the presence of real images in the training process. The development dataset includes both artificial and real images of lung tuberculosis patients. Our team submitted two methods for evaluation: a CNN model achieved an F1 score of 0.4804, while the SIFT-KNN approach obtained an F1 score of 0.449. In addition to the models stated above, the pHash approach was applied which enabled to map real images to their corresponding synthetic counterparts. These findings provide valuable insights into detecting real image characteristics in synthetic biomedical datasets, highlighting the importance of addressing security concerns when employing GANs for medical image generation.
ImageCLEF is an evaluation forum organized annually that encompasses research tasks oriented
towards image analysis and cross-language annotation. ImageCLEF 2023 [
Since the discovery of x-rays, medical imaging has achieved significant advancements in
medicine and fundamentally revolutionised the way diseases are identified and treated[
Deep learning [
For the ImageCLEF 2023, for the task Controlling the Quality of Synthetic Medical Images created via GANs , the organizers shared us the benchmarking image set collected from 8000 lung tuberculosis patients. The imaging modality is CT, and the dataset comprises of axial slices of 3D CT scans obtained from those patients. Amongst them, there are normal and as well with lung lesions, or even more serious stages. The images are of 256×256 pixels and are in PNG ifle format. Difuse neural networks are employed in the generation of synthetic images which are of same resolution similar to the real images. The initial development dataset comprised 500 synthetic images, around 80 images which were not used for the training of generative models and 80 real images used for training the generative models. However, the test dataset also included images using the generative models. In the test dataset, the synthetic images are 10,000 and the real images composed of 200 images, where the images are not segregated as used for generative and not used for generative training. Figure 1 illustrates a sample image from all three classes of generated, used and not used.
With the development dataset shared as a part of the ImageCLEF Medical task we analysed
the fingerprints between the two images, i.e generated image and the real images. The unique
ifngerprints were identified using the Perceptual Hashing algorithm illustrated in Figure 2.
The pHash algorithm [
The first model devised for this task is based on a Convolutional Neural Network (CNN) architecture. CNNs are widely used for image classification tasks due to their ability to efectively capture spatial features from images. The model begins by loading and preprocessing the images. Resizing the images to a uniform size ensures that all images have the same dimensions, which is essential for further processing. The images are resized to a specific width and height, in this case, 64×64 pixels. By resizing the images, we eliminate any variations in size that could hinder the training process. After resizing, the training images are divided into two categories: used images and unused images. These categories are represented by the labels 1 and 0, respectively and are finally concatenated as train labels. This prepares the data for training the CNN model. The CNN model architecture consists of three convolutional layers, each followed by a max pooling layer. Convolutional layers apply filters to the input images, capturing local patterns and features. The max pooling layers downsample the output of the convolutional layers, reducing the spatial dimensions and retaining the most important features. This helps in reducing the computational complexity of the model. After the convolutional and max pooling layers, the output is flattened and passed through two dense layers. The flattened output is fed into the dense layers, which are fully connected layers. The activation function used in the convolutional layers and the first dense layer is the Rectified Linear Unit (ReLU) function, which introduces non-linearity and helps in capturing complex relationships in the data. The final dense layer uses a sigmoid activation function, which squashes the output to a value between 0 and 1, representing the probability of the image belonging to the used category. During training, the model is fine-tuned by attempting to minimize the binary cross entropy loss. This loss function measures the dissimilarity between the predicted probabilities and the true labels. The optimizer used for minimizing the loss is ADAM, which is an eficient optimization algorithm commonly used for training deep neural networks. In the testing phase, a threshold of 0.5 is used to convert the predicted probabilities into binary predictions. If the predicted probability is greater than or equal to 0.5, the image is classified as used (1), otherwise, it is classified as unused (0). These predictions are then printed out. The use of a CNN model along with appropriate preprocessing techniques and activation functions allows for the efective classification of images into used and unused categories. By training the model on a labeled dataset and fine-tuning it with optimization techniques, the model can learn to distinguish between the features of used and unused images. The resulting predictions provide insights into whether an unknown image was used to generate GAN images or not. The proposed models are depicted in Figure 3.
The SIFT + KNN model is a powerful technique for image recognition that combines the ScaleInvariant Feature Transform (SIFT) algorithm with the K-Nearest Neighbors (KNN) classifier. This model leverages the strengths of both algorithms to achieve accurate and eficient image classification. The SIFT algorithm plays a crucial role in the model by extracting robust and distinctive features from images. It identifies key points that are invariant to scale, rotation, and afine transformations. These keypoints are characterized by their scale, orientation, and local descriptors that capture the image’s appearance in their local neighborhoods. The SIFT algorithm provides robustness to various challenges such as changes in lighting conditions, partial occlusions, and viewpoint variations. The KNN classifier, on the other hand, serves as the classification mechanism in the model. Given a set of labeled training samples with their corresponding feature vectors extracted using SIFT, the KNN classifier assigns labels to new, unseen samples based on their proximity to the training samples in the feature space. It employs the majority voting scheme among the K nearest neighbors to determine the label of the input sample. In the training phase, the SIFT algorithm was first applied to the GAN images, used images and the unused images. The features were clustered accordingly to create a vocabulary of visual words. Similarly SIFT features were extracted for the test images. This was then passed through our classifier (KNN) to perform the required classification.
An improvement in the given tasks can be made by finding the source image from the set of real images for the GAN images provided. This can be done using hashing techniques, one such being perceptual hashing. This model starts by loading in the given test images, both real and GAN. Perceptual hashing is a technique where Discrete Cosine Transform is used to convert the image into a frequency domain. The low frequency components are then used to construct hash values. This approach makes pHash robust to changes in scale, rotations and minor modifications in the image. This ideally aims at generating a unique fingerprint (hash value) for the images which allows eficient searching and matching of similar images from our dataset. The hash values generated by pHash are binary strings that represent the perceptual content of the images. Once the hash values are computed and stored, hamming distance between the GAN image hash and Real image hash is performed. The Hamming distance measures the number of bit positions at which the corresponding bits in two hash values difer. A lower Hamming distance indicates a higher similarity between the images, while a higher Hamming distance suggests greater dissimilarity. As mentioned earlier in the paper, a threshold of 50 was used to check if it is a used or unused image. Upon identification the source images are found for the Used images. This approach allows for studying the connections between the generated and real images, shedding light on the efectiveness of the SIFT+KNN and CNN models in determining the source images
In this section, the proposed models are implemented, and the corresponding performance metrics are discussed.
The proposed model is executed on a workstation with a six core Intel processor of 3.9GHZ, 32GB DDR4 RAM and NVIDIA GEFORCE RTX 11GB GPU.
The obtained results are illustrated in Table 1. The submission 1 is based on the CNN model, which involves a 3 layered CNN to classify the used and unused images. The model is trained for 10 epochs using the Adam optimizer, and the loss function used is binary cross entropy.. With this model, we achieved an accuracy of 0.535 and a precision of 0.544. The specificity and recall are obtained as 0.64 and 0.43. For submission 2, the SIFT + KNN model was used and this model achieved an accuracy of 0.61, precision of 0.512, specificity of 0.6 and a recall of 0.4 as their performance metrics. Apart from the above models, as mentioned earlier in the paper, pHash technique was used. This technique helped us with the mapping of source images to their generated counterparts as in Figure 4. Below are some of the outputs from the pHash model. The first one was the output when the used set of images from the training dataset was used along with the GAN images. The second one shows the same content for the real images. The images with “None” value show that they were unused in the generation of images illustrated in Figure 5.
In this paper, we proposed two approaches using CNN and SIFT+KNN to classify the used and not used images for the generation of synthetic images. The F1 score of the first approach is 0.4804 while that of the second approach is 0.449. In terms of accuracy we find that the SIFT and KNN approach has an value of 0.61 while that of the CNN model is 0.535. Along with that, an image hashing technique, which identifies the fingerprint of the images through hash values is discussed. The hamming distance is calculated and based on the value, the generated images can be mapped to their appropriate real images. This assists us to segregate the used and unused images for the training of the difusion network. In future work, these images can be experimented with using one-shot or few-shot learning, which assists in achieving faster and more eficient classification of generated images using diferent imaging modalities.