This paper provides an notebook for the ImageCLEFmedical GANs lab at CLEF 2023. Using generative adversarial networks for medical image generation is a standard data-expanding method. However, the quality of the generated data is not high, and finding the source based on the generated images is a new task worth investigating, as finding the source of the real data can ensure the reliability of the generated data. The GANs task is a completely new challenge in the ImageCLEFmedical track. The task is focused on examining the existing hypothesis that GANs are generating medical images that contain the "fingerprints" of the real images used for generative network training. In this paper, our team(one ifve one zero) use contrastive learning to find real images with high response values through similarity calculations based on the natural similarity between real images and generated images. We use a triplet loss function for optimization.
Generating medical images is a fundamental problem in medical imaging and can be used
for applications such as data enhancement and model training[
However, due to the unclear data sources in the process of false data generation, the quality
and reliability of incorrect data have also been widely questioned[
The GANs task is an entirely new task in the ImageCLEFmedical track[
To address the requirements of this task, this paper uses similarity calculations with the generated data as the target, the images used for a generation as positive examples, and the real images unused for a generation as negative examples. The model learns the data distributions of the three by learning them through contrastive learning, and since the generated images are output through the positive examples, the nature of generative adversarial networks is to learn the data distribution, so the generated data has a higher similarity to the data distribution than the positive examples, so contrastive learning is used as a way to distinguish the distribution of the data.
Investigate the hypothesis that GANs are generating medical images that are in some way similar to the ones used for the GAN training. The task is related to the problem of the security of personal medical image data in the context of generating and using artificial images in diferent real-life scenarios.
The objective of the task is to detect “fingerprints” within the synthetic biomedical image data to determine which real images were used in training to produce the generated images. The task is to analyze test image datasets and assess the probability with which certain images of real patients were used for training image generators and which were not.
The benchmarking image data are the axial slices of 3D CT images of about 8000 lung tuberculosis patients. This particularly means that some of them may appear pretty “normal,” whereas the others may contain certain lung lesions, including severe ones. These images are stored in the form of 8-bit/pixel PNG images with dimensions of 256x256 pixels.
The published development dataset for the task includes 500 artificial images, 80 real images which were unused for training generative neural networks as well as 80 real images taken from the image set which has been used for training the corresponding generative model. The test dataset was created in a similar way. The only diference is that the two subsets of real images are mixed, and no proportion of non-used and used ones has been disclosed. Thus, a total of 10,000 were generated, and 200 real images were provided.
The process of generating a model is essentially learning the distribution of accurate data. By analyzing the real training data, you can learn the distribution of the data, and having learned the distribution of the data, you can generate a lot of data that fits this data distribution based on the distribution. To find out the real images based on the generated fake images, the good idea is to fit the model, that is, to fit the generative ability of the model, because diferent models learn to learn diferent features; if you can know the network model and the parameters, you can achieve the inverse process, through the generated fake images reverse inference to the real images.
However, as the task did not tell us strictly what network was used, and the number of images given in the training set was very small, with only 80 of them used, it was dificult to train the generative adversarial network to achieve a simulation of the network model. Therefore, we can only focus on the distribution of the data, which is close to the distribution of the used images and difers significantly from the distribution of the unused data, so we can see the use of contrastive learning for the model’s training.
Contrastive learning uses data that has been augmented as a positive sample and data that is
not of the same category as a negative sample for comparison training[
This paper adopts a pre-trained model, which can be pre-trained on large-scale data to learn universal and robust feature representations[8]. These features can be migrated to various specific tasks, thereby improving the generalization ability and performance of the model. By utilizing pre-trained models, knowledge from large-scale data can be transferred to specific tasks, thereby reducing training time and data volume. This can greatly improve the eficiency and accuracy of the model. In some application scenarios, insuficient or incomplete data annotation makes it dificult to train models directly.
Pre-trained models can utilize large-scale unlabeled data to learn the potential structure and patterns of the data, thus addressing the problem of insuficient or incomplete data annotation. This paper uses various pre-training models, including Inception V3, ResNet, and EficientNet.[9].
The triplet loss function describes the requirements of this task well[10]. In the design of the above model, the original image of the target is found by similarity by calculating the distance between the target and the positive and negative examples. This is consistent with the ternary loss function, as shown in Figure 2.
In this paper, our goal is to make the distance between the target and the positive example smaller than the distance between the target and the negative examples[11]. Therefore, we need to calculate the distance between the target, positive and negative examples, and use it as an input to the loss function. Specifically, we use the Euclidean distance to calculate the distance, as shown in Equation (1).
loss = max(0, || () − (+)||2 − || () − (− )||2 + ) (1) Where () denotes the generated image, (+) denotes the positive example, the used image, and (− ) denotes the negative example, the unused image, and is the hyperparameter.
In existing experiments, training data was used for training and reasoning was done in test data. However, by analyzing the data, it can be seen that 500 images were generated in the training data, which were generated from 80 images. The data generated in the test data provided 10000 images. At this time, using training data for training and testing data for reasoning seemed inappropriate[12]. Due to the connection between the training data and the test data, it can be inferred that the 10000 images provided in the test have the same data distribution as the images in the training. Therefore, the generated images from the test data are also included in the training.
If the test data is placed in training, there will be imbalanced data. The generated data in the given test data has 10000 images, while the test data in the training data only has 500 pieces. Therefore, copying 500 pieces of the training data is equivalent to giving greater weight[13]. Thus achieving a balance between training and testing data. Secondly, there are only 80 used and unused images in the training, which means there are only 80 positive and negative images, respectively, which is a significant diference from the target’s data volume. In comparative learning, it is necessary to take a target image, a positive image, and a negative image separately. After pre-processing, the number of data sets is shown in Table 1.
Due to the large size of the target data, the positive data was enhanced to 1000 images and replicated 10 times to maintain consistency with the target data. In this way, data can be taken from the target and then from positive and negative, respectively. The process of contrastive learning can be completed. The reverse update is completed through the triplet loss function, making the distance between the target and the positive closer and closer and the distance between the target and the negative farther and farther.
This task has three evaluation metrics: Accuracy, Recall, and F1 score. In this experiment, a total of three results were submitted, and three pre-trained models were used, i.e., Inception V3, ResNet and EficientNet. The experimental equipment used in this experiment is RTX 3070, with a learning rate of 1e-4. The final submitted result scores are shown in Table 2.
As can be seen from Table 2, the best results were achieved by pre-training the model based on EficientNet, which optimises the depth, width and resolution of the network at the same time and is highly versatile and scalable, and therefore performs better.
In this paper, to complete the task of finding the original image according to the generated image, from the perspective of the similar data distribution between the generated image and the original image, we use the contrastive learning architecture, combined with transfer learning, and use the pre-trained feature extraction module to find the target with large response value through the similarity calculation method, so as to find the original image. Future research can attempt more fine-grained networks, such as those in face detection, as the generated medical images are highly similar and difer mainly in some fine-grained features. This is in line with the networks in face detection, and further research can be considered from this perspective. [6] T. Kipf, V. der Pol et al., Contrastive learning of structured world models, arXiv preprint arXiv:1911.12247 (2019). [7] C.-Y. Chuang, R. et al., Debiased contrastive learning, Advances in neural information processing systems 33 (2020) 8765–8775. [8] X. Qiu, S. et al., Pre-trained models for natural language processing: A survey, Science
China Technological Sciences 63 (2020) 1872–1897. [9] H. Chen, W. et al., Pre-trained image processing transformer, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12299– 12310. [10] W. Deng, Z. et al., Rethinking triplet loss for domain adaptation, IEEE Transactions on
Circuits and Systems for Video Technology 31 (2020) 29–37. [11] B. Yu, L. et al., Correcting the triplet selection bias for triplet loss, in: Proceedings of the
European Conference on Computer Vision (ECCV), 2018, pp. 71–87. [12] S. Kotsiantis, K. et al., Handling imbalanced datasets: A review, GESTS international transactions on computer science and engineering 30 (2006) 25–36. [13] N. V. Chawla, Data mining for imbalanced datasets: An overview, Data mining and knowledge discovery handbook (2010) 875–886.