This paper provides a detailed explanation of the approaches proposed by the Biomedical Imaging Goa Lab for the ImageCLEFmedical GANs tasks. The proposed approaches use feature vectors extracted from the penultimate layer of a pre-trained CNN to represent CT scan images. For the first task, -means clustering is first performed on the extracted features. Then, 1-nearest neighbour classifiers are used to determine if an input real CT scan image is used to generate the synthetic CT scan images. The best-performing model using this approach produced an 1-score of 0.5315. For the second task, a Gaussian Mixture Model is used to perform clustering of the input deep features extracted from the CNN. This produced an Adjusted Rand Index of 0.63812.
Artificial intelligence has brought about significant improvements in the fields of medicine and health care. The use of machine learning in automating diagnosis, content-based image retrieval (CBIR), and treatment recommendation systems has attempted to lessen the burden on existing healthcare professionals. However, the development of such automated systems requires enormous amounts of training data. Such training data, especially that involving medical images is often dificult to obtain. This could be due to privacy concerns among patients as well as a lack of homogeneity in the techniques used to capture the images. In the absence of such data, it is dificult to train machine learning models to give accurate results.
This issue could be solved by creating synthetic medical images using Generative Adversarial
Networks (GANs), in which synthetic medical images can be generated from real ones. If this generation
process could be performed without afecting the privacy of patients, it could be ideal for producing
large amounts of medical data with ease. However, if the original medical image could be recovered from
the synthetic image, it could raise privacy concerns. The ImageCLEF 2024 challenge is an initiative that
tackles such issues [
In this paper, we have proposed to use deep features extracted from a convolutional neural network (CNN) to represent the visual characteristics of the CT scan images. For the first task, we have assumed that the test images are similar to the training images. Based on this assumption, we propose to use a 1-nearest neighbour classification approach in which the test image is compared to cluster centers obtained after performing -means clustering on the training data. The second task aims to find image ifngerprints in the synthetic CT scan images. To do this, we propose to extract deep features from these CT scan images and then use a Gaussian mixture model to perform clustering. This is based on the assumption that the data follows a Gaussian distribution in a GMM. This is a preliminary study on possible approaches to solve these problems and we will investigate other solutions in the future.
Each CT scan image is given as input to a CNN and the output of one of the layers of the CNN is considered as the feature vector for the input image. Identifying whether or not, a test image is used to generate synthetic CT scan images is done using a 1-nearest neighbour-based classification approach using the feature vectors obtained from the CNN. For the clustering task, a GMM is used to perform clustering on the extracted feature vectors.
In this section, a description of the method used for feature extraction is provided. This is followed by an explanation of the technique used for the classification of the CT scan images. Lastly, the clustering approach used to identify the fingerprints in the generated images is elucidated.
A CNN performs two main functions: (1) feature extraction from the input, and (2) classification of the
extracted features to certain classes. A CNN consists of diferent layers that each perform a distinct
operation. An image when passed through a CNN undergoes a series of transformations across these
layers. The final output layer of the CNN performs the classicfiation task. Once an image is passed
through a CNN, the output of any of the internal layers can be used to represent the given image. For
a CNN to efectively capture the visual characteristics of the input images, the CNN must be trained
on a large number of images. However, in the absence of suficient training images, a CNN that is
pre-trained on images from a similar dataset can be used. As the number of CT scan images available
for both tasks was not adequate to train a CNN from scratch, we have used a CNN that is pre-trained
on natural images from the ImageNet dataset [
The proposed approach to determine if a given CT scan image is used to generate a set of synthetic
CT scan images is based on the assumption that the images used for generation in the training and
test datasets are similar. As shown in Figure 1, each image in the training dataset is passed through
the pre-trained ResNet50 CNN, and the 2048-dimension output ′ of the average pooling layer after
the last convolutional layer is extracted as the input feature vector. If the training dataset contains
training images = {1, . . . , } that were used to generate the synthetic images, the corresponding
feature vectors are ′ = {1′ , . . . , ′}. If the training dataset contains images = {1, . . . , }
that were not used to generate the synthetic images, then their corresponding feature vectors are
′ = {1′, . . . , ′}. Both sets of feature vectors are separately clustered using the -means clustering
approach where the value of is varied [
Scans
O Images Not Used to Generate Synthetic CT
Scans
Pre-Trained
CNN Backbone Pre-Trained
CNN Backbone
Feature Vectors
U' centers = {1 , . . . , } are obtained. Clustering the feature vectors present in ′ results in the formation of clusters with cluster centers = {1, . . . , }.
For the testing phase, each test image is also passed through the same pre-trained ResNet50 CNN,
and the 2048-dimension output is extracted as ′ to represent the test image . To identify the closest
cluster center, the Manhattan distance is used as a similarity measure [
The Manhattan distance between ′ and each of the cluster centers in and is calculated as ′ and ′ , respectively. If the smallest distance in ′ is less than the minimum distance in ′ , the test image is classified as used and is considered to be used to generate the synthetic images. Otherwise, the test image is considered not to be used to generate the synthetic CT scan images. In this way, the class label of the 1-nearest neighbour cluster center is assigned to the test image .
For this task, the feature vectors for all the images in the training and test datasets are extracted from
the penultimate layer pre-trained ResNet50 CNN that is pre-trained on the ImageNet dataset. The
2048-dimension vectors are then clustered using a Gaussian Mixture Model (GMM) [
Training/Test
Dataset
Pre-Trained
CNN Backbone
This section consists of a description of the datasets used for both tasks followed by a detailed explanation of the experiments conducted on the datasets.
Both datasets consist of axial slices of three-dimensional CT scan lung images. These images were of 256 × 256 pixels in size. Since the first task involves detecting which real CT scan images were used to generate the synthetic CT scan images, this dataset consists of a mix of real and synthetic CT scan images. The details of the dataset used for Task 1 Identify training data ‘fingerprints’ are as follows: • The training dataset consists of two sets of image data. The first set contains 100 real images used for the generation task, 100 images not used for the generation task, and 10, 000 generated images that are generated from the real CT scan images. The second set consists of 3, 000 real images used for the generation task, 3, 000 images that are not used for the generation task, and 10, 000 CT synthetic CT scan images generated from the real images. • The test dataset also consists of two sets of image data. The first set contains 5, 000 synthetic CT scan images and 4, 000 unknown real CT scan images. The second set consists of 7, 200 synthetic CT scan images and 4, 000 unknown real CT scan images. The main aim of the task involving this dataset is to determine if the unknown real CT scan images are used to generate synthetic CT scan images.
The goal of the second task is to identify the image fingerprints in the generated CT scan images in the dataset. The details of the dataset used for Task 2 Detect generative models’ ‘fingerprints’ are as follows: • The training data contains three sets of synthetic CT scan images with each set having 200 images each.
• The test dataset consists of 3, 000 synthetic CT scan images.
The training data consists of images that are used to generate the synthetic CT scan images and images that are not used to generate the synthetic images. Once the feature vectors ′ and ′ are extracted after passing the images through the CNN, they are separately clustered using the -means clustering method to obtain cluster centers each in and , respectively. During testing, each test image is compared with the 2 × cluster centers in and , and the label of the closest cluster is assigned to the test image. For this task, we have submitted 6 runs for diferent values of used for -means clustering. The values of are powers of 2, i.e. 1, 2, 4, 8, 16, 32. The results of this task are provided in Table 1.
The results indicate that the 1-scores vary with the value of . The best result of an 1-score of 0.5315 from the submitted runs was obtained by the model where = 4. The second best result was obtained by the model with = 8 with an 1-score of 0.5145. The submitted run where = 2 gave the worst performance. This shows that an optimal value of must be chosen.
Once the features are extracted from the CNN, they are clustered using a Gaussian Mixture Model
(GMM). During the training process, we experimented with GMMs that had 3 components. This was
based on our observations of the training data. During the test phase, we assumed that the GMMs have
4 components. The 4 submitted runs difer based on the methods used to initialize the means of the
components in the GMMs. The 4 methods used for initializing are as follows:
1. -means: Uses the -means clustering algorithm for initialization [
The results of this task are provided in Table 2. The performance of the proposed approach indicates that initialization of the GMM components using the -means algorithm gives the best results while random initialization of the GMM components is not efective. However, the proposed approach is based on the assumption that the data follows a Gaussian distribution. If this is not true, then the proposed approach may not be an ideal solution to this problem.
This paper describes the methods used by the Biomedical Imaging Goa group for the ImageCLEFmedical GANs task. The proposed approach uses features extracted from the penultimate layer of a CNN that is pre-trained on natural images. For the classification task, the test image is compared with the cluster centers formed from the training data and labeled according to a 1-nearest neighbour approach. For the clustering task, the feature vectors are clustered using a GMM-based approach.
Since this is a preliminary study conducted by our group, in the future we could explore other possible methods to improve eficiency. This could involve using feature vectors from other CNNs as well as diferent classification and clustering methods. Further, the proposed approaches are based on certain assumptions that could be investigated to provide a better explanation of other possible solutions.