Automated interpretation of medical images promises to revolutionize traditional diagnostic approaches, rendering them not only more eficient but also significantly faster. For the Concept Detection task, a Multi-label CNN (Convolutional Neural Network) model is proposed which is capable of mapping a single image to multiple highly probable concepts. This model's versatility lies in its ability to discern various features within medical images, facilitating a nuanced understanding of complex visual data. For the Caption Prediction task, a CNN-LSTM (Convolutional Neural Network - Long Short-Term Memory) model is proposed to predict accurate captions for given images. This model harnesses the power of CNNs to extract salient visual features and combines it with the sequential processing capabilities of LSTM networks to generate contextually relevant and accurate descriptions. This working note paper presents the results of the Kaprov team at ImageCLEFmedical 2024 Image Captioning and its subtasks of concept detection and caption prediction.
Radiology images, including X-rays, Computed Tomography scans (CT scans) and Magnetic Resonance Imaging (MRI), and ultrasounds, are fundamental tools in medical diagnostics. They provide detailed visual information that helps healthcare professionals diagnose, monitor, and treat various medical conditions. Interpreting these images requires extensive training and expertise, as radiologists must identify subtle patterns and anomalies that can indicate the presence of diseases or injuries.
Traditionally, classifying and captioning radiology images is a manual process carried out by experienced radiologists. This involves a systematic approach where radiologists review images, analyze • Statistical Models: Apply statistical techniques to model the relationships between image features and diagnostic outcomes. These models can help in identifying probable conditions based on image data
Recent advancements in the field of medical image captioning have made significant strides by
leveraging deep learning techniques. Among these techniques, CNNs and LSTM networks are
particularly notable [
However, there are several challenges to address. One major issue is the lack of large, annotated datasets for training. Another is understanding how these complex models make decisions, known as model interpretability. Additionally, training and deploying these models require significant computational resources.
To address these challenges, the proposed approach integrates deep learning models with
domainspecific knowledge bases, such as the Unified Medical Language System (UMLS) [
There are two tasks that have been worked upon, the first one being the Images Concept Detection and the second one being the Caption Prediction Task.
The dataset used for these two tasks is from ImageCLEF 2024 [
This dataset dives deep into the intricate world of medical image analysis. It encompasses various imaging modalities such as X-rays, CT scans, MRIs, and ultrasounds. Each modality ofers a unique perspective on the internal structures of the body, providing crucial information for diagnoses. This diversity not only reflects the breadth of clinical imaging techniques employed in medical practice but also challenges machine learning models to generalize across diferent imaging modalities. Moreover, many images in the dataset depict scenes where multiple medical concepts coexist, requiring models to discern and accurately label complex visual content. Furthermore, the captions linked with these images show a broad range in length and complexity, reflecting the detailed observations and nuanced interpretations by healthcare professionals.
Figure 1 depicts a representative image from the dataset, sourced from Muacevic et al. (2023) under CC BY license.
Figure 2 illustrates a frequency histogram of Concept Unique Identifiers (CUIs) used in the dataset, highlighting the distribution of concepts across the medical images.
This section outlines the process of preparing data for concept detection and caption prediction.
In concept detection, converting multi-label dataset into a machine learning-friendly format is crucial, as each image can be associated with multiple concepts.
Each instance can have multiple labels for diferent concepts, so it needs to be converted into a binary matrix format. The MultiLabelBinarizer from sklearn.preprocessing module is an efective tool for this task. It processes a list of lists, where each sublist contains the labels for a sample, and converts it into a binary matrix. In this matrix, each column represents a unique concept, and each row corresponds to a sample, with binary entries indicating the presence (1) or absence (0) of a concept.
The data pre-processing method for concept detection and image captioning involves several essential
steps. Firstly, the VGG16 (Visual Geometry Group) [
In parallel, the text captions related to these images are cleaned and standardized. This involves converting each caption to lowercase, removing non-alphabetical characters, and consolidating multiple spaces into one. Additionally, start and end tags are added to mark the beginning and end of each caption. The Tokenizer from keras.preprocessing.text module is used to tokenize the cleaned captions, convert words to unique integer indices and calculate the vocabulary size.
CNNs are well-suited for medical concept detection in image analysis [
The Concept Detection Model is designed for multi-label classification tasks [
The Caption Prediction Model combines image and text data to generate captions. First, it extracts image features using a dense layer with ReLU activation. At the same time, it converts text sequences into dense vectors to capture their meanings.
Next, it merges the image and text data by concatenating them, allowing the model to understand both inputs together. This combined data goes through an LSTM layer, which captures the sequence of words needed to create coherent captions. Dropout is used to prevent overfitting and improve the model’s performance on new data.
The output from the LSTM is enhanced with the original image features, creating a strong link between the image and text.
This model is chosen for the seamless integration of visual and textual data. It begins with a dense layer that captures intricate details from medical images, ensuring efective feature extraction. The LSTM layers are essential for managing sequential data, creating coherent and logical text descriptions. Additionally, dropout regularization reduces overfitting, making the model more reliable with new data. This blend of techniques results in high-quality captions that are accurate and contextually meaningful.
The Concept Detection Model is a Convolutional Neural Network (CNN) [Fig. 3], designed to classify images into 1945 diferent categories. The model utilizes a sequential architecture implemented with TensorFlow and Keras. It starts with a convolutional layer employing 16 filters of size 3 × 3, followed by a batch normalization layer to stabilize and accelerate the training process. This pattern of convolution, batch normalization, and pooling is repeated with more filters—32, 64, and 128 in the next convolutional layers—to capture increasingly complex features at each stage. Max pooling layers (2 × 2 size) reduce the size of the feature maps, making the model simpler and faster. Dropout layers with a rate of 0.25 are incorporated after each max pooling layer to prevent overfitting by randomly deactivating a fraction of neurons during training. After the convolutional and pooling layers, the model transitions to fully connected (dense) layers. The first dense layer has 64 units, followed by another batch normalization and dropout layer. This is followed by a dense layer with 128 units, again followed by batch normalization and dropout. The final dense layer uses a softmax activation function to calculate the probabilities for each of the 1945 classes. The model is compiled using the Adam optimizer and the categorical cross-entropy loss function. The model is trained with a batch size of 32 for 20 epochs. This extended training duration facilitates gradual adjustments to the model’s weights using computed gradients from the training data. The EarlyStopping mechanism halts training if validation loss stagnates over three consecutive epochs, thereby mitigating overfitting and enhancing the model’s ability to generalize to new data.
The Caption Prediction Model used for image caption prediction is the CNN-LSTM Fusion Caption Generator [Fig. 4], which integrates both dense and sequential data processing through a structured series of layers. It begins with two input layers: one for fixed-length dense data and another for variable-length sequential data. The dense data from the first layer passes through a dense layer with 256 units, serving as a feature extractor. Simultaneously, sequential data from the second layer undergoes embedding, converting each element into a 256-dimensional vector. These processed outputs are concatenated and fed into an LSTM layer with 256 units to capture temporal dependencies. Dropout layers are applied after the LSTM and subsequent dense layers to prevent overfitting. An additive connection merges the LSTM output with the initial dense layer’s output to aid gradient flow. The final dense layer produces predictions tailored to the specific output requirements with 12,574 units. This model is implemented with a batch size of 32 and trained over 10 epochs. A batch size of 32 means that in each training iteration, the model processes 32 samples concurrently. This choice optimizes the utilization of computational resources like GPU memory while ensuring that the model’s weight updates are stable and based on gradients computed from each batch.
Figure 3: Concept Detection Model architecture.
Figure 4: Caption Prediction Model architecture.
The Concept Detection Model, as shown in Table 1, demonstrated significant accuracy in identifying relevant concepts within radiology images. Evaluation metrics, including F1-score and F1-score manual, indicated that the model efectively handled the multi-label classification challenge, accurately mapping images to multiple concepts.
The Caption Prediction Model, detailed in Table 2 , exhibited strong capabilities in generating coherent and contextually relevant captions. Performance metrics such as BERTScore and ROUGE scores were used to evaluate the quality of the generated captions. The results showed that the model produced captions with high linguistic richness and relevance, closely aligning with human annotations.
In both the Concept Detection and Caption Prediction tasks, the models showcased strong performance, demonstrating its ability to efectively analyze radiology images. For the Concept Detection task, it accurately identified relevant concepts, and endeavors to provide valuable support in medical diagnostics by mapping multiple concepts to each image. In the Caption Prediction task, the model generated meaningful and contextually appropriate descriptions, aligning well with the visual content of the images. Various metrics, including F1-Score, BERTScore, ClinicalBLEURT, CLIPScore, RefCLIPScore, BLEU-1, BLEURT, CIDEr, METEOR, and ROUGE underscored the models’ comprehensive performance in producing linguistically rich and accurate captions.
We thank the CSE department of SSN College of Engineering for letting us utilize the GPU machine extensively to implement this task.