The action of understanding and interpretation of medical images is a very important task in the medical diagnosis generation. However, manual description of medical content is a major bottleneck in clinical diagnosis. Many research studies were devoted to develop automated alternatives to this process, which would have enormous impact in terms of eficiency, cost and accuracy in the clinical workflows. Diferent approaches and techniques have been presented in the literature ranging from traditional machine learning methods to deep learning based models. Inspired by the outperforming results of the later techniques, we present in the current paper, our team participation (RomiBed) to the ImageCLEF medical caption prediction task. We addressed the challenge of medical image captioning by combining a CNN encoder model with an attention-based GRU language generator model whereas a multi-label CNN classifier is used for the concept detection task. Using the provided data in the training, validation and test subsets, we obtain an average F_measure of 14.3% and a BLEU score of 0.243 on the ImageCLEF concept detection and the caption prediction challenges, respectively.
With the increasing number of medical images generated worldwide from diferent
modalities in hospitals and health centers, the need to analyse and discover their content is crucial.
Indeed, medical images ofer a safe environment to explore patient’s health state without the
need for a surgery or any other invasive procedures [
Motivated by the recent advances made in deep neural networks in diferent tasks of computer vision and NLP, especially due to their promising results in the machine language translation models, we present in this paper our contribution to the ImageCLEF 2021 medical task under the team name ’RomiBed’. We proposed a multi-label classification CNN model for the first sub-task after applying an augmentation technique based on the center cropping of the medical images. Features were extracted using a pre-trained model, while the classification is performed using a CNN network. For the second sub-task, we proposed an encoder-decoder model with an attention layer where the encoder is based on a CNN feature extractor and the decoder is composed of a GRU network with an attention mechanism.
This paper is organized as follows. First, we briefly review the related medical image captioning studies from the literature in Section. 2. In Section. 3, we provide a brief description of the ImageCLEF dataset used in this study. Next, we detail the methodology we followed to construct the concept detection model as well as the caption prediction model in Section. 4. We discuss each step of the process and deliver the results in terms of F_measure for the concept detection and BLEU for the caption prediction. Finally, we finish with a conclusion where we highlight some key insights and future directions.
Automatic image captioning (AIC) in the medical field has gained a particular attention from
researchers due to its importance and its huge impact on health care centers by allowing
instantaneous understanding of medical images for doctors as well as patients. In addition,
the significant progress made to date in artificial intelligence due to deep learning models
contributed greatly to the AIC task [
The data used for this year edition is shared between both the ImageCLEFmed Caption and the ImageCLEF-VQAMed tasks. The dataset include three sets: the training set composed of the VQA-Med 2020 training data with 2756 medical images; the validation set consisting of 500 radiology images and the test set consisting of 444 radiology images. In addition, for the concept detection task, an excel file containing the medical image ID and the corresponding concepts CUIs is given to map each medical image onto its related concepts. Similarly, an excel ifle containing the captions of each medical image is provided for the caption prediction task. We present in Fig. 1 sample images with their underlying concepts and in Fig. 2 samples of images with their captions.
On further analysis of the datsaet, we illustrate in Fig. 3 the number of images per each CUI concept. It is obvious that the most frequent CUI is the ’C0040398’ corresponding to ’Tomography, Emission-Computed’ with 1159 images. Moreover, Fig. 4 presents the number of images by the number of CUIs associated to each image. We can see from this image that most of the medical images are attached to 2 to 3 concepts whereas the maximum number of concept CUIs per image is 10.
Moreover, we noticed that the maximum number of sentences per image caption is 5 and the maximum length of any caption is 47 words before pre-processing whereas it is 33 words after the pre-processing.
For this year edition of the ImageCLEFmed Caption challenge [
Text Pre-processing: We apply a pre-processing scheme on the image to concept matching ifle to organize the concepts of each medical image into a list and create data-frames from image IDs and their underlying CUIs. In addition, we pre-process the captions using the NLTK package, by performing tokenization, punctuation and stop-words removal (using the default NLTK’s "english" stopword list), lower casing each token and finally applying the stemming to obtain filtered sentences for each medical image using the NLTK’s Snowball stemmer. Morover, for the RNN decoder, we add two tokens: ’<start>’ and ’<end>’ to identify the beginning and the end of each caption.
Image Pre-processing: Image data generators are created for the three sets of data to preprocess the images before feature extraction. These generators iterate over the data subsets and normalize the images to facilitate the features calculation. Then we apply horizontal and vertical flip in addition to a crop-center based data augmentation technique that we implement with a fraction of 87.5%. The implemented data augmentation approach allowed us to expand the training data without altering the visual content of the image. Then, the images are resized to fit in the feature extractor size which is 224 × 224 × 3 in our case.
As we mentioned before, the first task of concept detection aims at identifying and localizing the relevant concepts present in each medical image. Therefore, we exploit the visual image content to extract significant visual features that allow us to distinguish the underlying concepts. These concepts are used further to construct image captions and could as well be utilized for the context-based images and information retrieval purposes.
To achieve the first step towards caption prediction, we performed a multi label classification. In the first run, we extracted image features using the pre-trained MobileNet-V2 and then performed the classification using a GRU network [ 21]. In the second run, we performed feature extraction using the pre-trained Inception-V3 model followed by a classification using a CNN network. ImageNet weights were used for both models and features were extracted from the last convolutional layer.
As illustrated by Fig. 5, the features extracted from the radiology images using the pre-trained MobileNet-V2 model are passed through a GRU layer, a flatten layer and then a fully connected layer with a Relu activation function and dropout. Finally, the labels of each medical image are predicted using a fully connected layer with a Sigmoid activation function.
Likewise, the features extracted from the radiology images using the pre-trained Inception-V3 model are passed through a flatten layer, a fully connected layer with batch normalization, Relu activation function, and dropout. Then, the probability of each class is calculated using a fully connected layer with a Sigmoid activation function (as shown by Fig. 6). If the probability is greater than 20%, we assert the input image belongs to that class. This probability was fixed to 20% after experimenting with diferent thresholds. If this threshold is fixed to a higher value, we would get a lot of false negatives where many images are not classified in their correct classes. However, if it is fixed to a smaller value, we would get a lot of false positives where many images are categorized into incorrect classes. Finally, we map the labels to their corresponding concepts.
The second sub-task relies on the concept vocabulary detected in the first sub-task in addition to the visual features extracted from the medical images to establish relationships between them and predict descriptive caption for each medical image. We attempted to address the issue of caption generation using an encoder-decoder architecture with attention mechanism.
The visual features are extracted from the medical images using a pre-trained model ’InceptionV3’ where weights from the ImageNet were employed. Then, these features are passed through a CNN model that is composed of a fully connected layer to flatten the feature vector. Next, an attention mechanism is employed to focus on the most important parts of the image and a context vector is constructed. Captions are pre-processed as we mentioned before and passed to an embedding layer. A concatenation layer is farther used to merge the context vector with the resulting embedding vector and the output is passed to a GRU layer. A flatten layer and two fully connected layers with dropout and a Relu for the first one and a Sigmoid activation function for the second were employed. Finally, relevant captions are generated word by word until the ’<end>’ token is met. Figure. 7 illustrates the attention-based encoder-decoder architecture we used to construct new captions for the medical images. Moreover, we exploit the teacher forcing during the training by using the ground truth sequences at every step rather than the sequence of newly generated words at previous steps.
We used the data provided by the ImageCLEF medical task to evaluate the performance of our models. Three subsets are used for the training, the validation and the test respectively. We report in this section, the performance metrics calculated as well as the results we obtained for both models.
We calculate the F_Measure, using the default ’binary’ averaging method, for the concept detection task as follows: ∑︁
∑︁ = ∈{} _∈ ∑︁ ∑︁ (_)
(_′) ′∈{} _′∈′
= · exp(∑︁ · log ) =1
Where BP refers to the brevity penalty, N refers to the number of n_grams (uni-gram, bi-gram, 3-gram and 4-gram), refers to the weight of each modified precision and refers to the modified precision. By default N=4 and = 1/N = 1/4.
Brevity Penalty (BP) allows us to pick the candidate caption which is most likely close in length, word choice and word order to the reference caption. It is an exponential decay and is calculated as follows: = {︃1
> exp(1− /) ⩽
Where r refers to the count of words in the reference caption and c refers to the count of words in the candidate caption.
Modified precision is computed for each n_gram as the sum of clipped n_gram counts of the candidate sentences in the corpus divided by the number of candidate n_grams as shows "(6)" [22]. It allows us to compute the adequacy and the fluency of the candidate translation to the reference translation.
· _ = 2 · +
Where the recall and the precision are calculated as follow and TP, FN, TN, FP correspond to true positive, false negative, true negative and false positive respectively.
=
+ = (3)
+
For the caption prediction task, we calculate the BLEU score by assuming that each caption is a single sentence even if it is actually composed of several sentences. For that we use the default implementation of the Python NLTK based on [22]: (1) (2) (4) (5) (6)
We obtained an average F_measure of 24.49% during the training process and a value of 14.3% during the inference process for the concept detection task using the MobileNetV2 as a feature extractor and the GRU network as a classifier.
Similarly, we obtained an average F_measure of 23.28% during the training process and a value of 13.7% during the inference process using the InceptionV3 model as a feature extractor and a CNN network as a classifier. However, our F_measure results are comparatively lower compared to the leading group (IALab_PUC) with a score of 50.5 %. Results are illustrated by Table. 1 where the run ID 136011 corresponds to the first configuration and the 136025 corresponds to the second configuration. Figure. 8 shows the evolution of the multi-label classification model accuracy across the epochs.
For the caption prediction task, we obtained a BLEU score of 0.287 during the training process and a value of 0.243 during the inference. Results are illustrated by Table. 2. In addition, Figure. 9 shows the loss calculated during the training process of the encoder-decoder model. We noticed the decrease of the cross-entropy loss across the epochs.
Finally, we show an example of a random medical image from the validation set with its real caption and the newly generated caption in Fig. 10. We observed a BLEU score of 0.339 for this image, where 8 words were correctly generated but the order of the words in the generated caption is diferent.
We presented in this paper our contribution to the ImageCLEF 2021 medical task where we proposed a CNN based multi-label classification model for the concept detection task and an attention-based encoder-decoder model for the caption prediction task. For both models, a transfer learning is used to extract signicfiant features from the real radiology images and a data augmentation based on center-cropping is applied to expand the used training subset. The evaluation of the caption detection task is conducted using the mean F_measure for which we obtained a score of 14.3%. Furthermore, BLEU score is used to evaluate the reliability of the generated captions for the caption prediction task for which we obtained a score of 0.243. We believe that we did not obtain promising results due to the small amount of data used and the fact that we did not explore more fine-tuned parameters for both models for time constraints. In addition, we did not include the textual features substituted by the medical concepts to generate the new captions. In future work, we will integrate the textual features of the images to the visual information to obtain more relevant captions. We will also investigate more advanced deep learning algorithms inline with more fine-tuned parameters. In addition, we will investigate our model performance on larger scale dataset for medical image captioning.
This work is supported by the Academy of Finland Profi5 DigiHealth project ( #326291), which is gratefully acknowledged. [20] Y. LeCun, Y. Bengio, et al., Convolutional networks for images, speech, and time series,
The handbook of brain theory and neural networks 3361 (1995) 1995. [21] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder-decoder for statistical machine translation, arXiv preprint arXiv:1406.1078 (2014). [22] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a Method for Automatic Evaluation of Machine Translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318.