Introduction: Medical image captioning is an important AI task in healthcare, automating the generation of text descriptions to support the management and interpretation of medical images. Our team, UIT-2Q2T, participated in the second task of the ImageCLEFmedical 2024 Caption challenge using the ROCOv2 dataset with the Bootstrapping Language-Image Pre-training (BLIP) approach. Methods: Our approach leveraged the BLIP architecture for multimodal medical image captioning. This architecture employs a Vision Transformer (ViT) as the image encoder and a Bidirectional Encoder Representations from Transformers (BERT) as the text model. Results: We ranked 5th according to BERTscore and placed 3rd with ROUGEScore, BLEURTScore, and RefCLIPScore. Additionally, we achieved 2nd place for BLEU-1, METEOR, and CIDEr scores. Notably, we obtained the top position with a CLIPScore of 0.827074, demonstrating the efectiveness of our approach in medical image captioning. Conclusion: Our participation in the ImageCLEFmedical 2024 Caption challenge demonstrated the efectiveness of the BLIP architecture for medical image captioning, achieving a high CLIPScore of 0.82707. This result demonstrates the model's potential to generate accurate and informative textual descriptions from medical images, thereby aiding diagnosis and assisting non-experts in understanding medical images.
Image captioning, a well-established field in artificial intelligence (AI), finds applications across diverse domains. In healthcare, the increasing availability of medical imaging equipment and the eficiency of diagnosis based on visual data have fueled the popularity of image-based patient diagnosis. Medical image captioning models address this need by automatically analyzing and describing medical images. These models generate textual descriptions that assist doctors in diagnosing diseases, understanding physiological processes, and enabling non-experts to interpret medical imagery.
This field integrates computer vision and natural language processing, demanding an understanding
of image components and their relationships [
To successfully deploy image captioning in healthcare, it is essential to integrate efective algorithms
and use a suficiently large and diverse training dataset. Our team participated in ImageCLEF 2024
[
Our approach for the caption prediction subtask is based on Bootstrapping Language-Image
Pretraining (BLIP) [
At ImageCLEFmedical 2024, we participated in the image captioning task. This is the 8th edition of the ImageCLEFmedical caption task. In this section, we will introduce the task in the ImageCLEFmedical 2024 Caption and the dataset used for this challenge.
ImageCLEFmedical 2024 Caption [
In this study, we focus on the second sub-task based on the provided dataset ROCOv2 [
The dataset for this task is ROCOv2 [
• Test Set: Consists of 17,237 radiology images.
As shown in Figure 1, the majority of captions in the dataset range from 50 to 150 words in length. Similarly, Figure 2 illustrates that among the six imaging modalities represented in the dataset, CT scans and X-rays are predominant, accounting for 24,227 and 19,363 samples in the training set, respectively.
In this study, the BLIP model was employed to tackle the image captioning task. This approach involves ifnetuning the BLIP model on the competition dataset, which consists of diverse and challenging imagecaption pairs. The pipeline of our method is illustrated in Figure 3, showcasing the steps involved in adapting the BLIP model for our specific image captioning task.
Bootstrapping Language-Image Pre-training (BLIP) [
The model uses Vision Transformer (ViT) [
N X Image Encoder
Feed Forward Self Attention
CC BY-NC [Bajracharya et al. (2021)] TEenxctoder
N X
Cross Attention Feed Forward
Bi Self-Att "[CLS] + Feed Forward Cross Attention
Bi Self-Att Feed Forward Cross Attention
Causal Self-Att Image-grounded Text encoder "[Encode] +
Image-grounded
Text decoder "[Decode] + Axial image of contrast-enhanced computed tomography (CECT) of lower abdomen shows a mixed solid and cystic mass....
To be able to train or pre-train the model for understanding and generation tasks, a multimodal mixture of an encoder and decoder is used, integrating three functionalities and three objectives, as illustrated in Figure 3. The functionalities include: • Unimodal Encoder: Encodes either the image or the text separately without considering the other modality. This helps in understanding individual representations. • Image-grounded Text Encoder: Encodes text while being conditioned on the image, allowing the model to capture relationships between visual and textual information. • Image-grounded Text Decoder: Generates text based on the given image, useful for tasks like image captioning where the output is text describing the input image.
The objectives are: • Image-Text Contrastive Loss (ITC): Ensures that paired image and text representations are closer together in the embedding space compared to unpaired ones. This helps the model learn strong associations between images and their corresponding texts. • Image-Text Matching Loss (ITM): Assesses whether a given image and text pair match or not, promoting accurate image-text alignment in the embedding space. • Language Modeling Loss (LM): Focuses on generating coherent and contextually accurate text based on given inputs, improving the model’s language generation capabilities.
These functionalities and objectives together enable the BLIP model to perform both vision-language understanding and generation tasks efectively.
Following the guidelines provided by the competition organizers, we employed two main metrics:
BERTScore [
As the organizers’ instructions, the captions underwent preprocessing through three steps: conversion to lowercase, replacement of numbers with a special token, and removal of punctuation. This preprocessing aimed to standardize the text inputs and enhance the quality of evaluation result.
In this section, we present our experimental setup and results for evaluating the BLIP model in the ImageCLEFmedical 2024 Caption challenge. The experiments were designed to test the model’s performance across diferent configurations and metrics, aiming to generate accurate and informative captions for medical images. We describe the setup in detail and discuss the results obtained from various test scenarios.
In our experiments, we employed BLIP model (base/large) from pre-trained checkpoints. For BLIP base, weights were utilized from the checkpoint "Salesforce/blip-image-captioning-base"2. Training was conducted over 15 epochs with an initial learning rate of 1e-5. A StepLR scheduler was used to decrease the learning rate by a factor of 10 every 3 epochs. For the BLIP large model, weights were utilized from the checkpoint "Salesforce/blip-image-captioning-large"3. Training was conducted over 5 epochs with an initial learning rate of 1e-5 and was stopped when the loss ceased to decrease. Throughout all experiments, the AdamW optimizer [21] was used. Input images were resized to 224x224, and the maximum length of text input was set to 200 tokens. To facilitate model training, a single GPU A100 PCIE 40GB was used.
For each model, experiments were conducted with four diferent generation settings using no_repeat_ngram_size = 3 to prevent the model from repeating any n-gram of size 3 within the generated text. The generation settings are as follows: (1) Greedy Search: This setting selects the token with the highest probability at each step, ensuring a straightforward and fast generation process but potentially missing out on more diverse or optimal sequences. (2) Beam Search with beam_size = 3: Beam search keeps the top 3 most probable sequences at each generation step, allowing for more exploration of potential sequences compared to greedy search. (3) Beam Search with beam_size = 4: Similar to the previous setting, but with a beam size of 4, which balances between exploration and computational eficiency. (4) Beam Search with beam_size = 5: This setting further increases the beam size to 5, allowing for more comprehensive exploration while balancing computational eficiency. (5) Beam Search with beam_size = 10: With a beam size of 10, this setting aims for a broader exploration of possible sequences, potentially improving the quality of text generation at the cost of higher computational resources.
The source code for our experiments is available on GitHub4.
To evaluate the efectiveness of our approach using the BLIP model on the image captioning task, we conducted a series of experiments on the competition dataset. This section presents the results, highlighting the model’s performance in generating captions. We compared the experimental results of the model using diferent configurations and also benchmark our model’s performance against other teams’. 2https://huggingface.co/Salesforce/blip-image-captioning-base (Last accessed: May 17, 2024) 3https://huggingface.co/Salesforce/blip-image-captioning-large (Last accessed: May 17, 2024) 4https://github.com/QuangHoang059/DS312
CC BY-NC [Trowbridge et all. (2022)]
As shown in Table 1, the BLIP large model outperforms the BLIP base model. Both models show generative capabilities, with beam search outperforming greedy search across ROUGEScore, as well as BERTScore and BLEUScore. Specifically, the BLIP base model achieves its highest BERTScore and ROUGE score with a beam size of 5, and its best BLEU score with a beam size of 3. The BLIP large model attains optimal results across all three metrics with a beam size of 4. Additionally, as illustrated by the two examples in Figure 4, the model accurately identifies objects and colors (white arrow), as well as diferent imaging modalities (CT and sagittal T2-weighted MRI).
According to the private test results announced by the organizing committee and partially presented in Table 2, our team ranked 5th based on the BERTScore metric. We achieved 3rd place with ROUGE, BLEURT, and RefCLIPScore metrics. For BLEU-1, METEOR, and CIDEr scores, we achieved 2nd place. Notably, we attained 1st place with a CLIPScore of 0.827074. These results demonstrate the model’s expected performance.
In this paper, we implemented and experimented with the BLIP model for the task of medical image captioning in the ImageCLEFmedical 2024 Caption challenge. The experimental results across various configurations showed promising outcomes, with the model achieving a CLIPScore of 0.82707 on the test set of the ROCOv2 dataset. Despite these achievements, there is still room for improvement in our research. The primary weakness of the model is its pre-training on a dataset significantly diferent from the medical domain, resulting in considerable bias.
Moving forward, we aim to enhance the model’s accuracy by utilizing pre-trained models with datasets that are more closely aligned with medical and diagnostic domains. Additionally, we plan to apply preprocessing methods tailored to diferent types of medical images to further improve the performance. Exploring domain-specific augmentation techniques and integrating more diverse medical datasets could also provide substantial gains. By addressing these areas, we hope to develop a more robust and accurate medical image captioning model, which can be a valuable tool in clinical settings for aiding diagnosis and assisting non-experts in understanding medical imagery.
Furthermore, future research will involve a detailed analysis of the model’s errors to understand the underlying reasons for its mispredictions. This analysis will guide the development of more efective strategies for fine-tuning and enhancing the model’s capabilities. Our ultimate goal is to contribute to the advancement of AI in healthcare by providing reliable and interpretable models that can support medical professionals and improve patient outcomes.
This research is funded by University of Information Technology-Vietnam National University HoChiMinh City under grant number D4-2024-01.
URL: https://aclanthology.org/W05-0909. [17] R. Vedantam, C. L. Zitnick, D. Parikh, Cider: Consensus-based image description evaluation, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4566–4575. doi:10.1109/CVPR.2015.7299087. [18] J. Hessel, A. Holtzman, M. Forbes, R. Bras, C. Yejin, Clipscore: A reference-free evaluation metric for image captioning, 2021, pp. 7514–7528. doi:10.18653/v1/2021.emnlp-main.595. [19] L. Jin, G. Luo, Y. Zhou, X. Sun, G. Jiang, A. Shu, R. Ji, Refclip: A universal teacher for weakly supervised referring expression comprehension, 2023, pp. 01–10. doi:10.1109/CVPR52729. 2023.00263. [20] A. Ben Abacha, W.-w. Yim, G. Michalopoulos, T. Lin, An investigation of evaluation methods in automatic medical note generation, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, Toronto, Canada, 2023, pp. 2575–2588. URL: https://aclanthology.org/2023.findings-acl.161. doi: 10. 18653/v1/2023.findings-acl.161. [21] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. URL: https://openreview.net/forum?id=Bkg6RiCqY7.