The rapid growth of deep learning techniques has profoundly influenced various sectors, notably medical imaging [1]. Among these advancements, using neural networks in radiology has garnered considerable attention due to its potential to enhance diagnostic accuracy and efficiency [2]. A particularly intriguing development in this field is the automatic generation of medical captions from radiology images [3]. This innovation aims to assist radiologists by providing preliminary interpretations and streamlining clinical documentation. Medical caption generation transforms visual information from radiological images into coherent, clinically valuable language descriptions. This process is inherently challenging due to the complexity and diversity of medical images, the need for precise and context-aware descriptions, and the necessity to incorporate domain-specific knowledge [3,4,5].
Traditional systems often fall short of these requirements, leading to the development of advanced attention mechanisms that can more effectively capture and interpret the intricate details found in radiological images. Recent research shows that integrating concept detection into caption generation algorithms improves performance [6,7]. Concept detection involves identifying and categorizing critical visual elements in an image, such as anatomical structures, pathological findings, and medical devices. By incorporating these detected concepts into the caption generation process, models can produce more accurate and contextually relevant descriptions. One of the advancements in this field is the ImageCLEF campaign, an annual multimodal machine learning competition established in 2003.
ImageCLEF [8] fosters advancements in multimedia processing, including computer vision, image analysis, classification, and retrieval in multilingual and multimodal contexts. In ImageCLEF 2024 [8], participants engaged in the ImageCLEFmedical Caption task [9], which included two subtasks: concept detection, aiming to identify critical elements within medical images, and caption prediction, focused on generating descriptive texts based on identified concepts. Concept detection aims to associate biomedical images with relevant medical concepts, thereby enhancing diagnostic notes by identifying key concepts that should be included in preliminary reports. Moreover, it facilitates the efficient organization and retrieval of medical images by indexing them according to related concepts. Caption prediction, or diagnostic captioning, remains a complex research challenge intended to support the diagnostic process by providing preliminary reports, rather than replacing physicians. This approach aids experienced clinicians in managing high volumes of daily medical examinations more swiftly and efficiently, while also reducing the likelihood of clinical errors among less experienced clinicians.
Our findings underscore that integrating concept detection enhances the efficacy of attention mechanisms and yields more coherent and diagnostically valuable captions. This research advances the development of intelligent technologies aimed at supporting radiologists in clinical practice, thereby elevating the standard of patient care. Section 2 provides a comprehensive review of pertinent literature. Section 3 outlines our dataset, while Section 4 describes our proposed methodology and presents experimental results. In Section 5, we discuss the conclusions drawn from our findings and outline avenues for future research. Our objective is to contribute to the fields of medical imaging and natural language processing by enhancing the capabilities of medical caption generation, thus paving the way for further advancements in automated reporting and medical data interpretation.
Medical imaging has been a focal point in the application of deep learning, benefiting from the availability of comprehensive datasets. Early datasets such as the NIH ChestX-ray14 [10] provided a large collection of chest radiographs annotated with disease labels, facilitating advancements in image classification and disease detection tasks. The MIMIC-CXR dataset [11], developed by Johnson et al., further enriched the field by offering not only radiographic images but also paired radiology reports, enabling research in image-to-text generation. These datasets have been pivotal in training and validating deep learning models, providing the groundwork for more sophisticated tasks such as medical caption generation and concept detection.
Concept detection in medical imaging involves identifying and categorizing essential visual elements such as anatomical structures, pathological findings, and medical devices. This task is crucial for generating accurate and contextually relevant medical captions. Early methods primarily relied on traditional machine learning techniques, which often struggled with the complexity and variability of medical images (e.g., SVMs (support vector machines), random forests, and k-nearest neighbors). However, recent advancements in deep learning, particularly CNNs (convolutional neural networks), have improved the accuracy of concept detection. Notable CNN architectures such as ResNet50 [12] and EfficientNet [13] have demonstrated substantial improvements in detecting and classifying visual elements in medical images.
Recently, Transformer-based models have been increasingly applied to concept detection due to their ability to capture long-range dependencies and contextual information. Notable examples include ViT (Vision Transformer) [14], BEiT (Bidirectional Encoder representation from Image Transformers) [15], and Swin Transformer [16]. These models provide robust feature representations and have shown promise in enhancing the accuracy and interpretability of medical image analysis.
Caption prediction, or diagnostic captioning, involves generating descriptive text that accurately summarizes the medical content of an image. This task extends beyond simple image annotation, requiring models to produce coherent and clinically meaningful narratives. Traditional captioning methods often used template-based approaches, which lacked flexibility and adaptability to different medical contexts. With the advent of deep learning, particularly sequence-to-sequence models and attention mechanisms, more sophisticated captioning systems have been developed.
For example, Jing et al. proposed a hierarchical LSTM (Long Short-Term Memory) [17] model combined with a co-attention mechanism to generate detailed radiology reports from medical images. Their model effectively captured the hierarchical structure of medical reports, producing more detailed and contextually appropriate captions [3].
The introduction of Transformer models specifically designed for the medical domain has advanced the field of medical image captioning. Transformers, particularly models like BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) [18], have demonstrated exceptional capabilities in understanding and generating biomedical text due to their ability to handle complex medical terminology and contexts. Recent research has leveraged these models to improve medical captioning. Additionally, LLMs (large language models) such as BioGPT [19] have been explored for their potential to generate coherent and diagnostically valuable medical captions, further pushing the boundaries of automated reporting in radiology.
ImageCLEF has included medical tasks annually since 2004. Since 2019, it has focused each medical task on a specific issue but combined them into a single task with multiple subtasks. Four tasks are proposed for 2024: Image Captioning, Image Question Answering for Colonoscopy Images, MEDIQA-MAGIC, Quality Control of Synthesized Medical Images Generated by GANs. In ImageCLEF 2024 [8], we engage in the Image Captioning task [9], simultaneously participating in two subtasks: Concept Detection Task and Caption Prediction Task, each crucial in the holistic process of generating informative captions for medical images.
• Concept Detection Task: The Concept Detection Task involves using a refined subset of the UMLS 2022 AB version for concept generation. This subset is carefully selected to enhance the accuracy of concept detection by filtering concepts based on their semantic types. Moreover, to optimize concept detection from images, a stringent exclusion criterion is applied to remove low-frequency concepts, based on insights from previous iterations. • Caption Prediction Task: In the Caption Prediction Task, a series of meticulous preprocessing steps are undertaken to ensure the integrity and coherence of the captioning process. Specifically, the removal of embedded hyperlinks within captions is performed as a fundamental preprocessing step. This careful action helps maintain data cleanliness and consistency, thereby supporting subsequent analytical processes and enabling accurate caption prediction outcomes.
The data for the captioning task will consist of images selected from medical literature, including annotations and related UMLS terms manually curated as metadata. For the development dataset, Radiology Objects in COntext Version 2 (ROCOv2) [20], an updated and expanded version of the Radiology Objects in COntext (ROCO) dataset [21], is used for both subtasks. As in previous versions, this dataset originates from biomedical articles in the PMC OpenAccess collection [22], with the test set comprising a set of previously unseen images.
• Training Dataset: Includes 70,108 images.
• Validation Dataset: Includes 9,972 images.
• Test Dataset: Includes 17,237 images.
We aim to extract features from images by carefully examining and testing a variety of pretrained models that fall into three main architectural paradigms, which are shown in Table 1. The list that follows summarizes the particular models that are being examined:
• CNN-based architectures: Microsoft/ResNet-50 [23], an archetype of conventional convolutional neural network (CNN) models, characterized by its utilization of residual blocks to mitigate the challenges associated with gradient vanishing, thereby enhancing model performance within computationally tractable bounds. • Transformer-based architectures:
-ViT (Vision Transformer) [14]: Pioneering the paradigm shift in image data processing, ViT adopts a transformative approach by encoding images into patch embeddings, followed by feature extraction using a Transformer encoder, reminiscent of text data processing methodologies. -DeiT (Data-efficient Image Transformers) [24]: An evolution of ViT, DeiT emphasizes data efficiency, facilitating training with reduced data volumes while preserving commendable performance metrics. -Swin-v2 (Shifted Window Transformer v2) [25]: Distinguished by its innovative utilization of self-attention mechanisms within shifted windows, Swin-v2 ameliorates computational complexity and augments performance across a spectrum of tasks, including image classification and segmentation. -BEiT (Bidirectional Encoder representation from Image Transformers) [26]: At the confluence of Transformer and BERT architectures, BEiT excels in capturing robust image features through bidirectional encoding methodologies. -BiomedCLIP [27]: A domain-specific adaptation of ViT tailored for biomedical applications, leveraging the CLIP architecture to enhance performance in medical domain tasks.
• Model Ensembles: In our ensemble framework, we leverage sophisticated fusion techniques to harness the collective predictive power of multiple models. A key method employed is weighted averaging, where predictions from each member model are aggregated based on their respective weights derived from validation performance.
- Following the feature extraction step, the retrieved features pass via a linear layer and classifier, where they are transformed and classified to provide outputs that correspond to the chosen class categories. This key step emphasizes the thorough orchestration of feature transformation and classification to produce predictions specific to the required class taxonomy.
A certain process must be followed while using the BEiT (Bidirectional Encoder Representations from Image Transformers) model in order to carry out idea filtering and modify the output threshold to detect variations in the outcomes. The following are the steps to follow: On a given dataset, do inference using the BEiT model and modify the output threshold to filter the ideas or classes.
Setting various threshold values and watching the ensuing outcomes allows for this modification. We may adjust and assess how different thresholds affect the model's performance using this procedure. Given the primary focus on Image Captioning in this research, the architectural design must effectively extract salient features from both the image and its corresponding text, combining them to generate the final caption. Our carefully curated multimodal fusion architecture incorporates essential components like an image encoder for pertinent feature extraction, a text encoder for eliciting semantic information from text, and a decoder to synthesize insights from the textual context. Additionally, the fusion mechanism integrates image features and output classifications from concept detection, synergistically blending them with textual input to decode and generate the caption output. The proposed approach leverages the pretrained Bidirectional Encoder Representations from Transformers (BEiT) model for image feature extraction. Boasting a symmetric Transformer architecture, BEiT can comprehend image representational features by concurrently considering both surrounding image patches and global context. With its extensive training on copious data, BEiT can be fine-tuned and achieve state-of-the-art results across several computer vision and image processing benchmarks.
To encode the input text captions, this research employs two domain-specific language models:
BioBART (Bidirectional and Auto-Regressive Transformers for Biomedical Text) and ClinicalT5 (Textto-Text Transfer Transformer fine-tuned on clinical data).
• BioBART [28] is a version of the BART model [29] adapted and further pre-trained on biomedical text data such as medical literature, case reports, and genomic analysis documents. Leveraging its bidirectional Transformer architecture, BioBART can effectively encode both general and biomedical domain-specific text, enabling the extraction of rich semantic representations for tasks like text summarization, medical question-answering, and report generation. • ClinicalT5 [30] is the T5 model [31] additionally fine-tuned on clinical text data including patient records and consultation reports. Harnessing its text-to-text transfer learning capability for multi-task modeling, ClinicalT5 can be applied to various natural language processing tasks in the healthcare domain, such as treatment classification, medical information extraction, and summarization of patient records.
For the process of encoding text concepts, we utilize the output from the BEiT model, which is specifically trained for the concept detection task. During this process, we apply a threshold of 0.5 to selectively retain predictions with a confidence score higher than 0.5, while discarding predictions with lower confidence scores. This discriminative process aids in capturing the semantic essence of the detected concepts, thereby facilitating their seamless integration into the multimodal fusion architecture for further processing and analysis.
Several experiments have been conducted to assess the efficacy of the proposed methodologies in addressing the ImageCLEFmedical Caption 2024 challenge. Specifically, each pre-trained vision model has been instantiated and evaluated, as detailed in Table 2, which offers a comprehensive overview of the pre-trained models employed in this study, encompassing their respective vision model designations, versions, and parameter counts for each fusion model. These experiments serve to elucidate both the potential and limitations inherent in each model with regard to the Image Captioning task, thereby facilitating the selection of the optimal approach for generating final predictions on the private test dataset of the competition.
• Concept Detection Task: For the concept detection subtask, the optimization criterion utilized during training is the AdamW optimizer [32]. The models are trained for 5 epochs with a batch size of 30 and an initial learning rate of 5e-5. During training, the BCEWithLogitsLoss function, which combines a Sigmoid layer and BCELoss, is applied, and a threshold value ranging from 0.45 to 0.5 is predominantly used to process the model's output. To ensure meaningful comparison results, consistent hyperparameters are maintained across all experiments. • Caption Prediction Task: During the training process for the caption prediction task, the CrossEntropyLoss criterion is applied with the ignore_index parameter set to the pad token index of the tokenizer. This setup helps mitigate the influence of pad tokens on loss computation, ensuring more precise training outcomes. For optimization, the AdamW optimizer is utilized with a learning rate of 1e-4 and a weight decay rate of 0.01, chosen to balance training efficiency and model generalization [32]. To leverage the benefits of Mixed Precision Training [33], the Gradient scaler is integrated into the training pipeline. This scaler adjusts the gradient scale, enhancing training efficiency and convergence speed of the models. Additionally, the LinearScheduleWithWarmup is employed to adjust the learning rate over time during training. This scheduling mechanism requires pre-defining the number of warmup steps and total training steps to optimize the learning rate schedule effectively. During each training iteration, a batch size of 16 is utilized. Overall, these training configurations and optimizations contribute to the performance and stability of the training process, leading to superior model performance.
The hardware utilized for computation included both NVIDIA Tesla T4 and NVIDIA Tesla P100 GPUs.
Our evaluation consists of two tasks: Concept Detection and Caption Prediction. Each task uses specific metrics to measure performance.
• Concept Detection Task: We assess the performance of concept identification using Accuracy, Precision, Recall, and F1 score. These metrics measure overall correctness, positive prediction accuracy, relevant concept capture, and balanced precision and recall, respectively [34]. • Caption Prediction Task: We evaluate the quality and coherence of generated captions using BERTScore (Bidirectional Encoder Representations from Transformers Score) [35], BLEU (Bilingual Evaluation Understudy, 1-4) [36], ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [37], and METEOR (Metric for Evaluation of Translation with Explicit ORdering) [38]. These metrics assess semantic similarity, fluency, relevance, coherence, informativeness, and lexical/syntactic aspects.
Using this diverse set of metrics, we ensure a comprehensive understanding of the model's performance and facilitate informed decision-making for further refinement.
As detailed in Table 2, the comparative evaluation of various concept detection models on the development validation set yields valuable insights into their performance across diverse evaluation metrics. Among these models, Swin-v2 emerges as the frontrunner, exhibiting the highest accuracy (0.16366), recall (0.47114), and F1 score (0.58944). This underscores Swin-v2's effectiveness in not only accurately identifying pertinent instances but also striking a harmonious balance between precision and recall, rendering it well-suited for concept detection endeavors. Ensemble methodologies, which predictions from multiple models, demonstrate promising outcomes as well. Notably, the Ensemble-2 model showcases commendable precision (0.94501) and a noteworthy F1 score (0.58581), suggesting that leveraging diverse models can augment predictive efficacy, particularly in precision-oriented tasks. While the Ensemble-4 model marginally surpasses Ensemble-2 in precision (0.94508), it exhibits a slightly lower F1 score (0.58460), implying a subtle trade-off in recall when employing additional models. BEiT-L and BiomedCLIP also manifest robust performance metrics. BEiT-L achieves an accuracy of 0.16145 and an F1 score of 0.58418, while BiomedCLIP demonstrates balanced performance with an accuracy of 0.15975 and an F1 score of 0.58319. These findings underscore the efficacy of these models in maintaining high precision and achieving a favorable balance with recall.
Other models such as BEiT-B, DeiT-B, and ViT-B exhibit commendable performance, albeit slightly trailing the top performers. For instance, BEiT-B records an accuracy of 0.15554 and an F1 score of 0.57662, indicating respectable yet not leading-edge performance. Similarly, DeiT-B and ViT-B attain comparable results, with DeiT-B registering an accuracy of 0.15674 and an F1 score of 0.57641, and ViT-B yielding an accuracy of 0.15413 and an F1 score of 0.57439. Conversely, ResNet-50 demonstrates notably inferior performance across all metrics, with an accuracy of 0.11412 and an F1 score of 0.51566. This underscores its relatively limited efficacy in the concept detection task.
In summation, the Swin-v2 model emerges as the most dependable choice for concept detection owing to its superior accuracy, recall, and F1 score. Ensemble methodologies, particularly Ensemble-2, exhibit robust performance, underscoring the advantages of model amalgamation. BEiT-L and BiomedCLIP offer balanced performance, rendering them viable alternatives. Meanwhile, ResNet-50's diminished performance suggests its lesser suitability for this specific task, underscoring the strides made by newer architectural advancements. As detailed in Table 3, the comparative analysis of various model configurations on the validation set reveals insights into the efficacy of incorporating concepts and post-processing techniques in caption generation tasks. The models evaluated include BEiT+BioBart and BEiT+Clinical-T5, with configurations either incorporating concepts derived from the Concept Detection subtask or excluding them, and applying post-processing to mitigate repetition in the output captions. The results indicate that for the BEiT+BioBart model, the inclusion of concepts and the application of post-processing do not result in any variation in performance across all evaluated metrics, including BERTScore, BLEU (from 1 to 4), ROUGE, and METEOR. This suggests that for BEiT+BioBart, the post-processing step does not impact the model's ability to generate captions when concepts are included, maintaining consistent performance.
In contrast, the BEiT+Clinical-T5 model demonstrates a more nuanced response to the incorporation of concepts and post-processing. When concepts are included without post-processing, there is a slight decline in BERTScore compared to the configuration without concepts. However, BLEU, ROUGE, and METEOR scores show an improvement with the inclusion of concepts, highlighting the potential benefits of concept integration in enhancing the model's performance in these specific metrics. Notably, when post-processing is applied, the BEiT+Clinical-T5 model exhibits substantial improvements across all metrics, irrespective of the presence of concepts. This improvement underscores the critical role of post-processing in refining output quality, with the highest METEOR score observed in the configuration without concepts but with post-processing. Comparing the two models, BEiT+Clinical-T5 generally outperforms BEiT+BioBart in BLEU, ROUGE, and METEOR scores. This superior performance is particularly evident when post-processing is applied, suggesting that BEiT+Clinical-T5 is more responsive to post-processing enhancements. However, BEiT+BioBart achieves a higher BERTScore when concepts are included, indicating a potential strength in semantic similarity measures.
In conclusion, the analysis underscores the importance of model selection, the strategic inclusion of concepts, and the application of post-processing in optimizing caption generation performance. BEiT+Clinical-T5 emerges as a more robust model with gains from post-processing, while BEiT+BioBart maintains consistent performance with concept inclusion. These findings provide valuable insights for future research and development in automated caption generation systems, emphasizing tailored approaches for different model architectures.
As detailed in Table 4, the performance evaluation of different models on the validation and private test sets provides a comprehensive understanding of their effectiveness across various configurations and datasets. For concept detection, three configurations were assessed: Concept BEiT-B with a threshold of 0.45, Detection BEiT-B with a threshold of 0.5, and Swin-v2 with a threshold of 0.5. The results reveal For caption prediction, four configurations were evaluated: BEiT+Clinical-T5 without concepts and without post-processing, BEiT+Clinical-T5 with concepts and without post-processing, BEiT+Clinical-T5 with concepts and with post-processing, and BEiT+BioBart with concepts and with post-processing. The BEiT+Clinical-T5 model without concepts and post-processing scored 0.46001 on the validation set and 0.4433 on the private test set, while adding concepts slightly improved the private test set score to 0.4453. However, the most considerable performance boost was observed when post-processing was applied to the BEiT+Clinical-T5 model with concepts, raising the scores to 0.57597 on the validation set and 0.558 on the private test set. This highlights the substantial role of post-processing in enhancing model performance.
Moreover, the BEiT+BioBart model with concepts and post-processing achieved the highest scores among all configurations, with 0.60589 on the validation set and 0.5794 on the private test set. This underscores the effectiveness of combining concepts with post-processing in the BioBart architecture, suggesting that such integration can improve caption generation quality. Overall, the analysis emphasizes the critical influence of model configuration, the integration of concepts, and the application of post-processing on the performance outcomes. The superior performance of the Swin-v2 model for concept detection and the BEiT+BioBart model for caption prediction indicates that different models may excel in specific sub-tasks, advocating for a nuanced approach in model selection and optimization based on the task requirements and dataset characteristics. As detailed in Table 5, when employing the BEiT model in conjunction with ClinicalT5 for medical image analysis, several notable errors have been observed across various dimensions. These errors include incorrect identification of regions or image types, omissions in providing specific details, and inaccuracies in context, thereby impacting the overall reliability of the model's results. The model occasionally encounters difficulties in accurately identifying regions of interest within the images. 5. a) ImageCLEFmedical_Caption_2024_valid_009001 is an example of Table 5. b) ImageCLEFmedical_Caption_2024_valid_009698 is an example of Table 5.
For instance, it might misinterpret an anteroposterior X-ray of the pelvis as indicating bilateral tibial fractures. Similarly, it might incorrectly classify a cross-sectional, contrast-enhanced CT scan of the larynx as a left renal tumor.
Omissions in providing specific details have become evident in the model's predictions. The model often fails to provide the complex details necessary for comprehensive clinical interpretation. For example, it may overlook critical features such as the eccentric position of a metallic head in an Xray or the presence of stratified bile in a CT scan. Moreover, contextual inaccuracies are common, leading to misleading or entirely incorrect descriptions. The model sometimes struggles to grasp the broader context of medical images, resulting in descriptions that do not align appropriately with the actual content of the images. Similarly, when utilizing the BEiT model in combination with BioBART, analogous errors have been observed across various aspects. These include incorrect identification of regions or image types, omissions in providing specific details, and contextual inaccuracies. Comparing BEiT with ClinicalT5 and BEiT with BioBART, although both models exhibit similar error patterns, there are minor differences in their performance. BEiT combined with ClinicalT5 demonstrates slightly better performance in certain aspects, such as providing more accurate descriptions and better contextual understanding. Conversely, BEiT combined with BioBART shows a slight advantage in specific scenarios, particularly in identifying anatomical structures or image types. However, both models have room for improvement, highlighting ongoing challenges in developing robust and reliable automated methods for medical image analysis. In both models, conceptual errors frequently occur, indicating a mismatch between the predicted concept and the actual content of the medical images. These errors underscore the challenges in accurately interpreting and classifying medical images based on their content.
To enhance the accuracy of medical image analysis models, a range of strategies must be employed to improve data quality, model architecture, and training processes. Firstly, the use of high-quality, well-annotated datasets is crucial. Combining this with data augmentation techniques such as rotation, zooming, flipping, and color adjustment can help increase the size and diversity of the training dataset, thereby enhancing the model's generalization capabilities. In terms of model architecture, employing models pre-trained on domain-specific datasets or state-of-the-art (SOTA) models that achieve superior results is essential. Furthermore, incorporating additional feature extraction from image data, such as bounding-boxes, segmentation, or advanced features, can help the model better understand the structure and context of the images. Finally, regularly testing and re-evaluating the model using diverse datasets will help in early detection of errors and timely adjustment of the model, ensuring the reliability and accuracy of medical image analysis results.
In this study, an enhanced approach to medical caption generation was introduced by integrating concept detection into attention mechanisms. The method improved performance metrics, with the Swin-v2 model achieving an F1 score of 0.58944 on the validation set and 0.61998 on the private test set, earning 3rd place in concept detection. For caption prediction, the BEiT+BioBart model, augmented with concept integration and post-processing, achieved a BERTScore of 0.60589 on the validation set and 0.5794 on the private test set, securing 9th place. These results underscore the effectiveness of concept-aware systems in generating precise and contextually relevant medical descriptions.
Future work will focus on enhancing model performance through several avenues unrelated to data expansion. First, optimizing model architectures and training protocols can further improve accuracy and efficiency. Second, incorporating more advanced attention mechanisms and fine-tuning hyperparameters may yield better contextual understanding and caption quality. Third, integrating explainability techniques will ensure that model predictions are interpretable and trustworthy for healthcare professionals. Additionally, exploring transfer learning and domain adaptation techniques could enhance model performance across various medical imaging modalities. Furthermore, leveraging large language models (LLMs) such as GPT-3 and BioGPT for their potential to generate coherent and diagnostically valuable medical captions will be explored [39] [19]. Finally, developing robust post-processing algorithms to further refine generated captions, ensuring they meet clinical standards, is planned. These efforts aim to advance the capabilities of medical image analysis and automated reporting systems, contributing to more sophisticated and reliable tools for the healthcare industry.
This research is funded by University of Information Technology-Vietnam National University HoChiMinh City under grant number D4-2024-01.