Purpose: Our study presents an enhanced approach to medical image caption generation by integrating concept detection into attention mechanisms. Method: This method utilizes sophisticated models to identify critical concepts within medical images, which are then refined and incorporated into the caption generation process. Results: Our concept detection task, which employed the Swin-v2 model, achieved an F1 score of 0.58944 on the validation set and 0.61998 on the private test set, securing the third position. For the caption prediction task, our BEiT+BioBart model, enhanced with concept integration and post-processing techniques, attained a BERTScore of 0.60589 on the validation set and 0.5794 on the private test set, placing ninth. Conclusion: These results underscore the eficacy of concept-aware algorithms in generating precise and contextually appropriate medical descriptions. The findings demonstrate that our approach considerably improves the quality of medical image captions, highlighting its potential to enhance medical image interpretation and documentation, thereby contributing to improved healthcare outcomes.
The rapid growth of deep learning techniques has profoundly influenced various sectors, notably medical
imaging [
Traditional systems often fall short of these requirements, leading to the development of advanced
attention mechanisms that can more efectively capture and interpret the intricate details found in
radiological images. Recent research shows that integrating concept detection into caption generation
algorithms improves performance [
ImageCLEF [
Our findings underscore that integrating concept detection enhances the eficacy of attention mechanisms and yields more coherent and diagnostically valuable captions. This research advances the development of intelligent technologies aimed at supporting radiologists in clinical practice, thereby elevating the standard of patient care. Section 2 provides a comprehensive review of pertinent literature. Section 3 outlines our dataset, while Section 4 describes our proposed methodology and presents experimental results. In Section 5, we discuss the conclusions drawn from our findings and outline avenues for future research. Our objective is to contribute to the fields of medical imaging and natural language processing by enhancing the capabilities of medical caption generation, thus paving the way for further advancements in automated reporting and medical data interpretation.
Medical imaging has been a focal point in the application of deep learning, benefiting from the availability
of comprehensive datasets. Early datasets such as the NIH ChestX-ray14 [
Concept detection in medical imaging involves identifying and categorizing essential visual elements
such as anatomical structures, pathological findings, and medical devices. This task is crucial for
generating accurate and contextually relevant medical captions. Early methods primarily relied on
traditional machine learning techniques, which often struggled with the complexity and variability
of medical images (e.g., SVMs (support vector machines), random forests, and k-nearest neighbors).
However, recent advancements in deep learning, particularly CNNs (convolutional neural networks),
have improved the accuracy of concept detection. Notable CNN architectures such as ResNet50 [
Recently, Transformer-based models have been increasingly applied to concept detection due to their
ability to capture long-range dependencies and contextual information. Notable examples include ViT
(Vision Transformer) [
Caption prediction, or diagnostic captioning, involves generating descriptive text that accurately summarizes the medical content of an image. This task extends beyond simple image annotation, requiring models to produce coherent and clinically meaningful narratives. Traditional captioning methods often used template-based approaches, which lacked flexibility and adaptability to diferent medical contexts. With the advent of deep learning, particularly sequence-to-sequence models and attention mechanisms, more sophisticated captioning systems have been developed.
For example, Jing et al. proposed a hierarchical LSTM (Long Short-Term Memory) [
The introduction of Transformer models specifically designed for the medical domain has advanced
the field of medical image captioning. Transformers, particularly models like BioBERT (Bidirectional
Encoder Representations from Transformers for Biomedical Text Mining) [
ImageCLEF has included medical tasks annually since 2004. Since 2019, it has focused each medical task
on a specific issue but combined them into a single task with multiple subtasks. Four tasks are proposed
for 2024: Image Captioning, Image Question Answering for Colonoscopy Images, MEDIQA-MAGIC,
Quality Control of Synthesized Medical Images Generated by GANs. In ImageCLEF 2024 [
• Concept Detection Task: The Concept Detection Task involves using a refined subset of the UMLS 2022 AB version for concept generation. This subset is carefully selected to enhance the accuracy of concept detection by filtering concepts based on their semantic types. Moreover, to optimize concept detection from images, a stringent exclusion criterion is applied to remove low-frequency concepts, based on insights from previous iterations. • Caption Prediction Task: In the Caption Prediction Task, a series of meticulous preprocessing steps are undertaken to ensure the integrity and coherence of the captioning process. Specifically, the removal of embedded hyperlinks within captions is performed as a fundamental preprocessing step. This careful action helps maintain data cleanliness and consistency, thereby supporting subsequent analytical processes and enabling accurate caption prediction outcomes.
The data for the captioning task will consist of images selected from medical literature, including
annotations and related UMLS terms manually curated as metadata. For the development dataset,
Radiology Objects in COntext Version 2 (ROCOv2) [
• Training Dataset: Includes 70,108 images.
We aim to extract features from images by carefully examining and testing a variety of pretrained
models that fall into three main architectural paradigms, which are shown in Table 1. The list that
follows summarizes the particular models that are being examined:
• CNN-based architectures: Microsoft/ResNet-50 [23], an archetype of conventional
convolutional neural network (CNN) models, characterized by its utilization of residual blocks to mitigate
the challenges associated with gradient vanishing, thereby enhancing model performance within
computationally tractable bounds.
• Transformer-based architectures:
– ViT (Vision Transformer) [
– Ensemble-2 model (Swin-v2 + BEiT): The symbiotic fusion of Swin-v2 and BEiT engenders a collaborative synergy, capitalizing on the distinctive strengths of each constituent model to surpass individual model performances. – Ensemble-4 model (Swin-v2 + BEiT + DeiT + ViT): Comprising a composite quartet of models, this ensemble fortifies accuracy and generalization capabilities through the combination of representatives from Transformer-based models.
Following the feature extraction step, the retrieved features pass via a linear layer and classifier, where they are transformed and classified to provide outputs that correspond to the chosen class categories. This key step emphasizes the thorough orchestration of feature transformation and classification to produce predictions specific to the required class taxonomy. Concept Filtering A certain process must be followed while using the BEiT (Bidirectional Encoder Representations from Image Transformers) model in order to carry out idea filtering and modify the output threshold to detect variations in the outcomes. The following are the steps to follow: On a given dataset, do inference using the BEiT model and modify the output threshold to filter the ideas or classes. Setting various threshold values and watching the ensuing outcomes allows for this modification. We may adjust and assess how diferent thresholds afect the model’s performance using this procedure.
Given the primary focus on Image Captioning in this research, the architectural design must efectively extract salient features from both the image and its corresponding text, combining them to generate the ifnal caption. Our carefully curated multimodal fusion architecture incorporates essential components like an image encoder for pertinent feature extraction, a text encoder for eliciting semantic information from text, and a decoder to synthesize insights from the textual context. Additionally, the fusion mechanism integrates image features and output classifications from concept detection, synergistically blending them with textual input to decode and generate the caption output. The proposed approach leverages the pretrained Bidirectional Encoder Representations from Transformers (BEiT) model for image feature extraction. Boasting a symmetric Transformer architecture, BEiT can comprehend image representational features by concurrently considering both surrounding image patches and global context. With its extensive training on copious data, BEiT can be fine-tuned and achieve state-of-the-art results across several computer vision and image processing benchmarks.
To encode the input text captions, this research employs two domain-specific language models: BioBART (Bidirectional and Auto-Regressive Transformers for Biomedical Text) and ClinicalT5 (Textto-Text Transfer Transformer fine-tuned on clinical data).
• BioBART [28] is a version of the BART model [29] adapted and further pre-trained on biomedical text data such as medical literature, case reports, and genomic analysis documents. Leveraging its bidirectional Transformer architecture, BioBART can efectively encode both general and biomedical domain-specific text, enabling the extraction of rich semantic representations for tasks like text summarization, medical question-answering, and report generation. • ClinicalT5 [30] is the T5 model [31] additionally fine-tuned on clinical text data including patient records and consultation reports. Harnessing its text-to-text transfer learning capability for multi-task modeling, ClinicalT5 can be applied to various natural language processing tasks in the healthcare domain, such as treatment classification, medical information extraction, and summarization of patient records.
For the process of encoding text concepts, we utilize the output from the BEiT model, which is specifically trained for the concept detection task. During this process, we apply a threshold of 0.5 to selectively retain predictions with a confidence score higher than 0.5, while discarding predictions with lower confidence scores. This discriminative process aids in capturing the semantic essence of the detected concepts, thereby facilitating their seamless integration into the multimodal fusion architecture for further processing and analysis.
Several experiments have been conducted to assess the eficacy of the proposed methodologies in addressing the ImageCLEFmedical Caption 2024 challenge. Specifically, each pre-trained vision model has been instantiated and evaluated, as detailed in Table 2, which ofers a comprehensive overview of the pre-trained models employed in this study, encompassing their respective vision model designations, versions, and parameter counts for each fusion model. These experiments serve to elucidate both the potential and limitations inherent in each model with regard to the Image Captioning task, thereby facilitating the selection of the optimal approach for generating final predictions on the private test dataset of the competition.
• Concept Detection Task: For the concept detection subtask, the optimization criterion utilized during training is the AdamW optimizer [32]. The models are trained for 5 epochs with a batch size of 30 and an initial learning rate of 5e-5. During training, the BCEWithLogitsLoss function, which combines a Sigmoid layer and BCELoss, is applied, and a threshold value ranging from 0.45 to 0.5 is predominantly used to process the model’s output. To ensure meaningful comparison results, consistent hyperparameters are maintained across all experiments. • Caption Prediction Task: During the training process for the caption prediction task, the CrossEntropyLoss criterion is applied with the ignore_index parameter set to the pad token index of the tokenizer. This setup helps mitigate the influence of pad tokens on loss computation, ensuring more precise training outcomes. For optimization, the AdamW optimizer is utilized with a learning rate of 1e-4 and a weight decay rate of 0.01, chosen to balance training eficiency and model generalization [ 32]. To leverage the benefits of Mixed Precision Training [33], the Gradient scaler is integrated into the training pipeline. This scaler adjusts the gradient scale, enhancing training eficiency and convergence speed of the models. Additionally, the LinearScheduleWithWarmup is employed to adjust the learning rate over time during training. This scheduling mechanism requires pre-defining the number of warmup steps and total training steps to optimize the learning rate schedule efectively. During each training iteration, a batch size of 16 is utilized. Overall, these training configurations and optimizations contribute to the performance and stability of the training process, leading to superior model performance. The hardware utilized for computation included both NVIDIA Tesla T4 and NVIDIA Tesla P100 GPUs.
Our evaluation consists of two tasks: Concept Detection and Caption Prediction. Each task uses specific metrics to measure performance.
• Concept Detection Task: We assess the performance of concept identification using Accuracy, Precision, Recall, and F1 score. These metrics measure overall correctness, positive prediction accuracy, relevant concept capture, and balanced precision and recall, respectively [34]. • Caption Prediction Task: We evaluate the quality and coherence of generated captions using BERTScore (Bidirectional Encoder Representations from Transformers Score) [35], BLEU (Bilingual Evaluation Understudy, 1-4) [36], ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [37], and METEOR (Metric for Evaluation of Translation with Explicit ORdering) [38]. These metrics assess semantic similarity, fluency, relevance, coherence, informativeness, and lexical/syntactic aspects.
Using this diverse set of metrics, we ensure a comprehensive understanding of the model’s performance and facilitate informed decision-making for further refinement.
As detailed in Table 2, the comparative evaluation of various concept detection models on the development validation set yields valuable insights into their performance across diverse evaluation metrics. Among these models, Swin-v2 emerges as the frontrunner, exhibiting the highest accuracy (0.16366), recall (0.47114), and F1 score (0.58944). This underscores Swin-v2’s efectiveness in not only accurately identifying pertinent instances but also striking a harmonious balance between precision and recall, rendering it well-suited for concept detection endeavors. Ensemble methodologies, which predictions from multiple models, demonstrate promising outcomes as well. Notably, the Ensemble-2 model showcases commendable precision (0.94501) and a noteworthy F1 score (0.58581), suggesting that leveraging diverse models can augment predictive eficacy, particularly in precision-oriented tasks. While the Ensemble-4 model marginally surpasses Ensemble-2 in precision (0.94508), it exhibits a slightly lower F1 score (0.58460), implying a subtle trade-of in recall when employing additional models.
BEiT-L and BiomedCLIP also manifest robust performance metrics. BEiT-L achieves an accuracy of 0.16145 and an F1 score of 0.58418, while BiomedCLIP demonstrates balanced performance with an accuracy of 0.15975 and an F1 score of 0.58319. These findings underscore the eficacy of these models in maintaining high precision and achieving a favorable balance with recall.
Other models such as BEiT-B, DeiT-B, and ViT-B exhibit commendable performance, albeit slightly trailing the top performers. For instance, BEiT-B records an accuracy of 0.15554 and an F1 score of 0.57662, indicating respectable yet not leading-edge performance. Similarly, DeiT-B and ViT-B attain comparable results, with DeiT-B registering an accuracy of 0.15674 and an F1 score of 0.57641, and ViT-B yielding an accuracy of 0.15413 and an F1 score of 0.57439. Conversely, ResNet-50 demonstrates notably inferior performance across all metrics, with an accuracy of 0.11412 and an F1 score of 0.51566. This underscores its relatively limited eficacy in the concept detection task.
In summation, the Swin-v2 model emerges as the most dependable choice for concept detection owing to its superior accuracy, recall, and F1 score. Ensemble methodologies, particularly Ensemble-2, exhibit robust performance, underscoring the advantages of model amalgamation. BEiT-L and BiomedCLIP ofer balanced performance, rendering them viable alternatives. Meanwhile, ResNet-50’s diminished performance suggests its lesser suitability for this specific task, underscoring the strides made by newer architectural advancements.
As detailed in Table 3, the comparative analysis of various model configurations on the validation set reveals insights into the eficacy of incorporating concepts and post-processing techniques in caption generation tasks. The models evaluated include BEiT+BioBart and BEiT+Clinical-T5, with configurations either incorporating concepts derived from the Concept Detection subtask or excluding them, and applying post-processing to mitigate repetition in the output captions. The results indicate that for the BEiT+BioBart model, the inclusion of concepts and the application of post-processing do not result in any variation in performance across all evaluated metrics, including BERTScore, BLEU (from 1 to 4), ROUGE, and METEOR. This suggests that for BEiT+BioBart, the post-processing step does not impact the model’s ability to generate captions when concepts are included, maintaining consistent performance.
In contrast, the BEiT+Clinical-T5 model demonstrates a more nuanced response to the incorporation of concepts and post-processing. When concepts are included without post-processing, there is a slight decline in BERTScore compared to the configuration without concepts. However, BLEU, ROUGE, and METEOR scores show an improvement with the inclusion of concepts, highlighting the potential benefits of concept integration in enhancing the model’s performance in these specific metrics. Notably, when post-processing is applied, the BEiT+Clinical-T5 model exhibits substantial improvements across all metrics, irrespective of the presence of concepts. This improvement underscores the critical role of post-processing in refining output quality, with the highest METEOR score observed in the configuration without concepts but with post-processing. Comparing the two models, BEiT+Clinical-T5 generally outperforms BEiT+BioBart in BLEU, ROUGE, and METEOR scores. This superior performance is particularly evident when post-processing is applied, suggesting that BEiT+Clinical-T5 is more responsive to post-processing enhancements. However, BEiT+BioBart achieves a higher BERTScore when concepts are included, indicating a potential strength in semantic similarity measures.
In conclusion, the analysis underscores the importance of model selection, the strategic inclusion of concepts, and the application of post-processing in optimizing caption generation performance. BEiT+Clinical-T5 emerges as a more robust model with gains from post-processing, while BEiT+BioBart maintains consistent performance with concept inclusion. These findings provide valuable insights for future research and development in automated caption generation systems, emphasizing tailored approaches for diferent model architectures.
As detailed in Table 4, the performance evaluation of diferent models on the validation and private test sets provides a comprehensive understanding of their efectiveness across various configurations and datasets. For concept detection, three configurations were assessed: Concept BEiT-B with a threshold of 0.45, Detection BEiT-B with a threshold of 0.5, and Swin-v2 with a threshold of 0.5. The results reveal that the Swin-v2 model performs the best, achieving scores of 0.58944 on the validation set and 0.61998 on the private test set, suggesting superior capability in accurately detecting concepts compared to the BEiT-B models. The Concept BEiT-B model with a threshold of 0.45 also shows strong performance, though slightly lower than Swin-v2, indicating the threshold setting’s impact on model eficacy.
For caption prediction, four configurations were evaluated: BEiT+Clinical-T5 without concepts and without post-processing, BEiT+Clinical-T5 with concepts and without post-processing, BEiT+ClinicalT5 with concepts and with post-processing, and BEiT+BioBart with concepts and with post-processing. The BEiT+Clinical-T5 model without concepts and post-processing scored 0.46001 on the validation set and 0.4433 on the private test set, while adding concepts slightly improved the private test set score to 0.4453. However, the most considerable performance boost was observed when post-processing was applied to the BEiT+Clinical-T5 model with concepts, raising the scores to 0.57597 on the validation set and 0.558 on the private test set. This highlights the substantial role of post-processing in enhancing model performance.
Moreover, the BEiT+BioBart model with concepts and post-processing achieved the highest scores among all configurations, with 0.60589 on the validation set and 0.5794 on the private test set. This underscores the efectiveness of combining concepts with post-processing in the BioBart architecture, suggesting that such integration can improve caption generation quality. Overall, the analysis emphasizes the critical influence of model configuration, the integration of concepts, and the application of post-processing on the performance outcomes. The superior performance of the Swin-v2 model for concept detection and the BEiT+BioBart model for caption prediction indicates that diferent models may excel in specific sub-tasks, advocating for a nuanced approach in model selection and optimization based on the task requirements and dataset characteristics.
As detailed in Table 5, when employing the BEiT model in conjunction with ClinicalT5 for medical image analysis, several notable errors have been observed across various dimensions. These errors include incorrect identification of regions or image types, omissions in providing specific details, and inaccuracies in context, thereby impacting the overall reliability of the model’s results. The model occasionally encounters dificulties in accurately identifying regions of interest within the images.
For instance, it might misinterpret an anteroposterior X-ray of the pelvis as indicating bilateral tibial fractures. Similarly, it might incorrectly classify a cross-sectional, contrast-enhanced CT scan of the larynx as a left renal tumor.
Omissions in providing specific details have become evident in the model’s predictions. The model often fails to provide the complex details necessary for comprehensive clinical interpretation. For example, it may overlook critical features such as the eccentric position of a metallic head in an Xray or the presence of stratified bile in a CT scan. Moreover, contextual inaccuracies are common, leading to misleading or entirely incorrect descriptions. The model sometimes struggles to grasp the broader context of medical images, resulting in descriptions that do not align appropriately with the actual content of the images. Similarly, when utilizing the BEiT model in combination with BioBART, analogous errors have been observed across various aspects. These include incorrect identification of regions or image types, omissions in providing specific details, and contextual inaccuracies. Comparing BEiT with ClinicalT5 and BEiT with BioBART, although both models exhibit similar error patterns, there are minor diferences in their performance. BEiT combined with ClinicalT5 demonstrates slightly better performance in certain aspects, such as providing more accurate descriptions and better contextual understanding. Conversely, BEiT combined with BioBART shows a slight advantage in specific scenarios, particularly in identifying anatomical structures or image types. However, both models have room for improvement, highlighting ongoing challenges in developing robust and reliable automated methods for medical image analysis. In both models, conceptual errors frequently occur, indicating a mismatch between the predicted concept and the actual content of the medical images. These errors underscore the challenges in accurately interpreting and classifying medical images based on their content.
To enhance the accuracy of medical image analysis models, a range of strategies must be employed to improve data quality, model architecture, and training processes. Firstly, the use of high-quality, well-annotated datasets is crucial. Combining this with data augmentation techniques such as rotation, zooming, flipping, and color adjustment can help increase the size and diversity of the training dataset, thereby enhancing the model’s generalization capabilities. In terms of model architecture, employing models pre-trained on domain-specific datasets or state-of-the-art (SOTA) models that achieve superior results is essential. Furthermore, incorporating additional feature extraction from image data, such as bounding-boxes, segmentation, or advanced features, can help the model better understand the structure and context of the images. Finally, regularly testing and re-evaluating the model using diverse datasets will help in early detection of errors and timely adjustment of the model, ensuring the reliability and accuracy of medical image analysis results.
In this study, an enhanced approach to medical caption generation was introduced by integrating concept detection into attention mechanisms. The method improved performance metrics, with the Swin-v2 model achieving an F1 score of 0.58944 on the validation set and 0.61998 on the private test set, earning 3rd place in concept detection. For caption prediction, the BEiT+BioBart model, augmented with concept integration and post-processing, achieved a BERTScore of 0.60589 on the validation set and 0.5794 on the private test set, securing 9th place. These results underscore the efectiveness of concept-aware systems in generating precise and contextually relevant medical descriptions.
Future work will focus on enhancing model performance through several avenues unrelated to
data expansion. First, optimizing model architectures and training protocols can further improve
accuracy and eficiency. Second, incorporating more advanced attention mechanisms and fine-tuning
hyperparameters may yield better contextual understanding and caption quality. Third, integrating
explainability techniques will ensure that model predictions are interpretable and trustworthy for
healthcare professionals. Additionally, exploring transfer learning and domain adaptation techniques
could enhance model performance across various medical imaging modalities. Furthermore, leveraging
large language models (LLMs) such as GPT-3 and BioGPT for their potential to generate coherent
and diagnostically valuable medical captions will be explored [39] [
This research is funded by University of Information Technology-Vietnam National University HoChiMinh City under grant number D4-2024-01. Philadelphia, Pennsylvania, USA, 2002, pp. 311–318. URL: https://aclanthology.org/P02-1040. doi:10.3115/1073083.1073135. [37] C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013. [38] S. Banerjee, A. Lavie, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, in: J. Goldstein, A. Lavie, C.-Y. Lin, C. Voss (Eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Association for Computational Linguistics, Ann Arbor, Michigan, 2005, pp. 65–72.
URL: https://aclanthology.org/W05-0909. [39] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Curran Associates Inc., Red Hook, NY, USA, 2020.