This paper presents the development and evaluation of two deep learning models for the ImageCLEFmedical 2025 challenge, targeting automated medical concept detection and diagnostic captioning. The primary goal was to create practical models that enhance the accuracy and clinical relevance of automated radiological reporting. For the concept detection task, a novel dual-branch architecture named MedCSRA (Medical Class-Specific Residual Attention) was proposed. It integrates a Class-Specific Residual Attention branch with a Global branch, using a ResNet-101 backbone, to efectively balance localized and global visual features. For the caption prediction task, a pre-trained BLIP (Bootstrapping Language-Image Pre-training) model was fine-tuned on the competition's dataset to adapt its powerful vision-language capabilities to the medical domain. Both submissions achieved strong results in the oficial competition. The MedCSRA model secured a fourth-place ranking in the concept detection task, with a primary F1-score of 0.5613. In the captioning task, the fine-tuned BLIP model also achieved a fourth-place ranking, with a competitive overall score of 0.3554 and strong semantic coherence, as measured by a BERTScore (Recall) of 0.5951. Our results demonstrate two key findings. First, a custom dual-branch architecture that explicitly balances local and global context is a highly efective strategy for multi-label concept detection. Second, standard fine-tuning of a large pre-trained vision-language model like BLIP is suficient to achieve a top-tier ranking in semantic captioning metrics, though challenges in clinical factuality, measured by metrics like UMLS (Unified Medical Language System) Concept F1, remain. These findings validate our approaches as competitive solutions for complex biomedical image analysis tasks.
In recent years, machine learning—particularly deep learning—has been a driving force behind substantial advancements in biomedicine. As the volume of medical imaging data continues to grow rapidly, there is a rising need for intelligent systems that can extract meaningful information, assist with clinical decision-making, and streamline workflows in healthcare environments.
One key area in this domain is diagnostic captioning, which involves generating descriptive, diagnostic-level text based on medical images. This task holds great promise in assisting clinicians by improving reporting eficiency and reducing the risk of human error—especially for less experienced practitioners. Rather than replacing human expertise, these systems are designed to augment and support the diagnostic process by providing preliminary insights that guide medical professionals toward faster and more accurate decisions.
ImageCLEF [
Our team, UIT-Oggy, participated in the ImageCLEFmedical 2025 Caption task, which consists of two complementary subtasks: medical concept detection and caption prediction. In the concept detection task, the goal is to identify and extract medically relevant terms from an image, which can enhance indexing, retrieval, and diagnostic support. The caption prediction task—also referred to as diagnostic captioning—focuses on generating coherent, informative, and medically accurate descriptions of patient conditions and anatomical structures visible in the image.
While diagnostic captioning remains a challenging problem due to the complexity of medical language and visual interpretation, it represents a transformative tool in modern clinical workflows. By providing initial report drafts and highlighting important image features, these models can reduce report turnaround times and help clinicians manage increasing workloads. At the same time, concept detection ensures that critical medical terms are not overlooked and serves as a foundation for structured reporting and semantic image understanding.
In this paper, we detail our methods and results from the ImageCLEFmedical 2025 challenge [
This section reviews prior work in two key areas relevant to our participation in the ImageCLEFmedical 2025 challenge. We begin with a broad overview of deep learning’s impact on medical imaging before delving into task-specific advancements.
A comprehensive evaluation by Nguyen-Tat et al.[
The task of medical image captioning is distinguished from its general-domain counterpart by its
stringent requirements for clinical accuracy and domain-specific knowledge [
A pivotal advancement was the integration of attention mechanisms, enabling models to focus on
salient image regions. In an influential study, Jing et al.[
The current state-of-the-art is increasingly driven by Large Language Models (LLMs) and
pretrained vision-language models. Notably, models like Med-PaLM (Medical Pathways Language Model)
have demonstrated impressive capabilities by achieving a passing score on the US Medical Licensing
Examination [
Parallel to captioning, the automatic annotation of medical images with multiple concepts is a critical
multi-label classification task. As documented in the survey by Litjens et al. [
A key development in this area was the release of the large-scale ChestX-ray14 dataset, which
enabled the creation of high-performance models for thoracic disease detection [
The ImageCLEF 2025 [
Our work utilizes the oficial dataset provided for the ImageCLEFmedical 2025 Caption task. This
dataset is a curated version of the Radiology Objects in Context (ROCO) v2 dataset[
As shown in Figure 2, the distribution of caption lengths in the training set is highly right-skewed. Most captions are concise, typically fewer than 30 words, while a small number are extremely long, with the longest exceeding 800 words. This long-tailed behavior suggests that, while many image descriptions are brief, models must also be robust enough to handle verbose and complex medical text.
In Figure 3, we visualize the number of concepts annotated per image. The majority of images contain 1 to 4 concepts, with a sharp decline in frequency beyond that. Only a few cases involve 10 or more concepts. This confirms the multilabel nature of the task and indicates that models must be capable of identifying both sparse and dense sets of medical concepts depending on the image complexity.
Image preprocessing is a pivotal step in preparing medical images for captioning tasks, ensuring that input data is standardized and optimized for model performance. The approach focuses on transforming raw images into a format suitable for a vision-language model, balancing computational eficiency with the preservation of essential visual information.
The images are loaded from the dataset directory, specifically handling JPEG files, using a computer vision library. Each image is resized to a uniform resolution of 224x224 pixels. This fixed size aligns with the input requirements of the vision-language model, ensuring compatibility with its pre-trained vision component. Standardizing image dimensions reduces computational complexity and facilitates consistent feature extraction in diverse medical images.
A specialized processor, designed for the vision-language model, is used to prepare both images and their corresponding captions. This processor handles the following tasks: • Image Normalization: Pixel values are scaled and normalized to match the expected input range of the pre-trained model, ensuring consistent processing across varied image sources. • Text Tokenization: Captions are tokenized using the model’s tokenizer, with padding applied to a maximum length of 200 tokens and truncation using a strategy that prioritizes longer sequences.
This ensures that variable-length captions are uniformly formatted. • Tensor Conversion: Both images and tokenized captions are converted into tensors, with unnecessary dimensions removed to match the model’s input requirements. The resulting data includes image tensors, tokenized caption IDs, and attention masks to indicate valid tokens, which are critical for both training and inference.
For the caption prediction task, the BLIP model was employed as the foundational architecture [
The BLIP architecture integrates a Vision Transformer (ViT) [
The captioning model was implemented using the PyTorch framework and the Hugging
Face Transformers library [
This model was subsequently fine-tuned on the oficial ImageCLEFmedical 2025 Caption [
For the medical concept detection task, a novel dual-branch architecture named MedCSRA is proposed
for multi-label image classification. The core innovation of MedCSRA lies in its hybrid approach, which
integrates both global and local features to create a more comprehensive image representation. As
illustrated in Figure 5, the model consists of a shared visual backbone and two parallel processing
branches: a Global Branch and a Class-Specific Residual Attention (CSRA) Branch. The CSRA branch is
inspired by the attention mechanism introduced by Fang et al. [
The MedCSRA model employs a ResNet-101 architecture as its visual feature extraction backbone [
MedCSRA processes spatial features through two branches: the Class-Specific Residual Attention
Branch, adapted from reference [
The Global Branch, our original contribution, enhances the model’s ability to capture global semantic patterns across the entire image. Spatial features extracted from the ResNet-101 backbone are subjected to Global Average Pooling, reducing the spatial dimensions to a compact vector of size [batch_size, in_features]. This vector encapsulates overarching contextual information, which is then passed through a fully connected layer to generate global logits of size [batch_size, num_classes]. The use of Global Average Pooling ensures computational eficiency while preserving critical global context, making it efective for detecting medical concepts with difuse patterns. The design leverages the backbone’s hierarchical features, complementing the localized focus of the CSRA Branch.
Let ℋ = {ℎ1, ℎ2, . . . , ℎ} represent the spatial feature maps extracted by the backbone, where ℎ ∈ R1024× × and and are the height and width, reflecting the output channels of ResNet101’s last convolutional block. Global Average Pooling is applied to compute a global feature vector: global =
1 · =1 =1 ∑︁ ∑︁ ℎ[:, , ] global = fc · global + fc where global ∈ R1024 is the pooled vector for each sample in the batch. This vector is then passed through a fully connected layer to produce global logits: where fc ∈ R1024× and fc ∈ R are the weights and bias of the linear layer, and is the number of classes.
Inspired by prior work [
This attention map is used to compute attention-weighted features through a tensor operation, implemented via torch.einsum, resulting in a vector of shape [batch_size, 1024]. These weighted features are then mapped to logits of size [batch_size, num_classes] using another fully connected layer. This process allows CSRA to emphasize class-specific regions, enhancing the model’s ability to detect subtle pathological patterns. The resulting logits are subsequently combined with those from the Global Branch to produce the final predictions.
The final logits are computed by fusing the outputs of the two branches using a weighted combination: combined = (1 − ) · global + · csra, where = 0.1, a value set based on the model implementation. This weighted fusion balances the global context from the Global Branch and the localized attention from the CSRA Branch.
A threshold is applied to the fused logits to determine the predicted labels. This threshold is automatically selected by evaluating performance on the validation set across a range of values from 0.05 to 0.55. The optimal threshold determined during training was 0.35, corresponding to the highest achieved F1-score.
Given that this is a multilabel classification task, we use the Binary Cross Entropy Loss (BCE) computed independently for each class: = − ∑︁ · log(ˆ) + (1 − ) · log(1 − ˆ) =1 (1) where is the number of medical concepts, and ∈ {0, 1} is the ground truth label for class . This loss function ensures the model learns to predict multiple labels accurately, aligning with the task’s requirements.
The MedCSRA model was implemented in PyTorch. For optimization, the Adam optimizer was used with an initial learning rate of 1 × 10− 4, adjusted via a CosineAnnealing scheduler over a maximum of 100 epochs. A weight decay of 1 × 10− 5 was applied to mitigate overfitting. To ensure reproducibility, all experiments were conducted with a fixed random seed of 42.
The model was trained with a batch size of 16 on a single NVIDIA A100 GPU with 40 GB of VRAM. For the ResNet-101 backbone, only the final convolutional block was unfrozen for fine-tuning, while earlier layers retained their pre-trained ImageNet weights. An early stopping mechanism was employed with a patience of 5 epochs, monitoring the F1-score on the validation set. Based on this criterion, the training process for the ResNet-101-based model concluded at epoch 34.
This section details the performance of our proposed models on the oficial test sets of the ImageCLEFmedical 2025 challenge. We present the results for the concept detection and caption prediction subtasks separately.
For the caption prediction subtask, a fine-tuned BLIP model was submitted (Team: UIT-Oggy, Run ID: 1914). The comprehensive evaluation results, provided by the challenge organizers across multiple metrics, are summarized in Table 1. The model achieved an overall score of 32.11%, indicating a moderate level of performance. While the high Similarity (87.98%) and BERTScore (59.51%) suggest a good semantic alignment with the reference captions, other metrics revealed significant challenges. The low UMLS Concept F1 (16.72%) and Factuality Average (13.46%) scores highlight that the model struggled to maintain clinical accuracy and factual correctness, which is a critical limitation. This suggests that while pre-trained vision–language models like BLIP provide a strong foundation, extensive domain-specific adaptation is required to overcome the nuances of medical reporting.
For the concept detection subtask, our primary submission utilized the MedCSRA model with a ResNet101 backbone. The oficial performance of this submission (Team: UIT-Oggy, Run ID: 1892) on the final test set, as evaluated by the challenge organizers, is presented in Table 2.
The model achieved a primary F1-score of 0.5613, securing a fourth-place ranking among all international teams participating in the challenge. This strong competitive result serves as a direct validation of our proposed dual-branch architecture. The high performance suggests that the model’s ability to fuse global contextual features (from the Global Branch) with localized, class-specific details (from the CSRA Branch) is a highly efective strategy for this complex multi-label classification task. It demonstrates that explicitly balancing these two types of information allows the model to successfully identify a wide range of medical concepts, from difuse abnormalities to subtle, localized findings, within a single unified framework.
In this paper, we presented the methods and results of the UIT-Oggy team’s participation in the ImageCLEFmedical 2025 challenge.
For the medical concept detection task, we introduced MedCSRA, a novel dual-branch architecture designed to integrate both global and localized visual features. Our experiments demonstrated that this approach is highly efective, with the MedCSRA model using a ResNet-101 backbone achieving a fourthplace ranking in the oficial competition (F1-score: 0.5613). This result validates that balancing classspecific attention with global context is a robust strategy for multi-label medical image classification.
For the caption prediction task, we adapted and fine-tuned a pre-trained BLIP model. While the model showed a strong capability for generating semantically coherent text (Similarity: 87.98%), our analysis revealed significant limitations in its clinical and factual accuracy (UMLS Concept F1: 16.72%). This ifnding underscores a key challenge: despite the power of large-scale pre-training, achieving clinical reliability in generative models requires more advanced domain-specific adaptation techniques.
Overall, our work contributes an efective architecture for concept detection and provides a critical analysis of the current state of vision-language models in the context of diagnostic captioning, paving the way for future research directions.
Based on the findings of this study, several promising research directions are identified. For our
medical concept detection model, MedCSRA, future work will focus on addressing the challenge of class
imbalance, a common issue in medical datasets where rare but critical concepts are underrepresented.
We plan to incorporate Focal Loss [
For medical image captioning, our analysis revealed that while the fine-tuned BLIP model achieves
strong semantic coherence, its clinical and factual accuracy remains a significant limitation. To bridge
this gap, a primary direction is to infuse external medical knowledge into the model. Furthermore,
to improve the low factuality score, we propose introducing a fact-checking module, inspired by
claim verification systems [
Finally, to enhance the model’s ability to ground its descriptions in visual evidence and improve
the AlignScore, we intend to explore multi-scale attention mechanisms like the Convolutional Block
Attention Module CBAM [
During the preparation of this work, we used Gemini 2.5 Pro and ChatGPT-4o in order to check grammar and sentence structure. After using these tools, we reviewed and edited the content as needed and take full responsibility for the publication’s content.
This research is funded by University of Information Technology-Vietnam National University HoChiMinh City under grant number D4-2025-04. To ensure full reproducibility of our results, the source code for both models developed in this study is publicly available on GitHub. The repositories are organized as follows: • BLIP Fine-tuning, • MedCSRA.