Accurate interpretation of medical images is essential for intelligent diagnostic systems. This project presents a unified deep learning framework that tackles two key challenges in medical image analysis: skin lesion segmentation and closed-ended visual question answering (VQA). For lesion segmentation, we introduce a model based on the Multi-Scale Feature Fusion Network (MSFNet), enhanced with boundary and reverse attention modules. This design improves the detection of irregular and low-contrast lesions. Tested on 314 dermatology images, the model achieved a mean Dice coeficient of 0.7021, a mean Jaccard index of 0.5410, and a maximum Dice score of 0.7512, supporting its efectiveness in aiding early melanoma detection. In parallel, our closed-ended VQA system combines visual feature extraction with language embeddings to answer structured questions-such as "yes/no," object types, and numeric values. On a set of 56 question-image pairs, it achieved 56.98% overall accuracy, with high scores in categories CQID012 (74.80%) and CQID035 (74.00%). Together, these results showcase the promise of deep learning in multi-modal medical image understanding. The integration of segmentation and VQA in a single pipeline highlights its potential for real-world applications, including clinical decision support, assistive tools, and automated medical interpretation.
Skin lesion segmentation is crucial for the early detection of melanoma and other dermatological
conditions [
To address these challenges, we adopt the Multi-Scale Feature Fusion Network (MSFNet), which ofers
an efective balance between precision and computational eficiency through its integration of attention
modules and hierarchical feature fusion[
The remainder of this paper is organized as follows: Section 2 reviews related work on skin lesion segmentation. Section 3 describes the proposed MSFNet architecture and methodology. Section ?? details the training strategy used, while Section ?? presents the evaluation metrics employed for performance assessment. Section ?? discusses the experimental results. Finally, Sections 6 provide the conclusion and future scope.
Closed Visual Question Answering (Closed-VQA) is a vision-language task where systems answer
image-based queries using a fixed set of responses [
U-Net [
Other alternatives like Vision Transformers[
Visual Question Answering (VQA) is a complex task requiring joint understanding of images and
language, with Closed VQA framing it as a multi-class classification problem for consistent evaluation
[
We employ the original MSFNet architecture[
Boundary Attention (BA) modules operate on intermediate layers (Conv2, Conv3) to emphasize lesion edges. Parallel Partial Decoder (PPD) processes deep features from Conv4 and Conv5 to produce a coarse semantic prediction. Reverse Attention (RA) modules then iteratively refine this prediction by re-focusing on uncertain boundary regions.
Finally, the outputs of BA, RA, and PPD modules are fused using element-wise addition and convolution layers to generate the final lesion mask. A sigmoid activation produces a binary probability map for segmentation.
As this architecture is reused without structural modification, detailed formulations are omitted here
and can be found in[
The proposed VQA framework for the ImageCLEF 2025 challenge [
For segmentation, all training was conducted using the gold-standard masks provided in the
MEDIQAMAGIC 2025 dataset [
Here, ℒWBCE denotes the Weighted Binary Cross-Entropy Loss, which mitigates class imbalance by assigning greater importance to underrepresented lesion pixels, while ℒIoU is the Intersection-overUnion Loss that directly promotes region overlap between prediction and ground truth. The coeficients were empirically set as = 1 and = 1 in all experiments.
To enhance generalization and prevent overfitting, standard data augmentation techniques such as lfipping, rotation, and contrast adjustment were applied during training. The Adam optimizer was employed with an initial learning rate of 1 × 10− 4. A learning rate decay strategy was incorporated, reducing the learning rate by a factor of 0.1 if the validation loss did not improve for 10 consecutive epochs. Training was conducted over a maximum of 100 epochs using a batch size of 8, with early stopping enabled based on validation loss. During inference, predicted masks were upsampled to the original image resolution using bilinear interpolation.
For evaluation, segmentation performance was quantified using the Dice coeficient and mean Intersection over Union (mIoU). The Dice score is computed as:
Dice(, ) = 2 (2)
2 + + where , , and represent the true positives, false positives, and false negatives, respectively. The mIoU metric averages IoU across all instances:
mIoU = 1 ∑︁
=1 + +
For Visual Question Answering (VQA), the model was trained in a supervised manner to classify inputs into one of 12 answer categories. The input comprises a fused feature vector of dimension 1024, created by concatenating a 512-dimensional text embedding (extracted using CLIP) and a 512-dimensional visual embedding (obtained from a Vision Transformer and Local Binary Patterns).
The classifier outputs logits over the answer space, which are converted into class probabilities using the softmax function: (3) (4) (5) (6) The model was trained to minimize the categorical cross-entropy loss:
^ ( = |x) = ∑︀=1 ^ , ∈ {1, 2, . . . , }
1 ∑︁ log (|x) ℒ( ) = −
=1 where is the number of training samples, is the ground truth label, and (|x) is the predicted probability for the true class. Optimization was performed using the AdamW optimizer with a fixed learning rate of 1 × 10− 4, and training was run for 1000 epochs with a batch size of 32. Mixed precision training on CUDA-enabled GPUs was used to accelerate computation and reduce memory usage.
Evaluation was conducted according to the ImageCLEF VQA-Med 2024 protocol. The performance was measured using macro-averaged accuracy across grouped question types. Let ℰ represent the set of encounter IDs and the set of grouped question IDs. For each encounter-question pair (, ), with gold-standard answers , and predictions ,, instance-level accuracy is defined as: Accuracy(, ) =
|, ∩ ,| max(|,|, |,|) Group-level accuracy is computed as: And the final macro-averaged accuracy across all question types is given by:
Accuracy = |ℰ | ∈ℰ Accuracyoverall = || ∈ 1 ∑︁ Accuracy (7) (8)
The evaluation process involved parsing both gold and predicted JSON files, grouping responses by question and encounter ID, and computing the aforementioned accuracy metrics. Predictions with missing instances were assigned an accuracy of zero to ensure consistency and fairness across all model submissions.
We evaluated the segmentation model on the MEDIQA-MAGIC 2025 DermaVQA-DAS dataset [
We also evaluated our closed-ended Visual Question Answering (VQA) model on a test dataset containing 56 image-question pairs as part of the ImageCLEF VQA-Med 2024 task. Submitted under the team name KLE1 (Rank 12), the model was assessed based on per-question-type accuracy and overall performance. As shown in Table 2, the model achieved an overall accuracy of 56.98%. It performed particularly well in CQID012 (74.80%) and CQID035 (74.00%), indicating strength in those question categories. However, it showed lower performance in CQID034 (39.00%) and CQID036 (35.00%), suggesting scope for improvement in those areas. These results confirm the model’s efectiveness in handling diverse closed-form visual questions while identifying areas for future refinement.
Our MSFNet-based segmentation framework demonstrated strong performance on the MEDIQA-MAGIC 2025 dataset, achieving a mean Dice coeficient of 0.7021 and Jaccard index of 0.5410 across 314 skin lesion instances. The hybrid loss formulation, combining Weighted Binary Cross-Entropy and IoU losses, was efective in handling class imbalance and enhancing boundary delineation. These results highlight the model’s robustness and generalization capability in binary lesion segmentation. For future enhancements, we aim to extend the framework to multi-class segmentation, integrate advanced attention modules, explore alternative loss strategies, and optimize the model for real-time clinical or mobile deployment.
In parallel, our closed-ended Visual Question Answering (VQA) model attained an overall accuracy of 56.98% on a 56-pair test set, performing notably well in specific categories such as CQID012 and CQID035. This reflects the model’s potential in accurately interpreting visual inputs and providing consistent answers to domain-specific questions. However, lower performance in certain categories also reveals areas for improvement. Future work will focus on expanding training data diversity, incorporating larger-scale multimodal datasets, and experimenting with transformer-based and attention-driven fusion architectures. These eforts aim to boost model generalization and accuracy across broader medical VQA tasks.
During the preparation of this work, we have used generative AI tool (ChatGPT) for tasks such as grammar checking and paraphrasing. All AI-generated content was reviewed and edited by the authors, who take full responsibility for the final version of the manuscript.
The sources for the ceur-art style are available via • GitHub, • Overleaf template.