The integration of AI into dermatological diagnostics is rapidly transforming clinical practice, with crucial applications in precise lesion segmentation and intelligent Visual Question Answering (VQA). Our team, H3N1, participated in ImageCLEF MAGIC 2025, tackling both these core challenges. We proudly announce our significant achievements: a top 4 finish in the Dermatology Segmentation Task and the winner position in the Dermatology VQA Task. These results validate the power of two advanced systems. Firstly, we use DermoSegDif , a system proposed by A. Bozorgpour et al. which revolutionizes skin lesion segmentation by leveraging a Denoising Difusion Probabilistic Model (DDPM) with novel boundary detection through weighted loss and 'Boundary Attention' for unparalleled contour precision (Jaccard: 0.514, Dice: 0.679). The modified U-Net with twopath feature extraction strategy helps capture diferent features, therefore providing the model with a more comprehensive view. Simultaneously, our developed DermKEM (Dermatology Knowledge-Enhanced Ensemble Model) system for Dermatology VQA excels with a knowledge-augmented multi-model ensemble. It employs a Genetic Algorithm for image enhancement, enriches captions via BLIP and external knowledge (Gemini 2.5 Flash), and feeds this into an ensemble of baseline models such as MUMC, and Gemini 2.5 Flash for highly accurate, contextually rich answers (Accuracy: 0.758). These synergistic systems not only demonstrate state-of-the-art capabilities but also redefine intelligent clinical support in dermatology.
The integumentary system, primarily comprising the skin, stands as the human body’s largest organ
and serves as a crucial interface with the external environment. Its multifaceted roles include providing
a protective barrier against pathogens and physical insults, regulating body temperature, facilitating
sensory perception, and contributing to immune responses. Consequently, dermatological diseases,
which encompass a wide array of conditions ranging from common inflammatory disorders such as
eczema and psoriasis to potentially life-threatening malignancies like melanoma, represent a significant
global health burden. The accurate and timely diagnosis of these conditions is paramount for efective
management and improved patient outcomes. Traditionally, dermatological diagnosis heavily relies
on visual inspection by trained clinicians, often augmented by non-invasive imaging techniques like
dermatoscopy. While this approach remains fundamental, its eficacy can be influenced by factors
such as inter-observer variability, the subtlety of early-stage lesion characteristics, and the clinician’s
experience level. Furthermore, the increasing demand for accessible dermatological care, particularly in
remote or underserved regions, has spurred the growth of teledermatology, where automated clinical
feedback is essential. In this context, objective and reliable automated decision support systems are
becoming increasingly vital. Artificial intelligence (AI) has demonstrated considerable potential in
augmenting diagnostic capabilities within various medical imaging domains, including dermatology.
Specifically, two key areas of AI research, dermatology image segmentation and Closed Visual Question
Answering (CVQA) for dermatology, are poised to revolutionize dermatological image analysis. Medical
image segmentation aims to precisely delineate regions of interest from surrounding healthy tissue
by generating masks that identify the afected areas. This provides quantitative data crucial for lesion
characterization and monitoring. Concurrently, VQA systems, which integrate computer vision with
natural language processing, enable clinicians to pose specific questions about an image (e.g., "How
much of the body is afected?") and receive contextually relevant, evidence-based answers. Such systems
can enhance diagnostic accuracy, streamline workflows, and facilitate more efective communication
between healthcare providers and AI tools. However, the development of robust AI models relies on
the availability of large-scale, high-quality, and diverse datasets, as well as rigorous benchmarking
to ensure quality and diversity of data. To address these challenges and foster advancements in AI
for dermatology, the ImageCLEFmed MEDIQA-MAGIC 2025 [
Our team, H3N1, participated in both dermatological tasks. For Task 1 (Segmentation), we investigated
advanced deep learning architectures, potentially exploring difusion-based models like DermoSegDif
[
Recent works about AI in medical field have increased significantly[
In the medical field, segmentation is a crucial step for identifying and delineating abnormal skin
regions such as lesions, malignancies, or infected areas. A widely adopted architecture for this task is
U-Net[
In this work, we addressed two primary tasks. For Task 1, we employed DermoSegDif,a boundaryaware system for skin lesions segmentation. For Task 2, we developed DermKEM, an advanced visual question answering (VQA) system tailored for dermatology. The architectures of these two systems are illustrated in Figure 1 and Figure 2, respectively, and will be elaborated upon in subsequent sections.
We noticed that there is no image enhancement method in the proposed model DermoSegDif, so we provided our own Genetic Algorithm-based image enhancement. We generate a population of 20 through 10 generations. For each image, we randomly modify the constrast and the brightness and calculate fitness using SSIM. Next, we select the top half by fitness and create new population by averaging and from two randomly selected top individuals, adding small random variation in the process. After 10 generations, we extract the best enhanced image based on SSIM with contrast and brightness . Figure 3.1.1 shows that there is significant improvement around skin lesions, making it clearer and easier to detect the boundary.
Due to the inconsistency of the images provided in the dataset, we apply a resize image method proposed
in DermoSegDif[
To generate descriptive image captions during preprocessing, we employed the BLIP (Bootstrapping
Language-Image Pre-training) model [
In the Medical VQA task, image-generated captions often pose challenges for models that are not
extensively trained on specialized medical contexts. These captions are typically short and may fail to
accurately describe lesions or pathological signs. We leverage Gemini 2.5 Flash [
For the skin lesion segmentation task, we employed DermoSegDif [
Difusion Process: DermoSegDif based on a standard DDPM framework that includes a forward
process - where Gaussian noise is gradually added to the ground truth segmentation mask —
and a reverse denoising process, which reconstructs the mask step-by-step with guiding image.
Rather than predicting the mask directly, the network learns to estimate the noise added at
each timestep, enabling more accurate reconstruction. Loss Function: To address the challenge of
fuzzy and ambiguous lesion boundaries, DermoSegDif introduces a novel boundary-aware loss function:
ℒ = (1 + Θ) ‖ − (, , )‖2
where Θ ∈ ( 1) is a dynamic weight map derived from a distance transform of the mask
boundary. This term increases the weight of pixels near the lesion boundary and decreases when
further from the boundary. The dynamic nature of Θ, based on the current time step , ensures the
progressive refinement of the boundary regions as the denoising proceeds. A gamma correction is
applied to the distance map to control the sharpness of attention around the boundary.
Denoising Network Architecture: The denoising network is a modified U-Net [
Each path contains ResNet blocks with separate time embeddings to capture distinct temporal
characteristics. The features of both paths are fused at multiple stages of the U-Net[
The decoder reconstructs the estimated noise ( ) by utilizing enriched skip connections from the encoder, which contains both semantic and boundary-focused information. An additional skip connection from the initial noisy input to the output layer ensures the retention of noise characteristics critical for accurate reconstruction.
Inference Strategy with Sampling-based Ensemble: During inference, DermoSegDif adopts a sampling-based ensemble strategy to enhance segmentation robustness. Specifically, the model generates nine segmentation predictions for each test image by running the difusion sampling process multiple times. These predictions are then averaged pixel-wise, followed by thresholding to produce the final segmentation mask (threshold of 0). This ensemble method mitigates the stochastic nature of difusion models and improves result consistency.
We experiment two kinds of model: traditional model and Vision Language model to discover the
performance of baseline models on dermatology visual question answering. For traditional model, we
ultilize MUMC[
MUMC: MUMC[
Gemini 2.5 Flash: Gemini 2.5 Flash[
Answer Shufle: For Vision-Language Models (VLMs), we experimented with shufling the order of answer options to evaluate model consistency. Typically, answer options are mapped as follows: option A to 1, option B to 2, option C to 3, and option D to 4. Subsequent to shufling, an example mapping could be A to 3, B to 1, C to 4, and D to 2. The model is queried multiple times with these shufled mappings, and the final selected option is determined by the answer choice that achieves the highest frequency of selection. In instances where multiple answer choices receive the same highest frequency, one is selected randomly, although this scenario was observed to occur infrequently. Few-shot Learning: For VLMs, we implemented few-shot learning. For each input sample, we provided a set of ground-truth examples randomly selected from the training dataset as in-context learning prompts. The number of such examples was equivalent to the total number of questions in the evaluation set, as detailed further in Section 5.
Ensemble: We employed a hard-voting ensemble strategy, combining the outputs from multiple model inferences. The final output was determined as the answer most frequently selected by the constituent models. In cases of a tie for the most frequent answer, one was selected randomly.
We used the oficial ImageCLEFmed MEDIQA-MAGIC 2025 dataset [
This dataset facilitates a segmentation task and is structured as follows: The training set, constituting 85%, consists of 2474 images, 7448 masks, and 842 queries. The validation set contains 157 images, 472 masks, and 56 queries, while the test set is composed of 314 images, 944 masks, and 100 queries. Mask files are stored as binary TIFF files, adhering to the naming convention IMG_{ENCOUNTERID}_{IMAGEID}_mask_{ANNOTATOR#}.tiff. Corresponding image files are available in PNG or JPG format, named as IMG_{ENCOUNTERID}_{IMAGEID}.png or IMG_{ENCOUNTERID}_{IMAGEID}.jpg. For the Closed QA task, the dataset includes closed questions, associated images, A dictionary of all possible closed questions and the option values associated with them, and a predefined list of 27 questions provided in the closedquestions_definitions_imageclef2025.json file, with both English and Chinese translations. Distribution of types of questions related to illnesses as show in Figure 5.
The second edition of the MEDIQA-MAGIC task focuses on multimodal dermatology response generation. Building upon the previous year’s challenge, this task introduces more complex reasoning by combining clinical narratives with associated dermatology images. The task is divided into two sub-tasks:
• Definition: Given a clinical history and an associated dermatological image, participants are required to generate segmentation masks that identify regions of interest related to the described dermatological condition.
• Definition: Closed VQA Participants are provided with a dermatology-related query (clinical narrative), one or more related images, and a multiple-choice question. The goal is to select the correct answer from the provided options.
Our experiments for the ImageCLEF MAGIC 2025 challenge were conducted on the Kaggle platform, utilizing two NVIDIA T4 GPUs and one NVIDIA P100 GPU.
For our segmentation task, we used the DermoSegDif framework as the core model. To improve the quality of our training data, we applied a Genetic Algorithm (GA) before training the segmentation model. Since DermoSegDif doesn’t have built-in image enhancement, this step helped create clearer and more informative images. The enhanced images were used to build new versions of the training, validation, and test datasets. We trained the model with the following settings: • Batch size: 8 • Input image size: 128 × 128 pixels • Difusion settings: – Timesteps: 250 – Beta schedule: Linear – start: 0.0004 – end: 0.08 For optimization, we used the Adam optimizer with: • Learning rate: 0.0001 • Betas: (0.7, 0.99) • Weight decay: 0.0 We also included a learning rate scheduler (ReduceLROnPlateau) to help the model train more eficiently. If the validation score didn’t improve after 5 epochs, the learning rate was cut in half. Each run lasted up to 40,000 iterations, or ended earlier if the validation results stopped improving. No other parameters was tested due to time constraint, as we believed the original parameter is the most optimal for the task. For evaluation, we used an ensemble approach. Each test image was passed through the model 5 times, and we averaged the outputs pixel by pixel. The final result was turned into a binary mask using a threshold. This method helped reduce noise and made the predictions more stable and accurate.
MUMC: We utilized the MUMC model. The model’s pre-training stage involved a version pre-trained
on three datasets: ROCO [
Ensemble: The ensemble method aggregated outputs from multiple model inferences. Final predictions were determined using hard voting, where the answer option with the highest frequency among the collected outputs was selected. We experimented with three distinct ensemble configurations: • MUMC combined with two Gemini 2.5 Flash inference runs (one using the original answer order, and one using a single shufled answer order). • MUMC combined with three Gemini 2.5 Flash inference runs (one original, and two using distinct shufled answer orders). • An ensemble of three Gemini 2.5 Flash inference runs (one original, and two using distinct shufled answer orders).
We used the oficial evaluation metrics defined by the ImageCLEFmed MEDIQA-MAGIC 2025 organizers for each task.
Task 1: Segmentation Performance was measured by the Jaccard Index (IoU) and Dice Coeficient. IoU = || ∩∪|| ; Dice = 2|×| |+ ∩||| Where is the predicted mask and is the ground truth mask. The ground truth was constructed using a ’Majority Vote’ from four annotators, where a pixel is considered positive if marked by at least two annotators. The final reported scores are macro-averaged IoU and Dice across the test set. Task 2: Closed Visual Question Answering The task was evaluated using Overlap-based Accuracy, suitable for multi-label answers. The accuracy for each question i is calculated as: Accuracy_ = |∩| max(||,||) Where and are the ground truth and predicted label sets, respectively. The final score is the mean of these per-question accuracies across the entire test set. 5.2.2. Results Task 1: Segmentation This section presents the performance of our submission for the MEDIQAMAGIC 2025 segmentation task. Our approach utilized the DermoSegDif model, a state-of-the-art difusion-based architecture for medical image segmentation. The model was evaluated on the unseen private test set using the oficial competition metrics. The oficial results are summarized in the Table 1. Task 2: Closed Visual Question Answering For private test phase, we contributed 4 piplines, which detailed in Table 3. Gemini 2.5 Flash shows the eficientness that the ensemble contains three Gemini inference runs achieved the highest private test score.
Gemini 2.5 Flash (+Preprocessing) Ensemble (MUMC + two Gemini 2.5 Flash runs) (+Preprocessing) (+Shufling) Ensemble (MUMC + three Gemini 2.5 Flash runs) (+Preprocessing) (+Shufling) Ensemble (three Gemini 2.5 Flash runs) (+Preprocessing) (+Shufling)
Limitations Despite the efectiveness of the proposed model, it exhibits notable limitations in handling very small lesions or subtle variations in texture and color, which may be imperceptible to the human eye or not be suficiently captured by the model without specialized training. The model can also struggle to distinguish between lesion types with similar surface characteristics and may fail to fully capture multi-attribute lesions (e.g., those presenting multiple colors or patterns). Additionally, the encoder stage in the proposed difusion network may not suficiently extract comprehensive features from both the input images and corresponding segmentation masks, potentially limiting segmentation accuracy in complex cases.
Future Works Future improvements can be focused on combining MUMC and Gemini 2.5 Flash in a hybrid or ensemble model to improve both image analysis and language understanding, making the system more accurate and reliable. Fine-tuning the models on a larger and more diverse medical dataset, especially with rare skin conditions, could help improve generalization. Adjusting Gemini 2.5 Flash’s prompt format to better handle medical terms may also boost performance. To reduce bias and increase realism, the dataset could be expanded with more context-based questions and images from people of various races and age groups. Evaluation can be strengthened by using metrics such as BLEU, ROUGE, and METEOR for free-form responses, as well as expert feedback from dermatologists. For the segmentation part, testing stronger encoder models like TransUNet [31] and applying more data augmentation can lead to better learning from the available data.
The MUMC (Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive
Losses) [
Stage 1: Pre-training This stage aims to learn robust and highly generalizable representations from large-scale medical image-caption datasets.
1. Input Data Preparation: • Image: Each image is divided into 16×16 patches. During training, 25% of the patches are randomly masked, and only the unmasked patches are input to the image encoder. This masked image modeling encourages the model to learn from partial visual information. • Text (Caption): TAssociated text descriptions are tokenized using a WordPiece tokenizer. 2. Architecture: A dual-encoder setup based on Momentum Contrast (MoCo) is employed, consisting of online and momentum encoders that stabilize contrastive training. 3. Optimization Objectives: The model is jointly optimized with four self-supervised losses: • Contrastive Loss (UCL and MCL): The core objective, which structures the latent space by pulling similar pairs together and pushing dissimilar pairs apart. Unimodal Contrastive Loss (UCL) operates within a single modality (image-to-image, text-to-text), while Multimodal Contrastive Loss (MCL) learns the alignment between modalities (image-to-text). • Image-Text Matching (ITM) Loss: A binary classification loss that determines whether a given image-caption pair is matched or randomly paired. This reinforces semantic alignment between modalities beyond just contrastive structure. • Masked Language Modeling (MLM) Loss: A language modeling objective where 15% of input tokens in the caption are randomly masked and the model must predict them using the remaining text and associated image features, thereby strengthening contextual understanding Stage 2: Fine-tuning The pretrained weights are transferred to a VQA model with a Transformerbased answering decoder (6 layers) that generates free-text answers from fused image-question embeddings. The model is fine-tuned using a standard cross-entropy loss over the target answer sequences.
This prompt utilizes a few-shot learning technique, wherein the model is provided with 27 complete examples from the training set before it is asked to process a new case. The primary objective of this structured prompt is to strictly constrain the model’s output to a single integer corresponding to the chosen answer’s index, thereby simplifying the results parsing process and increasing output consistency.
The prompt is constructed following a "guidance - examples - task" architecture. The full prompt template is provided below.
You are an expert dermatologist. Your task is to answer a multiple-choice question about a clinical case based on the provided clinical context and image(s).
You will be shown several examples first, followed by a new case to solve.
Your response MUST be a single integer representing the index of the correct answer.
DO NOT include any other text, explanation, or formatting. JUST THE NUMBER. --- EXAMPLE 1 START --[IMAGE(S) ARE PROVIDED HERE] Clinical Context: {Example 1 Context} Question: {Example 1 Question} Options: 0: {Option 0} 1: {Option 1} ...
Correct Answer Index: {Example 1 Correct Index} --- EXAMPLE 1 END ----- EXAMPLE 2 START --[IMAGE(S) ARE PROVIDED HERE] Clinical Context: {Example 2 Context} Question: {Example 2 Question} Options: 0: {Option 0} 1: {Option 1} ...
Correct Answer Index: {Example 2 Correct Index} --- EXAMPLE 2 END --... (25 more examples follow the same structure) ... --- EXAMPLE 27 START --[IMAGE(S) ARE PROVIDED HERE] Clinical Context: {Example 27 Context} Question: {Example 27 Question} Options: 0: {Option 0} 1: {Option 1} ...
Correct Answer Index: {Example 27 Correct Index} --- EXAMPLE 27 END ----- YOUR TASK START --Now, analyze the following new case and provide your answer as a single integer. [IMAGE(S) ARE PROVIDED HERE] Clinical Context: {Query Case Context} Question: {Query Case Question} Options: 0: {Query Option 0} 1: {Query Option 1} 2: {Query Option 2} 3: {Query Option 3} Answer Index: B.2. Explanation of Prompt Components • Role Definition : The initial line, "You are an expert dermatologist," establishes a specific persona and domain expertise for the model. This helps the model to approach the problem from the perspective of a medical specialist, potentially activating more relevant reasoning paths. • Strict Output Instruction: The lines "Your response MUST be a single integer..." are capitalized and emphasized to capture the model’s attention and ensure it understands the precise output format required. This is the most critical change compared to previous prompt versions, as it eliminates conversational or explanatory text that would complicate automated evaluation. • Few-shot Examples: A total of 27 examples (from ‘EXAMPLE 1‘ to ‘EXAMPLE 27‘) are included to "teach" the model the desired input-output format. Each example provides a complete instance, including the full context, the image (passed as an image object, represented by ‘[IMAGE(S) ARE PROVIDED HERE]‘), the question, the multiple-choice options, and the correct answer index.
This in-context learning is crucial for guiding the model’s behavior without updating its weights. • Query Task: The final section, beginning with ‘— YOUR TASK START —‘, is where the actual data for the case to be predicted is inserted. Placeholders such as ‘Query Case Context‘ are dynamically replaced with the real data. The final line, ‘Answer Index:‘, acts as a direct cue for the model to complete the sequence and provide its numerical answer.
The core of the enrichment process is to design sophisticated prompt for a powerful generative model (Gemini 2.5 Flash). The full prompt text is provided below to illustrate the detailed instructions given to the model.
ROLE: AI Medical Concept Enrichment Specialist CONTEXT: You are tasked with processing text content, typically image captions or descriptions (query_content_en field) related to medical observations, often within a Visual Question Answering (VQA) context for medicine, especially but not limited to Dermatology. Your goal is to enhance this text by identifying specific medical terms and appending concise, accurate definitions. This prompt is designed for the Gemini 2.5 Flash model.
OBJECTIVE: To enrich the input text by identifying relevant medical terms (including but not limited to Dermatology) and appending their brief, accurate definitions immediately after the term, formatted as Term [Definition]. Definitions for Dermatology-specific terms should prioritize consistency with DermNet NZ (site:dermnetnz.org). Definitions for other medical terms (diseases, symptoms, findings, anatomical locations, procedures, relevant medications) should be consistent with standard medical knowledge and ontologies like SNOMED CT (site:https://www.snomed.org/) or UMLS (site: https://www.nlm.nih.gov/research/umls/index.html) or wikidoc (site: https://www.wikidoc.org/index.php/Main_Page), use your crawl skills to get information.
INPUT: A single string of text representing the value of a query_content_en field or similar medical text description.
OUTPUT: The modified string of text, with relevant medical terms enriched as specified. The overall structure and non-relevant parts of the original text must remain unchanged.
CONSTRAINTS: 1. Scope: Enrich specific medical terms. This includes: - Names of diseases or conditions (e.g., psoriasis). - Specific symptoms or clinical findings (e.g., macule, erythema). - Relevant anatomical locations (e.g., epidermis, dermis). - Medical or surgical procedures (e.g., biopsy).
- Commonly referenced medications (e.g., methotrexate). 2. Exclusion: Do NOT enrich: - Highly general terms (e.g., ’disease’, ’patient’). - Common non-medical words (e.g., ’tired’, ’left’, ’right’).
- Terms already adequately explained by the context. 3. Source Prioritization: - For Dermatology: Prioritize DermNet NZ (site:dermnetnz.org).
- For other medical terms: Use SNOMED CT, UMLS, Wikidoc. 4. Definition Format: Term [Concise, clear definition]. 5. Definition Content: Brief, 1-2 short sentences. 6. Accuracy: Ensure definitions are medically accurate. 7. Case Sensitivity: Identify terms regardless of case, but preserve
original capitalization in the output. 8. No Modification Otherwise: Do not alter any other part of the text.
INSTRUCTIONS: 1. Receive the input text string. 2. Scan the text to identify potential medical keywords. 3. For each term: a. Verify it meets enrichment criteria (CONSTRAINT 1 & 2). b. Determine if it is dermatological or general medical. c. Generate a concise definition per Source Prioritization. d. Format the definition as specified (CONSTRAINT 4 & 5).
e. Append the formatted definition to the term. 4. If no relevant terms are found, return the original text. 5. Return the fully processed text string.
EXAMPLES: Input Text (Dermatology Focus): The patient presented with severe psoriasis and was prescribed methotrexate.
Output Text: The patient presented with severe psoriasis [a common, chronic inflammatory skin disease characterized by red, itchy, scaly patches] and was prescribed methotrexate [an immunosuppressant drug...].
Input Text (General Medical): Image shows pitting edema on the lower leg.
Output Text: Image shows pitting edema [swelling, typically in the limbs, where pressing the skin leaves a temporary indentation] on the lower leg.
Input Text (Exclusion): Doctors predict that he has some kind of infection, he feels tired.
Output Text: Doctors predict that he has some kind of infection [invasion and multiplication of microorganisms...], he feels tired. (Note: ’predict’, ’tired’ are not enriched).
Now, process the following input text based on these instructions:
The prompt’s design is multi-faceted, aiming to transform a powerful but general-purpose language model into a precise and reliable medical annotation tool. Each component plays a strategic role in achieving this goal.
• Role-Playing (‘ROLE‘): The prompt assigns the model the persona of an ‘"AI Medical Concept Enrichment Specialist"‘. This technique is crucial for setting the context. It shifts the model from a general conversational mode to a professional, domain-specific one, encouraging it to access and utilize its training data related to medical science and formal writing styles. • Zero-Shot, Instruction-Following (‘INSTRUCTIONS‘): The core of the prompt is a detailed, algorithmic set of instructions. Rather than relying on the model to infer the task from examples alone (few-shot), it explicitly defines the procedure: scan, identify, verify, generate, format, and return. This converts a potentially ambiguous creative task into a more deterministic, rule-based process, significantly increasing the reliability and consistency of the output. • Strict Constraints and Negative Logic (‘CONSTRAINTS‘): A key to high-precision output is defining not only what to do, but also what not to do.
– Inclusion Scope: By listing categories of terms to enrich (diseases, symptoms, anatomy), the prompt focuses the model’s attention on high-value, specific medical concepts that carry significant diagnostic weight. – Exclusion Scope: The negative constraints (e.g., excluding ’patient’, ’doctor’, ’left’) are vital for preventing "over-enrichment". Without these rules, the model might define common words, cluttering the output and diluting the importance of the truly significant medical terms. This improves the signal-to-noise ratio of the enriched caption. • Knowledge Grounding and Source Prioritization: To prevent model "hallucination" or inaccurate definitions, the prompt explicitly grounds the required knowledge in authoritative external sources. By instructing the model to prioritize definitions consistent with DermNet NZ for dermatology and established ontologies like SNOMED CT or WikiDoc for general medicine, we guide the model to generate factually accurate and contextually appropriate information. The instruction to ‘"use your crawl skills"‘ leverages the model’s ability to access and synthesize information from its vast training data, which includes these reliable web sources. • Format Enforcement via Examples: While the prompt is primarily instruction-based, it includes a set of clear ‘EXAMPLES‘. These serve a critical function: they demonstrate the exact implementation of all the preceding rules, especially the strict output format of ‘Term [Definition]‘. The examples cover diverse cases, including dermatology-specific terms, general medical terms, and a case showing correct exclusion, leaving no ambiguity about the expected output. This combination of explicit instructions and illustrative examples is a powerful technique for ensuring the model adheres to the desired schema.
In conclusion, the efectiveness of our knowledge enrichment pipeline is not merely due to using a powerful LLM, but is a direct result of this carefully crafted prompt. It strategically combines role-playing, explicit instructions, positive and negative constraints, knowledge grounding, and clear examples to transform a general-purpose tool into a specialized, reliable, and highly efective component of our medical VQA system.
During the preparation of this work, we used Gemini 2.5 Flash and ChatGPT-3.5 in order to: check grammar and sentence structure. After using these tools, we reviewed and edited the content as needed and take full responsibility for the publication’s content.
This research is funded by University of Information Technology-Vietnam National University HoChiMinh City under grant number D4-2025-04. [28] J. J. Lau, S. Gayen, D. Demner, A. Ben Abacha, Visual question answering in radiology (VQA-RAD), 2022. [29] X. He, Z. Cai, W. Wei, Y. Zhang, L. Mou, E. Xing, P. Xie, Towards visual question answering on pathology images, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Association for Computational Linguistics, Online, 2021, pp. 708–718. URL: https://aclanthology.org/2021.acl-short. 90/. doi:10.18653/v1/2021.acl-short.90. [30] B. Liu, L.-M. Zhan, L. Xu, L. Ma, Y. Yang, X.-M. Wu, Slake: A semantically-labeled knowledgeenhanced dataset for medical visual question answering, 2021. URL: https://arxiv.org/abs/2102. 09542. arXiv:2102.09542. [31] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, Y. Zhou, Transunet: Transformers make strong encoders for medical image segmentation, 2021. URL: https://arxiv.org/abs/2102.04306. arXiv:2102.04306.