This paper presents an overview of the Medical Visual Question Answering task (VQA-Med) at ImageCLEF 2019. Participating systems were tasked with answering medical questions based on the visual content of radiology images. In this second edition of VQA-Med, we focused on four categories of clinical questions: Modality, Plane, Organ System, and Abnormality. These categories are designed with di erent degrees of di culty leveraging both classi cation and text generation approaches. We also ensured that all questions can be answered from the image content without requiring additional medical knowledge or domain-speci c inference. We created a new dataset of 4,200 radiology images and 15,292 question-answer pairs following these guidelines. The challenge was well received with 17 participating teams who applied a wide range of approaches such as transfer learning, multi-task learning, and ensemble methods. The best team achieved a BLEU score of 64.4% and an accuracy of 62.4%. In future editions, we will consider designing more goal-oriented datasets and tackling new aspects such as contextual information and domain-speci c inference.
Recent advances in arti cial intelligence opened new opportunities in clinical
decision support. In particular, relevant solutions for the automatic interpretation
of medical images are attracting a growing interest due to their potential
applications in image retrieval and in assisted diagnosis. Moreover, systems capable of
understanding clinical images and answering questions related to their content
could support clinical education, clinical decision, and patient education. From a
computational perspective, this Visual Question Answering (VQA) task presents
an exciting problem that combines natural language processing and computer
vision techniques. In recent years, substantial progress has been made on VQA
with new open-domain datasets [
However, there are challenges that need to be addressed when tackling VQA
in a specialized domain such as Medicine. Ben Abacha et al. [
Inspired by the success of visual question answering in the general domain,
we conducted a pilot task (VQA-Med 2018) in ImageCLEF 2018 to focus on
visual question answering in the medical domain [
In VQA-Med 2019, we selected radiology images and medical questions that
(i) asked about only one element and (ii) could be answered from the image
content. We targeted four main categories of questions with di erent di culty
levels: Modality, Plane, Organ system, and Abnormality. For instance, the rst
three categories can be tackled as a classi cation task, while the fourth category
(abnormality) presents an answer generation problem. We intentionally designed
the data in this manner to study the behavior and performance of di erent
approaches on both aspects. This design is more relevant to clinical decision
support than the common approach in open-domain VQA datasets [
In the following section, we present the task description with more details and examples. We describe the data creation process and the VQA-Med-2019 dataset in section 3. We present the evaluation methodology and discuss the challenge results respectively in sections 4 and 5. 2
In the same way as last year, given a medical image accompanied by a clinically relevant question, participating systems in VQA-Med 2019 are tasked with answering the question based on the visual image content. In VQA-Med 2019, we speci cally focused on radiology images and four main categories of questions: Modality, Plane, Organ System, and Abnormality. We mainly considered medical questions asking only about one element: e.g., \what is the organ principally shown in this MRI?", \in what plane is this mammograph taken?", \is this a t1 weighted, t2 weighted, or air image?", \what is most alarming about this ultrasound?").
All selected questions can be answered from the image content without requiring additional domain-speci c inference or context. Other questions including these aspects will be considered in future editions of the challenge, e.g.: "Is this modality safe for pregnant women?", "What is located immediately inferior to the right hemidiaphragm?", "What can be typically visualized in this plane?", "How would you measure the length of the kidneys?" 3
We automatically constructed the training, validation, and test sets, by (i) applying several lters to select relevant images and associated annotations, and (ii) creating patterns to generate the questions and their answers. The test set was manually validated by two medical doctors. The dataset is publicly available4. Figure 1 presents examples from the VQA-Med-2019 dataset. 3.1
We selected relevant medical images from the MedPix5 database with lters based on their captions, modalities, planes, localities, categories, and diagnosis methods. We selected only the cases where the diagnosis was made based on the image. Examples of the selected diagnosis methods: CT/MRI Imaging, Angiography, Characteristic imaging appearance, Radiographs, Imaging features, Ultrasound, Diagnostic Radiology. 3.2
We targeted the most frequent question categories: Modality, Plane, Organ system and Abnormality (Ref:VQA-RAD).
1) Modality: Yes/No, WH and closed questions. Examples: { was gi contrast given to the patient? { what is the mr weighting in this image? { what modality was used to take this image? { is this a t1 weighted, t2 weighted, or air image?
2) Plane: WH questions. Examples: { what is the plane of this mri? { in what plane is this mammograph taken?
3) Organ System: WH questions. Examples: { what organ system is shown in this x-ray? { what is the organ principally shown in this mri?
4) Abnormality: Yes/No and WH questions. Examples: { does this image look normal? 4 github.com/abachaa/VQA-Med-2019 5 https://medpix.nlm.nih.gov (a) Q: what imaging method was used? A: us-d - doppler ultrasound (b) Q: which plane is the image shown in? A: axial (c) Q: is this a contrast or noncontrast ct? A: contrast (d) Q: what plane is this? A: lateral (e) Q: what abnormality is seen in the image? A:nodular opacity on the left#metastastic melanoma (f) Q: what is the organ system in this image? A: skull and contents (g) Q: which organ system is shown in the ct scan? A: lung, mediastinum, pleura (h) Q: what is abnormal in the gastrointestinal image? A: gastric volvulus (organoaxial) { are there abnormalities in this gastrointestinal image? { what is the primary abnormality in the image? { what is most alarming about this ultrasound? Planes (16): Axial; Sagittal; Coronal; AP; Lateral; Frontal; PA; Transverse; Oblique; Longitudinal; Decubitus; 3D Reconstruction; Mammo-MLO; MammoCC; Mammo-Mag CC; Mammo-XCC.
Organ Systems (10): Breast; Skull and Contents; Face, sinuses, and neck; Spine and contents; Musculoskeletal; Heart and great vessels; Lung, mediastinum, pleura; Gastrointestinal; Genitourinary; Vascular and lymphatic.
Patterns: For each category, we selected question patterns from hundreds of
questions naturally asked and validated by medical students from the VQA-RAD
dataset [
The training set includes 3,200 images and 12,792 question-answer (QA) pairs, with 3 to 4 questions per image. Table 1 presents the most frequent answers per category. The validation set includes 500 medical images with 2,000 QA pairs. 3.4
A medical doctor and a radiologist performed a manual double validation of the test answers. A total of 33 answers were updated by (i) indicating an optional part (8 answers), (ii) adding other possible answers (10), or (iii) correcting the automatic answer. 15 answers were corrected, which corresponds to 3% of the test answers. The corrected answers correspond to the following categories: Abnormality (8/125), Organ (6/125), and Plane (1/125). For abnormality questions, the correction was mainly changing the diagnosis that is inferred, by the problem seen in the image. We expect a similar error rate in the training and validation sets that were generated using the same automatic data creation method. The test set consists of 500 medical images and 500 questions. 4
The evaluation of the systems that participated in the VQA-Med 2019 task
was conducted based on two primary metrics: Accuracy and BLEU. We use an
adapted version of the accuracy metric from the general domain VQA6 task
that strictly considers exact matching of a participant provided answer and the
ground truth answer. We calculate the overall accuracy scores as well as the
scores for each question category. To compensate for the strictness of the
accuracy metric, BLEU [
Out of 104 online registrations, 61 participants submitted signed end-user
agreement forms. Finally, 17 groups submitted a total of 90 runs, indicating a notable
interest in the VQA-Med 2019 task. Figure 2 presents the results of the 17
participating teams. The best overall result was obtained by the Hanlin team,
6 https://visualqa.org/evaluation.html
achieving 0.624 Accuracy and 0.644 BLEU score. Table 2 gives an overview of
all participants and the number of submitted runs7. The overall results of the
participating systems are presented in Table 3 to Table 4 for the two metrics in
a descending order of the scores (the higher the better). Detailed results of each
run are described in the ImageCLEF 2019 lab overview paper [
Turner.JCE JUST19 Team Pwc Med Techno deepak.gupta651 ChandanReddy Dear stranger abhishekthanki IITISM@CLEF 26913 27142 26941 27079 27232 26884 26895 27307 26905
Similar to last year, participants mainly used deep learning techniques to build
their VQA-Med systems. In particular, the best-performing systems leveraged deep
convolutional neural networks (CNNs) like VGGNet [
Analyses of the question category-wise8 accuracy in Table 3 suggest that in general, participating systems performed well to answer modality questions, followed by plane and organ questions because the possible types of answers for each of these question categories were nite. However, for the abnormality type questions, systems did not perform well in terms of accuracy because of the underlying complexity of open-ended 8 Note that the question category-wise accuracy scores are normalized (each divided by a factor of 4) so that the summation is equal to the overall accuracy. questions and possibly due to the strictness of the accuracy metric. To compensate for the strictness of the accuracy, we computed the BLEU scores to understand the similarity of the system generated answers and the ground-truth answers. The higher BLEU scores of the systems this year (0.631 best BLEU vs. 0.162 in 2018) further verify the e ectiveness of the proposed deep learning-based models for the VQA task. Overall, the results obtained this year clearly denote the robustness of the provided dataset compared to last year's task. 6
We presented the VQA-Med 2019 task, the new dataset, the participating systems, and o cial results. To ensure that the questions are naturally phrased, we used patterns from question asked by medical students to build clinically relevant questions belonging to our four target categories. We created a new dataset for the challenge9 following goal-oriented guidelines, and covering questions with varying degrees of di culty. A wide range of approaches have been applied such as transfer learning, multi-task learning, ensemble methods, and hybrid approaches combining classi cation models and answer generation methods. The best team achieved 0.644 BLEU score and 0.624 overall accuracy. In future editions we are considering more complex questions that might include contextual information or require domain-speci c inference to reach the right answer.
This work was supported by the intramural research program at the U.S. National Library of Medicine, National Institutes of Health.
We thank Dr. James G. Smirniotopoulos and Soumya Gayen from the MedPix team for their support. 9 www.crowdai.org/clef tasks/13/task dataset les?challenge id=53
github.com/abachaa/VQA-Med-2019