=Paper=
{{Paper
|id=Vol-2380/paper_112
|storemode=property
|title=Leveraging Medical Visual Question Answering with Supporting Facts
|pdfUrl=https://ceur-ws.org/Vol-2380/paper_112.pdf
|volume=Vol-2380
|authors=Tomasz Kornuta,Deepta Rajan,Chaitanya Shivade,Alexis Asseman,Ahmet Ozcan
|dblpUrl=https://dblp.org/rec/conf/clef/KornutaRSAO19
}}
==Leveraging Medical Visual Question Answering with Supporting Facts==
https://ceur-ws.org/Vol-2380/paper_112.pdf
Leveraging Medical Visual Question Answering
with Supporting Facts
Tomasz Kornuta, Deepta Rajan, Chaitanya Shivade,
Alexis Asseman, and Ahmet S. Ozcan
IBM Research AI, Almaden Research Center, San Jose, USA
{tkornut,drajan,cshivade,asozcan}@us.ibm.com,alexis.asseman@ibm.com
Abstract. In this working notes paper, we describe IBM Research AI
(Almaden) team’s participation in the ImageCLEF 2019 VQA-Med com-
petition. The challenge consists of four question-answering tasks based
on radiology images. The diversity of imaging modalities, organs and
disease types combined with a small imbalanced training set made this a
highly complex problem. To overcome these difficulties, we implemented
a modular pipeline architecture that utilized transfer learning and multi-
task learning. Our findings led to the development of a novel model
called Supporting Facts Network (SFN). The main idea behind SFN is
to cross-utilize information from upstream tasks to improve the accu-
racy on harder downstream ones. This approach significantly improved
the scores achieved in the validation set (18 point improvement in F-
1 score). Finally, we submitted four runs to the competition and were
ranked seventh.
Keywords: ImageCLEF 2019 · VQA-Med · Visual Question Answering
· Supporting Facts Network · Multi-Task Learning · Transfer Learning
1 Introduction
In the era of data deluge and powerful computing systems, deriving meaningful
insights from heterogeneous information has shown to have tremendous value
across industries. In particular, the promise of deep learning-based computa-
tional models [15] in accurately predicting diseases has further stirred great
interest in adopting automated learning systems in healthcare [2]. A daunt-
ing challenge within the realm of healthcare is to efficiently sieve through vast
amounts of multi-modal information and reason over them to arrive at a differen-
tial diagnosis. Longitudinal patient records including time-series measurements,
text reports and imaging volumes form the basis for doctors to draw conclusive
insights. In practice, radiologists are tasked with reviewing thousands of imag-
ing studies each day, with an average of about three seconds to mark them as
Copyright c 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0). CLEF 2019, 9-12 Septem-
ber 2019, Lugano, Switzerland.
anomalous or not, leading to severe eye fatigue [24]. Moreover, clinical work-
flows have a sequential nature tending to cause delays in triage situations, where
the existence of answers to key questions about a patient’s holistic conditions
can potentially expedite treatment. Thus, building effective question-answering
systems for the medical domain by bringing advancements in machine learning
research will be a game changer towards improving patient care.
Visual Question Answering (VQA) [17, 1] is a new exciting problem domain,
where the system is expected to answer questions expressed in natural language
by taking into account the content of the image. In this paper, we present results
of our research on the VQA-Med 2019 dataset [3], an open challenge associated
with the ImageCLEF 2019 initiative [10]. The main issue here, in comparison
to the other recent VQA datasets such as TextVQA [23] or GQA [9], is dealing
with scattered, noisy and heavily biased data. Hence, the dataset serves as a
great use-case to study challenges encountered in practical clinical scenarios.
In order to address the data issues, we designed a new model called Support-
ing Facts Network (SFN) that efficiently shares knowledge between upstream
and downstream tasks through the use of a pre-trained multi-task solver in com-
bination with task-specific solvers. Note that posing the VQA-Med challenge as
a multi-task learning problem [4] allowed the model to effectively leverage and
encode relevant domain knowledge. Our multi-task SFN model outperforms the
single task baseline by better adapting to label distribution shifts.
2 The VQA-Med dataset
The VQA-Med 2019 [3] is a Visual Question Answering (VQA) dataset embedded
in the medical domain, with a focus on radiology images. It consists of:
– a training set of 3,200 images with 12,792 Question-Answer (QA) pairs,
– a validation set of 500 images with 2,000 QA pairs, and
– a test set of 500 images with 500 questions (answers were released after the
end of the VQA-Med 2019 challenge).
In all splits the samples were divided into four categories, depending on the main
task to be solved:
– C1: determine the modality of the image,
– C2: determine the plane of the image,
– C3: identify the organ/anatomy of interest in the image, and
– C4: identify the abnormality in the image.
Our analysis of the dataset (distribution of questions, answers, word vocabular-
ies, categories and image sizes) led to the following findings and system-design
related decisions:
– merge of the original training and validation sets, shuffle and re-sample new
training and validation sets with a proportion of 19:1,
– use of weighted random sampling during batch preparation,
– addition of a fifth Binary category for samples with Y/N type questions,
– focus on accuracy-related metrics instead of the BLEU score,
– avoid label (answer classes) unification and cleansing,
– consider C4 as a downstream task and exclude it from the pre-training of
input fusion modules,
– utilization of image size as an additional input cue to the system.
3 Supporting Facts Network
Typical VQA systems process two types of input, visual (image) and language
(question), that need to undergo various transformations to produce the answer.
Fig. 1 presents a general architecture of such systems, indicating four major
modules: two encoders responsible for encoding raw inputs to more useful rep-
resentations, followed by a reasoning module that combines them and finally, an
answer decoder that produces the answer.
Image Image
Encoder
Reasoning Answer Answer
Module Decoder
Question Question
Encoder
Fig. 1. General architecture of Visual Question Answering systems
In the early prototypes of VQA systems, reasoning modules were rather sim-
ple and relied mainly on multi-modal fusion mechanisms. These fusion tech-
niques varied from concatenation of image and question representations, to more
complex pooling mechanisms such as Multi-modal Compact Bilinear pooling
(MCB) [6] and Multi-modal Low-rank Bilinear pooling (MLB) [12]. Further,
diverse attention mechanisms such as question-driven attention over image fea-
tures [11] were also used. More recently, researchers have focused on complex
multi-step reasoning mechanisms such as Relational Networks [21, 5] and Mem-
ory, Attention and Composition (MAC) networks [8, 18]. Despite that, certain
empirical studies indicate early fusion of language and vision signals significantly
boosts the overall performance of VQA systems [16]. Therefore, we explored the
finding of an ”optimal” module for early fusion of multi-modal inputs.
3.1 Architecture of the Input Fusion Module
One of our findings from analyzing the dataset was to use the image size as ad-
ditional input cue to the system. This insight triggered an extensive architecture
search that included, among others, comparison and training of models with:
– different methods for question encoding, from 1-hot encoding with Bag-of-
Words to different word embeddings combined with various types of recur-
rent neural networks,
– different image encoders, from simple networks containing few convolutional
layers trained from scratch to fine-tuning of selected state-of-the-art models
pre-trained on ImageNet,
– various data fusion techniques as mentioned in the previous section.
Question Question Word Question
Tokenizer Embeddings Encoder
Image Feature Map Fusion I Question
Extraction Question Tokenizer Classifier Category
Image Image Size Fusion II Fused Word Question
size Encoder Inputs Embeddings Encoder
(a) Input Fusion (b) Question Categorizer
Fig. 2. Architectures of two modules used in the final system
The final architecture of our model is presented in Fig. 2a. We used GloVe
word embeddings [20] followed by Long Short-Term Memory (LSTM) [7]. The
LSTM outputs along with feature maps extracted from images using VGG-
16 [22] were passed to the Fusion I module, implementing question-driven
attention over image features [11]. Next, the output of that module was con-
catenated in the Fusion II module with image size representation created by
passing image width and height through a fully connected (FC) layer.
Note that the green colored modules were initially pre-trained on external
datasets (ImageNet and 6B tokens from Wikipedia 2014 and Gigaword 5 datasets
for VGG-16 and GloVe models respectively) and later fine-tuned during training
on the VQA-Med dataset.
3.2 Architectures of the Reasoning Modules
During the architecture search of the Input Fusion module we used the model
presented in Fig. 3, with a simple classifier with two FC layers. These models were
trained and validated on C1, C2 and C3 categories separately, while excluding
C4. In fact, to test our hypothesis we trained some early prototypes only on
samples from C4 and the models failed to converge.
After establishing the Input Fusion module we trained it on samples from
C1, C2 and C3 categories. This served as a starting point for training more
complex reasoning modules. At first, we worked on a model that exploited in-
formation about 5 categories of questions by employing 5 separate classifiers
which used data produced by the Input Fusion module. Each of these clas-
sifiers essentially specialized in one question category and had its own answer
Question
Image Input Classifier Answer
Fusion
Image
size
Fig. 3. Architecture with a single classifier (IF-1C)
label dictionary and associated loss function. The predictions were then fed to
the Answer Fusion module, which selected answers from the right classifier
based on the question category predicted by the Question Categorizer mod-
ule, whose architecture is shown in Fig. 2b. Please note that we pre-trained the
module in advance on all samples from all categories and froze its weights during
the training of classifiers.
Question Support C1 Classifier C1
Categorizer
Question Support C2 Classifier C2
Image Input Support C3 Classifier C3 Answer Answer
Fusion Fusion
Image
size Classifier C4
Fusion III
Classifier Y/N
Fig. 4. Final architecture of the Supporting Facts Network (SFN)
The architecture of our final model, Supporting Facts Network is pre-
sented in Fig. 4. The main idea here resulted from the analysis of questions about
the presence of abnormalities – to answer which the system required knowledge
on image modality and/or organ type. Therefore, we divided the classification
modules into two networks: Support networks (consisting of two FC layers) and
final classifiers (being single FC layers). We added Plane (C2) as an additional
supporting fact. The supporting facts were then concatenated with output from
Input Fusion module in Fusion III and passed as input to the classifier spe-
cialized on C4 questions. In addition, since Binary Y/N questions were present
in both C1 and C4 categories, we followed a similar approach for that classifier.
4 Experimental Results
All experiments were conducted using PyTorchPipe [14], a framework that fa-
cilitates development of multi-modal pipelines built on top of PyTorch [19]. Our
models were trained using relatively large batches (256), dropout (0.5) and Adam
optimizer [13] with a small learning rate (1e − 4). For each experimental run, we
generated a new training and validation set by combining the original sets, shuf-
fling and sampling them in proportions of 19 : 1, thereby resulting in a validation
set of size 5%.
Resampled Valid. Set Original Train. Set Original Valid. Set
Model Prec. Recall F-1 Prec. Recall F-1 Prec. Recall F-1
IF-1C 0.630 0.435 0.481 0.683 0.497 0.545 0.690 0.499 0.548
SFN 0.759 0.758 0.758 0.753 0.692 0.707 0.762 0.704 0.717
Table 1. Summary of experimental results. All columns contain average scores achieved
by 5 separately trained models on resampled training and validation sets. We also
present scores achieved by the models on original sets (in the evaluation mode).
In Tab. 1 we present a comparison of average scores achieved by our baseline
models using single classifier (IF-1C) and the Supporting Facts Networks (SFN).
Our results clearly indicate the advantage of using ’supporting facts’ over the
baseline model with a single classifier. The SFN model by our team achieved a
best score of (0.558 Accuracy, 0.582 BLEU score) on the test set as indicated
by the CrowdAI leaderboard. One of the reasons for such a significant drop in
performance is due to the presence of new answers classes in the test set that
were not present both in the original training and validation sets.
5 Summary
In this work, we introduced a new model called Supporting Facts Network
(SFN), that leverages knowledge learned from combinations of upstream tasks
in order to benefit additional downstream tasks. The model incorporates domain
knowledge that we gathered from a thorough analysis of the dataset, resulting in
specialized input fusion methods and five separate, category-specific classifiers.
It comprises of two pre-trained shared modules followed by a reasoning module
jointly trained with five classifiers using the multi-task learning approach. Our
models were found to train faster and to deal much better with label distribution
shifts under a small imbalanced data regime.
Among the five categories of samples present in the VQA-Med dataset, C4
and Binary turned out to be extremely difficult to learn, for several reasons.
First, there were 1483 unique answer classes assigned to 3082 training samples
related to C4. Second, both C4 and Binary required more complex reasoning
and, besides, might be impossible to conclude by looking only at the question
and content of the image. However, our observation that some of the information
from simpler categories might be useful during reasoning on more complex ones,
we refined the model by adding supporting networks. Given, modality, imaging
plane and organ typically help narrow down the scope of disease conditions
and/or answer whether or not an abnormality is present. Our empirical studies
prove that this approach performs significantly better, leading to an 18 point
improvement in F-1 score over the baseline model on the original validation set.
References
1. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., Parikh,
D.: VQA: Visual Question Answering. In: Proceedings of the IEEE international
conference on computer vision. pp. 2425–2433 (2015)
2. Ardila, D., Kiraly, A.P., Bharadwaj, S., Choi, B., Reicher, J.J., Peng, L., Tse,
D., Etemadi, M., Ye, W., Corrado, G., Naidich, D.P., Shetty, S.: End-to-end lung
cancer screening with three-dimensional deep learning on low-dose chest computed
tomography. Nature Medicine (2019)
3. Ben Abacha, A., Hasan, S.A., Datla, V.V., Liu, J., Demner-Fushman, D., Müller,
H.: VQA-Med: Overview of the medical visual question answering task at image-
clef 2019. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-
WS.org < http : //ceur − ws.org/ >, Lugano, Switzerland (September 09-12 2019)
4. Caruana, R.: Multitask learning. Machine learning 28(1), 41–75 (1997)
5. Desta, M.T., Chen, L., Kornuta, T.: Object-based reasoning in VQA. In: 2018 IEEE
Winter Conference on Applications of Computer Vision (WACV). pp. 1814–1823.
IEEE (2018)
6. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multi-
modal compact bilinear pooling for visual question answering and visual grounding.
In: EMNLP (2016)
7. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation
9(8), 1735–1780 (1997)
8. Hudson, D.A., Manning, C.D.: Compositional attention networks for machine rea-
soning. In: CVPR (2018)
9. Hudson, D.A., Manning, C.D.: Gqa: a new dataset for compositional question
answering over real-world images. arXiv preprint arXiv:1902.09506 (2019)
10. Ionescu, B., Müller, H., Péteri, R., Cid, Y.D., Liauchuk, V., Kovalev, V., Klimuk,
D., Tarasau, A., Ben Abacha, A., Hasan, S.A., Datla, V., Liu, J., Demner-Fushman,
D., Dang-Nguyen, D.T., Piras, L., Riegler, M., Tran, M.T., Lux, M., Gurrin, C.,
Pelka, O., Friedrich, C.M., de Herrera, A.G.S., Garcia, N., Kavallieratou, E., del
Blanco, C.R., Rodrı́guez, C.C., Vasillopoulos, N., Karampidis, K., Chamberlain,
J., Clark, A., Campello, A.: ImageCLEF 2019: Multimedia retrieval in medicine,
lifelogging, security and nature. In: Experimental IR Meets Multilinguality, Mul-
timodality, and Interaction. Proceedings of the 10th International Conference of
the CLEF Association (CLEF 2019), LNCS Lecture Notes in Computer Science,
Springer, Lugano, Switzerland (September 9-12 2019)
11. Kazemi, V., Elqursh, A.: Show, ask, attend, and answer: A strong baseline for
visual question answering. arXiv preprint arXiv:1704.03162 (2017)
12. Kim, J.H., On, K.W., Lim, W., Kim, J., Ha, J.W., Zhang, B.T.: Hadamard product
for low-rank bilinear pooling. In: ICLR (2017)
13. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 (2014)
14. Kornuta, T.: PyTorchPipe. https://github.com/ibm/pytorchpipe (2019)
15. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436 (2015)
16. Malinowski, M., Doersch, C.: The Visual QA devil in the details: The impact of
early fusion and batch norm on CLEVR. In: ECCV’18 Workshop on Shortcomings
in Vision and Language (2018)
17. Malinowski, M., Fritz, M.: A multi-world approach to question answering about
real-world scenes based on uncertain input. In: Advances in neural information
processing systems. pp. 1682–1690 (2014)
18. Marois, V., Jayram, T., Albouy, V., Kornuta, T., Bouhadjar, Y., Ozcan, A.S.: On
transfer learning using a MAC model variant. In: NeurIPS’18 Visually-Grounded
Interaction and Language (ViGIL) Workshop (2018)
19. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z.,
Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)
20. Pennington, J., Socher, R., Manning, C.: Glove: Global vectors for word repre-
sentation. In: Proceedings of the 2014 conference on empirical methods in natural
language processing (EMNLP). pp. 1532–1543 (2014)
21. Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia, P.,
Lillicrap, T.: A simple neural network module for relational reasoning. In: Advances
in Neural Information Processing Systems. pp. 4967–4976 (2017)
22. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recognition. arXiv preprint arXiv:1409.1556 (2014)
23. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D.,
Rohrbach, M.: Towards vqa models that can read. arXiv preprint arXiv:1904.08920
(2019)
24. Syeda-Mahmood, T., Walach, E., Beymer, D., Gilboa-Solomon, F., Moradi, M.,
Kisilev, P., Kakrania, D., Compas, C., Wang, H., Negahdar, R., et al.: Medical
sieve: a cognitive assistant for radiologists and cardiologists. In: Medical Imaging
2016: Computer-Aided Diagnosis. vol. 9785, p. 97850A. International Society for
Optics and Photonics (2016)