In this working notes paper, we describe IBM Research AI (Almaden) team's participation in the ImageCLEF 2019 VQA-Med competition. The challenge consists of four question-answering tasks based on radiology images. The diversity of imaging modalities, organs and disease types combined with a small imbalanced training set made this a highly complex problem. To overcome these di culties, we implemented a modular pipeline architecture that utilized transfer learning and multitask learning. Our ndings led to the development of a novel model called Supporting Facts Network (SFN). The main idea behind SFN is to cross-utilize information from upstream tasks to improve the accuracy on harder downstream ones. This approach signi cantly improved the scores achieved in the validation set (18 point improvement in F1 score). Finally, we submitted four runs to the competition and were ranked seventh.
In the era of data deluge and powerful computing systems, deriving meaningful
insights from heterogeneous information has shown to have tremendous value
across industries. In particular, the promise of deep learning-based
computational models [
Visual Question Answering (VQA) [
In order to address the data issues, we designed a new model called
Supporting Facts Network (SFN) that e ciently shares knowledge between upstream
and downstream tasks through the use of a pre-trained multi-task solver in
combination with task-speci c solvers. Note that posing the VQA-Med challenge as
a multi-task learning problem [
The VQA-Med 2019 [
In all splits the samples were divided into four categories, depending on the main task to be solved: { C1: determine the modality of the image, { C2: determine the plane of the image, { C3: identify the organ/anatomy of interest in the image, and { C4: identify the abnormality in the image.
Our analysis of the dataset (distribution of questions, answers, word vocabularies, categories and image sizes) led to the following ndings and system-design related decisions: { merge of the original training and validation sets, shu e and re-sample new training and validation sets with a proportion of 19:1, { use of weighted random sampling during batch preparation, { addition of a fth Binary category for samples with Y/N type questions, { focus on accuracy-related metrics instead of the BLEU score, { avoid label (answer classes) uni cation and cleansing, { consider C4 as a downstream task and exclude it from the pre-training of input fusion modules, { utilization of image size as an additional input cue to the system. 3
Typical VQA systems process two types of input, visual (image) and language (question), that need to undergo various transformations to produce the answer. Fig. 1 presents a general architecture of such systems, indicating four major modules: two encoders responsible for encoding raw inputs to more useful representations, followed by a reasoning module that combines them and nally, an answer decoder that produces the answer.
Image Question
Image Encoder Question Encoder
Reasoning Module
Answer Decoder
In the early prototypes of VQA systems, reasoning modules were rather
simple and relied mainly on multi-modal fusion mechanisms. These fusion
techniques varied from concatenation of image and question representations, to more
complex pooling mechanisms such as Multi-modal Compact Bilinear pooling
(MCB) [
Image
The nal architecture of our model is presented in Fig. 2a. We used GloVe
word embeddings [20] followed by Long Short-Term Memory (LSTM) [
Note that the green colored modules were initially pre-trained on external datasets (ImageNet and 6B tokens from Wikipedia 2014 and Gigaword 5 datasets for VGG-16 and GloVe models respectively) and later ne-tuned during training on the VQA-Med dataset. 3.2
Architectures of the Reasoning Modules During the architecture search of the Input Fusion module we used the model presented in Fig. 3, with a simple classi er with two FC layers. These models were trained and validated on C1, C2 and C3 categories separately, while excluding C4. In fact, to test our hypothesis we trained some early prototypes only on samples from C4 and the models failed to converge.
After establishing the Input Fusion module we trained it on samples from C1, C2 and C3 categories. This served as a starting point for training more complex reasoning modules. At rst, we worked on a model that exploited information about 5 categories of questions by employing 5 separate classi ers which used data produced by the Input Fusion module. Each of these classi ers essentially specialized in one question category and had its own answer Question
Image Image size
Image label dictionary and associated loss function. The predictions were then fed to the Answer Fusion module, which selected answers from the right classi er based on the question category predicted by the Question Categorizer module, whose architecture is shown in Fig. 2b. Please note that we pre-trained the module in advance on all samples from all categories and froze its weights during the training of classi ers.
Support C1 Support C2 Support C3 Fusion III
Classifier C1 Classifier C2 Classifier C3 Classifier C4 Classifier Y/N
Answer Fusion
The architecture of our nal model, Supporting Facts Network is presented in Fig. 4. The main idea here resulted from the analysis of questions about the presence of abnormalities { to answer which the system required knowledge on image modality and/or organ type. Therefore, we divided the classi cation modules into two networks: Support networks (consisting of two FC layers) and nal classi ers (being single FC layers). We added Plane (C2) as an additional supporting fact. The supporting facts were then concatenated with output from Input Fusion module in Fusion III and passed as input to the classi er specialized on C4 questions. In addition, since Binary Y/N questions were present in both C1 and C4 categories, we followed a similar approach for that classi er. 4
All experiments were conducted using PyTorchPipe [
Prec.
In Tab. 1 we present a comparison of average scores achieved by our baseline models using single classi er (IF-1C) and the Supporting Facts Networks (SFN). Our results clearly indicate the advantage of using 'supporting facts' over the baseline model with a single classi er. The SFN model by our team achieved a best score of (0:558 Accuracy, 0:582 BLEU score) on the test set as indicated by the CrowdAI leaderboard. One of the reasons for such a signi cant drop in performance is due to the presence of new answers classes in the test set that were not present both in the original training and validation sets. 5
In this work, we introduced a new model called Supporting Facts Network (SFN), that leverages knowledge learned from combinations of upstream tasks in order to bene t additional downstream tasks. The model incorporates domain knowledge that we gathered from a thorough analysis of the dataset, resulting in specialized input fusion methods and ve separate, category-speci c classi ers. It comprises of two pre-trained shared modules followed by a reasoning module jointly trained with ve classi ers using the multi-task learning approach. Our models were found to train faster and to deal much better with label distribution shifts under a small imbalanced data regime.
Among the ve categories of samples present in the VQA-Med dataset, C4 and Binary turned out to be extremely di cult to learn, for several reasons. First, there were 1483 unique answer classes assigned to 3082 training samples related to C4. Second, both C4 and Binary required more complex reasoning and, besides, might be impossible to conclude by looking only at the question and content of the image. However, our observation that some of the information from simpler categories might be useful during reasoning on more complex ones, we re ned the model by adding supporting networks. Given, modality, imaging plane and organ typically help narrow down the scope of disease conditions and/or answer whether or not an abnormality is present. Our empirical studies prove that this approach performs signi cantly better, leading to an 18 point improvement in F-1 score over the baseline model on the original validation set. 16. Malinowski, M., Doersch, C.: The Visual QA devil in the details: The impact of early fusion and batch norm on CLEVR. In: ECCV'18 Workshop on Shortcomings in Vision and Language (2018) 17. Malinowski, M., Fritz, M.: A multi-world approach to question answering about real-world scenes based on uncertain input. In: Advances in neural information processing systems. pp. 1682{1690 (2014) 18. Marois, V., Jayram, T., Albouy, V., Kornuta, T., Bouhadjar, Y., Ozcan, A.S.: On transfer learning using a MAC model variant. In: NeurIPS'18 Visually-Grounded Interaction and Language (ViGIL) Workshop (2018) 19. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z.,
Desmaison, A., Antiga, L., Lerer, A.: Automatic di erentiation in pytorch (2017) 20. Pennington, J., Socher, R., Manning, C.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532{1543 (2014) 21. Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia, P., Lillicrap, T.: A simple neural network module for relational reasoning. In: Advances in Neural Information Processing Systems. pp. 4967{4976 (2017) 22. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014) 23. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. arXiv preprint arXiv:1904.08920 (2019) 24. Syeda-Mahmood, T., Walach, E., Beymer, D., Gilboa-Solomon, F., Moradi, M., Kisilev, P., Kakrania, D., Compas, C., Wang, H., Negahdar, R., et al.: Medical sieve: a cognitive assistant for radiologists and cardiologists. In: Medical Imaging 2016: Computer-Aided Diagnosis. vol. 9785, p. 97850A. International Society for Optics and Photonics (2016)