This paper describes the participation of the University of Massachusetts Medical School in the ImageCLEF 2019 Med-VQA task. The goal is to predict the answers given the medical images and the questions. The categories of the questions are provided for the training and validation datasets. We implemented long-short-term memory (LSTM) for question textual feature extraction and transfer learning followed by the co-attention mechanism for image feature extraction. Due to the provided category information, we implemented the SVM model the predict the question category which is used as another feature for our system. In addition, we applied the embedding based topic model (ETM) to generate question topic distribution as one more feature for our system. To efficiently integrate different types of features, we employed the multi-modal factorized high-order pooling (MFH). For the answer prediction, we developed a two-channel framework to handle different categories of questions through single-label classification and multi-label classification respectively. We submitted 3 valid runs, and the best system achieved the accuracy of 0.566 and the BLEU score of 0.593, ranking the 5th place among 17 participating groups.
Given an image and a natural language question about the image, visual question answering (VQA) task is to provide an accurate natural language answer. This task combines computer vision (CV) and natural language processing (NLP). One of the challenges for VQA task is how to fuse different types of features. Various methods, like LinearSum and Multi-modal Factorized Bilinear Pooling (MFB), have been designed and practiced on the VQA task.
A lot of studies of VQA task are in the general domain. With increasing implementations of deep learning to support clinical decision making and improve patient engagement, some studies begin to focus on the VQA task in the medical domain. ImageCLEF 2019 [1] organized the inaugural edition of the Medical Domain Visual Question Answering (Med-VQA) Task [2]. Given a medical image with a clinically relevant question, the system is tasked with answering the question based on the visual image content. The dataset of this year is different from the last year. The categories of the questions are provided. For the questions in the first three categories, there are a limited number of answer candidates. And the answers to the questions in the last category are narrative.
In this work, we introduced the question category information and the question topic distribution as two additional information during the information fusion process. To develop an integrated system that is able to handle all four categories of questions, we developed two-channel structures for the answer prediction. One channel is to classify the image-question pairs into the close set of answer candidates. The other channel is to generate a narrative answer given an image-question pair. 2
Our system consists of 6 components: transfer learning for image feature extraction, LSTM for question textual feature extraction, other features (including question category information and question topic distribution), co-attention mechanism, MFH for feature fusion, and answer generation. Fig. 1 shows the architecture of our system. A pre-trained biomedical word embedding (dimension of 200) is used as the embedding layer. After the word embedding layer, a bidirectional LSTM network is used to extract textual features of the question. During the training process, the embedding of the “unknown” token is first initialized randomly and then learned. The textual features are transformed to predict the attention weight of different grid locations, which generates the attentional features of the question. 2.2
We applied transfer learning to extract image features. The pre-trained ResNet-152 model of ImageNet (excluding the last 2 layers, pooling layer and fully-connected layer) is the image feature extractor. The parameters of the last 2 convolutional blocks of ResNet-152 model are fine-tuned during the training process. Then we applied the co-attention mechanism to generate the attentional features of the image. 2.3
ETM [3] is applied to generate several topics from the questions. 10 topics are generated by applying ETM on the questions in the training dataset. Each question is assigned a vector of topic distribution according to the frequencies of topics’ words appearing in the question. The topic distribution is used as another input feature of the MFH. According to the instruction of the 2019 Med-VQA, the questions are from 4 categories. SVM is used to classify the category of the question. We applied TF-IDF based unigram vectorization to extract textual features, and trained a support vector machines (SVM) model using the questions from the training dataset. The accuracy of the SVM model on validation dataset is 100%, which shows that the language used in different categories of questions are relatively unique and less ambiguous. The category information of the question is used as an additional input of the MFH. 2.5
MFH [4] contains multiple dependent MFB blocks. The output from the expand stage of the previous MFB block is fed into the next MFB block as additional input, and the output from multiple MFB blocks are merged together as a final fused feature representation.
We applied a 2-block MFH model to fuse 4 types of features including image attentional features, question attentional features, question topic distribution, and question category, which is shown in Fig. 2. 2.6
According to the instruction of 2019 Med-VQA, for the questions of the first 3 categories, the corresponding answers are in a limited number of certain candidates. We regarded this case as a single-label classification task. On the other hand, the questions of “abnormality” category are the narrative type. This case is regarded as a multi-label classification task. So, we built two-channel structures which are shown in Fig. 3. One is for the single-label classification task and the other one is for the multi-label classification task. For multi-label classification, each unique word in the answer sentence is considered an answer label for the corresponding image-question pair. Based on the distribution of all the answer labels, the narrative answer is generated using the sampling method.
Therefore, both a classification result and a distribution of words in the answer are predicted through our system for each pair of image-question. If the classification result is one of the certain candidates of the answers, the final answer is that candidate. Otherwise, the final answer is a combination of words generated by the sampling method.
The loss function of our system is an integration of two loss functions. We applied
the cross-entropy loss function for the single-label classification structure and
Kullback–Leibler divergence loss function for the multi-label classification structure. Given
an image-question pair, the loss L is calculated as follows:
= (1 − ) ∗ + ∗
(
We experimented with 4 settings of the pre-trained ResNet-152 model on ImageNet:
(
A pre-trained word-embedding (dimension of 200) on PubMed and the clinical notes
from MIMIC-III Clinical Database is used as the word embedding layer. We
experimented with 2 settings to handle “unknown” token in the questions: (
We submitted 3 runs on the test dataset and the settings and results are shown in Table 1. The accuracy of our best system is 0.566 and the BLEU score is 0.593 on the test dataset, ranking fifth place. The team ranking first place of this competition obtained an accuracy of 0.624 and a BLEU score of 0.644. We experimented with three different settings of deep learning structures for MEDVQA task 2019, where we introduced two more types of features and we constructed two-channel structures for answer prediction. Due to the limited time, we did not implement Bidirectional Encoder Representations from Transformers (BERT) [6] which will be explored in the future to extract question textual features. 5
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.