This paper describes our approach for Medical Visual Question Answering (VQA-Med 2020) Task of ImageCLEF 2020. We used an encoder-decoder architecture for generating answers given the question and image. The encoder takes two inputs: the first is a feature vectors of image obtained from VGG16, and the second is a vector representation for each question using BERT. The question features are self-attended to get attention features. The question attention features, and image features are fused using multi-modal factorized bilinear pooling (MFB). The fused features are further self-attended to get fuse attention features. The thought vectors are obtained by concatenation of fuse attention and encoder LSTM hidden states. The decoder generates answer word by word for the input question and the image. The decoder consists of LSTM layer, Bahdanau Attention, and dense layer of answer vocabulary size with SoftMax. The answers are embedded using GLOVE word vectors before being passed to the decoder LSTM. Our best model achieves 37.8% accuracy and BLEU score of 0.439.
Visual question answering requires understanding of Computer Vision and NLP simultaneously. VQA has gained lot of attention from academicians and researchers due to introduction of new language models like BERT, new feature fusion techniques based on bilinear pooling and attention mechanisms. The recent advancements in artificial intelligence have encouraged the healthcare sector in storing large number of health records electronically. VQA can be used for automatic interpretation of radiology images, thereby helping make clinical decisions
In this paper, we present our method to build a deep learning model for ImageCLEF
VQA-Med 2020 Task [
This paper is organized in the following manner: Section 2 provides a brief information regarding similar works done on VQA-Med which inspired our work. Section 3 presents a brief description of dataset used. Section 4 describes our model architecture based on encoder-decoder system. Section 5 describes our model evaluation and results achieved on test set. Section 6 presents conclusions and future work. 2
The dataset used here is VQA-Med-VQA 2020 [
The training set: 4,000 radiology images with 4,000 associated QuestionAnswer (QA) pairs.
The validation set: 500 radiology images with 500 QA pairs.
The VQA test set: 500 radiology images with 500 associated questions. Additional data for training was used from abnormality section of VQA-Med 2019 dataset which constitutes 3817 images and 3817 Question-Answer (QA) pairs.
Fig 1 shows few examples of questions and answers from dataset.
Fig 1 Examples from VQA-Med-2020 dataset
In past few years, many interesting approaches are reported on VQA and most of these approaches consider VQA as a classification problem. However, the generative models are the only better options if the answers are long. In particular, the best performing VQA models reported in VQA-Med 2019 task were classification based [3,4,5 and 6].
In this paper, we aim to explain our generative approach for VQA on the VQA-Med
2020 task. We have used VGG16 [
Fig 2 Model Structure
Fig 2 shows the proposed model based on encoder-decoder system. Different parts of this model are described in the following paragraphs.
The encoder has two main components. The first component is a Pretrained
VGG16 [
The decoder consists of LSTM network and attention network that takes the thought vector as initial state and encoder LSTM sequence output and try to predict the answer word by word. 4.1
The encoder takes image and the question as input and returns thought vectors and sequence output.
The image features are extracted using VGG16 Pretrained model from last pooling layer with size of 7*7*512. This image feature vector is then passed to the MLP block having two fully connected layers with 2048 and 768 hidden nodes respectively along with dropout. The main purpose of this MLP block is to decrease the feature vector dimension to half of the LSTM output vectors.
The semantic meanings of the questions are extracted using 12-layer, 768 hidden BERT Pretrained model. For each question, we will get a feature vector of size t*768, where t is the max sequence length of the question.
It has been reported that the attention mechanism allows the VQA models to effectively learn which regions of the image are important for given question. It’s always better to employ an effective attention mechanism to improve the performance of VQA models. Our model uses question image fuse attention based on glimpse attention networks. We have used two glimpses each for question and image as an optimal choice based on Fukui et. al. 2016 [11].
For question self-attention, the output from Pretrained BERT (t*768) is passed to question attention layer giving output attention feature vector of size 768. The image features from MLP block (1*768) and question attention features are fused using multimodal bilinear factorize pooling (MFB) [12] giving an output vector of size 1*768. Finally, the MFB output is passed to image attention layer with two glimpses giving an output of size 1*768.
Long Short-Term Memory networks or LSTM are a special type of RNN that are designed to avoid long range dependency problem of vanilla RNN networks. The LSTM cell carries an extra memory state (other than hidden state h) to store the context information. An LSTM has three different gates (i.e. Input gate, forget gate and output gate) which decide flow of information across time steps.
At each time step, LSTM cell has three inputs viz. current word (xt), previous hidden state (ht-1) and previous memory state (ct-1), and three outputs viz. output hidden state (ht), output memory state (ct) and encoded sequence.
In our model, the input to encoder LSTM are obtained by concatenation of question features extracted from BERT and tiled fuse attention from MFB. The Encoder LSTM has hidden nodes of 768. The outputs are hidden state h (768), memory state c (768) and encoded sequence of size t*768. 4.2
The decoder generates answer word by word for the input question and the image.
The decoder consists of LSTM layer, Bahdanau Attention Layer and dense layer of
answer vocabulary size with SoftMax. The answers are embedded using GLOVE [
At first time step, decoder takes four inputs viz. the start of sequence <SOS>, hidden state of the encoder, memory state of encoder and encoder sequence output. The output will be the first word of the answer which is obtained with highest probability using SoftMax layer. This word will be the input to LSTM at second time step to predict the next word of the answer. This process keeps on going until the decoder predicts end of sequence <EOS> token. 5
A total of 5 runs were submitted to ImageCLEF VQA-Med-VQA 2020. The
training was done on NVIDIA GeForce 940 MX GPU Device. The CUDA
implementation of LSTM was employed from Tensorflow 2.0. Evaluation of the model
was conducted using two different metrics: BLEU and Strict Accuracy, based on
VQAMed-VQA 2020 competition [
There are two main models: model1 has hidden size of 1024 with approx. size of 36M and model2 with hidden size of 768 and approx. size of 26M. Model1 is trained with Adam optimizer while model2 is trained with RMSprop optimizer. Experiments were also done with different dropouts. 6
In this paper we describe the model we submitted in ImageCLEF 2020 VQAMed-VQA task. Our proposed model VGG+BERT+MFB+GLOVE employs an encoder-decoder architecture with an advanced feature fusion technique MFB with question image fuse attention. The image and question features were extracted using VGG16 and BERT Base networks respectively. The answers were encoded using GLOVE word embeddings. Our model achieved the accuracy of 37.8% and BLEU score of 0.439 on the test set. 7 10. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. 11. A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847, 2016. 2, 3, 4, 5, 6, 7, 8 12. Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1839{1848, 2017.