This paper briefly describes our model for the ImageCLEF Medical Visual Question Answering Task 2021 (ImageCLEF VQA-Med task 2021). Our method is based on a universal VQA framework and consists of image feature extraction module, question feature extraction module and feature fusion module. We employ the modified ResNet-34 as the backbone to construct an image feature extractor, which efectively extracts pixel-level features and enhances the model performance in a deep network. For question feature extraction, we firstly use word embedding to map question tokens to high dimension vectors, and then input them to a long-short-term memory (LSTM) to extract high-level question features. In addition, we leverage Multi-modal Factorized Bilinear Pooling (MFB) with a co-Attention mechanism to fuse these features to predict final answers. Our model achieves the accuracy score of 0.222 and bleu score of 0.255, ranking at the eighth among all participating teams in the VQA-Med task.
In recent years, the applications of deep learning in Computer Vision (CV) and Natural Language
Processing (NLP) have gained remarkable progress. The development of deep learning in single
modality facilitates researchers to explore multimodal studies, e.g., image-text retrieval [
In the ImageCLEF VQA-Med task 2021 [
As for the universal VQA benchmark datasets, we can utilize the object-level and pixel-level
information of images as well as the entity information of questions for answer prediction.
This prior knowledge can greatly enhance the performances of the VQA models. Due to the
shortages of ImageCLEF VQA-Med task 2021 dataset, we utilize the modified ResNet-34 [
(a) Data example of ImageCLEF 2021 VQA-Med dataset.
(b) Data example of common VQA tasks
CNN has been widely used in image feature extraction throughout computer vision. Since LeNet
[
The researches of NLP tasks have been greatly promoted by the proposal of transformer.
Transformer introduces the full-connected layer (FC) to build the self-attention mechanism,
which replaces RNN to learn contextual information of a long-length sentence. Later, Devlin J
et al. [
The simple methods of feature fusion in deep neural networks include concatenating the diferent kinds of feature in channel dimension, making element-wise sum or producting with same feature map sizes. But these might not be expressive enough to fully capture the complex associations between the two diferent modalities. The Multimodal Compact Bilinear pooling (MCB) [21] projects the images and text representations to a higher dimensional space, and then convolves both vectors by using element-wise product in Fast Fourier Transform (FFT) space. The Multi-modal Factorized Bilinear Pooling (MFB) [22] introduces co-Attention mechanism to jointly learn both image and question attention. The co-Attention mechanism can efectively learn which regions are important for the images related to the questions.
The overview of the architecture of our proposed model is shown in Figure 2. Our model consists of three components: A image feature extraction module, a question feature extraction module and an attention feature fusion module.
The main component of the original ResNet-34 is convolution neural network (CNN). With the characteristics of translation invariant, translation equivalence, scale invariance and rotation invariance, CNN can learn strong data bias with a small number of data. Besides, the residual structure has a large receptive field and it keeps gradient from vanishing in model training process. Benefitting from these strengthens, this image feature extraction module efectively learns pixel features and spatial features of the medical images. Although it could be easy to overfit because of the small number of data, we alleviate this situation with a dropout operation. The output of the original ResNet-34 is a 1000-dimension vectors used for 1000 classification, which is not suitable in this task.
Compared to the original ResNet-34, we remove the global average pooling layer (GAP) and full-connected layer (FC), as showed in Figure 3. Before the images are fed to this module, each image is resized to 128×128 with the INTER_AREA algorithm. In order to fit the input size of the feature fusion module, we reshape the shape of output image feature maps from B×512×16×16 to a new shape of B×64×2048, where B represents the batch size in training. We take this reshaped feature maps as the extracted image features.
In this task, we introduce the word embedding and LSTM to extract the question features. Given the questions of the raw data, the preprocessing of them includes two steps: tokenizing the words of questions and fixing the length of sentences to 12. After that, the tokenized sequences are sent to the word embedding module and generate the embedded word vectors in the dimension of B×12×600 (Note the tokens are encoded using GLOVE word embeddings). These embedded vectors are fed into the LSTM module to acquire high-level question features. The input layer dim and hidden layer dim are set to 600 and 1024 respectively. The number of LSTM unit in feature extraction module is set to 1. In this task, we use the whole sequence output features instead of the last token output features to guarantee the information integrity of sentence structure.
While inputting the word vectors to the LSTM, the forget gate of LSTM determines whether the information flows from previous moment can pass through to the next moment with a sigmoid function, which prompts LSTM to keep useful information and filter useless ones. The input gate of LSTM determines which information needs to be updated at current moment, and the output gate outputs updated information to next moment or as final output. With the gate units, LSTM has a strong ability in storing state information, which benefits to catch contextual correlation in long-time sequences.
After extracting the image features and question features, we feed them to the Multi-modal Factorized Bilinear Pooling (MFB) with the co-Attention mechanism together to obtain the fused features.
The MFB keeps the robust expressive capacity when compacts the features from diferent modalities by using the matrix factorization tricks. The co-Attention mechanism consists of a self-attention mechanism (SA) and a guided-attention (GA). Given two modalities features and , first makes the self-attention operation to generate attention features, denoted as . Furthermore, the is used to guide the attention learning to obtain attention features of , denoted as . This operation enhances the connection of two modalities in the learning process. By introducing the co-Attention mechanism to the MFB, the joint feature representation learning can be more accurate and efective.
In this task, we input the image features and question features (denoted as and ) extracted from the modified ResNet-34 and LSTM to the MFB with the co-Attention mechanism, as showed in Figure 4. We firstly made a self-attention operation on to obtain question attention features _, then we used the MFB to fuse the _ and to guide the image features attention learning to generate image attention features _. After that, we used the MFB to fuse the _ and _ with vector multiplication and projected the fused features to a linear dimension. At last, we employed a softmax function on the fused features to predict the probability of each answer.
This dataset of ImageCLEF VQA-Med task 2021 contains a training set of 4000 image-question pairs, a validation set of 500 pairs and a test set of 500 pairs. The triplet data of images, questions and answers are one-to-one associated. The training set of ImageCLEF VQA-Med task 2021 is totally the same as the training set of ImageCLEF VQA-Med task 2020. The organizers replaced the samples of image, question and answer in validation set and test set in ImageCLEF VQA-Med task 2021. Therefore, we merged the validation set of ImageCLEF VQA-Med task 2020 with the training set of ImageCLEF VQA-Med task 2021 in order to extend the training set to 4500 paired image-question-answer data.
By analyzing the dataset, we found that the data can be divided into two types: open-ended, i.e., the answers are “yes/no” or the questions start with “Is/Does/. . . ”, or close-ended, i.e., the questions do not have limited structures and could have multiple correct answers. Through statistics, there are 88 close-ended paired data and the rest of them are open-ended in the training set. However, there is no close-ended data in the validation set and test set. The total classes of answers are 332 in the training set and 236 (subset of training set) in the validation set. The statistical result is summarized in Table 1.
Our model was optimized with BCE loss function on RTX2080 GPU devices by training 800 epochs. We trained the network from the scratch without using any pretrained model weights. We employed Adam with 1 = 0.9 and 2 = 0.999 as the optimizer, and used dynamic learning rate to update the weights. In terms of convergence, we visualized the optimizations of the objective of our proposed model in Figure 5, from which we could observe that the training loss was decreasing, the accuracy score and bleu score were increasing. We utilized the model weight that generated the highest validation accuracy score as the final model weight. We submitted the result generated by this trained model on the test set and achieved the accuracy score of 0.222 and bleu score of 0.255. The top 10 result of this competition is shown in Table 2. (a) Train loss.
(b) Validation accuracy score.
(c) Validation bleu score.
An intuitive observation of Figure 5 (b) and (c) is that the performance of our method on validation set is much better than on test set. The reason could be summarized as follows: The sampling method of the test set was diferent from that of the training and validation set. Besides, in order to fit the input size of the image feature extraction model, the input images were resized to 128×128 roughly, which might lose some spatial information and introduce the noises. In addition, the questions of the dataset provided little entity information of the related classes.
This paper describes the model designed in the ImageCLEF VQA-Med task 2021 competition. We proposed a modified ResNet-34 + LSTM + MFB with a co-Attention mechanism to predict ifnal answers of VQA-Med. In image features extraction process, CNN was employed to learn the image bias and dropout function was introduced to suppress the over-fitting situation. For question feature extraction module, LSTM was utilized to extract the question features that did not rely on a large number of data. The two modalities features were fused by the MFB with co-Attention mechanism to generate the fused features for answer prediction. The best result of our model is 0.222 in accuracy score and 0.255 in bleu score. 1073. doi:10.1002/grl.50288. [20] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, arXiv preprint arXiv:2005.14165 (2020). [21] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, M. Rohrbach, Multimodal compact bilinear pooling for visual question answering and visual grounding, arXiv preprint arXiv:1606.01847 (2016). [22] Z. Yu, J. Yu, J. Fan, D. Tao, Multi-modal factorized bilinear pooling with co-attention learning for visual question answering, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 1821–1830.