Intelligent learning and understanding of image and text information are important research directions for the successful application of deep learning in computer vision (CV) and natural language processing (NLP). This paper takes medical images and questions as the research objects, by extracting the feature information contained in the medical images and questions and combining with the attention mechanism makes the computer system to more accurately obtain the information expressed by the images. Then, the model predicts the answers to the questions about the images. This paper proposes a novel model for the ImageCLEF VQA-Med 2020 task [1]. In this model, we use the improved pre-trained VGG16 to extract image features, and GRU module to extract text features of the questions. Then the structure of Seq2seq, including encoding and decoding parts, is applied to obtain the predicted answers. Our team gets the seventh rank in the ImageCLEF VQA-Med 2020 challenge, and our model achieves accuracy score and BLEU score of 0.376 and 0.412 respectively, in the competition.
With the rapid development of CV and NLP, visual question answering
(VQA) has become one of the increasingly popular research areas in deep
learning [
The next chapter briefly describes the relevant work and summarizes the methods used in this model. Chapter 3 describes our proposed method and dataset. Chapter 4 introduces our model in detail. Chapter 5 describes the experimental results and model evaluation results, and Chapter 6 is the summary of this paper. 2
After reading a lot of literature, we found that the implementation of VQA
technology is generally based on deep learning, and deep neural networks are
also the most effective method to achieve VQA tasks. VQA tasks are roughly
divided into two aspects. First of all, CNN is generally used to extract image
features, such as VGGNet, Resnet, Inception, Googlenet, and so on. Pre-trained
deep learning networks by ImageNet [
The dataset in this paper is from the ImageCLEF VQA-Med 2020 task, which is divided into three parts, training set, validation set and test set as shown in Table 1. Compared to last year’s data set, the pattern for this year’s data set is one image for multiple questions instead of one image for one question last year, and this year’s data set is not divided into four categories last year’s data set, but the type of this year’s questions is closer to last year’s abnormality class questions. It is also the hardest class of questions to deal with, because the answers to the corresponding questions are not very regular.
Images Questions Answers
In continuation of the two previous editions, this year’s task on VQA-Med
consists in answering natural language questions from the visual content of
associated radiology images, it focuses particularly on questions about abnormalities
[
There are two examples of medical images and associated questions and answers from the training set of ImageCLEF VQA-Med 2020, as shown in Figure 1:
This paper proposes an Encoder-Decoder model. The answers from the training set are extracted to form a candidate answer set. There are a total of 333 candidate answers. All we have to do is to let the model assign a predicted probability value to each answer word in this candidate answer set. The output module consists of GRU network that takes the thought vector which includes question and image features as initial state. < SOS > token is taken as input in the first time step, then the GRU network tries to predict the answer using softmax layer. This method can be expressed as a mathematical formula: y = argmax P (ajq; i; m); (1) where y is the candidate answer word option with the highest probability predicted by the model, q is the answer to the question, i is the image corresponding to the question, m provides all parameters of the model. 4.2
The model we propose in this paper uses the sequence to sequence method. The general structure of this method is composed of an encoding module and a decoding module, as shown in the Figure 2:
The encoder is responsible for compressing the input sequence into a vector of a specified length. This vector can be regarded as the semantics of the sequence. This process is called encoding. As shown in the Figure 2, the simplest way to obtain the semantic vector is to directly use the hidden state of the last input as semantic vector C. It can perform a transformation on the last hidden state to obtain a semantic vector, and also perform a transformation on all the hidden states of the input sequence to obtain a semantic vector. The calculation formula is:
C = q(h1; h2; h3; htx) = htx; (2) where hi represents the output of each hidden layer, C is the state of the last input htx.
The decoder is responsible for generating the specific sequence based on the semantic vector. This process is called decoding. As shown in the Figure 2, the simplest way is to input the semantic variables obtained by the encoder as the initial state into the decoder’s RNN to obtain the output sequence. It can be seen that the output of the previous moment will be used as the input of the current moment, and the semantic vector C only participates in the operation as the initial state, and the subsequent operations are independent of the semantic vector C. The calculation formula is: yi = g(yi 1; h′i; C); (3) where yi 1 is the output of the previous step, h′i is the output of the hidden layer, and g represents the nonlinear activation function.
The following symbols are represented as inputs at the decoding stage: < P AD >: Complete characters. < EOS >: End-of-sentence identifier on the decoder side. < U N K >: Low-frequency words or some words have not encountered so on. < SOS >: The start identifier of the sentence on the decoder side. 4.3
Encoder In terms of image feature extraction in the encoding module, we use
an improved VGG16 model [
Extracting image features refers to inputting an image preprocessed in form
of pixels into a feature vector with high-level semantic information.
Convolutional neural networks as feature extractors are all standard models proposed in
ImageNet image recognition tasks, and CNN models can be used to indirectly
used a large amount of training data on ImageNet to perform better feature
extraction on images. This paper uses the pre-training VGG-16 model as the
visual feature extractor of the images. Since the last two layers have entered
the classification step, we need the complete output image features, so the last
two layers are removed, and the 4096-dimensional features are extracted from
the fully connected layer, and then the output feature vector passes through
an attention module [
In terms of text feature extraction, we input the question text into RNN after
Glove Embedding [
ht 1; xt]) ht = (1 zt) ht 1 + zt het where zt is the update gate, which is the logic gate when updating activation, rt is the reset gate, whether to give up the previous activation ht when deciding on candidate activation, het is candidate activation, receive [xt,ht 1], ht is activate gate, which is the hidden layer of GRU, receive [ht 1,het].
The following Figure 5 is the model structure of encoding part we used. (4) Decoder The semantic variables obtained by the encoder are input into the GRU of the decoder as the initial state to obtain the output sequence. The output of the previous moment will be used as the input of the current moment, and finally the predicted answer will be output.
The following is the model structure of decoding part we used.
As shown in Figure 6, in this stacked GRU network, the red curve represents the hidden state information of the previous moment as the input of the next moment, the model input < SOS > represents the start of decoding, the model prediction output < EOS > represents the end of prediction. This is the decoder part of our model. 5
ImageCLEF VQA-Med 2020 competition implements two evaluation
methods, Accuracy (Strict) and BLEU [
The implementation of the BLEU method is to calculate the N-grams
model of the candidate sentence and the reference sentence [
As shown in the Table 3, the Xception+GRU model was proposed in last year’s competition by our team. The traditional CNN and RNN models were used in image processing and text processing respectively, which did not perform very well in this year’s dataset. This year, we mainly introduced the encoding and decoding structure of seq2seq and made ablation experiments based on last year’s model. We can see that the model with seq2seq construct achieves better accuracy. The VGG16+GRU+seq2seq model proposed in this paper is improved based on the traditional CNN model to reduce the number of parameters and improve the accuracy. The hyperparameters of this model are set as follows: we set the learning rate to 0.0001 in ADAM optimizer, with dropout = 0.5, epoch = 80 and batchsize = 64. The following is a comparison of the results of all the experiments we performed. It can be seen that the VGG16-seq2seq model is the best in this paper.
This paper describes the model we use in the ImageCLEF VQA-Med 2020 competition. We use the seq2seq framework to input feature and predict answers. The image feature extraction part uses the improved VGG16 model. The text feature extraction uses the GRU model, and finally achieves the accuracy score of 0.376, and BLEU score of 0.412. We will improve the model and combine the attention mechanism in the Seq2seq structure to continuously improve the accuracy. Besides, our future work includes: (1) Image and natural language are signals of two modalities. How to fully integrate these two modalities belongs to the task of multi-modality fusion, which requires us to design a type that can fully learn the relationship between different modalities. (2) If the image visual features and the text features of the questions are directly fused, there will be a semantic level mismatch, so we will design a model to handle this question and improve the accuracy of VQA system.
This work was supported by the Natural Science Foundations of China under Grants 61463050, the NSF of Yunnan Province under Grant 2015FB113.