This paper describes the submission of the Yunnan University team in the Visual Question Answering task of the ImageCLEF 2021 VQA medical image challenge. According to the analysis of the dataset, we regard this task as a classification task. Firstly, we use the pre-trained VGG16 model, Global Average Pooling (GAP), and image enhancement technology to process and extract the image features. Secondly, we use BioBERT, which is pre-trained with biomedical text, to extract all the semantic features. BioBERT and BERT have the same model structure, but BioBERT have better performance in extracting medical text features. Thirdly, the semantic features and image features are fused by Multi-modal Factorized High-order (MFH) Pooling. Finally, the fused features are input into a fully connected layer for classification. Our method achieved an accuracy score of 0.362 and a BLEU score of 0.402 and ranked 2nd among all the participating teams in the VQA-Med task at ImageCLEF 2021. Our code is publicly available1.
The visual question answering (VQA) task aims to answer questions according to the content of the corresponding image. It involves data processing technology in the field of computer vision(CV) and natural language processing(NLP). The dataset of the VQA task is composed of medical images and related question-answer pairs. The main task of the VQA system is to input images and questions into the system and predict an answer according to the questions.
For the general domain VQA, there are a large number of datasets, many advanced models, and technologies to solve this task. With the increasing interest in the application of artificial intelligence technology in the medical field, VQA has attracted people's attention in medical field, because it can support doctors' clinical decisions and enhance patients' understanding of their symptoms from medical images, especially in patient-centered medical care.
Compared with VQA for the general domain, VQA for the medical domain is a more challenging task. Firstly, due to the high cost of collecting valid data, the available medical data for training is limited. For the general domain VQA, we can easily obtain thousands of images with guaranteed quality. Secondly, the words used in question and answer matching or medical report are quite different from the language used in daily life, and they are more professional.
In the following, we first describe the work related to the VQA Med task in Section 2. Then the dataset provided by ImageCLEF 2021 is described in Section 3. In Section 4, we describe the details of our proposed method, and then we describe the experiments in Section 5. We finally conclude this paper in Section 6. 2
For the medical field VQA, this task is more challenging because it requires specialized
medical datasets and expert doctors to understand the data. The VQA-Med competition
started in 2018, and since then, it has provided a medical dataset for VQA tasks every year. In
2019, the Zhejiang University team [
In the third edition of the VQA-Med challenge in 2020, the AIML team [
ImageCLEF 2019 dataset [
In this section, we introduce the task model we submitted to the ImageCLEF VQA-Med 2021 competition. Our model consists of four parts: image feature extractor, text feature extractor, Multi-modal Factorized High-order (MFH) Pooling feature fusion with the Co-attention mechanism, and classification model. In this paper, we regard the ImageCLEF VQA-Med 2021 task as a classification task with C categories. C is the result of removing all the repeated answers in the task. Fig.2. shows the structure of our model. what is abnormal in the ct scan?
Image Feature extraction Question Feature extraction
Feature Fuse
with Co-attention
Answer
Classify
Fig.2. Our model architecture
unicornuate uterus
hiatal hernia
In our model, we use the pre-trained VGG16 network on the Imagenet dataset to extract
features, and the GAP strategy is applied to the VGG16 network to prevent overfitting. GAP
averages the output feature of the convolution layer with different channel numbers. The
shape of the input model is 224 * 224 * 3, and the shape of the VGG16 network output is 224
* 224 * 64,112 * 112 * 128, 56 * 56 * 258, 28 * 28 * 512, 14 * 14 * 512, 7 * 7 * 512. After
averaging the output according to the channel, we get five vectors, 1 * 1 * 64, 1 * 1 * 128, 1 *
1 * 258, 1 * 1 * 512, 1 * 1 * 512, and finally we concatenate these five vectors to get a 1 * 1 *
1472 dimensional vector, and input it to the next network.
BioBERT [
Feature fusion
Multi-modal feature fusion is one of the most important technologies to improve the
performance of the VQA model. For multi-modal feature fusion, most existing methods use a
simple linear model to combine image visual features with text semantic features. This paper
uses the multi-modal factorized high order pooling (MFH) [
Answer prediction According to the analysis of the dataset of this competition, we regard this task as a classification task. In this part, the received fusion features of 2000 dimensions are first input into a dropout layer with P = 0.3, and then connected to a fully connected layer for final classification prediction.
Our model trained 350 epochs on GTX2080Ti for about 7 hours. In this part, we will introduce the training process and parameters in detail. 5.1 Train data extension In addition to the dataset provided by ImageCLEF 2021 VQA-Med task, we also add the abnormal subset of the ImageCLEF 2019 VQA-Med task training set, which contains 473 images and question-answer pairs as the training set of this task. In the ImageCLEF 2019 VQA-Med training set data, only the problem that exists in the ImageCLEF 2021 VQA-Med task test set will be added to this training task as training data.
In order to achieve the best performance of the model in the validation set, the parameters are adjusted several times. Finally, we use binary cross-entropy loss function, Adamax optimizer, dropout with P = 0.1, and initial learning rate of 1e-3. Secondly, the default super parameter setting of MFH (with Co-attention) is used in the multi-modal feature fusion part.
In VQA-Med2021, accuracy and BLEU are used as the evaluation criteria. The accuracy represents the correct sample in all samples, and the BLEU score measures the similarity between the real answer and the predicted answer. The maximum accuracy we achieved in the validation set is 66.8%. Fig.3 shows the change curve of train accuracy and valid accuracy in the training process. In a total of ten valid submissions, the VGG16 (with GAP) + BioBERT + MFH pooling (with Co-attention) + linear classification layer model proposed in this paper finally achieves an accuracy score of 0.362 and a BLEU score of 0.402. The entries of this paper won second place in this competition. The results of the competition have been shown in Table 1. Our team ID is Zhao_Ling_Ling.
Accuracy 0.382 0.362
BLEU 0.416 0.402
TeamS jeanbenoit_delbrouck In this paper, we describe the model we submitted to the ImageCLEF 2021 VQA-Med challenge. We use BioBERT to extract text features. BioBERT has a better performance than BERT in biomedical text extraction. In addition, the application of Multi-modal Factorized High-order (MFH) Pooling with the Co-attention mechanism also makes the model get better performance in this task. For future work, we will continue to improve the current network, introduce some more advanced methods and apply them to other data sets and tasks. Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018). [12] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016. [13] Staudemeyer, Ralf C., and Eric Rothstein Morris. "Understanding LSTM--a tutorial into Long Short-Term Memory Recurrent Neural Networks." arXiv preprint arXiv:1909.09586 (2019). [14] Cho, Kyunghyun, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. "Learning phrase representations using RNN encoder-decoder for statistical machine translation." arXiv preprint arXiv:1406.1078 (2014). [15] Vu, Minh, Raphael Sznitman, Tufve Nyholm, and Tommy Löfstedt. "Ensemble of streamlined bilinear visual question answering models for the imageclef 2019 challenge in the medical domain." In CLEF 2019-Conference and Labs of the Evaluation Forum, Lugano, Switzerland, Sept 9-12, 2019, vol. 2380. 2019. [16] A. Ben Abacha, M. Sarrouti, D. Demner-Fushman, S. A. Hasan, H. Müller, Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain, in: CLEF 2021 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org,Bucharest, Romania, 2021. [17] B. Ionescu, H. Müller, R. Peteri, A. Ben Abacha, M. Sarrouti, D. Demner-Fushman, S.
A.Hasan, V. Kovalev, S. Kozlovski, V. Liauchuk, Y. Dicente, O. Pelka, A. G. S. de Herrera,J. Jacutprakart, C. M. Friedrich, R. Berari, A. Tauteanu, D. Fichou, P. Brie, M. Dogariu, L. D.Ştefan, M. G. Constantin, J. Chamberlain, A. Campello, A. Clark, T. A. Oliver, H. Moustahfid,A. Popescu, J. Deshayes-Chossart, Overview of the ImageCLEF 2021: Multimedia retrieval in medical, nature, internet and social media applications, in: Experimental IR MeetsMultilinguality, Multimodality, and Interaction, Proceedings of the 12th International Conference of the CLEF Association (CLEF 2021), LNCS Lecture Notes in ComputerScience, Springer, Bucharest, Romania, 2021.