The recent advancement and digitalization in the medical domain requires an image based question answering system to support clinical decisions. This system also helps the patients to know about their present conditions rapidly with more information. As an effort to promote the development, ImageCLEF 2020 organizes third edition of the Visual Question Answering (VQA) Task. In this task, the abnormality related questions are to be answered for the given set of radiology images. In the proposed system, VGGNet based on transfer learning approach and LSTM are used to extract the image and text feature vectors respectively in the encoder stage. Then, both feature vectors are combined and given as input to the decoder for predicting the answer. The purpose of selecting VGGNet and LSTM are: (i). VGGNet is able to extract medical image features effectively in small dataset (ii). LSTM is capable to accommodate significant information of the text. Moreover, the proposed model is evaluated for three datasets namely original dataset (4500 samples), reduced dataset (4348 samples) and augmented reduced dataset (4626 samples). The proposed model resulted in an accuracy of 0.282 and a BLEU score of 0.330 for augmented reduced dataset, which is ranked ninth among all participating group in ImageCLEF 2020 VQAMED task.
The amount of data generated and used in this era are increasing exponentially and medical domain is not an exception. Also, everyone wants to know the answer for everything they come across in the internet world. Multiple search engines are working towards satisfying their knowledge thirst, unfortunately very few image based search engines are available in the market. However, these search engines are generalized and not suitable for medical domain. 1
The medical domain is wide, and it needs prior knowledge and analysis to answer
the questions. These issues can be addressed by improving the medical image based
question answering system. To enhance this research further, ImageCLEF is
conducting VQA task in medical domain since 2018 [
The Visual Question Answering (VQA) in medical domain helps people (especially
partially sighted) in better understanding of their condition and supports clinical
decision. The challenges of VQA in medical domain includes: (i). Parameter selection and
feature extraction for medical dataset which differs from the real time and abstract
dataset (ii). Specific VQA model which works for all medical category is in developing
stage. For example in [
The remaining part of the paper spans across following subsections. In Sect. 2, literature survey related to automation in medical domain, inference from VQA task for real world dataset and its recent advancement in medical domain are discussed. Sect. 3 gives brief description about the ImageCLEF VQA-Med 2020 dataset and two other proposed datasets used for analysis and validation. In Sect. 4, the design of the proposed VQA model based on inference attained and its implementation are explained. A brief summary about the result and the respective evaluation of all five runs are given in Sect. 5 and conclusion is given at the end. 2
The recent studies shows a tremendous advancement in the medical domain. One of the
best advancement is that the medical data in structured, semi-structured and
unstructured formats are digitized. From the last decades, Artificial intelligence (AI) utilizes
the digitization advancement and enhances an automation in the medical domain. In
[
From 2018, ImageCLEF is conducting VQA task in medical domain. The
VQAMed 2018 and VQA-Med 2019 dataset contains organ, plane, modality and abnormality
related visual question answer pairs. In these tasks, most of the researcher applied
pretrained models like VGGNet, ResNet, etc., to encode medical images and Recurrent
y
c
a
r
u
c
c
A
Neural Networks (RNN) to generate question encodings. Some of the researchers
applied attention based mechanism to extract relevant image features to answer the
questions. The highest BLEU, WBSS and CBSS scores obtained in 2018 tasks are 0.162,
0.186 and 0.338 respectively [
From the overall inference, VGGNet and LSTM are selected for the implementation
of given task on VQA 2020. The advantages of selecting VGGNet [
The advantages of using LSTM [
In this section, three medical VQA dataset are discussed along with its description. The three datasets are: ImageCLEF VQA-Med 2020 dataset (Original Dataset (OD)) and two other datasets used with modification (Reduced Dataset (RD) and Augmented Reduced Dataset (ARD)). In Original dataset, (ImageCLEF VQA-Med 2020 dataset), the dataset is divided into three subsets namely training set, validation set and test set as 4000, 500 and 500 with equivalent number of question answer pairs. In addition the dataset consists of abnormality related visual questions for different organs (e.g. lung, skull, spine, gastrointestinal, musculoskeletal), planes (e.g. axial, sagittal and corona) and modalities (e.g. CT, X-ray, MRI). For better learning, the training set and validation set (as a total 4500 samples) are used for training.
The Reduced Dataset (RD) consists of 4348 samples (from training and validation set) for training and 500 samples for testing. The reduced dataset is generated by two ways namely (i). Eliminate the least contributing samples (ii). Identify and reduce the number of samples of similar class, when the count deviates much from the remaining classes. These samples degrade the overall performance of the system and hence both approaches are applied.
The Augmented Reduced Dataset (ARD) consists of 4626 samples (from training and validation set) for training and 500 samples for testing. The dataset is augmented by collecting samples from VQA-Med 2018 and 2019. The collected samples are merged with RD to generate Augmented Reduced Dataset. Augmenting the training set improves the learning rate and as a result generates better model. The OD, RD and ARD contains 330, 316, 316 classes respectively. 4
In this VQA task, the VGGNet and LSTM techniques are used to answer the medical visual questions. The system design of the proposed model is shown in Fig. 1. In this, the feature information from the medical image and its question-answer pairs are extracted and concatenated by encoder. Then, the concatenated feature vector is decoded by timestamp to generate the answer, with post-processing at the end. The proposed model consists of five modules namely, (i). Pre-processing (ii). Encoder (iii). Decoder, (iv). Post-processing and (v). Answer prediction. In the pre-processing stage, the input samples are converted to required format for effective image and text processing. As a first step, the images are reshaped to (229, 229) dimension (preferable input size of VGG16/VGG19 network). In text processing, comma is the best separator and hence the question-answer pairs are converted to comma separated file. The already existing comma within the field are converted to related special symbol (here we used semicolon). Otherwise, these commas within the field are encountered as separator, and end up with an imbalanced fields. Encoder transforms the feature vectors into the required format for the model to answer the questions. This transformation is required because the type and dimension of features extracted from image and its respective text are different. Hence a dimensionality mapping is required to bridge the gaps and then the features vectors are concatenated. To perform this, encoder has three sub-modules namely (i). Image processing (ii). Text processing (iii). Concatenation. The system architecture of the encoder is given in Fig.2 shows the each encoding stages along with the size of feature vector before and after concatenation.
In the proposed system, the image features are extracted by VGG16/VGG19. The last layer of the VGGNet is frozen and the resulted model is used for image feature extraction as transfer learning approach. The last layer is frozen because VGGNet is trained for ImageNet dataset (1000 classes) but we required the output dimension to be 1024. For this reason, after the last before layer, the dense and fully connected layers are used to adjust the dimension of the image feature vector.
Text processing, computes the dependency between the words and derives the information from the sequence of input words. The LSTM (an advanced type of RNN) is used to generate the text feature vector. The input text is tokenized into individual words and the minimum and maximum length of question and answer are computed. The LSTM computes the question embedding (using the Glove vector), timestamp by timestamp for the respective samples. This vector is given to the fully connected layer to project it to the same dimensional shape as image feature vector.
The computed feature vectors (image and text feature vector) are combined using element wise multiplication and are later used by decoder for model creation. 4.3
Both visual and textual features are merged into three dimensional vector (2048-dimensional space) which is a sequence of vectors. As both the image and textual features are represented as sequence of vectors (not as single vector), LSTM is required to feed the concatenated vector to the softmax layer. The system architecture of this sub-module, decoder is shown in Fig.3. In post-processing, the generated answer needs to be converted to the required format as in training set. In this, semicolon in the generated answers are converted back to comma format. In this stage, encoder-decoder model based on VGG-LSTM is generated. The answer for the test set can be predicted by the model. Further, the result can be analysed and evaluated using performance metrics like accuracy and BLEU score. 5
The proposed model is executed on three datasets (as discussed in Section 3) and analysed using five different combination of techniques, such as: (i). VGG16 (excluding last layer) followed by LSTM for original dataset (ii). Same as (i) for reduced dataset (iii). VGG16 (excluding last layer) followed by LSTM and post-processing at the end for Augmented Reduced Dataset (iv). Same as (iii), but the pre-trained model is VGG19 (v). Similar to first combination but post-processing is included at the end. From the results it is inferred that proposed model with post-processing is included at the end for Augmented Reduced Dataset gives better performance than the other combinations. In Table 2, OD, RD and ARD represents Original dataset, Reduced Dataset and Augmented Reduced Dataset respectively.
The performance of the model depends on appropriate parameter selection also. In this model, RMSPROP optimizer is used with a learning rate of 0.001 and the batch size, epoch and dropout are set to 256, 400 and 0.2 respectively. For training the model using these hyper parameters, each run took approximately 180 minutes in GPU. Among the five runs, third run achieved a better accuracy score of 0.282 and the BLEU score of 0.330. The final result of the leaderboard is given in Table 3 where our team achieved 9th place in the listed ranks.
In this paper, an approach for Visual Question Answering (VQA) on medical domain is implemented for ImageCLEF VQA-Med 2020 dataset and further analysed using two different types of proposed datasets namely: Reduced Dataset (RD) and Augmented Reduced Dataset (ARD). The proposed model has five stages namely: (i). Pre-processing (ii). Encoding (iii). Decoding (iv). Post-processing and (v). Answer prediction. In pre-processing, the dataset has been converted to the specific input format as required for VGGNet and LSTM. Then the image and text features are extracted and concatenated. The concatenated feature vector is decoded for next level. In post-processing, the answer is converted to the format as in the training dataset. Finally, the generated model predicts the answer for the test set. Among the five runs of the proposed model the better result is achieved for augmented reduced dataset with an accuracy score of 0.282 and BLEU score of 0.330.
In medical VQA domain, large amount of information needs to be extracted and hence it has more memory constraint. This can be addressed with the help of GPU and the selection of optimal hyper parameters. In future, the proposed VQA model can be improvised by developing a design of Convolutional Neural Network (CNN) for medical images and fixing the appropriated hyper parameters with visualization of layers. In addition, the advanced text processing approach like BERT, which represent each sentence in 768-d question feature vector can be included. 7
Our profound gratitude to SSN College of Engineering, Department of CSE, for allowing us to utilize the High Performance Computing Laboratory and GPU Server for the execution of this challenge successfully.