This paper describes the contribution of the Department of Software Engineering at the Azrieli College of Engineering, Jerusalem, Israel to the ImageCLEF VQA-Med 2019 task. This task was inspired by the recent ever greater success of visual question answering (VQA) in the general domain. Given medical images accompanied by clinically relevant questions, participating systems were tasked with answering questions based on the image content.We explored and implemented a two-stage model. The rst stage predicts the category of the textual question, while the second stage is comprised of 5 sub-models. Each sub-model is a classic VQA deep learning module with two branches for feature extraction, the rst using CNN to extract image features, and the second using embedding (and optionally LSTM) to extract textual features. The network then combines the two feature branches to predict the appropriate answer. We found that most sub-models didnt need LSTM to achieve high scores on the validation and test data-sets. We submitted 10 models for the challenge, our best submission overall ranked 9th out of 17. All source codes are available at https://github.com/turner11/VQA-Med Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2019, 9-12 September 2019, Lugano, Switzerland.
The ever increasing demand for automated computer systems (AI) to assist clinical medical practice addresses two main audiences: Doctors who use these systems to get a second opinion on their diagnosis; and patients who increasingly have easy access to comprehensive and detailed medical data which they nd bewildering. Thus, addressing patients, the systems motivation is to help them have a better understanding of their medical condition, by providing detailed explanations of the results of their medical tests and scans, which is something that doctors, naturally, are unable to do for each data item of each patients le. The current access to ones detailed medical le without explanation leads to the unfortunate situation that patients turn to searching the Internet and online forums to better understand their condition, reaching misleading information and false conclusions. Consequently, this often worries patients, either because insu ciently speci c details of their health are considered, or even worse, because irrelevant, false, inexpert information is found.
Visual question answering (VQA) [
A VQA [
The VQA COCO-QA Challenge is studying a problem very similar to the
VQAMed task. VQA [
In this study, we took a di erent approach: our objective was to use classic
VQA methods [
The evaluation of the participant systems of the VQA-Med 2019 task was
conducted based on two metrics: BLEU, and Accuracy (Strict). Accuracy (Strict)
is an adapted version of the accuracy metric from the general domain VQA task
that considers exact matching of a participant provided answer and the ground
truth answer. BLEU [
The input to our method is an image and a questions referring to it. The output
is an answer for the question in the context of the given image. Fig 2. The
system is comprised of two stages: The rst predicts the category of the textual
question Fig 3, while the second is a classic VQA module which combines the
question and image to predict a relevant answer (see Fig 4 below). The rst stage
classi es the question into 5 question categories: Modality, Plane, Organ System
and 2 Abnormality categories: Note, we subdivide the tasks given Abnormality
class into two categories: Questions with a Yes or No answer, and All Other
Questions. This stage uses embedding and an MLPClassi er, and it optimizes
the log-loss function using LBFGS or stochastic gradient descent. We used the
sklearn package [
The second stage is a classic VQA deep learning module, which takes the
question and image as a combined input and predicts the appropriate answer.
The text undergoes preprocessing and embedding the output of this branch
is treated as features. We investigated whether LSTM was needed at the next
stage or not. Image features are extracted using a CNN network (VGG 19) [
We will review and analyze the submissions in order of the scores they achieved on the test set, looking particularly at the following features of the submitted models: 1) Optimization Function, 2) Activation Function, 3) Loss Function, 4) batch size, 5) size of fully-connected layers, 6) number of units in the LSTM layer, and 7) whether Class Weights were used.
Since this paper is focused on nding whether LSTM is needed for VQA
tasks, and we wanted the e ect and contribution of LSTM to be highlighted,
we chose to work with a very simple convolution network, and used the VGG
network. when evaluating optimization functions we found that RMSprop
produced the best results across all the submitted models. Results clearly indicated
using Softmax for Optimization alongside Categorical Crossentropy as the Loss
Function was the best option, which was expected as they are the most natural
choice for a task like this [
Lets turn to the last three parameters: Size of fully-connected layers Number of units in the LSTM layer
Whether Class Weights were used 5.1
Submissions Details We will review our ten submissions in order of their Challenge test set results. Note that each submission is comprised of ve sub-models { one per question category.
In the tables presented per model, each row represents a sub-model (for a question category), the columns are: { Column 1 { question category that sub-model was trained for { Column 2-3 { sub-models validation set scores (strict-accuracy and BLEU) { Column 4 { size of fully-connected layers { Column 5 { number of units in the LSTM layer (LM in short) { Column 6 { Loss Function used { Column 7 { Activation Function used (act, in short) { Column 8 { batch size { Column 9 { number of epochs (epo, in short) { Column 10 { whether Class Weights were used Best Performing Submission The submission with the highest test set score had the following characteristics: Fully-connected layer size of 14 for all submodels This is the highest number used among our submissions, and our ndings indicate that a higher number of fully-connected layers was more successful in generalizing from the validation set. An LSTM layer was used only in the submodel handling the Abnormality { Other question category. In the training and validation datasets Yes and No answer frequencies were not balanced for the Abnormality Yes/No category. We therefore investigated whether class weights would improve accuracy and found that they did. See test set results in Table 4 and validation results in Table 1 2nd and 3rd best performing submissions The di erences between these models and the best submission were not great, nor were the di erences in the scores they both achieved 0.53 strict and 0.55 BLEU. Compared to the best Submissions 4,5,6,8,9 These submissions either did not use Class Weights at all or did not use them exclusively for the Abnormality { Yes/No category, and the size of fully-connected layers was smaller, emphasizing the importance of these elements to the network. See test set results in Table 4, Submissions performing 7th and 10th These submissions did not include LSTM, their low scores proving the importance of this layer for handling complex tasks such as the Abnormality { Other question category. See test set results in Table 4, 6
This paper presents research done in the context of participation in the VQAMed Challenge. We analyzed VQA classi ers and feature extraction methods for image and text classi cation in the context of medical images in the VQA-Med 2019 task. We found that none of the sub-models needed LSTM, except the one handling the Abnormality Other questions category, the most complex task, which also required fully-connected layers of size 21, unlike all the other categories, for which fully-connected layers of size 14 were su cient. Class weights are needed only in cases were a signi cant imbalance between answer class frequency exists as there was in this challenge in the Abnormality Yes/No question category. We submitted 10 models, our best submission ranking 9th out of 17. All source codes are available at https://github.com/turner11/VQAMED, 7
This paper focused on the question of whether and when LSTM may be useful for VQA tasks. We therefore chose to work with a very simple convolution network, the VGG network. Further research on the e ect and contribution of the LSTM module is needed in order to look at a broader range of convolution networks, including more advanced versions, such as ResNet and Inception, and their e ect on results. We intend to investigate the e ects of using larger size fully-connected layers and more epochs. Looking at batch size, found that our best performing submissions had a batch size of 32, with 75 for the Abnormality Yes/No submodel, while all lower batch sizes produced less accurate results. The batch size was limited by computing resources, and we intend to examine larger batch sizes with stronger processors.