In this paper, we describe our contribution to the 2020 ImageCLEF Medical Domain Visual Question Answering (VQA-Med) challenge. Our submissions scored rst place on the VQA challenge leaderboard, and also the rst place on the associated Visual Question Generation (VQG) challenge leaderboard. Our VQA approach was developed using a knowledge inference methodology called Skeleton-based Sentence Mapping (SSM). Using all the questions and answers, we derived a set of classi able tasks and inferred the corresponding labels. As a result, we were able to transform the VQA task into a multi-task image classi cation problem which allowed us to focus on the image modelling aspect. We further propose a class-wise and task-wise normalization facilitating optimization of multiple tasks in a single network. This enabled us to apply a multi-scale and multi-architecture ensemble strategy for robust prediction. Lastly, we positioned the VQG task as a transfer learning problem using the VGA task trained models. The VQG task was also solved using classi cation.
Visual question answering (VQA) [
In the 2020 VQA challenge, our AIML team participated in, the dataset [
Our initial attempt at the 2020 VQA-Med challenge was to ne-tuning of the
Pythia [
For the associated Medical Domain Visual Question Generation (VQG-Med) challenge, we considered the task as a transfer learning problem, where we applied the VQA-Med data trained models as non-trainable feature extractors. The answer generation is also formed as a classi cation task. Our best submission scored 0.348 in BLEU score which won the rst place at the VQG challenge.
In the rest of the paper, we give explanations on our VQA and VQG approaches. Each approach is a self-contained section to avoid cluttering. 2 2.1
We will rst introduce the general domain VQA methods followed by an introduction to the methods that have been applied speci cally in the medical domain VQA.
General domain VQA: the goal of a VQA method is to produce an answer
from a given image-question pair. Early VQA works [
Medical domain VQA: a noticeable di erence between the medical domain and general domain VQA is the size of the dataset. The general domain VQA can accumulate a sizable dataset due to the fact that a common-sense knowledge is su cient for generating question and answers. On the other hand, the necessity of clinical expertise imposes a huge di culty in the medical domain VQA data collection.
In the 2018 VQA-Med challenge, the leading3 three participating teams [
The CBSS can result a di erent ranking.
medical article pretrained embedding [
In the 2019 VQA-Med challenge, the top three teams [
The VQA-Med 2020 dataset has a composition of 4,000 radiology images for training, 500 for validation, and 500 for testing. Each image has exactly one Question-Answer (QA) pair from the abnormality question category.
We followed the o cial suggestion to use the VQA-Med 2019 dataset4 as additional training data. The VQA-Med 2019 dataset has 3,200 medical images for training, 500 for validation, and another 500 for testing. For training and validation sets, there are 12,792 and 2,000 QA pairs, giving most images exactly one QA pair in each question category (i.e., imaging modality, imaging plane, organ systems, and abnormality). For the test set, each question category has 125 images. In addition, the yes-no questions appear only in the imaging modality and abnormality question categories. 2.3
As mentioned in Sec. 1, Pythia [
Skeleton variables Candidates this alts this, the ct alts ct, ct scan, mri, pet, x-ray, image, . . . normal alts normal, abnormal imaged alts imaged, displayed, seen, shown, . . .
is being alts is, is being
Before applying SSM, we rst removed the duplicated questions in the dataset, resulting in 266 unique questions. After then, we applied word-level edit distance (i.e., levenshtein distance) to pairs of questions, nding groups of questions with 1-distance and 2-distance. For example, in Table 1, the corresponding questions of each question backbone mostly have either 1-distance or 2-distance within the group, and the highest 4-distance is between \what is shown in the x-ray?" and 5 The naming was determined by choosing the a representative candidate from candidates for each skeleton variable; by ignoring the \alts" su x, a question backbone becomes readable. \what is seen in this ct scan?". The grouped questions were manually checked to see if the dissimilar parts can be described by a uni ed skeleton variable. If so, the generated backbone would replace the group of question and enter the next iteration of edit distance computation. The rst iteration was able to detect most of the easy question groups, leave the later iterations with a small number of questions.
The process was ran until all questions were skeletonized, resulting in 68 question backbones. We labeled the question backbones in the four aforementioned question categories, partially based on the corresponding answers. In addition, we also determined two sub categories under the imaging modality category, namely the MR modality category and the contrast imaging type category. Next, we compared our own question category annotation with the o cial question category annotation for the VQA-Med 2019 test set (only available in this set), which is equivalent. The SSM was able to populate dynamic question variations (with some rule based restrictions, e.g., changing \ct scan" in \is the ct scan normal?" to other candidates except \ct" and \image" results in a fallacious judgement of the image modality, hence is not allowed) and the same Pythia model trained with the augmented questions was able to rectify the yes-no and wh-question cross answering errors. Nevertheless, we found the SSM method rendered language modelling trivial. With its help, we can solve the VQA task as an image classi cation task.
Label inference from question backbones: based on the question category annotation, we were able to record the paired answer annotation as the label for each mapped task. In addition, we could also extract labels from the skeleton variables. For example, for the rst question \is the ct scan normal?" in Table 1, \ct" is capturable by $fct altsg and \normal" is capturable by $fnormal altsg; hence producing a coarse modality label \ct", and also produce a binary abnormality label \normal" if the answer is a \yes". We found the same can also be generalized to infer task labels from the wh-questions.
An issue with the question backbone derived modality labels is that the
detailed modality (e.g., ct with contrast or not) is unknown. To address this issue,
we treat the coarse modality labels as an independent task. The answer derived
modality labels were mapped back to the coarse labels following the information
provided in [
At the end of the process, we were able to produce six classi cation tasks: 1) ne imaging modalities; 2) coarse imaging modalities; 3) imaging plane; 4) organ systems; 5) binary abnormality, and 6) categorical abnormality. 2.4
The schematic of an exemplar image classi cation network we used is illustrated in Fig. 1, sketched with the knowledge inference process. The two important tasks are the binary and categorical abnormality classi cation tasks while the rest four can be thought as regularization tasks. We believe that all the tasks should have strong correlation to each other, i.e., the correct imaging modality and organ judgements should be strong prior knowledge for correct recognition of abnormality.
Q: “what is most alarming about this ct
scan?” A: “pancreatic carcinoma”
Knowledge Inference Matched Backbone
what is most alarming about ${this_alts} ${ct_alts}? Categorical Abnormality
Binary Abnormality Coarse Imaging Modality pancreatic carcinoma abnormal
ct
Backbone Network (e.g., ResNet, DenseNet, VGG)
Fine Imaging Modality Coarse Imaging Modality
Imaging Plane
Organ System Binary Abnormality Input Image
Categorical Abnormality Shared Feature Space
Class-wise and task-wise normalization: since only the 2019 challenge images have (almost) complete four QA pairs per image, a large number of images in the joint 2019 and 2020 dataset do not have a complete label set (mainly the 2020 images). Hence when all six tasks are jointly optimized via a mini-batch gradient method, a conventional normalization by the batch size e ectively assigns a lower weight to a less populated task, e.g., for a batch with 12 images, a task that has 3 labeled images e ectively has 0.25 weighting. In addition to the incomplete label problem, we also observed imbalanced class distributions within the tasks. For example in the categorical abnormality question category, the number of samples per abnormality class ranges from 4 to 104. We propose to solve both issues together by a class-wise and task-wise normalization in order to jointly optimize all six tasks together. Assume that t 2 fcoarse modality; : : :g represents a task, for a set of images X and the label set Yt, the mini-batch training loss L is computed as: L = X t
1 Pct 1(ct 2 Yt)
X ct
1 Pyt2Yt 1(yt = ct)
X x2X;yt2Yt 1(yt = ct) `t(x; yt)
; where x 2 X and yt 2 Yt represent individual image and label, 1(:) denotes an indicator function, and ct denotes a candidate class of t (e.g., ct 2 fct, . . . , x-rayg, if t = coarse modality ).
We adopted a multi-scale learning technique, using 128, 256, 384, and 512 as candidate image resize options. After applying the resize operation, we randomly crop the network input image at a ratio of 87.5% along both dimensions from a resized image. Random a ne transformations and horizontal ip were used. The initial learning rate is set to 1e-3, linearly reduced 1e-6 after 100 epochs using Adam optimizer.
On the other hand, ResNets [
We show the validation results from all trained models in Table 3, the corresponding training volume includes 2019-ftrain, val, testg and 2020-train. Based on these results, we made decisions of which models to be trained for test evaluation. Note that the training volume was changed to all of the 2019-ftrain, val, testg and 2020-ftrain, val, testg sets for training the testing-use models. We included the 2020-test set because some amount of partial coarse imaging modality labels (i.e., from $fct altsg) and binary abnormality labels (i.e., only the abnormal ones from wh-question abnormality) were extractable by SSM from only the questions, which served as a form of weak regularization for the test images. Finally, for the categorical abnormality type questions, we only select a top prediction from the VQA-Med 2020 subset of the abnormality classes as the predictions.
Our submissions on the 2020 validation set are shown in Table 4. Our second submission was purposed to determine the exact category type of the last question backbone in Table 1 as the ve instances all appear in the 2020 test set. Although all other 2020 questions were in the abnormality question category (aligned with the o cial statement), we found the ve questions could also be interpreted as asking which organ is present. We treated the 5 questions as categorical abnormality questions in the rst submission and as organ questions in the second submission. Given the accuracy dropped, the ground truth should be the abnormality category.
From a post-challenge point of view, our third submission secured the leading position in the leaderboard. Our fourth submission was purposed to include more DenseNet-121 instances in the ensemble as the DenseNet-121-only multiscale ensemble showed the highest 0.6 accuracy in Table 3. Our fth submission added the two VGG multi-scale groups, presenting the nal ensemble result of all trained models. Nevertheless, these nal attempts only pushed up the performance marginally, suggesting a performance saturation in our approach.
The VQG-Med challenge dataset is a much smaller dataset compared to the VQA-Med datasets. The training set contains 780 radiology images with 2,156 associated QA pairs. The validation set has 141 images with 164 QA pairs. The test set has only 80 images. The goal of the VQG challenge is to generate between 1 to 7 answers for each test image. 3.2
The VQG challenge describes a question generation task which in concept is close to image captioning but our proposed solution continued as a classi cation approach. The main reason is that we found there were more than one ground truth questions tied to each image. Unlike a VQA task, a question can be considered as a prior knowledge on which the corresponding answer is conditionally dependent. Generating multiple questions while lacking such prior knowledge could be resolved by sampling approaches, but it can be di cult to associate a random state to a speci c ground truth question. Hence, we instead treated all observed questions for an image as its attributes and modelled the question generation task as again an image attributes classi cation task. A downside of the classi cation approach is not able to produce novel questions.
Our VQG approach was built upon our VQA-Med solution with the following settings.
{ Solving the question generation task as a classi cation task leads to a total
of 2,073 classes each as an unique observed question from the joint training
and validation sets.
{ We were concerned about netuning the entire image model by the
limited amount of data and the large number of class, which may end up
over- tting in a much faster rate, hence we did not choose to ne-tune the
backbones. However, as a compensation of non-linear capacity, we added a
2-layer batch-normalized and fully-connected (FC) (512 units each, ReLU
activation) multiple-level perceptron (MLP) model before the softmax layer.
The MLP model also avoided a direct mapping from the image features
(e.g., 2048 dimensional features) to the 2,073 classes which would result in a
computational expensive matrix multiplication and a large memory usage.
{ At the training hyper-parameter level, we kept the initial learning rate as
1e-3 but adjusted the nal learning rate to 1e-5. Finally, we shortened the
number of epochs to 40.
{ Each training image could be associated with more than one question,
resulting a multi-label problem. We used the Stochastic Ground Truth method
in [
These settings helped us to reuse most of the VQA-Med code base and models to develop a tangible solution within a very short time frame. Similar to the VQA-Med 2020 result presentation, we show the VQG-Med 2020 validation and test results separately in Table 5 and 6, respectively. While the o cial evaluation only has BLEU score, in our local evaluation, we used top-7 accuracy to evaluate the validation performance. For o cial testing, each of our submission generates seven questions according to the highest probabilities for each image.
The rst two submissions tested whether the large input size models should be continued. Given the lower top-7 accuracy on the validation set and the same BLEU value on the test set, we decided to not continue the 512 input size training. In the third submission, we tried to utilize the ground truth answer annotations by introducing the answer classi cation as an additional regularization task, but the result dropped by 0.009. In addition, the results from the rst three submissions suggested a low correlation between the validation top-7 accuracy and the test BLEU scores. Hence in our forth submission we made two decisions in order to push for a much larger margin on the local evaluation: 1) forgoing the low accuracy models from the ensemble (validation accuracy < 0.079); 2) including the DenseNet-121 architecture given its good performance in the VQA-Med challenge. The fourth submission scored 0.11 for the validation accuracy and 0.348 for the test BLEU score, secured our leading position in the VQG-Med challenge. Finally, in the fth submission, we further added the DenseNet-161 multi-scale models as a last-minute attempt. Given the local evaluation dropped by 0.012, the test performance drop was expected as well. 4
In this paper, we described our participation at the 2020 VQA-Med challenge and the associated VQG-Med challenge. The center of our approach is a knowledge inference method which we named Skeleton-based Sentence Mapping (SSM). In the VQA-Med challenge, the SSM method was useful on multiple fronts: 1) it mapped questions to a set of backbones which were useful to populate dynamic question instances; 2) it replaced the need of the language modelling and was able to provide the direct selection to the corresponding answer predictor; and 3) it was used to infer six image classi cation tasks and corresponding training labels. Bypassing the development of language modelling allowed us to focus on tweaking the image classi cation model so that we devoted more time and resource on the multi-scale and multi-architecture ensemble learning. At last, we developed a class-wise and task-wise normalization technique for balancing the class and task populations, allowing the tasks with incomplete labels to be jointly optimized in one network.
The main inspiration of SSM came from [
We are aware of the fact that SSM is not fully automated which requires further development. In addition, we understand SSM is a form of explicit reasoning model and its e ciency highly depends on the question regularity and dataset size which may not generalize well for VQA datasets containing free-form questions.