This paper describes the details of our participation in the image caption prediction task at CLEF 2018. We explored and implemented an encoder-decoder framework to generate caption given a medical image. For the encoder, we compared two variations of convolutional neural networks (CNN) architectures: ResNet-152 and VGG-19, and for the decoder we used the long short term memory (LSTM) recurrent neural network. The attention mechanism was also experimented on a smaller sample to evaluate its impact on the model tting and prediction performance. We submitted 4 valid runs and the best result achieved 0.18 BLEU score, which ranked second among all participant teams.
Automatically making computer understand the content of an image and o ering reasonable description in natural language has gained more and more attentions from the computer vision and natural language processing community. In clinical practice, medical specialist and medical researchers usually write diagnosis reports to record microscopic ndings from images, so automatic captioning on medical images will bene t healthcare providers with valuable insights and reduce their burden across the overall clinical work ow.
Due to its importance, ImageCLEF 2018 [
In participating the CLEF caption prediction task, we adopted the proven successful Encoder-Decoder architecture of neural networks, where we investigated two state-of-the-art image encoder variants (VGG-19 and ResNet-152) integrated with LSTM recurrent neural networks. Most existing studies use image encoders which were pre-trained on ImageNet, a very large dataset containing daily images from general domain. However, this pre-trained model is not available in medical domain. Therefore, we performed netuning during end-to-end training of our model. 2
The whole training dataset of Image CLEF caption prediction contains 222,314 image-caption pairs, and the length of captions varies from 1 word to 816 words. The average caption length is 20 words. Images are mainly from medical domain, but some of them are generic pictures such as animals or geography pictures. Caption itself is also noisy, e.g. some copy right or author information is also included. The o cial test data contains 9,938 images which are used to evaluate the system's performance. 3
In the encoder-decoder framework, the encoder extracts e ective image features, based on which the decoder generates a sequence of textual terms for summarizing the image's content. As image understanding is crucial for caption prediction, we built two systems: VGG Attention Model using VGG-19 architecture as encoders integrated with the attention mechanism, and ResNet Model using residual networks as encoders. Those two encoders have shown promising results on image classi cation and are expected to work well for caption prediction. Both systems depend on LSTM model for decoding. 3.1
To improve generality of model, we used random crop method to augment more data into original dataset. As we found that only 0.28%(628 instances) of the training data contain captions with length larger than 145, we removed these data to reduce the sequence length required in LSTM model which can speed up the training process without losing much data representativeness. 3.2
We adopted the architecture from the "show,attend, and tell" [
Encoder: VGG-19 Attention mechanism can guide the LSTM decoder to dynamically focus on speci c parts of the image when generating the next caption term at each time step. Therefore we choose the output from the relatively lower convolution layer of the VGG net as image feature representation, which enables the model to attend to salient regions of the image. Speci cally, we extracted features from the conv5 3 layer, which provides the 14 x 14 x 512 feature map (512 is the number of feature maps). The attened 196 x 512 encoding will be fed in the subsequent LSTM decoder.
Decoder: Attention Based LSTM As shown in Fig. 1, the LSTM encoder receives the current word embedding and a dynamic image representation at each time step through the attention module. The attention module calculates di erent weights for feature map from di erent image locations, so that the LSTM encoder will know which location should be focused more for producing the next word. The weights are based on the hidden state of the LSTM at previous time step and the feature map extracted from the aforementioned VGG encoder(Fig. 1). 3.3
Another model we explored is to use ResNet(residual neural networks) architecture in place of VGG network as image encoders. The motivation to swap VGG with ResNet is that the latter achieved state-of-the-art results in ImageNet classi cation tasks in general domain. It indicates that ResNet has the potential to produce a better encoding on image features. By adding shortcut of each block, residual network architecture also accounts for the vanishing gradients problem when training very deep convolution neural networks.
As shown in Fig 2, we adopted ResNet-152 encoder which directly connects
to a standard LSTM decoder. Due to the depth of the architecture, we didn't
add attention in this model as it is tricky to pick up an appropriate layer to
attend to, which we will investigate in the future work. For implementation,
we used pre-trained models' parameters to initialize ResNet-152 and performed
end-to-end netuning. Note that for both systems, the LSTM decoder doesn't
use any pre-trained model.
One of the challenges for this caption prediction task is that the caption length
varies largely, and the number of words per caption ranges from 1 to 145 even
with our preprocessing step. In last year's challenge, ISIA [
We use the BLEU score as a metric to measure whether the model can over t the small data properly. All the experiments are using Adam gradient descent method, and the learning rate is set as 0.0001. As we can see from Table 1, the capacity of the model to handle long length captions is more sensitive to hidden size than word embedding size, i.e. when the hidden size is not large enough(256), increasing the word embedding size doesn't help much(run1 vs. run2); when the hidden size is increased to 512, the model can t well with embedding size of 256 or 512(run3 or run4). As expected, when embedding size is further reduced to 128, the capacity is adversely a ected(run4 vs. run5). Based on this result, we choose the hidden size of 512 and the word embedding size of 256 in our systems. We also did this experiment on the attention LSTM model with VGG encoder and similar results were observed. We used Adam stochastic gradient descent method for model training, with learning rate set as 0.0001, the maximum epoch set as 20, and the batch size set as 16. All our models were trained on two Tesla M60 GPUs in an Amazon EC2 server. However, attention based model training process is extremely slow, so we have no time to get a usable model before the challenge deadline. Therefore, our submitted runs were based on ResNet model.
We delivered totally 6 runs, and 4 of them are valid, so we only describe those 4 runs. As mentioned before, those 4 runs are all based ResNet encoder with standard LSTM decoder with di erent epochs of training. Table 2 shows our o cial test results.
{ best results among teams: the best results among all participants for caption prediction. { run1 epoch 6: the run with 6 epochs of training { run2 epoch 13: the run with 13 epochs of training { run3 epoch 13 remove UNK: unknown words (UNK) were removed from run2 { run4 epoch 13 remove UNK netune: same setting with run3, except for allowing whole ResNet architecture to be netuned in the training (other runs only netuned the top 2 layers) We can see that run1 delivered an under- tted model obtaining the worst performance. With the same training epochs, no di erences were found in terms of removing UNK or not, which may be due to the pre-processing step embedded in the evaluation script. Run4 achieved the best result of 0.18 demonstrating fully netuning is bene cial. CLEF image caption prediction dataset is very di erent with COCO or Flickr 10k dataset. Some captions of this dataset are extraordinary long. Fig. 3 and Fig. 4 show two examples, one is from CLEF image caption prediction dataset, which has ve sentences, and the other one is from COCO dataset, which only has one sentence.
The caption in Fig. 3 is very complicated, covering di erent semantics: a summary at the beginning followed by other descriptions as well as reasoning/inferring which can't be seen in the picture at all. In contrast, the caption from COCO dataset(Fig. 4), it only describes what can be seen from the picture. Thus, compared with the general domain VQA task, CLEF caption prediction is Fig. 3. Example: image of CLEF training dataset, caption: (1)Summary of key interactions in 'atypical' WD-repeats 6 and 7 of RACK1, illustrated with the structure of yeast Asc1p (PDB: 3FRX). (2)RACK1 proteins are characterised by a signi cant sequence extension between blades 6 and 7, leading to a knob-like protrusion from the upper face of the propeller. (3)The P0 Asp in blade 7 forms a salt bridge to an Arg at the P+4 position and this is packed against a Tyr (or in some orthologues Phe) in the P-2 position. (4) The P0 Asp is absent in blade 6. (5)In principle, these features together with the absence of the GH signature dipeptide on the loops between blades 5 and 6 and between blades 6 and 7 may facilitate structural transitions in this region of the protein that are important for function. more challenging, not to mention the additional added complexity due to di cult medical terms. To address this, the model needs to be more intelligent with adequate reasoning ability, which may require more complex and hierarchical text modeling structure with support of background knowledge.