In this paper, we describe our caption prediction and concept detection systems submitted for the ImageCLEF 2017 challenge. We submitted four runs for the caption prediction task and three runs for the concept detection task by using an attention-based image caption generation framework. The attention mechanism automatically learns to emphasize on salient parts of the medical image while generating corresponding words in the output for the caption prediction task and corresponding clinical concepts for the concept detection task. Our system was ranked rst in the caption prediction task while showed a decent performance in the concept detection task. We present the evaluation results with detailed comparison and analysis of our di erent runs.
Automatically understanding the content of an image and describing in natural
language is a challenging task which has gained a lot of attention from
computer vision and natural language processing researchers in recent years through
various challenges for visual recognition and caption generation [
leverage the clinical concept vocabulary created in the concept detection task towards generating a coherent caption for each medical image.
The recent advances in deep neural networks have been shown to work well for large scale image processing, classi cation and captioning tasks. Speci cally, the combined use of deep convolutional neural networks (CNN) with recurrent neural networks (RNN) has demonstrated superior performance for these tasks [5{11] based on the use of sequence to sequence learning and encoder-decoderbased frameworks for neural machine translation [12{14].
Motivated by the success of such prior works, we use an encoder-decoder
based deep neural network architecture for the caption prediction task [
Experimental Setup
The training data contains 164,614 biomedical images with associated clinical concepts or captions extracted from PubMed Central4. Furthermore, 10K images per task are provided as the validation set while 10K additional images are provided as the test set for both tasks. 2.2
We use an encoder-decoder-based framework that uses a CNN-based architecture
to extract the image feature representation and a RNN-based architecture with
an attention-based mechanism to translate the image feature representation to
relevant captions [
The decoder uses a long short-term memory (LSTM) network [
Our models are trained with stochastic gradient descent using Adam [
For the caption prediction task, we submitted four runs as follows:
{ Run1: This run does not consider any semantic pre-processing of the
captions; the entire training and validation set are used to train the model as
described in Section 2.2.
{ Run2: This run considers semantic pre-processing of captions using MetaMap
[
For the concept detection task, we submitted three runs as follows: { Run1: In this run, we consider the task as a sequence-to-sequence generation problem similar to caption generation, where the CUIs associated with an image are simply treated as a sequence of concepts; the entire training and validation set are used to train the model as described in Section 2.2. { Run2: This run is created by simply transforming the generated captions (for the test set) from Run1 of the caption prediction task by replacing clinical terms with the best possible CUIs from the UMLS metathesaurus. { Run3: This run is created by simply transforming the generated captions (for the test set) from Run2 of the caption prediction task by replacing clinical terms with the best possible CUIs from the UMLS metathesaurus. 2.4
The evaluation for the caption prediction task is conducted using the well-known
metric for machine translation, BLEU [
We can see that for the caption prediction task, Run4 and Run1 achieved high scores denoting the e ectiveness of our approach. Overall, our system was ranked rst in the caption prediction task. Run4 is better as it includes all possible terms from the ontologies in the generated caption but trades-o the coherence of the caption. Hence, this approach increases the BLEU scores, which essentially computes exact word overlaps between the generated caption and the ground truth caption. Run2 likely su ered from the limited training data whereas Run3 has a lower score as it accepts only the longest possible clinical term as a replacement for a CUI in the caption.
For the concept detection task, Run1 performed reasonably well, but shows that there is still room for improvement. We may consider treating the task as a multi-label classi cation problem to achieve possible improvements. Run2 and Run3 were limited due to the 2-step translation of clinical terms to CUIs from the generated captions of the other task, which potentially indicates propagation of errors in learning the captions to the downstream task. We presented the details of our participation in the caption prediction and concept detection tasks of the ImageCLEF 2017 challenge. Our system was ranked rst in the caption prediction task and showed decent performance in the concept detection task. Overall, evaluation results showed the e ectiveness of our approach. We highlighted potential reasons for errors in our submissions and discussed future work to consider for improved results.