Manual image annotation is a major bottleneck in the processing of medical images and the accuracy of these reports varies depending on the clinician's expertise. Automating some or all of the processes would have enormous impact in terms of e ciency, cost and accuracy. Previous approaches to automatically generating captions from images have relied on hand-crafted pipelines of feature extraction and techniques such as templating and nearest neighbour sentence retrieval to assemble likely sentences. Recent deep learning-based approaches to general image captioning use fully di erentiable models to learn how to generate captions directly from images. In this paper, we address the challenge of end-to-end medical image captioning by pairing an imageencoding convolutional neural network (CNN) with a language-generating recurrent neural network (RNN). Our method is an adaptation of the NICv2 model that has shown state-of-the-art results in general image captioning. Using only data provided in the training dataset, we were able to attain a BLEU score of 0.0982 on the ImageCLEF 2017 Caption Prediction Challenge and an average F1 score of 0.0958 on the Concept Detection Challenge.
Generating a textual summary of the insights gleaned from a medical image is
a routine, yet nonetheless time-consuming task requiring much human e ort on
the part of highly trained clinicians. Prior e orts to automate this task relied
on hand-crafted pipelines, employing manually designed feature extraction and
techniques such as templating and sentence retrieval to assemble likely sentences
[
In this paper we describe the submission of the University of Sydney's
Biomedical Engineering & Technology (BMET) group to the caption prediction and
concept detection task of the ImageCLEF 2017 caption challenge [
Image captioning, whereby the contents of an image are automatically described
in natural language, is challenging task in machine learning, requiring methods
from both image and natural language processing. Many early approaches to
this problem involved complex systems comprising of visual feature extractors
and rule based methods for sentence generation. Li et al. [
Recently, deep learning-based encoder-decoder frameworks for machine
translation [16] have been adapted and applied to problem of image captioning. By
replacing the language-encoding Long Short-Term Memory (LSTM) [
There has been limited work in adapting such methods to the medical
domain, despite the large volume of image and text data found in PACS. Schlegl
et al. [
To the best of our knowledge, only one published work exists for applying neural image captioning to a medical dataset [15]. In this work the authors employ an architecture similar to Vinyals et al. [19] to generate an array of keywords for a radiological dataset. 3
Unless otherwise speci ed, the same method was applied for both the caption prediction and concept detection tasks. The set of concepts assigned to an image in the concept detection task is considered to be a caption where each concept label is a word in the sentence. In both cases only the supplied training dataset was used to train the models. 3.1
In order to simplify the task, each image in the training set was preprocessed in
accordance with the task's evaluation preprocessing speci cations. This involved
converting the caption to lower case, removing all punctuation (some captions
contained multiple sentences, however, after this step each caption became a
single sentence), removing stopwords using the NLTK [
The images are rst resized to 324x324px, and a 299x299px crop is then
selected. During training this is a random crop, but during evaluation a central
crop is used. We apply image augmentation during training to regularise the
model [
A validation set was provided by the organisers of the task and it entirely reserved for validation. No part of it was used for training, and speci cally we did use it to build the dictionary of unique words. 3.2
Our method extends Vinyals et. al's NICv2 model [20] 1. The NICv2 model
consists of two di erent types of neural networks paired together to form an
imageto-language, encoder-decoder pair. A CNN, speci cally the InceptionV3 [18]
architecture, is used as an image encoder. InceptionV3 is one of the most accurate
architecture for general image classi cation according to the ImageNet [
An image is rst preprocessed as described above and then fed to the input of the CNN. The logits of the CNN are passed into a single layer fully-connected neural network which functions as an image embedding layer. This image embedding then the becomes to initial state of the LSTM network. As per [20] the embedding was passed only at the initial state and is not used subsequently. At each state subsequent to the initial state, then LSTM's output is passed to a word embedding layer and then to a softmax layer. At each time step the output of the softmax is the probability of each word in the dictionary. For two of the caption prediction experiments (PRED2 & PRED4) we modi ed the baseline language model to use a 3-layer LSTM model with a single dropout layer on the output. Increasing the number of LSTM layers improves the ability of the language model to represent complex sentences and long term dependencies. Industrial neural machine translation models have been demonstrated to use decoder layers with up to 8 layers [21].
In all our models we used 1024 units for both the image and word embedding layers. The CNN is initialised using weights from a model trained on the ImageNet dataset, while the weights for the LSTM are initialised from a random normal distribution with values between -0.08 and 0.08 as per [16]. For the nal caption prediction experiment (PRED4) we attempted to domain transfer the updated CNN weights from the DET2 caption detection model. This was attempted to avoid corruption of CNNs during end-to-end training (discussed in Sect. 4). 3.3
The loss optimised during training is the summed cross entropy of the output of the softmax compared to the one-hot encoding of the next word in the ground truth sentence. This loss was minimised with standard Stochastic Gradient Descent (SGD) using an initial learning rate of 1:0 and a decay procedure that reduced the learning rate by half every 8 epochs (there were 164541 examples in the training set and a batch size of 16, so each epoch contains 10284 minibatches). Gradients were clipped to 5:0 for all experiments. 1 As per the Tensor ow-Slim implementation https://github.com/tensor ow/models/tree/master/im2txt available
at: g m E
orthomorpha unk sp n holotyp b right
gonopod mesal later view respect cf distal part right
gonopod mesal later subor subcaud
view respect scale bar 02 mm
As suggested by Vinyals et al. [19] we use Beam Search to generate sentences at inference time. This avoids the non-trivial issue that greedily selecting the most probable word at each time-step may result in a sentence which is itself of low probability. Ideally we would search the entire space for the most probable sentence, however, this would have an exponential computational cost associated with it as a forward pass through the entire model must be made for each node of the search tree. Therefore some search procedure is required in order to nd the most probable sentence given limited computational resources. Our best results were achieved with a beam size of 3 and maximum caption length of 50. The sentence output of the concept detection task was converted to an ordered set of concept labels. Tables 3 & 4 detail the training, validation and test results of the various runs. Please note that due to the high cost of inference, the training scores are estimated based on a random sample of 10000 images from the training set.
We attempted a two-phase training procedure as suggested by Vinyals et al. [20] for DET2 and some unsubmitted experiments. In the rst phase we froze the CNN weights and trained only the LSTM and embedding layers. Then, once the language model had begun to converge we trained the entire model end-toend with a very small learning rate (1e 5). This is suggested by the Vinyals et al. as necessary as otherwise the CNN model will become corrupted and never recover. However, we found that despite training the LSTM for a very long time in the rst phase and using a very small learning rate in the second phase we would very quickly corrupt the CNN as evidenced by a sharp increase in dead ReLUs and a large decrease in BLEU score. We found that BLEU scores would eventually return to those achieved in the rst phase of training, however, the dead ReLUs did not revive. We believe that the underlying issue is that the degree of domain transfer required to go from general images to medical images is vastly greater than that required to go from one collection of general images to another (i.e. ImageNet to MSCOCO). Based on the small variance between our training and validation scores we do not believe that the models were over tting, however, the large variance between validation and test scores indicates that there was a large disparity between the training and validation data and the data in test set. Based on the non-over tting analysis we could potentially train a much larger vision model for a longer time and improve the overall performance.
Additionally the fact that we could not successfully train the vision model without corrupting the network was a major limiting factor in our experiments.
Future work will investigate the potential of larger language models and devising a training regime that allows true end-to-end training for medical images.
Actual
orthomorpha latiterga sp n holotyp b right gonopod mesal later view respect cf distal part right gonopod mesal later subor subcaud view respect scale bar 02 mm orthomorpha unk sp n holotyp b right gonopod mesal later view respect cf distal part right gonopod mesal later subor subcaud view respect scale bar 02 mm BLEU: 0.9304 BLEU1: 0.9629 BLEU2: 0.92 BLEU3: 0.9231
BLEU4: 0.9167 preoper later radiograph right knee
The authors are grateful to the NVIDIA Corporation for their donation of the Titan X GPU used in this research. 15. Shin, H.C., Roberts, K., Lu, L., Demner-Fushman, D., Yao, J., Summers, R.M.:
age annotation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2497{2506 (2016) 16. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger,
Curran Associates, Inc. (2014) 17. Syeda-Mahmood, T., Kumar, R., Compas, C.: Learning the correlation between images and disease labels using ambiguous learning. In: Medical Image Computing and Computer-Assisted Intervention { MICCAI 2015. pp. 185{193. Springer, Cham (Oct 2015) 18. Szegedy, C., Vanhoucke, V., Io e, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision (Dec 2015) 19. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator (Nov 2014) 20. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge. IEEE Trans. Pattern Anal. Mach.
Intell. (Jul 2016) 21. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun,
22. Yao, B.Z., Yang, X., Lin, L., Lee, M.W., Zhu, S.C.: I2t: Image parsing to text description. Proc. IEEE 98(8), 1485{1508 (Aug 2010)