This article describes the participation of the Computer Science Department of
Morgan State University, Baltimore, Maryland, USA in the ImageCLEFcaption
under ImageCLEF 2018. The problem of automatic image caption prediction
involves outputting a human-readable and concise textual description of the
contents of a figure appeared in a biomedical journal. It is a challenging
problem in biomedical literature as it requires techniques from both the fields of
Computer Vision to understand the visual contents of the image and Natural
Language Processing (NLP) to turn the understanding of the image into words
in the right order to generate the textual description for caption. Recently, deep
learning methods have achieved state-of-the-art results on this challenging
problem in general domain of natural photographic images. For the tasks of
caption prediction and concept detection, we used the merge architectures for
the encoder-decoder recurrent neural network (RNN) due to it’s success over
inject for caption generation. The merge model combines both the encoded
form of the image input with the encoded form of the text description generated
so far. The combination of these two encoded inputs is then used by a very
simple decoder model to generate the next word in the sequence. In this article, we
present main objectives of experiments, overview of these approaches,
resources employed, and describe our submitted runs and results with conclusions
and future directions.
This article describes our second year’s participation in ImageCLEF 2018 [
There has been a lot of interest recently from Computer Vision, Machine Learning
and NLP community in developing deep learning-based approaches for automatic
caption generation of natural photographic images [
The provided datasets have separate training and test sets, which are really just different groups of image identifiers in separate text files respectively (such as CaptionTraining2018-List.txt and CaptionTesting2018-List.txt in case of caption prediction task). From these file names, we extracted the photo identifiers and use these identifiers as keys to filter photos and descriptions for each set.
The merge model (Fig. 1) combines both the encoded form of the image input with
the encoded form of the text description generated so far, which is handled by
Recurrent Neural Networks (RNNs). This separates the concern of modeling the image
input, the text input and the combining and interpretation of the encoded inputs [
The entire process of the model generation is shown as a plot in Fig. 2 and can be described in three parts as follows:
We loaded all the images from a subset of the original training set, extract their
features using a pre-trained 16-layer VGG model [
The training dataset contains caption for each image, which requires some minimal cleaning. Each image name without the file extension (e.g., .jpg) is acted as a unique identifier in a processed text file of descriptions. The descriptions are cleaned and tokenized in order to reduce the size of the vocabulary of words; this means that each token is comprised of words separated by white space where words are transformed to lowercase, all punctuation are removed from tokens and tokens that contain one or fewer characters are also removed. Finally, we save the dictionary of image identifiers and descriptions to a new file named descriptions.txt, with one image identifier and description per line.
This is a word embedding layer for handling the text input, followed by a LSTM
layer. The model expects input sequences with a pre-defined length which are fed
into an embedding layer that uses a mask to ignore padded values. This is followed by
an LSTM layer with 256 memory units. Both the input models produce a 256-element
vector. Further, both input models use regularization in the form of 50% dropout. This
is to reduce overfitting the training dataset, as this model configuration learns very
fast. The Decoder model merges the vectors from both input models using an addition
operation. This is then fed to a Dense 256 neuron layer and then to a final output
Dense layer that makes a softmax prediction over the entire output vocabulary for the
next word in the sequence [
Both the feature extractor and sequence processor output a fixed-length vector. These are merged together and processed by a Dense layer to make a final prediction. The model we will develop will generate a caption given a photo, and the caption will be generated one word at a time. The sequence of previously generated words will be provided as input. Therefore, we will need a first word to kick-of the generation process and a last word to signal the end of the caption. We used the strings startseq and endseq for this purpose.
Each caption of an associated image is tokenized into words after performing some text processing and cleaning operations. The model is provided one word and the photo and generate the next word. Then the first two words of the description will be provided to the model as input with the image to generate the next word. For example, the input sequence “Mandibular true occlusal radiograph" would be split into five input-output pairs to train the model:
The Decoder model merges the vectors from both input models using an addition
operation. This is then fed to a Dense 256 neuron layer and then to a final output
Dense layer that makes a softmax prediction over the entire output vocabulary for the
next word in the sequence. Later, when the model is used to generate descriptions,
the generated words are concatenated and recursively provided as input to generate a
caption for an image. The Keras functional API [
For implementation of our deep learning model and submission of the runs, we used Keras (version 2.1.5) library in a Python 3 (version 3.5.2) SciPy environment on top of TensorFlow (version 1.1.0) backend. Keras provides a clean and convenient way to create a range of deep learning models on top of both Theano or TensorFlow backend which can be executed on both GPUs and CPUs given the underlying frameworks. Other necessary libraries, such as scikit-learn, OpenCV, Pandas, NumPy pickle, and Matplotlib are also installed and used to support visual and text feature extraction for model inputs as well as model generation, and saving and plotting the model. For training, the model was fitted for 30 epochs and given the amount of training data each epoch took around 3 hours in a window-based workstation.
This section provides descriptions of our submitted runs and analysis of the result. We (morgan_cs) tried to submit few runs for each task of caption prediction and concept detection. However, only a single run for each task was graded successfully, whereas others were failed due to processing errors.
Both runs are generated based on the deep learning method (merge model) described in previous sections. We almost implemented the same process of feature (textual caption and visual) extraction and model generation for training for both tasks. Results are generated based on the test sets (9,938 images) for captions and concepts. However, for training we only used only a small subset (around 4K images) of the original training set (> 222K images) as provided by the organizers. This limitation is due to the limited resources (both memory and computing power) that was available during the model generation and result submission steps in this evaluation campaign. UMMS morgan_cs 0.25 0.18
Our successful submission (morgan_cs) of caption prediction (ID: 6202) received a BLUE score of 0.1724, which is ranked third (group submission wise) as shown in Table 1. We also submitted mistakenly a run for caption prediction (which was graded successfully with a 0.0 score although it was meant for the submission of concept detection task). We failed to submit few other runs. However, there were some problems in our runs (result files) and received errors while the files were parsed by the online evaluation tool under crowdAI provided by the CLEF organizer.
For the sole submission (ID: 6216) of concept detection, we received a F1 score of 0.0417 (ranked 4th group wise) which is comparatively lower compared to the leading group (UA.PT_Bioinformatics) with a score of 0.11.
This article describes the strategies of the second year’s participation of the Morgan
CS group for the concept detection and caption prediction tasks of
ImageCLEFcaption [