This paper describes the details of our methods for participation in the caption prediction task of ImageCLEF 2017. The dataset we use is all provided by the organizers and indcoleusdne'tany external resources. The key components of our framework include a deep model part, an SVM part and a caption retrieval part. In deep model part, we use an end to end architecture with Convolutional neural network (CNN) and a Long Short-Term Memory (LSTM) to encode and decode images and captions. According to the statistics of training dataset, we train different models with different lengths of captions. Then in SVM part, we use Support Vector Machine (SVM) to determine which model to use when generating the description for a test image. In this way, we can combine these models from the previous deep model part. In caption retrieval part, we use the image feature extracted from CNN and apply Nearest Neighbor method to retrieve the most similar image with caption in the training dataset. The final description is the aggregation of the generated sentence and the caption retrieved from the training dataset. The best performance of our 10 submitted runs ranks the 3rd in group which doesn'utse external resources.
Over the past few years, there has been a huge interest in the task of automatically
generating captions for images. It is interesting to see how a machine can solve this
problem which is very easy to a person. Many progress [
There are three main approaches to generate image caption: one is using templates
[
The purpose of ImageCLEF 2017 caption task [
Our method can be separated into three parts. The first part is deep model part.
This part bases on deep model proposed by Vinyals [
The main structure of our method can be seen in the Fig. 1. The structure is composed of three parts: deep model, SVM and caption retrieval part. Each part will be introduced in detail below.
Fig. 1. Structure of our method
Preprocessing. In the preprocessing of data, we notice that training dataset is
different from the other well-known Image Caption dataset [
ent deep models in training stage. For each deep model, the input training data is a subset of dataset after preprocessing and the max length we set are shown in Table 2. Other initial parameters are the same in all the three deep models.
Training SVM classifier. We try to use SVM classifier combine the three models together to generate captions of images. So in SVM part, we attempt to train a SVM classifier which can predict the three kinds of sentence length 0-13, 13-30 and 30 over. We use all the images from training dataset and extract image features from fc7 layer in VGGNet to train a three class SVM classifier. This SVM classifier will be used to determine which deep model to use in the prediction stage. The accuracy of this SVM classifier in predicting the validation data is shown in Table 3. Caption retrieval: using Nearest Neighbor method. The performance of the model which only use deep model and SVM classifier is not optimal. Therefore, we attempt to use Nearest Neighbor method to retrieve the most similar image in caption retrieval part. If the Euclidean distance between the image CNN feature of predicted image and the image CNN feature of retrieved image is larger than a threshold, we will use the caption of retrieved image as a supplement in the final caption. The performance of model get an optimal when the threshold are 300 in CNN feature and 25 in normalized CNN features after many trials. In table 5, we can see that there is an improvement in the performance after applying the NN method.
Fig. 2. An illustration of prediction Prediction. As shown in Fig. 2, the input is an image and output is the generated sentences in the prediction stage. First, we extract the CNN feature from input image, then use the SVM classifier to determine which model to use. Next we use the trained deep model to generate a caption of this input image. Finally we apply the Nearest Neighbor method to retrieve a closest caption. If the Euclidean distance between image CNN feature of input image and image CNN feature of retrieved image is larger than a threshold, this caption will be used as a supplement of the caption. 3
The dataset we use in our method is all from the provided dataset. None of external datasets is used in our experiments. We first divide the ImageCLEF 2017 caption training data to three parts according the sentence length of training data captions. Then we use the divided subset to train three different deep models based on the CNN and LSTM model. After that, a three classes SVM classifier is trained, using 4096 dimensions vector extracted from VGGNet fc7 (fully connected) layer as feature and using label 0, 1, 2 represent the three kinds of sentence length. Finally, we use NN (Nearest Neighbor) method to find the most similar image in training dataset. The feature is the same feature used in training the SVM classifier and it is normalize from 0 to 1. The distance function is the Euclidean distance. If the similarity is larger than a threshold, we will use this retrieved image caption as a supplement in final caption.
out that whether the sentence length affects the final performance. Different sentence
lengths and their performance in the validation data are shown in Table 4. We use the
BLEU [
h =13,30,100)
SVM has some improvement in performance, the deep model cannot achieve an optimal result. So we conduct another experiment to explore whether using Nearest Neighbor method to retrieve image caption can help improve the performance.
We use normalized features and non-normalized features for NN method to compare the performance. And the results are shown in Table 5.The results demonstrate that adding the retrieved caption can lead to a better result and the result will be further improved when using normalized features. test_13_svm_3_nn_dist_25_normal_noUNK: use three classes SVM classifier and NN method. In NN method, we use normalized features and threshold is 25. Besides, we remove the UNK which is used to represent those word account didn’t achieve five times in training dataset.
test_5_svm_nn_dist_3000_nounk_modified_2: use two classes SVM classifier and NN method. In NN method, we e ndoidrmn’atlizeuds features and threshold is 3000. As mention before, this run also removes UNK.
test_12_svm_3_nn_dist_25_normal: use three classes SVM classifier and NN method. We use normalized features and threshold is 25 when applied NN method. The difference is that this run emdoidvne’tUNrK.
test_11_svm_2_nn_dist_25_normal_noUNK: use two classes SVM classifier and NN method. Normalized the image feature and threshold is 25. UNK is remain in this run.
test_10_svm_2_nn_dist_25_normal: use two classes SVM classifier and NN method. Use normalized features and threshold is 25. UNK is removed.
test_9_svm_three_nn_3000_noUNK: use three classes SVM classifier and NN method. In NN method, we didn’t use normaslizaendd thfreeasthuoreld is 3000. UNK is removed.
test_6_svm_three_parts: only use three lengths classes SVM classifier and deep model.
test_2_svm_two: only use two lengths classes SVM classifier and deep model. test_1_wc5sl70: only use CNN+LSTM deep model.
test_8_svm_two_remove_UNK: only use two lengths classes SVM classifier and UNK is removed.
We have submitted ten runs in the caption prediction subtask and the performances are shown in the Table 6. The performance of the best run is 0.2600 in Mean BLEU score. Compared to other runs which doesn’t including SVM classifier retrieval, the performance has been greatly improved. or caption
As shown in the Fig.3, the model can generate different lengths of captions according to the picture. The first example shows that the caption retrieved by NN method is similar to the caption generated by deep model and is also relate to the ground truth. Besides, the image of the first example contains images A and B. We are delighted to see the model can learn the pattern and generate a caption include alphabetic number and information in the two images. The third example shows that the distance between input image and the similar image retrieved using Nearest Neighborhood method is far from threshold, so the retrieved caption will not be used in the final caption. 4
In this paper, we describe our method in ImageCLEF 2017 caption prediction subtask. We use statistics of training dataset and divide the training dataset into three parts. Then in the training stage, we train three deep learning models and use CNN and LSTM to generate natural language sentences. A three classes SVM classifier is trained at the same time to determine which deep model to use when predicting image caption. Besides, Nearest Neighbor method is also applied to retrieve a similar image and its caption in the training data as a supplement in final caption. After performing the experiments above, we get the following conclusions. Firstly, the sentence length parameter in training can affect the performance. By training separately models and using SVM classifier, we achieve a better result compared to the model only use CNN+LSTM. Secondly, the similar image can provide useful information in caption. After applying the Nearest Neighbor method to retrieve a similar image and caption, the performance of the model can be greatly improved. However, we limit the sentence length and remove the words that only appear one or two times during the training. The removed words in the training dataset cannot be made full use of. This makes some generated captions lack of readability. In addition, the generated language model is too simple to generate a complex and fully descriptive caption. Compared with other participants in this task, the best performance of our 10 submitted runs is 0.2600 in Mean BLEU score, ranks the 3rd in group which doesn’t use external resources.
This work was supported in part by the National Natural Science Foundation of China under Grant 61532018 and 61322212, in part by the Beijing Municipal Commission of Science and Technology under Grant D161100001816001, in part by the Lenovo Outstanding Young Scientists Program, in part by National Program for Special Support of Eminent Professionals and National Program for Support of Top-notch Young Professionals.