In this paper we summarize our experiments in the ImageCLEF 2015 Scalable Concept Image Annotation challenge. The RUCTencent team participated in all subtasks: concept detection and localization, and image sentence generation. For concept detection, we experiments with automated approaches to gather high-quality training examples from the Web, in particular, visual disambiguation by Hierarchical Semantic Embedding. Per concept, an ensemble of linear SVMs is trained by Negative Bootstrap, with CNN features as image representation. Concept localization is achieved by classifying object proposals generated by Selective Search. For the sentence generation task, we adopt Google's LSTM-RNN model, train it on the MSCOCO dataset, and netune it on the ImageCLEF 2015 development dataset. We further develop a sentence re-ranking strategy based on the concept detection information from the rst task. Overall, our system is ranked the 3rd for concept detection and localization, and is the best for image sentence generation in both clean and noisy tracks.
This year we participated in all the subtasks, i.e., concept detection and
localization and image sentence generation, in the ImageCLEF 2015 Scalable Concept
Image Annotation challenge. In addition to the 500k web image set (webupv2015
hereafter) and the 2k develop set (Dev2k) provided by the organizers [
Next, we introduce in Section 2 our concept detection and localization system, followed by our image sentence generation system in Section 3. 2
Task 1: Concept Detection and Localization
We develop a concept detection and localization system that learns concept
classi ers from image-level annotations and localizes concepts within an image by
region-level classi cation, as illustrated in Fig. 1. Compared with our earlier
systems [
Training: Image-level concept modeling
web / Flickr images concept
Tag-based image search Test: Concept detection and localization
Positive Example
Selection by
HierSE
Object Selective Search proposals
CNN Feature Extraction
Refinement
Positive Training Examples Since hand labeled data is allowed this year, we
collect positive training examples from multiple sources of data, including
1. ImageNet. For 208 of the 251 ImageCLEF concepts, we can nd labeled
examples from ImageNet [
Negative Bootstrap
** ** Prediction
Since the annotations of webupv2015 and ickr2m are noisy, per concept we
collect its positive training examples by a two-step procedure. In the rst step,
we conduct tag-based image search using tags of the concept as the query to
generate a set of candidate images. For webupv2015, an image is chosen as a
candidate as long as its meta data, e.g., url and query logs, overlap the tags. In
the second step, these candidate images are sorted in descending order in terms
of their relevance to the concept, and the top-n ranked images are preserved
as the positive training set. To compute the relevance score between the given
concept and a speci c image, we embed them into a common semantic space
by the Hierarchical Semantic Embedding (HierSE) algorithm [
Since HierSE takes the WordNet hierarchy into account, it can resolve
semantic ambiguity by embedding a label into distinct vectors, depending on its
position in WordNet. Consider the label `basin' for instance. In the context of
ImageCLEF, it is `basin.n.01', referring to a bow-shaped vessel. On the other
hand, it can also be `basin.n.03', meaning a natural depression in the surface of
the land. In HierSE, a label with a given sense is represented by a convex
combination of the embedding vectors of this label and its ancestors tracing back to
the root. Fig. 2 shows the top-ranked images of `basin' from ickr2m, returned
by the tag relevance (tagrel) algorithm [
Given the HierSE ranked images per concept, we empirically preserve the top n = 1000 images as positive training examples. Since the number of genuine positives varies over concepts, this year we also consider an adaptive selection strategy. We train an SVM model using the top 20 images as positives and 20 images at the bottom as negatives. The previously ranked images are classi ed by the model and labeled as positive if the classi cation scores exceed 0. We denote this strategy as HierSE + SVM.
Negative Bootstrap To e ectively exploit an overwhelming number of (pseudo)
negative examples, we learn an ensemble of linear SVMs by the Negative
Bootstrap algorithm [
To evaluate the multiple methods for selecting positive and negative
examples, we split the ImageNet set into two disjoint subsets, imagenet-clef15train
and imagenet-clef15test, which contain positive examples for the 208
ImageCLEF concepts. Using imagenet-clef15test as the test set, the performance of the
individual methods is summarized in Table 1. For webupv2015, the combination
of HierSE + SVM and Negative Bootstrap performs the best. For ickr2m, the
combination of HierSE and Negative Bootstrap performs the best. Still, there is
a substantial gap between the model trained on auto-selected examples (MAP
0.375) and the model trained on hand labeled examples (MAP 0.567). So for
the 208 concepts, we combine all the models trained on imagenet-clef15train,
(a) basin by tagrel
(b) basin by HierSE
webupv2015, and ickr2m, while for the other 43 concepts, we combine all the
models trained on webupv2015 and ickr2m. Although learned weights are found
to be helpful according to our previous experiences [
Object Proposal Generation For each of the 500k images, we use Selective Search [16] to get a number of bounding-box object proposals. In particular, the image is rst over-segmented by the graph-based image segmentation algorithm [17]. The segmented regions are iteratively merged by hierarchical grouping as depicted in [16]. As the CNN feature is extracted per bounding box, the number of chosen object proposals is set to 20 given our computational power. Detection Given an image and its 20 object proposals, we classify each of them using the 251 concept models, and label it with the concept of maximum response. Notice that for each concept, we have two choices to implement its model. One choice is HierSE, as it was developed for zero-shot learning, and is thus directly applicable to compute cross-media relevance scores. The other choice is linear SVMs trained from the selective positives combined with Negative Bootstrap. HierSE is the same as used for positive example selection, while the SVMs used here are distinct from the SVMs in Section 2.1.
To reduce false alarms, we re ne the detection as follows. For object proposals labeled as the same concept, if their number is lower than a given Minimum Detection threshold md (we tried 1, 2, and 3), they are discarded, otherwise we sort them in descending order in terms of detection scores. We go through the ranked proposals, preserving a proposal if its bounding box has less than 30% overlap with the previously preserved proposals. 2.3
We submitted eight runs in the concept detection and localization task. ruc task1 hierse md1 is our baseline run, directly using HierSE to compute the relevance score between an object proposal and a concept, with the Minimum Detection threshold md set to 1, namely no removal. ruc task1 hierse md2 is the same as ruc task1 hierse md1, but with md = 2. ruc task1 hierse md3 is the same as ruc task1 hierse md1, but with md = 3. ruc task1 svm md1 uses SVMs for concept detection, with md = 1. ruc task1 svm md2 is the same as ruc task1 svm md1, but with md = 2. ruc task1 svm md3 is the same as ruc task1 svm md1, but with md = 3. ruc task1 svm md2 nostar is the same as ruc task1 svm md2, but empirically thresholding the detection results of two concepts `planet' and `star', as they tend to be over red. ruc task1 svm md3 nostar is the same as ruc task1 svm md3, but empirically thresholding the detection results of two concepts `planet' and `star'. Result Analysis The performance of the eight runs is shown in Fig. 3. We attribute the low performance of the HierSE runs to the following two reasons. First, HierSE is designed for zero-shot learning. It does not use any examples of the ImageCLEF concepts. Second, the current implementation of HierSE relies on the ImageNet 1k label set [18] to embed an image, while the 1k set is loosely related with the ImageCLEF concepts. We can see that the Minimum Detection threshold is helpful, improving MAP 0.5Overlap from 0.452 to 0.496. 0.8 0.7 0.6 p0.5 a l r e v .5O0.4 0 _ P AM0.3 0.2 0.1 0.00 runs form other teams ruc_task1_svm_md3 ruc_task1_svm_md3_nostar ruc_task1_svm_md2_nostar ruc_task1_svm_md2 ruc_task1_svm_md1 ruc_task1_hierse_md3 ruc_task1_hierse_md2 ruc_task1_hierse_md1 10 20 30
40
subtask1 runs
50
60
70
80
We have used the MSCOCO dataset [
In the training stage, we rst train the LSTM-RNN on the MSCOCO dataset. We then ne-tune the model on the ImageCLEF Dev2k dataset using a low learning rate. Beam search is used in text decoding as in [19]. We nally re-rank the hypothesis sentences utilizing the concept detection results. Given an image, our system rst generates k best sentences with con dence scores, meanwhile the concept detection system from task 1 provides m best detected concepts with con dence scores. If a detected concept appears in the hypothesis sentence, we call it a matched concept. We then compute the ranking score for each hypothesis sentence by matching the detected concepts with the hypothesis sentences as follows:
RankScore(hypoSent) = conceptScore + (1 ) sentenceScore; (1) where conceptScore refers to the average of the con dence scores of all the matched concepts in the hypothesis sentence (hypoSent), sentenceScore refers to the con dence score assigned to the hypothesis sentence by the RNN model. The parameter has been tuned on the Dev2k validation set. Fig. 5 showcases some examples of how re-ranking helps the system produce better image descriptions.
In the clean track, \golden" concepts, i.e., ground truth concepts by hand labelling, are provided. So we set the con dence score for each concept as 1. In
(a)
(b)
this condition, if the top-ranked sentence does not contain any of the \golden"
concepts, the word in the top-ranked sentence with the closest distance to any
one of the \golden" concepts will be replaced by that concept. The word distance
is computed using a pre-trained word2vec2 model [
3.2
We submitted six runs for the noisy track, and two runs for the clean track.
[Noisy track] RUC run1 dev2k is our baseline trained on the Dev2k dataset
without sentence re-ranking.
[Noisy track] RUC run2 dev2k rerank-hierse is trained on the Dev2k
dataset, using the 251 concepts predicted by HierSE [
Result Analysis The performance of the noisy track runs is shown in Fig. 6(a). We see that ne-tuning improves the system performance (RUC run1 dev2k 0.1659 versus RUC run4 netune-mscoco 0.1759). Sentence re-ranking helps as well (RUC run1 dev2k 0.1659 versus RUC run2 dev2k rerank-hierse 0.1781).
The performance of the clean track runs is shown in Fig. 6(b). It shows that the LSTM-RNN model trained on MSCOCO and ne-tuned on Dev2k is less e ective than the model trained on Dev2k alone. This is somewhat contradictory to the results of the noisy track. For a more comprehensive evaluation, Table 2 shows precision, recall, and F1 scores of the two runs, from which we see that 0.30
RUC run2 netune-mscoco obtains a higher precision score. This is probably because the reranking process based on the \golden" concepts makes the system more precision oriented.
Since the details of the test sets used in the two tracks are not available to us, we cannot do more in-depth analysis. For the concept detection and localization task, we nd Hierarchical Semantic Embedding e ective for resolving visual ambiguity. Negative Bootstrap improves the classi cation performance further. For the image sentence generation task, Google's LSTM-RNN model can be improved by sentence re-ranking.
Notice that for varied reasons including the limited number of submitted runs
and the unavailability of detailed information about the test data, our analysis
is preliminary, in particular a component-wise analysis is largely missing.
Acknowledgements. This research was supported by the National Science
Foundation of China (No. 61303184), the Fundamental Research Funds for the
Central Universities and the Research Funds of Renmin University of China
(No. 14XNLQ01), the Beijing Natural Science Foundation (No. 4142029), the
Specialized Research Fund for the Doctoral Program of Higher Education (No.
20130004120006), and the Scienti c Research Foundation for the Returned
Overseas Chinese Scholars, State Education Ministry. The authors are grateful to the
ImageCLEF coordinators for the benchmark organization e orts [