In this working note, we describe details of our method in ImageCLEF2014 Scalable Concept Image Annotation task. We are given images from the Web and some additional information, including web pages, in which images exist. Using this information, we must construct an annotation system that has high performance and scalability. To assign labels to each image, we use the page title and attributes of an image tag extracted from the web page. As visual features, we propose the use of the combination of two complementary features, which are Fisher Vector and deep convolutional neural network based feature. They are generative and discriminative feature respectively. We then train linear classi ers using Passive{Aggressive with Averaged Pairwise Loss. After training, we calculate the score of each concept for test data and label some concepts having the best scores. Results show that the combination of two features contributes to the improvement of recognition performance.
For the ImageCLEF2014 Scalable Concept Image Annotation task [
As visual features, we use a convolutional neural network (CNN) based
feature as well as the Fisher Vector (FV) [
In our experiment, we combine two types of features, which are FV and CNNbased features. Actually, FV is often used in image recognition tasks because of its recognition performance. However, many results of recent studies show that deep CNN achieves high performance on many tasks. Therefore, we expect that the feature, which is the neuron activation pattern in the hidden layers of the network, has high-representational ability.
These two features are extracted in completely di erent ways. We obtain FV by coding local descriptors considering their probabilistic distribution. Therefore FV can be regarded as the feature expressing generative information of an image. In contrast to FV because deep CNN based feature is extracted from the network trained for recognition task, we can regard it as a discriminative feature.
Assuming that these two types of features, which represent di erent kinds of information, mutually compensate for representational ability, we propose their combined use. Our contribution is the usage of a combination of features that have complementary properties to improve the performance of annotation systems.
The remainder of this working note is structured as follows. Section 2 presents a description of two types of visual features: FV and deep CNN based features. In Section 3, we explain details of how we obtain labels from training data. Then, in section 4, we introduce a multi-label linear classi er training method: PAAPL. In section 5, we present the results of experiments, using either or both of these visual features. Finally, in section 6, we discuss the analysis of the results obtained in our experiment. 2 2.1
As a visual feature, we use Fisher Vector (FV) because FV can achieve better recognition performance than Bag of Visual Words with a linear classi er. In general, the linear classi er is less costly than a nonlinear one such as kernelSVM when the amount of the training sample increases. Therefore, FV is suitable for this task, which requires scalability.
In our experiments, we extract four local descriptors: SIFT, GIST, LBP, and C-SIFT. The dimensions of all these local descriptors are reduced to 64 dimensions using Principal Component Analysis (PCA). These local descriptors are densely extracted from ve scales of patches (squares 16, 25, 36, 49, 64 pixels on a side) sliding with a step of six pixels. Using some of the obtained local descriptors, we train a Gaussian Mixture Model (GMM) with 256 components, which have diagonal matrices as covariance matrices. After training GMM, we extract FV from each image by calculating the gradient of log-likelihood of local descriptors with respect to parameters of GMM. Then we normalize it using the Fisher information matrix. Power normalization and L2 normalization are applied to the extracted FVs. To include spatial information, we divide images into 1 1, 2 2, and 3 1 cells, extract features from each region and concatenate them into one vector. The nal dimension of FV is 262,144 (64 256 2 8).
As described above, FV is obtained by coding local descriptors considering their probabilistic distribution. Therefore FV can be regarded as a feature expressing generative information of the image. 2.2
In addition to FV, we use a deep convolutional neural network (CNN) based feature extracted from a deep CNN model that had been pre-trained with ImageNet dataset.
In recent years, many studies that speci cally examine deep CNN have shown
that such models can perform better than conventional feature representation in
object recognition and other tasks. On ImageNet Large Scale Visual Recognition
Challenge, one system [
However, training deep architecture of CNN requires large-scale data to prevent over tting and to ensure generalization ability. Trained models will have low recognition performance if the training data are few.
For this task, because we must label training data automatically, the number of reliably labeled data we can obtain using our method is roughly 100,000 on the development set and about 200,000 on the test set. This fact implies that the approximate number we can use in training models is only 1,000 per concept, on average. Therefore, because of limitations in the amount of data in the given dataset, it is di cult to train deep CNN models to produce high recognition performance.
According to [
In consideration of the discussion presented above, we use feature
representations extracted from a deep CNN model pre-trained with the ImageNet dataset.
Following the method of [
Contrary to FV, because deep CNN-based features are extracted from the network, which is trained for recognition task, we can regard it as a feature that expresses discriminative information of an image.
In our experiment, we use DeCAF, an open source library produced by [
Because no explicit labels are assigned to training images, we must label them using additional information or external sources. In this section, we describe how to assign labels to images. The pipeline of label assignment is shown in Fig.1. We label images in the following two steps.
webpage!
WordNet Database
Parse XML!
WC= {C,synonym ( C ) , hyponym ( C )} T={words related to images}
Assign
Label! if T ∩ WC ≠ 0
In the rst step, we parse the xml les of the web page, in which an image exists. Then we extract page titles and attributes of the image tag, which include src, title, and alt. The hope is that these attributes include important information about what the image represents. Then we split them into a set of single words T . For example, if there is an image tag in an xml le shown in Fig.2, then we obtain
T = fQueen; Prince; corgi; family; abroadg: !" " !"!!# !"#$%&'"" # # # #####"$%&#'()*)+(&$,--.!/"01234567## #######89:;#*#<=::#>3+>(?<#
" #######>:9*@=::#>3+>(?A#9B;#CD;;E#>E?#6($E);##### #######F?G>(?#G$9B#+E;!#+"H!#9!B#;#H>%$:IJ'#)+(&$#KL# # # # # # " !"#$%#&"
D(:#+H#$%>&;" 89:;#+H#$%>&;" =:9;(E>8M;#G+(?#GB;E# $%>&;#)>EN9#3;#(;>?"
In the second step, we collect a set of synonyms and hyponyms for each
concept C using WordNet [
WC = fC; synonym(C); hyponym(C)g where synonyms (C) and hyponyms (C) respectively represent sets of synonyms and hyponyms of the concept C. For example, given a target concept \dog", we obtain
Wdog = fdog; puppy; corgi; :::g: We assign the concept C as the label to the image if at least one word in WC appears in T . 4
In this section, we introduce the multi-label linear classi er training method
Passive{Aggressive with Averaged Pairwise Loss (PAAPL) [
First, we describe the model update rule of PA. Given the t-th training sample, we designate the visual feature by xt. We de ne Yt as the set of labels assigned to the t-th sample, and Yt as the set of labels not assigned. The model (weight) of the linear classi er corresponding to concept label C before updating by the t-th sample is denoted by wtC . 1. Fetch the t-th training sample. Then compute scores for each label using current models. Because classi ers we are training are linear, scores are given by simple calculation of the inner product of weight and feature. 2. Based on scores, nd a combination of labels rt 2 Yt and st 2 Yt in the following way.
rt = arg min wtr xt
r2Yt st = arg max wts xt
s2Yt 3. For a combination of rt and st, compute the hinge-loss l l(wtrt ; wtst ; (xt; Yt)) = (0 1 if wtrt xt
wtst xt > 1 (wtrt xt wtst xt) otherwise 4. Update models using hinge-loss according to the following rule. wtr+t1 = wtrt + wts+t1 = wtst l l 2jxtj2 + D1 xt 2jxtj2 + D1 xt Therein, D is a Passive{Aggressive parameter that reduces the negative inuence of noisy labels.
Then we describe the method of training classi ers with PAAPL. 1. Pick the t-th training sample, compute scores for target labels using current models. 2. For a randomly selected combination of labels rt 2 Yt and st 2 Yt, hingeloss is calculated as PA and remove rt and st from Yt and Yt. Continue this process until jYtj = 0 or jYtj = 0. 3. For combinations satisfying the condition that the hinge-loss is not 0, update the models according to the update rule of PA.
In PAAPL, convergence of models is faster than in PA because PAAPL updates multiple pairs of models for one sample, whereas PA updates only one pair of models. 5
At the training phase, we rst extract visual features and assign labels. Then, linear classi ers are trained. At the test phase, we calculate scores for test images using the linear classi ers we trained. The concepts are labeled on those images. When training classi ers, the number of iterations is set to 5 and Passive Aggressive parameter D is set to 1:0 105. After the training process, we average scores from di erent models trained using di erent visual features. Also, we decide concepts by selecting those with scores in the top 4% of all given concepts to each test sample. In our experiments, we compare two types of visual features. The settings, except for the combinations of visual features, are xed throughout all of our experiments.
First, for each type of feature, we respectively search for the best combination. For FV, we use four local descriptors, SIFT, C-SIFT, GIST, and LBP. Because the respective properties of these four features di er, we try all possible combinations of them. As for deep CNN-based features, we extract them from sixth and seventh layers. From each layer, we obtain features of two types, such as activations and outputs of each unit. Therefore we also obtain four types of visual features. In contrast to the case of FV, we need not try all possible combinations because combinations of features from the same layer do not make sense theoretically: they are expected to have similar properties. As shown in Tables Table 1 and Table 2, in both cases, the results of combining all features achieves higher performance than the others. These results respectively correspond to our Run 1 and Run 2.
Finally, we combine these two types of visual features. Using the information presented above, we combine all four FVs and four deep CNN based features. The nal results are presented in a Table in Table 3, which correspond to all Runs we submitted. We achieved better performance using both features than by using either one.
As a result, we achieved the second score among all participants with our best run.
C-SIFT GIST LBP SIFT MF-samples
sixth (ReLU) sixth seventh (ReLU) seventh MF-samples X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X In this working note, we described our annotation method for the ImageCLEF 2014 Scalable Concept Image Annotation task. As visual features, we used FV and deep CNN based feature. Assuming that these two types of features mutually express di erent kinds of information complementarily, we tried combining them. In our experiment, we showed how the combination of features contributes to the improvement of recognition performance. Results show that the combination of generative features and discriminative features proved e ective in image recognition tasks.