In this paper, we describe our methods for ImageCLEF 2013 Personal Photo Retrieval Task. We devote our attention to making our system efficient in retrieving documents which have the similar topic with few query data. We train a ranking function using rankSVM. We extract Fisher Vectors (FVs) from several local descriptors as visual features, and use Bag-of-Words (BoW) as metadata features. The final similarity score is the weighted average of the visual similarities and the metadata similarities, where the weights are determined by relevance feedback. Results have shown that our system achieves the best performance in the Personal Photo Retrieval Task.
This paper describes our methods for the ImageCLEF 2013 Personal Photo Retrieval Task. Our objective is to investigate efficient methods for retrieving documents that have the similar topic with few query data. The dataset is obtained such that it imitates actual browsing data. It contains query by example(QBE) and browsing data, though some of them are unavailable. We compare three methods to measure a visual similarity between a document and a topic: SVM, rankSVM, and a similarity between a query and the nearest sample. We use the distance metric like radial basis function (RBF) kernel for a metadata similarity. We extract Fisher Vectors (FVs) as visual features, and use Bag-of-Words (BoW) as metadata features. The final similarity score is the weighted average of the visual similarities from several local descriptors and the metadata similarity, where the weights are determined by relevance feedback. Fisher Vectors. As a visual feature, we use Fisher Vectors (FVs) computed from four kinds of local descriptors: SIFT, C-SIFT, LBP, and GIST. Actually, GIST is usually used to describe a whole image, but we use it as a local descriptor. All local descriptors are reduced to 64 dimensions using Principal Component Analysis (PCA). Local descriptors are densely extracted from five scales of patches on a regular grid every six pixels and learn a Gaussian Mixture Model (GMM) with 256 components, which have a diagonal matrix as its covariance matrix. To use spatial information, we divide images into 1 1, 2 2, and 3 1 cells. Then FVs are calculated over each region as follows.
Let X = fx1; x2; ; xN g be a set of N local descriptors extracted from an image, and wi, i, i be the mixture weight, mean vector, covariance matrix of the i-th Gaussian, respectively. Then we difine, ui = vi =
1 ∑N N pwi n=1
1 N p 2wi n=1
N ∑ n(i) [
1 n(i) i 2 (xn
i) ; i 1diag((xn i)(xn i)T ) 1] ; where 1 is a column vector whose components are all 1 and diag(X) for matrix X is a column vector which is composed of diagonal components of X. n(i) is the soft assignment of xn to i-th Gaussian as n(i) = wiui(xn) K j=1wj uj (xn)
;
G = [u1T v1T : : : uTK vKT ]T ; where ui is the i-th Gaussian, and it is also known as the posterior probability. The FV representation is therefore given as where K is the number of GMM components.
Following [
As metadata features, we use Bag-of-Words (BoW) representation, which is based on the idea that each word in a text appears independently. BoW is obtained by counting the occurrence frequency of words in a text. In our method, each EXIF datum is represented as a component of the histogram separately. We use 10 Exif data in xml files as metadata features. Each feature and its dimension are shown in Table 1. We represent metadata features as xm. This section describes retrieving methods that we applied in the task. The dataset I for the task is composed of 5,555 images with metadata. Therefore, each image i 2 I has visual information and meta information. For a topic t, we represent the query by example(QBE) as qt, and browsing data as Bt = fb1; b2; :::g 2 I in which bi is browsed later than bj (i > j). QBE is an image which is admitted as the similar image for the topic. Browsing data is obtained while seeking QBE. A n-dimensional feature vector of image i is represented as xiv 2 Rn.
In our approach, we use rankSVM to ascertain the similarity between visual features, and use the distance metric as the similarity between metadata features. Finally, we obtain a similarity score of the document by summing the weighted similarity scores (rankSVM score for visual features + similarity score for metadata features). We use relevance feedback to obtain the weights for each score. Relevance is based on the variance between QBE and browsing data. For a comparison, we also adopt Nearest Neighbor and Support Vector Machine for visual features. where d(xv; xiv) is a distance between xv and xiv, e.g. a Euclidean distance. 3.2
Support Vector Machine (SVM) is a popular learning method that applies a maximum margin manner for binary classification. stv;i is a similarity score of an image i for topic t, and calculated as: For topic t, to train a linear model (wt; ht), SVM needs training data Dt = f(xiv; yiv;t)gjiI=j1. We choose a label of image i for topic t as: stv;i = wt xvi
ht: yiv;t = {1
(i 2 fqtg [ Bt) 1 (otherwise): Here, SVM is formulated as following optimization problem:
1 wt;f igjiD=1tj 2 jjwtjj2 + C∑ min i=1
i; jDtj s.t. yiv;t(wt xiv ht) i 1 0: i (i = 1; ; jDtj); 3.3
To learn a model for each topic from the QBE, browsing data, and other images,
we applied rankSVM proposed by T. Joachims[
t t t Pt = O1 [ O2 [ O3; O1t = f(i; j)ji = qt; j 2 I O2t = f(i; j)ji 2 Bt; j 2 I O3t = f(bi; bj )jbi; bj 2 Bt; i > jg: fqtgg; Bt
fqtgg; Here, rankSVM is formulated as following optimization problem: 1 wt;f igjiP=t1j 2 jjwtjj2 + C∑ min i=1
i; jPtj s.t. wt (xjv xvk) i 1 0: j (8(j; k) 2 Pt); 3.4
Similarity between metadata features is calculated by the distance metric of BoW feature vectors as: d(xim; xjm) = 1 e jjxim xjmjj2 ;
1 ci;j = 1 + d(xim; xjm) ; stm;i =
∑ k2Bt[qt ck;i; where xim and xjm are feature vectors i; j 2 I. d(xim; xjm) is the distance between two metadata features. In this method, we use 1 as . Here, ci;j is a similarity between xim and xjm. stm;i is a similarity score of metadata of image i for topic t, and is obtained by summing the scores with all of the query browsed images. Then we normalize the scores to zero mean and unit variance. 3.5
Relevance feedback is a technique of information retrieval that improves a user’s query and facilitates retrieval of information that a user needs. In this task, which feature is important for retrieval varies by topics. Therefore, the system must recognize the criteria for retrieval automatically from the query such as QBE document and browsing data. We calculate the weights of each feature. The final similarity score is the weighted average of the visual similarities and the metadata similarity.
In this method, the weights are calculated utilizing the process of the creation of the query data. Each query image is browsed sequentially. The weights of features are updated by chosen images. For example, the feature of which variance in query images is small should be important. We expand the weight formula as: !ln;tew = !l;t =
I Bl t ; l !ln;tew + (1 ) !lo;ltd; where is the updating weight. !l;t is the weight used in summing all of the similarity scores. lI is the variance of images about feature l. lBt is the variance of images in Bt. An initial value of !l;t is 1, and as the element count of Qt increases by one, !l;t is recalculated.
Finally, the score of image i is calculated by summing visual feature scores and metadata feature scores which are weighted with relevance feedback. The final score stf;iin is calculated as: where slv;t;i and slm;t;i are the visual and metadata scores of the image i for topic t. Lv and Lm are all of the visual and metadata features respectively. Nlv and N m l represent the number of combinations of the visual features and the metadata features respectively. In this paper, we set N m = 10. Nlv is 1, 2, 3, and 4. lv l and lm are the parameters to change ratio of the weight between visual and metadata features. 4
This section describes details of the comparison of learning methods and the result of visual feature’s combinations. 4.1
In our experiment, four kinds of local descriptors were extracted from each
image: SIFT, C-SIFT, LBP, and GIST. Each descriptor was sampled densely on
a regular grids (every six pixels). The dimensionalities of SIFT, C-SIFT, LBP,
and GIST were 128, 384, 1024, and 960 respectively. They were coded into two
state-of-the-art global feature representations (4 2 = 8 visual features in total).
One is FV, as explained in the previous section. First, the mixture model of 256
Gaussians was trained using a standard EM-algorithm. To use spatial
information, FVs were calculated respectively over 1 1, 2 2, and 3 1 cells. Thereby,
we obtained FVs of which the dimensionality was 64 256 8 2 = 262; 144.
The other is Locality-constrained Linear Coding (LLC) [
In this experiment, we examined which learning method is effective to train a ranking function for the task using only visual features. We investigated which visual feature should be combined to achieve the best performance through the next experiment. We use NN, SVM, and rankSVM to train the ranking function. Therefore, we examine 8 3 = 24 experiments in total. We use trec eval system published for evaluation. We use a ndcg cut 100 value in the evaluation of average users. The ndcg cut 100 value indicates the rate of compliance with the top 100 documents of correct data. This experiment addresses a combination of visual features and metadata features. First, we evaluated only metadata features for retrieval, then we got a score of 0:6228. With only visual features, the best score is 0:3861 using FV, LBP, and rankSVM.
The combination formula is given as: where lv and lm are the parameters to change ratio of the weight between visual and metadata features. We use lv = 3 and lm = 1.
Finally, we present the top six combinations of four visual features in two circumstances where 10 metadata features are used and not used respectively. We obtained the scores shown in Table 3 and Table 4. Without metadata features, the combination of three descriptors (C-SIFT, LBP, and GIST) got the best score among them. Additionally when metadata features were combined, the combination of all of the features (SIFT, C-SIFT, LBP, GIST, and Metadata) got the best score. We could not use this evaluation system before the submission. Therefore, we submitted the score of the experiment which uses four features (C-SIFT, LBP, GIST, and Metadata) we considered the best. 5
This paper explained our approach for ImageCLEF 2013 Personal Photo Retrieval Task. Results of comparative experiments show FVs as useful for visual representation and BoWs as metadata features. Retrieval is performed using rankSVM with visual features and the distance metric between metadata features. Using combinations of FVs from various local descriptors (C-SIFT, LBP, and GIST) and metadata features, we achieved the top score among all teams.