In this paper we describe our approach to the ImageCLEF2012 Photo Annotation task. We used both visual and textual modalities for all submissions. We described each image with a fixed length representation using different similarity measures. By this method we were able to combine, before the classification, a large variety of descriptors to improve the classification quality. This descriptor is a combination of several visual and textual similarity values between the actual image and a reference image set, containing well selected training images. We trained Gaussian Mixture Models (GMM) to define a generative model for low-level descriptors extracted from the training set using Harris-Laplacian point detection. We used two descriptors, a grayscale gradient and a color moment based one. In order to measure the visual similarity between two images, we extracted several dense Fisher vectors per image. Besides calculating visual features, we adopted a biclustering method to cluster the Flickr tags and the images at the same time. Additionally, we measured the similarity of images according to their Flickr tags using Jensen-Shannon divergence.
The GMM based Fisher gradient vector computed of SIFT [
We used low-level patch feature vectors to describe the visual content of an image by
approximately 15k descriptors per image per modality. Our sampling strategy included a dense grid
and a Harris-Laplace point detection [
To build an efficient soft codebook, we trained a Gaussian Mixture Model (GMM) for both
descriptors. The training procedure of GMM models took about 20 minutes using 3 million training
points per descriptor. We used our open-source CUDA GMM implementation. Our training method
was based on a standard Expectation Maximization algorithm with a non-hiherarchical structure.
We avoided the well-known vulnerability of the EM algorithm to underflow by computing the
condiditonal probabilities [
The final high-level dataset independent representation of images was the normalized Fisher gradient vector. We also calculated a separate Fisher vector on the Harris-Laplacian detected corner descriptors. As by our GMM implementation we were able to compute all the conditional probabilites for each feature vector without significant loss of time, which resulted a strongly dense Fisher vector even in fp32. 3
Our previous experiment [
The applied biclustering was an expansion of Dhillan’s information theoretic co-clustering
algorithm [
Efficient combination of different feature sets based on a wide range of visual modalities is one
of the main problems of image classification. This problem becomes more complex if we have
additional non-visual features such as Flickr tags. Our starting point was a widely used technique:
learning SVM models on textual and visual Bag-of-Words models [
2 i=1 j=1 (1) subject to P yiαi = 0 for all i with αi ≥ 0.
Having multiple number of kernels due the representations via different modalities with previously selected kernel functions, we can modify the dual form into a multiple kernel learning problem:
m M aximizeLDual(α, β) = X − i=1
2 i=1 j=1 n=1 (2) subject to P yiαi = 0 for all i with αi ≥ 0 , where N is the number of the basic kernels and Kn(xi, xj) is the nth kernel function.
The above problem is a special case of the Multiple Kernel Learning problems where the
kernels are computed on different feature sets. In comparison to Bach [
To avoid the computationally expensive MKL problem we used a feature transformation
method. Distance from the training set, as a feature transform for classification is a well-known
technique. Sch¨olkopf et al. showed that a class of kernels can be represented as norm-based
distances in Hilbert spaces [
We defined a dense representation combining modality adaptive similarity based feature transforms. Let us consider a set of documents D (we call it as reference set) and their corresponding representations Dr. We defined the uniform representation of a document X over the set of representations R of a reference set D as
R R LR(X, D) = [X βrsimr(Xr, Dr1), .., X βrsimr(Xr, Drd)]
r=1 r=1 where P βr = 1 and simr denotes the selected similarity measure on basic representation r and d is the size of the reference set. The dimensionality of this representation is the cardinality of the reference set. 4.1
The proper selection of the reference set could decrease significantly the demanding computational time of solving the standard dual problem. More precisely, we are seeking for the minimal set of documents without affecting significantly the quality of the learning procedure.
To determine the reference set we defined a ranking for the images according to their annotations. The rarer a concept, the higher the score of its positive instances will be. We cut the list where the training documents contain at least a specified quantity of positive samples for all categories. We set the minimal amount of positive samples to p ∗ N where N is the number of training images. If a category did not have the minimal amount of positive instances all the samples were included. The resulted subset of training images using p = 0.01 contained only 6260 images out of the original 15k training images. Since the dimension of the combined representation equals with the number of images in the reference set this selection reduced the dimension by more than 50%.
To identify the weight vector β of the basic representations per class we sampled the training
set. We used 5k images for training and 5k images for validation. We trained binary SVM classifiers
using the LibSVM package [
In order to determine the parameters of the combined representation we experimented on the basic features using a subset of the training set. It can be seen in Table 2 that color moment and HOG descriptors complement each other. Although the average number of keypoints detected by Harris-Laplacian was considerably less than for the rest of the poolings (average 2k vs. 15k descriptors per image), we measured small performance differences between them. For Flickr tags we tested three methods (Table 3). We selected the top 25,000 Flickr tags as vocabulary. The refined biclustering using visual similarity and Jensen-Shannon divergence outperformed JensenShannon divergence and the purely tag based biclustering. We experimented with the parameter p for proper reference set selection over the best combined representation including all visual similarity values and Jensen-Shannon divergence. It can be seen in Table 1 that the performance loss was negligible even using less than half of the features. If we left only the 11.9% of the training set as reference set the performance dropped significantly. In jch10ksep we used the ranked reference set with 6260 images and an annotation category based weighting scheme for the combination (19 different weight vectors). We trained binary SVM classifiers per class using a reduced training set containing only 10k images.
Addition to jch10ksep, in jchb10ksep we added a refined biclustering representation with 2k clusters to the common representations. Notice that by biclustering the dimension of the representation was significantly the lowest of all (Table 4).
Our best performing run jchf r15k used the entire training set as reference set and the binary SVM models were trained on the whole training set (15k) per class. The adopted weight vector β were the same for each class. It is worth to mention that we experienced increase in computational time in comparison to jch10ksep or jchb10ksep. The reason for the nonlinear increase is that by jchf r15k we used the whole training set as reference set and the binary SVM classifiers were trained on the entire training set.
Our second best performing run jchaggsep was a combination of jchf r15k and jch10ksep (Table 6). We simply averaged the predictions of the runs. In jchbicwelf we aggregated the output of the biclustering based classifier and the jchf r15k using a linear combination learned previously on the training-validation set. 6
Our approach for ImageCLEF 2012 Photo Annotation task employed various representations of the images based on different visual and textual modalities. We extracted several Ficher vectors using a grayscale and a color patch descriptor. We applied a biclustering method to cluster the images and their Flickr tags. We combined the different descriptors and representations before the classification. This combination procedure included a transformation, a feature aggregation and a selection step.