We describe in this paper the different approaches tested for the Photo Annotation task for CLEF 2011. We experimented state of the art techniques, by proposing late fusions of several classifiers trained on several features extracted from the images. The classifiers are SVMs and the late fusion is a simple addition of classification probabilities coming from the SVMs. The results obtained place our runs in the middle of the pack, with our best visual-based MAP at 0.337 We also integrated of Flickr human annotations, leading to a large increase of the MAP with a value of 0.377.
This paper aims at describing the proposal and results of the LIG-MRIM
research group at the Photo Annotation task for CLEF 2011. The
proposal of the group focused mainly on applying a late fusion on multiple
learners based on SVM. We also experimented some processes to reduce
the feature space dimensions, and we made use of a simple integration
with Flickr tags. The findings according to the official evaluations confirm
that: late fusion of multiple features lead to good result, that dimension
reduction on few features is an interesting direction to focus on, and that
a simple integration of human assigned tags improves results.
The corpus [
The outline of this paper is the following. In section 2, we begin describe the visual feature extracted and their representation. Section 3 presents the processing applied on Flickr tags. In section 4, we focus on the classification applied on the extraction, as on the fusion processed between the different learners results. In section 5, we list our results, and we conclude in section 6.
We focus here the feature extracted, as well some PCA-based dimension reduction on some features. The features considered cover most the common feature we find the in literature. 2.1
The features that were extracted are color-based as well as texture based. Some features are extracted globally from the whole image, and others are extracted from image regions, before being aggregated to represent one image. In the following, we give an identifier for each feature before explaining the extracted feature. Such identifiers will be reused in section 5.
Global features
– h3d 64: normalized RGB Histogram. Such color-based histogram is
64 dimensions large, using a simple 4 x 4 x 4 subsampling respectively
the R, G and B colors components;
– gab 40: normalized Gabor transform [
Local features The local features extracted are SIFT-like. They are
extracted for regions of the images, resulting from dense sampling of
harris-laplace region of interest detection. Each of these features are
represented as bag of visual word, similarly to [
Dimension reduction on features representations
As shown by [
As Image annotation collection is an excerpt from Flickr, the human generated tags are available. We know that such manually input tags are not always easy to process (typos, jokes, etc.), but we propose a simple way to handle some of them. First, for each image, we split the tags into words, and we apply a Porter stemmer in a way to group similar words into classes. In a second step, if one stemmed tag equals one of the 99 stemmed labels, then the label is selected for the image. The resulting representation is a 99 dimensions binary vector, with 1 if the label describes the image and 0 otherwise. 4 4.1
All the classification processes on the visual features use Multiple-SVM classifiers based on Radial Basis Funcion (RBF) kernels, since it was proved to be a good solution for data imbalance problems. Such problems occur for many labels in the collection under consideration here. So, for each label, we get positive and negative samples that are used as input for the learning of the Support Vectors.
During the classification, the image representation is input to the SVM
using each model, and a binary classification is processed. We assume
here that each classifier outputs a probability of classification in [
We present the MAP results obtained on a validation set. Our training set is composed of 2/3rd of the official training set, generated randomly with a post processing ensuring a similar distribution of the tags that on the official training set. This last point is important, especially for the labels that have only few samples (like skateboard with only 12 positive samples). The validation set in composed of the 1/3rd remaining images of the official training set.
The table 1 presents the results obtained feature by feature for each visual feature listed in section 2.1. This table shows that all the SIFTbased features with hard assignment behave consistently, with MAP values between 0.246 and 0.258. The soft assignment opp sift har unc 1000 outperforms slightly the hard assignments, but only marginally. We notice also that the hg 104 features behave surprisingly well compared to SIFT-like features.
The table 2 focuses on the results obtained when considering the dimension reduction process depicted in part 2.2. In the last column of this table, we list the percentage of increase compared to the original (i.e., not reduced) features. This table shows that the reduction of dimension proposed always outperforms the original features. This result is especially visible with the opponent sift features with strict assignment. In any cases, the dimension reduction seems effective for harris laplace features, and less for dense sampling-based features. For the “low dimansional” features, we notice also a large imrpovement with one one half reduction, leading to very good results for hg 104 pw0.375 54, which has onlyt 54 dimensions.
The last table, 3 of this section deals with the results obtained after fusing the results, according to the explanations of section 4. We chose three fusions, which correspond to the configuration of the official run submitted: – msvm: the late fusion of all the 20 visual features considered earlier in the paper; – msvm tags: the late fusion of the visual scores and the Flickr tags scores; – msvw two desc: the late fusion of the two best features according the table 2, but considering two different kinds of regions for the features (i.e. one Harris-Laplace based, and one dense sampling based) to ensure variability in the fused results: opp sift har unc 1000pw0.450p400 and opp sift dense 1000pw0.450p400.
The conclusions drawn from this table is that the fusion always outperforms each f its components (such result is well known in the community). We see here that Flickr tags integration, even is the processing is quite straightforward, leads to an important increase of the results. These three configurations are the ones used for the official submissions. Descriptor identifier MAP (increase vs. best visual feature in the fusion) msvm msvm tags msvw two desc We present here the official MAP results obtained from our runs in 4. This table shows also in the last column the rank obtained in comparable lists (i.e., list of visual results for msvm and msvw two desc, and list of multi-modal results for msvm tags). The results obtained place our best visual run, msvm with a MAP of 0.336, in the first tier of the list, and above the average and the median values respectively of 0.289 and 0.323. For the multimodal run, msvm tags with a MAP of 0.378, the rank is above the middle, and also above the average and the median values of respectively 0.370 and 0.371 . This paper presented the worjk of the LIG-MRIM team for the Photo Annotation task for CLEF 2011. We used a large set of 20 features, with or without strict bin assignment, dimension reductions, with and without integrating Flickr tags. The results obtained place our run in the first tier for the visual runs, and in the first half for the multimedia runs. In the future, we will focus on dimension reductions to find out what reductions are useful.
This work was partly supported by: a) the Quaero Programme, funded by OSEO, French State agency for innovation and b) the R´egion Rhones Alpes (projet LIMA).