We present the BossaNova scheme for the ImageCLEF 2012 Flickr Photo Annotation Task. BossaNova is a mid-level image representation, recently developed by our team, that enriches the Bag-of-Words representation, by keeping a histogram of distances between the descriptors found in the image and those in the codebook. Our scheme has the advantage of being conceptually simple, non-parametric, and easily adaptable. Compared to other schemes existing in the literature to add information to the Bag-of-Words model, it leads to much more compact representations. Furthermore, it complements well the cutting-edge Fisher Vector representations, showing even better results when employed in combination with them. In our participation, we submitted four purely visual runs. Our best result (MiAP = 34.37%) achieved the second rank by MiAP measure among the 28 purely visual submissions and the 18 teams.
The ImageCLEF 2012 Flickr Photo Annotation Task is a multi-label classification problem. The task can be solved by following three different approaches: i) automatic annotation with visual information only, ii) automatic annotation with textual information only, iii) multi-modal approaches that consider visual and textual information. We consider only the visual content for the feature extraction. The dataset consists of 25, 000 Flickr images, splitting into training (15, 000 images) and test (10, 000 images) subsets.
The image set is annotated with 94 concepts that are very diverse and range
across categories such as people (e.g., male, female), nature (e.g., lake, beach),
weather (e.g., rainbow, fog) and even sentiments (e.g., unpleasant, euphoric). A
detailed overview of the dataset and the task can be found in [
In our participation in the ImageCLEF 2012 Flickr Photo Annotation Task,
we present our BossaNova scheme. Our aim is to emphasize the performance of
the BossaNova representation, using a single low-level feature (SIFT descriptors)
and SVM classifiers. BossaNova is a mid-level image representation [
Bag-of-Words representations can be understood as the application of two
critical steps [
Bossa Nova focus on the pooling step, by keeping a histogram of distances
between the descriptors found in the image and those in the codebook. Our scheme
has the advantage of being conceptually simple, nonparametric and easily
adaptable. Additionally, it leads to much more compact representations, compared to
other schemes to add information to the Bag-of-Words representation.
Furthermore, it complements well the cutting-edge Fisher Vector representations [
Our BossaNova scheme is composed of the following three steps: (i) extraction
of local image features (by SIFT descriptors [
BossaNova is a mid-level image representation which offers a more
information-preserving pooling operation based on a distance-to-codeword
distribution. In order to preserve a richer portrait of the information gathered during the
coding step, the BossaNova pooling function produces a distance distribution,
instead of compacting all information pertaining to a codeword into a single
scalar, as performed by Bag-of-Words representations [
Figure 1 illustrates the BossaNova and the Bag-of-Words pooling functions. The BossaNova pooling (Figure 1a) represents the discrete (over B bins) density distribution of the distances between the codeword cm and the local descriptors of an image. For each center cm, we obtain a local histogram zm. The colors (green, yellow and blue) indicate the discretized distances from the center cm to the local descriptors shown by the black dots. For each colored bin zm;b, the height of the histogram is equal to the number of local descriptors, whose discretized distance to codeword cm fall into the bth bin. In Figure 1a, B = 3. We can note that if B = 1 (Figure 1b), the histogram zm reduces to a single scalar value Nm counting the number of feature vectors falling into center cm.
To form the whole BossaNova image representation, all local histograms
zm are then concatenated. In addition, since the occurrence rate of each
codeword cm in the image is lost, BossaNova representation incorporates an
additional scalar value Nm for each codeword, counting the number of local
descrip(a) BossaNova pooling
(b) Bag-of-Words pooling
tors close to that codeword. That value corresponds to a Bag-of-Words term,
accounting for a raw measure of the presence of the codeword cm in the image.
Thus, BossaNova image representation z can be written as [
The key parameters in our BossaNova representation are the number of codewords M , the number of bins B in each histogram zm, and the range of distances [αmmin, αmmax] – the minimum distance αmmin and the maximum distance αmmax in the Rd descriptor space that define the bounds of the histogram.
The bounds αmmin and αmmax define the range of distances for the histogram computation. Local descriptors outside those bounds are ignored. For αmmax, the idea is to consider only descriptors that are “close enough” to the center, and to discard the remaining ones. For αmmin, the idea is to avoid the empty regions that appear around each codeword, in order to avoid wasting space in the final descriptor.
In BossaNova, αmmin and αmmax are set up differently for each codeword cm. Since our codebook is created using k-means, we take advantage of the knowledge about the “size” of the clusters, given by the standard deviations σm. We set up the bounds as αmmin = λmin σm and αmmax = λmax σm. In practice, the three parameters of the BossaNova become B (M being fixed), λmin and λmax. 3
We first describe our experimental setup (Section 3.1). We then detail our submitted runs (Section 3.2). Finally, we analyze our results at ImageCLEF 2012 Flickr Photo Annotation Task (Section 3.3).
As low-level descriptors, we have extracted SIFT and Opponent SIFT
descriptors [
To learn the codebooks, we apply the k-means clustering algorithm with
Euclidean distance over one million randomly sampled descriptors. For Fisher
Vectors [
One-versus-all classification is performed by support vector machine (SVM)
classifiers. We use a linear SVM for Fisher Vectors, since it is well known
that nonlinear kernels do not improve performances for those representations,
see [
We have submitted four runs in total. All runs use only visual information.
Run 1 – ID 1341070721262: Combination of BossaNova and Fisher Vector
representations. We use only SIFT descriptors. BossaNova parameters values
are: 4096 codewords, 2 bins, 5-nearest codewords in semi-soft coding, [0.4
σm, 2.0 σm] (range of distances for the histogram computation), see [
Run 2 – ID 1341070953984: Combination of BossaNova and Fisher Vector representations. We use only SIFT descriptors. BossaNova parameters values are: 4096 codewords, 2 bins, 5-nearest codewords in semi-soft coding, [0.4 σm, 2.0 σm] (range of distances for the histogram computation). Fisher Vectors are obtained with 384 Gaussians. We apply a sigmoid function to map the SVM scores to probabilities, where A = 20 and B = 1. Run 3 – ID 1341348153832: BossaNova representation. We use only SIFT descriptors. BossaNova parameters values are: 4096 codewords, 2 bins, 5-nearest codewords in semi-soft coding, [0.4 σm, 2.0 σm] (range of distances for the histogram computation). We apply a sigmoid function to map the SVM scores to probabilities, where A = 10 and B = 1. This run achieved the third score by MiAP measure among all visual submissions.
Run 4 – ID 1341348523492: Combination of BossaNova and Fisher Vector representations. We use SIFT and Opponent SIFT (only for Fisher Vector) descriptors. BossaNova parameters values are: 4096 codewords, 2 bins, 5nearest codewords in semi-soft coding, [0.4 σm, 2.0 σm] (range of distances for the histogram computation). Fisher Vectors are obtained with 384 Gaussians (for SIFT descriptors) and 128 Gaussians (for Opponent SIFT descriptors). We apply a sigmoid function to map the SVM scores to probabilities, where A = 10 and B = 1. 3.3
In Table 1, we list the performance of our submitted runs. As detailed in [
Regarding the MiAP metric, we can notice that our best run reached 34.37%
by combining the BossaNova and Fisher Vector representations (Run 1),
achieving thus the second rank among the 28 purely visual submissions and the 18
teams. It is worthwhile to point out that, according to [
Also, our BossaNova scheme (Run 3) achieved the third rank reporting 33.64% MiAP. Moreover, from Table 1, we can observe that using opponent SIFT (Run 4) as supplementary features does not bring any improvement. However, we consider that result is particularly affected by the severe dimensionality reduction of Opponent SIFT, from 392 to 128 dimensions. 4
In this paper, we presented our BossaNova scheme for the ImageCLEF 2012 Flickr Photo Annotation Task. Our method has the advantage of being conceptually simple, non-parametric and easily adaptable.
In our participation, we submitted four purely visual runs. Our best result (MiAP = 34.37%), which applied the combination of BossaNova and Fisher Vector representations, achieved the second rank by MiAP measure among the 28 purely visual submissions, while our BossaNova method achieved the third rank (MiAP = 33.64%). The absolute difference between the first MiAP and our best MiAP is only 0.44%. We consider that those results are particularly noteworthy considering the fact we have not yet exploited the use of complex combinations of different low-level local descriptors.
Feature combinations in a kernel learning framework is currently investigated in order to take advantages of all the features together.
This work is partially supported by CAPES/COFECUB 592/08/10, CNPq 14.1312/2009-2, ANR 07-MDCO-007-03, FAPESP and FAPEMIG.