Introduction

Sample Selection, Category Speci c Features and Reasoning

Eugene Mbanya

Sebastian Gerke

Christian Hentschel

Patrick Ndjiki-Nya

patrick.ndjiki-nyag@hhi.fraunhofer.de 0 0 Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute

In this paper we present our approach to the 2011 ImageClef PhotoAnnotation task, which is based on the well known bag-of-words model. We investigated an approach for selecting the most informative training samples per concept for classi cation and the impact of fusing the OpponentSIFT feature with the GIST feature which calculates global image statistics, on scene-based concepts. We also incorporated a post-classi cation processing step, which re ned classi cation results based on rules of inference and exclusion between concepts. The di erent approaches provided classi cation gains when compared to the standard bag-of-words model using only the OpponentSIFT feature.

Introduction

The ImageClef Photo Annotation Task is an annual competition, which draws interest from the Computer Vision research community, with the aim of addressing the problem of e cient annotation of large scale image collections. The task achieves this by inviting competition from di erent research institutions to provide solutions for the automatic classi cation of photos taken from the Flickr1 community into di erent categories.

The 2010 ImageClef PhotoAnnotation Task [ 6 ] focused on providing annotations to a testset of 10000 images using 93 concepts. These concepts were mostly object-based e.g. dog, cat, scene-based e.g. landscape, beach, event-based e.g. work, travel, quality-based e.g. blurry, overexposed or representation-based e.g. portrait, art. Our experiments [ 5 ] in last year's task were based on the standard bag-of-words model for image classi cation. We performed a feature fusion of the OpponentSIFT [ 9 ] local feature and the no-reference objective image sharpness measure [ 3 ], which showed classi cation gains with the most gains occuring among the quality-based concepts. We also performed a post-classi cation step, which was based on the observation that many of the categories could be seen as being related to one other, thereby enabling the inference or exclusion of other categories.

In this year's task, the list of concepts was extended to 99 including sentimentbased concepts e.g. sad and happy. Our experiments in this year's task built on our approaches of last year. We performed a feature fusion of the OpponentSIFT descriptor and the GIST descriptor, with the target being to improve the classi cation performance of scene-based concepts, since the GIST descriptor has successfully been applied to scene classi cation and recognition [ 8 ]. Computation of the descriptor involves accumulating image statistics over the entire scene rather than over local regions. We also evaluated an approach to select the most informative training samples from the training set per category to train the classi ers, which resulted in qualitative as well as runtime performance gains. We will refer to this approach as Smart Sampling (SS) in the rest of this paper. Finally we used optimized versions of our post-classi cation processing algorithms of last year's task in this year's detection system.

In the following, section 2 describes our concept detection system in more detail and outlines the di erences to our system of last year while section 3 summarizes this paper by giving some results and providing an outlook into future work. 2

Concept Detection System

Our concept detection system for this year's task was an optimized version of last year's system, which integrated a fusion of the OpponentSIFT descriptor and the GIST descriptor, the SS approach for e cient training sample selection per category and an optimized post-classi cation processing step. For a detailed description of last year's system architecture, including details of the OpponentSIFT feature refer to [ 5 ].

The spatial envelope model was initially introduced in [ 8 ] as a low-dimensional representation of a scene and successfully applied for k-nearest-neighbor scene classi cation. The idea was to represent the dominant spatial structure of a scene rather than applying segmentation or any kind of processing of individual objects or regions. The authors propose a set of perceptual dimensions (naturalness, openness, roughness, expansion, ruggedness) that represent the dominant spatial structure of a scene. The spatial envelop model was later termed the GIST descriptor following Friedman's [ 4 ] de nition of a scene gist; an abstract representation of the scene that spontaneously activates memory representations of scene categories (a city, a mountain, etc.), which is essentially what is captured by the spatial envelope model [ 7 ]. Extraction of GIST is performed by ltering the image by a bank of Gabor lters. The image is split into a 4 4 regular grid and the Gabor lter responses are averaged over each block.

We use the implementation presented in [ 2 ] which takes a squared gray-level image of xed size as input. All images are rescaled initially to 256 256 irrespective of their aspect ratios. We obtain a GIST feature vector of 512 dimensions per image, which is signi cantly less than the bag of keypoints vectors at even the smallest grid resolution. Moreover, the GIST features can be computed much more e ciently as there are no codebook and histogram computation steps involved. Fusion of the GIST descriptor and the OpponentSIFT descriptor was done analogously to our fusion of the OpponentSIFT descriptor and the noreference objective image sharpness measure in last year's system.

We further applied a Smart Sampling optimization step to all classi ers. Smart Sampling selects the most informative training images from the training set in order to train a classi er faster and more e ciently [ 1 ]. This occurs in an iterative process, whereby the training set is divided into a number of subsets and the classi er is trained using one of the subsets and used to classify another subset. For every further iteration, the training set is composed of the previous iteration's training images and classi ed images having a classi er con dence of jcj < 1, and with classi cation performed on a new subset. Through this process, we select training samples from the whole training set which lie in close proximity to the separating hyperplanes of the classi er for each category. This led to an increase in runtime and qualitative performance.

Finally, we performed further tests to optimize the Exclusion and Inference rules, which we used in the post-classi cation processing step of last year's system, while adapting them to the newly added categories of this year's task. Changes in the category list of this year, led to the need for modi cations of the post-classi cation processing algorithms. Last year it was possible to identify groups of categories, with each image being able to belong to only one category in each group. With the exclusion of categories such as No Visual Season this year, the same groups as in last year could no longer be identi ed. Consequently, the following equation which was assumed to hold for all groups of excluding categories was invalidated. (1) (2) (3) The equation was modi ed to [ Cp = I p2P [ Cp p2P

I which further leads to di erent update rules for con dences. Rather than c0(i; p) = c(i; p) 0 if c(i; p) > c(i; q) else 8q 2 P n p where the maximum con dence for a category is maintained and all other con dences in a group of excluding categories are set to 0, the following update rule was used.

c0(i; p) = c(i; p) 0 if c(i; p) > c(i; q) and c(i; q) > 0:73105 else 8q 2 P n p (4) The treshold 0.73105 was chosen because it is the value of the sigmoid function applied to 1.0, i.e. the exclusion rule is only used if the maximum category condence for a given image is outside the tube around the separating hyperplane. For category inference, the system from last years submission was maintained. However, the rule con dence threshold (i.e. the threshold that toggles if a rule is used) was tuned. Last year a threshold of 0.99 was used, meaning that a rule was only used if its con dence in the training set was at least 0.99. We optimized this threshold to maximize the Example-based F-Measure. The maximum gain was reached using a value of 0.63 for this threshold. The update rule was modi ed analogously to the exclusion update rule given before, by applying rules only if the con dence of the rules' left-hand side category con dence was above 0.73105 (i.e. outside of the tube around the separating hyperplane). 3

Results and Summary

We submitted 5 di erent runs. All runs use the OpponentSIFT histograms as baseline. The rst run (OpSIFT ) uses the OpponentSIFT feature alone. Another run (OpSIFT+Excl+Inf+SS ) uses the Smart Sampling optimization and applies category inference and exclusion as a post-classi cation processing step. A third run (OpSIFT+Gist ) does no post-classi cation processing but uses the GIST feature as an additional feature. The fourth run (OpSIFT+SS ) uses the Smart Sampling optimization step in addition to the OpponentSIFT feature. A nal fth run (OpSIFT+SS+Inf ) uses the OpponentSIFT features with the Smart Sampling optimization step and applies the inference rule on the classi cation results.

Three di erent evaluation measures were computed. For evaluating the classi cation performance per concept the Mean Average Precision (MAP) was used. The evaluation per example was performed using the example-based F-Measure (F-Ex) and the Semantic R-Precision (SR-Precision).

Table 1 shows the average scores achieved for each measure. For all evaluation measures, all runs with extensions to the baseline OpponentSIFT method resulted in performance gains.

Run-Con guration MAP Avg. F-Ex SR-Precision OpSIFT 0.325111 0.579904 0.71262264 OpSIFT+GIST 0.335234 0.588042 0.71764547 OpSIFT+SS 0.326483 0.580804 0.71251965 OpSIFT+Excl+Inf+SS 0.325981 0.580729 0.71252900

OpSIFT+SS+Inf 0.325981 0.580729 0.71252900

In order to observe the in uence of the GIST feature on scene-based concepts, we compared the MAP for those concepts out of the overall 99 concepts we considered as scene-based concepts (`Building Sights', `Citylife', `Landscape/ Nature','Indoor',`Outdoor','Mountains', `Sunset/ Sunrise', `Park/ Garden', `Beach/ Holidays') with their baseline (using only the OpponenSIFT descriptor) values. This is depicted in table 2. We observe that using the GIST feature together with the OpponenSIFT feature yields a gain of 0.00807 compared to the baseline alone.

Category Park/Garden Sunset/Sunrise Mountains Outdoor Indoor Landscape/Nature Citylife Building Sights Beach/Holidays Mean Average Precision 0.578772

OpSIFT OpSIFT+GIST For those categories where the results di er, the best performing run is highlighted in the table. In terms of MAP per category, the exclusion of categories often performed worse than the other runs. For categories especially, where the average precision was already low, the exclusion rule worsened the results. We attributed this to a lack of reliability of the SVM con dence outputs. The SVM classi er of bad performing categories had a very small output range, e.g. values ranging from 0.94 to 0.96. In such cases, the number of support vectors used for the categories was usually near the total number of training samples. Using these categories for inference or exclusion of other categories yielded a signi cant decrease in performance, propagating the error introduced by one categorie's classi er to other categories. In the future, a check for the reliability of classi er outputs should be performed to avoid such error propagation. This also holds for the category inference post-processing rule, which also su ers from this problem.