This paper describes the method that was used for our two submission runs for the ImageCLEF Photo Annotation task. Image Features Global Features The following global features were each computed over the entire image. Each feature essential provides a histogram for the image. In features where a quantized HSV space was used, the following quantization parameters were employed: 12 Hue Bins, 3 Saturation Bins, 3 Value Bins. Bins were of equal width in each dimension. This results in a total of 108 bins. The choice of these parameters was motivated by [6] HSV Histogram - 108 dim
The ImageCLEF Photo Annotation 2009 task involved 53 concepts spanning from abstract concepts (aesthetics, blur) to visual elements (trees, people). Although an ontology was provided, our method did not rely on the ontology heavily, except for handling disjoint cases.
Our method follows the framework in [
Color Auto Correlogram (CAC) - 432 dim We computed the CAC over a quantized HSV space with 4 distances 1,3,5,7. This gives us a feature vector of 108*4 = 432 dimensions.
For each color & distance pair (c, d), we computed the probability of nding the same color
at exactly distance d away. Refer to [
Color Coherence Vector (CCV) - 216 dim
We computed the CCV over a quantized HSV space. Since there are two states - coherent and
incoherent, this gives us a feature vector of 108*2 = 216 dimensions. We set the tau parameter to
1% of the image size. Refer to [
Census Transform (CT) - 256 dim The CT histogram is a simple transformation of each pixel into a 8-bit value based on its 8 surrounding neighbors (two states, either >= or < its neighbor). This provides a feature vector of 256 dimensions (histogram of the CT values). Refer to [8] for details. 2.1.5
Edge Orientation Histogram - 37 dim We used the LTI-Lib's Canny Edge detector to compute the edge orientation histogram. Each pixel is assigned to either an edge (with orientation) or non-edge. Orientations are quantized into 5 degree angle bins, giving a total of 36 bins for 180 degrees. An additional bin is concatenated for non-edges. This gives a nal vector of 37 dimensions. 2.1.6
Interest Point Based SIFT - 500 dim
We used the SIFT binary provided by David Lowe [
Densely Sampled SIFT - 1500 dim
We densely sampled SIFT points at 10 pixel spacings, 4 scales (4, 8, 12, 16 px radius) and 1
orientation. The points are similarly quantized into 1500 visual words by k-Means. This scheme
follows that of [
For a number of concepts, the classi cation problem can be framed in the Multiple Instance Learning (MIL) framework. In essence, a concept (e.g. Mountain) exists if and only if a region within the image demonstrates the concept. Hence, one is motivated to consider whether it is possible to improve performance by nding appropriate region(s) to consider in an image.
We de ne a local region to be a bounding box. Given a bounding box in an image, one can compute image features similar to the global features. Hence, we de ne a local region feature to be a feature vector that is extracted based only on region in a bounding box. Therefore, the problem of nding local region features is reduced to one of nding good bounding boxes for each image. 2.2.1
To nd good local region selectors (bounding boxes), we frame the problem in a MIL setting. In this setting, each image is a bag-of-regions and regions are considered to be true i they contain the target concept. Furthermore, a bag is true i it contains a true region.
We used EM-Diverse Density [
For each concept, we learned a local region selector based on the densely sample SIFT features. From each local region, we extract only the HSV, interest point SIFT and densely sampled SIFT histograms. These three histograms form the (concept-speci c) local features for each image. 3 3.1
For the nal classi cation, we used LIBSVM [
K(Si; Sj ) =
X
1 f2features f 2(f (Si); f (Sj ))
Both local and global features were treated in the same manner. f is the average 2 distance for a particular feature; we used it to normalize the distances across di erent features.
We also performed experiments on cost-sensitive SVMs but the results did not vary that much. 3.2
Unsurprisingly, di erent features work well with di erent concepts. To combine di erent features,
one could incorporate weighting into the kernel function. This method is adopted by the INRIA
group in their VOC2007 submission [
We chose a simpler method for feature selection in which a greedy algorithm is used. Furthermore, we do employ any partial weighting scheme for the features. Using equal error rate (ERR) as out performance measure, the algorithm is described as follows:
1. F = all global features 2. For each feature f 2 F : Compute error rate if f is removed 3. Remove the feature which results in best improvement 4. Repeat (2-3) until removing any feature results in worse performance 5. Consider each feature f 2 All F eatures
F : Compute error rate if f is added 6. Consider each feature f 2 F : Compute error rate if f is removed 7. Add or remove the feature which gives best improvement 8. Repeat (5-7) until local optima is reached 9. Return F The appendix contains a list of concepts and the features that were selected for each of them. While there was a new hierarchical measure introduced for the task, we did not speci cally optimize for it. The lack of the annotator agreement values also made it more di cult to optimize for this new measure.
However, for the classes that were speci ed as disjoint, we did simple post processing on the probability estimates to ensure that exactly 1 of the concepts is 0:5. This was achieved by simply moving the probability estimates to 0:5 .
Two nal submissions were made for this task. One used all the features available while the other used only the global features. The o cial results of the two runs in terms of Average Equal Error Rate (Avg. EER), Average Area Under Curve (Avg. AUC), Average Annotation Score with Annotator Agreement (Avg. AS with AA) and Average Annotation Score without Annotator Agreement (Avg. AS without AA) are reported in Table 1.
Based on the Avg. EER measure, our run with all features was ranked 6 out of 74 submitted runs, while the run using only global features was ranked 11. Comparing the best runs from each group, were reported to be third in the list of twenty participating groups. Evaluating our performance based on the Average Annotation Score with and without the use of Annotator Agreement, our runs were ranked second (global features only) and third (all features).
Submission Run
Avg. EER
Avg. AUC
Avg. AS with AA
Avg. AS without AA All Features (CVIUI2R 22 2 1244628714641.txt) Global Features Only (CVIUI2R 22 2 1244629050173.txt) [8] Jianxin Wu and James M. Rehg. Where am i: Place instance and category recognition using spatial pact. Computer Vision and Pattern Recognition, IEEE Computer Society Conference on, 0:1{8, 2008.
Concept 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
Features1 gdsift1500 gsift500 gcac ldsift1500 gdsift1500 gsift500 gcac gccv gcthist gsift500 gcac gccv gcthist gdsift1500 gsift500 gcac gcthist gdsift1500 gsift500 gcac ldsift1500 gdsift1500 gsift500 gcac ghsv gdsift1500 gsift500 gcac gcthist gdsift1500 gsift500 gcac gdsift1500 gcac gccv ghsv gcthist gdsift1500 gsift500 gcac ghsv gcthist ldsift1500 gsift500 gcac gccv ghsv gcthist gdsift1500 gsift500 gcac gccv ghsv lsift500 ldsift1500 gedgehist gdsift1500 gsift500 ghsv gcthist gdsift1500 gsift500 ghsv ldsift1500 gdsift1500 gsift500 ghsv gcthist gdsift1500 gsift500 gccv gdsift1500 gsift500 gccv gcthist gdsift1500 gsift500 gcac gdsift1500 gsift500 gcac ghsv gedgehist gdsift1500 gsift500 ldsift1500 gdsift1500 gcac gdsift1500 gsift500 gccv gdsift1500 gsift500 gccv gcthist gedgehist gdsift1500 gsift500 gcac gccv gsift500 gcac ghsv gcthist gdsift1500 gcac gccv ghsv gcthist gdsift1500 lsift500 ldsift1500 lhsv gdsift1500 gsift500 gcac gsift500 gcac gccv ghsv gcthist gdsift1500 gsift500 gcac gdsift1500 gsift500 gcac gccv gcthist gdsift1500 gsift500 gcac gccv gdsift1500 gsift500 gcthist gdsift1500 gsift500 gcac gcthist gdsift1500 gccv gcthist lsift500 gdsift1500 gsift500 gcac lhsv gedgehist gdsift1500 gsift500 gccv ghsv gdsift1500 gsift500 gccv ghsv gdsift1500 gsift500 gccv gdsift1500 gsift500 gcac gdsift1500 gsift500 gccv ghsv gcthist gdsift1500 gsift500 gcthist gdsift1500 gsift500 gcthist gdsift1500 gsift500 gcac gccv gdsift1500 gsift500 gcthist gedgehist lsift500 lhsv gdsift1500 gsift500 gccv gcthist gdsift1500 gsift500 gcac gdsift1500 gsift500 gcac ldsift1500 gedgehist gdsift1500 gsift500 gcac gccv gcthist gedgehist gdsift1500 gsift500 gcthist ldsift1500 gdsift1500 gsift500 gcac gcthist gdsift1500 gsift500 gcac gccv ldsift1500 lsift500 gdsift1500 gsift500 gcac gccv ghsv ldsift1500 gedgehist