SZTAKI @ ImageCLEF 2008 Visual Concept Detection∗

Abstract

We describe our approach to the ImageCLEF-VisualConcept 2008 task. Our method is based on image segmentation, using a feature vector describing the visual content of image segments or the entire image. Logistic regression was used for classication. Images were segmented by a home developed segmenter. While in this preliminary report classication by global image features performed best, preliminary results suggest the importance of segmentation for certain classes. We are planning to provide improved analysis in the near future.

Keywords

• H
• 3 [Information Storage and Retrieval ]
• H
• 3
• 1 Content Analysis and Indexing
• H
• 3
• 3 Information Search and Retrieval
• H
• 3
• 4 Systems and Software
• H
• 3
• 7 Digital Libraries

Introduction

In this paper we describe our approach to the ImageCLEF Visual Concept 2008 evaluation campaign over the IAPR TC-12 Benchmark [ 6 ]. The Visual Concept Detection Task has the objective to identify visual concepts. Both the training and test-set was a part of the IAPR TC-12 database. 1,800 images were published, which were classied according to a small concept hierarchy with 17 concepts. The test database consisted of 1,000 images. For each of these images it was required to determine the presence or absence of the concepts.

Our method is based on our approach to the object classication track of the previous year ([ 4 ]). As a main dierence, we used global features in addition to segment based ones, since concepts such as night and day are characterized by the entire image. For this reason our CBIR method is based on segmentation of the image and on the comparison of features globally as well as segmentwise. While may of the existing CBIR systems rely on so called blobs, regions, or segments [ 3, 8, 2, 7 ], the specialty of our method is our special segmentation method and the combining of global and segment based approach.

∗This work was supported by the EU FP7 project JUMAS Judicial Management by Digital Libraries Semantics and by grants OTKA NK 72845 and NKFP-07-A2 TEXTREND.

Due to processing and classication costs we show preliminary results only that we plan to revise in the near future. As a main issue, we were not able to perform method selection and blending on a separate heldout set that, as expected, resulted in overtting both for our classicator combination and for our segment ltering methods. 2

Visual feature generation

Our CBIR system [ 4, 1 ] relies on so called blobs, regions or segments. Classes such as building or people are classied by extracting specic features from the segments. For segmentation we use the code of the Felzenszwalb and Huttenlocher [ 5 ] graph-based method. Global classes such as outdoor are classied by using the entire image as a single segment.

By the distinction of classes that characterize global and local features of the image, respectively, we experimented with the number and size of the segments starting from a single segment per image for global classes down to a very large number of segments. After resizing images to a size of maximum 500x500 by keeping the aspect ratio, we tuned the minimum segment size and the cut parameters of the FelzenszwalbHuttenlocher algorithm to select a small and a medium granularity segmentation. The small version resulted typically in more than 100 while medium in less than 100 segments per image. The minimum segment size is 50 pixels for small and 1500 for medium.

The runs submitted also dier in the features used to characterize the segments. We use mean color, RGB histogram and the 2D Fourier transform of the image in addition to shape values formed by converting segments to binary pattern, then resizing to 10x10 so that binary values are converted to grayscale values proportionally. glob1: 33 values per image for mean color (RGB) and a 10-bin histogram for all the 3 channels (RGB). No segmentation is performed. glob2: 173 values per image for mean color (RGB), a 20-bin histogram, 2x5 contrast (5 maximal and 5 minimal values of L-channel in HSL color-space) and 100 values of a 2D Fourier transform (sampled along zig-zag). No segmentation is performed. medium: 135 values per segments for mean color, 3x10 histogram and 10x10 shape. Segments are of medium size, i.e. less than 100 in number per image. small: Same as medium with small size segments, i.e. more than 100 segments per image. 3

Classication

We use logistic regression for classication with the global or segment features as input. The output real value is interpreted as the probability of the image or segment belonging to the specic class. For a single image we averaged the segment based predictions, which turned out more accurate than either the minimum or the maximum. Finally, a threshold of 0,5 was applied to get binary values. We did not use the logical information included in the class hierarchy, which could improve our method.

In our mixed run for each class we used the classier that performed best on the training data. Due to time constraints we did not use a heldout set, which resulted in overtraining for this run. By closer analysis the glob1 run was overtrained the most. By replacing glob1 by glob2 the combined performance improved over the best single run even in this overtrained scenario. The explanation for the overtraining for glob1 may lie in the low number of features used. 4

Results

EER AUC ppnn 36.07 67.15 EER AUC

Conclusion and future work

In summary we may observe best overall performance for the high dimensional global feature space, closely followed by the medium resolution segmentation. We also reached improvement (although not among the submitted runs) by combination. Results in this report are preliminary and we are planning to rerun all our classicators by using separate heldout sets for segment ltering and combination.

References

[1] AndrÆs Benczoer , IstvÆn Br, MÆtyÆs Brendel, CsalogÆny KÆroly, BÆlint Darczy, and DÆvid Siklsi. Cross-modal retrieval by text and image feature biclustering . In Working Notes for the CLEF 2007 Workshop , Budapest,Hungary, 2007 .

[2] Chad Carson , Serge Belongie, Hayit Greenspan, and Jitendra Malik . Blobworld: Image segmentation using expectation-maximization and its application to image querying . IEEE Trans. Pattern Anal. Mach . Intell. , 24 ( 8 ): 10261038 , 2002 .

[3] Yixin Chen and James Z. Wang . Image categorization by learning and reasoning with regions . J. Mach. Learn. Res. , 5 : 913939 , 2004 .

[4] Thomas Deselaers , Allan Hanbury, Ville Viitaniemi, AndrÆs Benczoer, MÆtyÆs Brendel, BÆlint Darczy, Hugo Jair Escalante Balderas, Theo Gever, Carlos Arturo HernÆndez Gracidas, Steven C. H. Hoi , Jorma Laaksonen, Mingjing Li , Heidy Marisol Marin Castro, Hermann Ney, Xiaoguang Rui, Nicu Sebe, Julian Stttinger, and Lei Wu . Overview of the imageclef 2007 object retrieval task . In Working Notes for the CLEF 2007 Workshop , Budapest,Hungary, 2007 .

[5] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Ecient graph-based image segmentation . International Journal of Computer Vision , 59 , 2004 .

[6] Michael Grubinger , Paul Clough, Henning Mller, and Thomas Deselears . The IAPR TC-12 benchmark - a new evaluation resource for visual information systems . In OntoImage, pages 1323 , 2006 .

[7] Qin Lv , Moses Charikar, and Kai Li . Image similarity search with compact data structures . In CIKM '04: Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management , pages 208217 , New York, NY, USA, 2004 . ACM Press.

[8] B. G. Prasad , K. K. Biswas , and S. K. Gupta . Region-based image retrieval using integrated color, shape, and location index . Comput. Vis. Image Underst . , 94 ( 1-3 ): 193233 , 2004 .