We introduce our participation at the ImageCLEF 2015 scalable concept detection and localization task. This edition focuses on generating not only annotations (concept detections) but also localizing concepts into a large image collection. Concept detection part of our runs is based on standard nonlinear support vector machines (SVMs). The localization part is blind and based on a priori learned statistics that generate multiple localization proposals. In spite of its blindness, the performance of this concept localization framework is promising.
The general problem of visual category recognition generally includes three
different tasks: concept detection (also known as image annotation) [1{3], concept
localization [
We are interested in this paper in concept detection and localization; we
present our solutions submitted to the ImageCLEF 2015 scalable concept
image annotation task [
Among existing object localization (and segmentation) methods those based on region proposals are currently receiving a particular attention. Their general principle consists in de ning multiple partitions of test images into sets of blobs that potentially correspond to actual objects. Only few of these partitions are scored and used to annotate and localize concepts in test images. Even though relatively successful, these approaches are highly dependent on the quality of image segmentation, which is known to be challenging especially when no a priori information is used about the statistics of these concept-localization proposals. Our proposed solution, discussed in this paper, avoids image segmentation and it is based on two steps: rst, we train SVM classi ers that detect concepts belonging to di erent test images. Afterwards, we use an priori (trained) statistical model in order to infer their most likely locations, without observing the content of these test images. We will show that in spite of the simplicity of this approach, the results are reasonably decent, and very promising, and this opens a new direction towards re ning these models and obtaining better performances by combining annotation and concept localization results.
The rest of this paper is organized as follows; rst, we describe our concept detection algorithm, based on SVMs and an e cient evaluation of the histogram intersection kernel. Then, we describe our a priori statistical model for blind concept localization, and we present and discuss our ImageCLEF 2015 results. Finally, we conclude the paper, with possible extensions for a future work.
conceptN SVMN concept1 SVM1
conceptN LocN concept1 Loc1 s c itirs u e h n o ti a z li a c o L 5 s n u R 2 x 5 concept1 SVM1 a itirr e c n o it c e t e d t p e c n o C 2 concept1 Loc1 conceptN SVMN conceptN
LocN TEST IMAGE conceptN conceptN conceptN concept1 concept1 concept1
TEST IMAGE conceptN concept1 concept1 concept1 concept1 concept1
Our concept detection and localization results are obtained according to the two following steps (see Fig. 2): i) Holistic concept detection: this step is achieved using global (holistic) visual and textual features. For that purpose, we train \one versus all" SVMs for each concept, in order to detect whether that concept exists in a given test image (see extra details in Section 2.1). ii) Blind concept localization proposals: in contrast to concept detection, concept localization is achieved blindly, i.e., without observing the content of a given test image. As will be shown subsequently, localization is achieved using a priori knowledge about possible locations of these bounding boxes. These knowledges correspond to learned localization statistics, of bounding boxes, taken from a training/dev set of concepts and their associated bounding boxes (i.e., from the le \imageclef2015.dev.bbox.v20150226"; see extra details in Section 2.2). 2.1
Holistic concept detection: training and classi cation We used only the holistic features provided in this ImageCLEF task including GIST, Color Histograms, SIFT, C-SIFT, RGB-SIFT, OPPONENT-SIFT, etc. We build 10 gram matrices (9 visual and 1 textual), based on e cient histogram intersection kernel, associated to these features. Then, we linearly combine those matrices into a single one. Notice that this combination does not result from multiple kernel learning but just a convex combination of kernels with uniform weights. We plug the resulting kernel into SVMs for training and testing. For each concept, we train \one-versus-all" SVM classi ers; we use many random folds (taken from training/dev data in \imageclef2015.dev.bbox.v20150226") for multiple SVM training and we use these SVMs in order to predict the concepts on the test set1. We repeat this training process, for each concept, through di erent random folds from the training set and we take the average scores of the underlying SVM classi ers. This makes classi cation results less sensitive to the sampling of the training set and also allows us to re-balance classi cation results mainly for concepts with unbalanced distributions of positive and negative data. 2.2
Blind concept localization proposals Several heuristics are tried in order to suggest multiple concept localization proposals. Given a test image and the list of concepts attached to it (see Section 2.1), concept localization is achieved without consulting the content of the test image (but only its detected concepts). Indeed, concept localization is blind and bounding boxes (BBs) are either xed (using test image dimensions) or based on statistics estimated o ine on the training/dev set (in \imageclef2015.dev.bbox.v20150226") as described subsequently.
In what follows, pc = (x; y; w; h) denotes the bounding box coordinates of a given detected concept c in a given test image; here (x; y) (resp. (w; h)) corresponds to the center (resp. dimensions; width and height) of the bounding box pc. Heuristic 1 ( xed BBs): for a detected concept c in a given test image, its bounding box pc is set to (W=2; H=2; W; H); here W and H respectively denote the width and the height of the test image. In what follows, we consider that all the test images are re-sized and have the same dimensions (i.e., W , H are constant for all the test images)2.
In the subsequent heuristics (2{5), we introduce the following notation: given a concept c, we consider Tc = fpicgi as the union of all the bounding boxes (in the training/dev set \imageclef2015.dev.bbox.v20150226") that belong to c. We also 1 A given test image is assigned to a given concept, i the underlying SVM score is positive. 2 Of course, the actual dimensions of the test images are taken into account in order to re-scale concept localization results. consider Nc as the average number of bounding boxes (par image) associated to c; Nc is evaluated from the training set. Prior to use the following heuristics (2{5), we consider an o ine step that clusters the coordinates in Tc (using kmeans) with a number of clusters xed to Nc (see Fig. 3, left).
Heuristic 2 (concept-dependent BBs): for a detected concept c in a given test image, we generate Nc bounding boxes whose coordinates correspond to the cluster centers obtained after applying k-means on Tc.
In the remaining three heuristics (3{5), we update the coordinates of the bounding boxes, by manipulating i) their dimensions in heuristic 3, ii) their centers in heuristic 4, and iii) both their centers and dimensions in heuristic 5. Heuristic 3 (re-scaled concept-dependent BBs): each bounding box pc = (x; y; w; h) generated in heuristic 2, is replaced by re-scaled BB. First, principal component analysis (PCA) is applied o ine to the BB dimensions f(wi; hi)gi in the training set that also belong to concept c, afterwards, the dimensions (w; h) of pc are moved towards the rst principal component of PCA3, with an amplitude proportional to its eigenvalue (and this corresponds a re-scale of the dimensions of pc). In this heuristic (x; y) remains unchanged (see Fig. 3, right). Heuristic 4 (shifted concept-dependent BBs): for each bounding box pc = (x; y; w; h) generated in heuristic 2, we generate two extra BBs, with shifted coordinates. Again, PCA is applied o ine to the BB coordinates f(xi; yi)gi in the training set that also belong to concept c, afterwards, the (x; y) coordinates of pc are shifted towards two opposite directions corresponding to the rst principal component of PCA. In this heuristic (w; h) remains unchanged (see Fig. 3, middle).
Heuristic 5 (shifted and re-scaled concept-dependent BBs): this heuristic corresponds to the combination of the two heuristics 3 and 4. 3 i.e., the eigenvector with the largest eigenvalue.
The targeted task is, again, concept detection and localization: given a picture, the goal is to predict which concepts (classes) are present into that picture and a proposal of bounding boxes surrounding these concepts. 3.1
ImageCLEF 2015 Collection A very large amount of images was gathered by the organizers, and using associated web pages, tags and meta-data were also provided. This set includes 500k images with only 2k images with known ground truth (i.e., labels and bounding boxes are given). These images belong to 251 concepts (see example in Fig. 1). Each image is again described with nine holistic visual features provided by the organizers, and we compute one extra textual feature using a normalized vector space model; rst, a vocabulary of keywords V is de ned4 in order to query the associated meta-data that include 500k textual descriptions. For each keyword ! 2 V, only images whose textual descriptions include ! have their ! vector entry set to non-zeros. 3.2
Submitted Runs All our submitted runs are based on SVM training and classi cation with the same kernel function (i.e., histogram intersection kernel), and the di erences reside in the used decision criteria for concept detection and localization. Our ten submitted runs correspond to the combination of the ve concept localization heuristics described earlier (see Section 2.2) and the two following concept detection criteria i) Criterion 1 (C1): the rst concept detection results are obtained by following the setting in Section 2.1. ii) Criterion 2 (C2): the second set of concept detection results is obtained using a slightly di erent criterion; more precisely, if an image has no detected concepts, i.e., all the SVM scores are negatives for all concepts, then we select the top 3 concepts (i.e., with the highest negative SVM scores) as annotations for that image. This makes it possible to increase the recall, with a possible impact on the precision.
Our runs are summarized in table 1. For all the submitted runs, performances are evaluated, by the organizers, using a variant of the Jaccard measure; the latter is de ned as the intersection over union of bounding boxes provided in the submitted runs and those in the ground truth. Mean average precision (MAP) measures based on di erent percentages of bounding box overlaps are given for each concept and also averaged through di erent concepts (see our results in 4 Including relevant keywords that are used in concept de nitions. 2X01X5 XreXsuXltsX)O.XveXrlaXp Runs # 5 (heuristic 3+C1) 9 (heuristic 5+C1) 3 (heuristic 2+C1) 7 (heuristic 4+C1) 1 (heuristic 1+C1) 6 (heuristic 3+C2) 10(heuristic 5+C2) 4 (heuristic 2+C2) 8 (heuristic 4+C2) 2 (heuristic 1+C2) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% { Di erent methods for \concept localization proposals" provide much better results when concept detection is relatively successful (see runs 1, 3, 5, 7, 9 vs runs 2, 4, 6, 8, 10 in table 2 for di erent overlap ratios). Following the spirit of our two-step method, these results clearly corroborate the fact that concept detection could be decoupled from localization as long as concept detection is achieved with a relative success. This clearly opens a direction towards enhancing the performances of localization proposals by further improving concept detection results. { From table 2, heuristic 3 (BB re-scaling) provides the best overall performances; indeed, even though shifting is important, it has less impact on performances compared to re-scaling. This is mainly due to the variability and non-rigidity of many concepts (such as animals), that require an adaptation of the dimensions of BBs, while shifting is already well captured by the statistical model (in heuristics 2, 4, 5); see again k-mean clustering in Section 2.2. 5 http://www.imageclef.org/2015/annotation. { From table 3, for almost all the concepts, statistical bounding box estimation (i.e., heuristics 2, 3, 4, 5) is very helpful in order to improve the quality of localization; for some concepts such as \frog", re-scaling and shifting are important, as this category is highly non-rigid while for other categories such as \bear", adaptation does not improve performances as \bear" localization is less predictable. Note also that for rigid (and man-made) objects, such as \cathedral" and \bicycle", re-scaling is more important than shifting as the proportions of the w-h dimensions, in these concepts, are very changing while for others (including natural objects and also some other man-made objects such as \airplane", \balloon", \bucket", \camera"), the adaptation of shift is more important than scale; as the variability of w-h proportions is small in these concepts. In sum, bounding box re-scaling and shifting is important for some concepts and less for others. This suggests, as a future extension, to mix di erent heuristics for di erent concepts (and we already observe this improvement in the \concept-by-concept" results). 4
We discussed in this paper, our participation at the ImageCLEF 2015 Scalable Concept Image Annotation Task. Our runs are based on a two-step process that decouples concept detection from localization. The former is achieved using SVMs trained with linear combination of elementary histogram intersection kernels, while the latter is accomplished blindly using a simple statistical model that allows us to generate multiple localization proposals (without image segmentation). Observed results show that i) the accuracy of concept detection has an impact on the performance of localization, and ii) the adaptation of scale and shift of concept localization is essential to improve performances mainly for concepts with a large variability in their extents.
A future possible extension, of this work, is to make concept localization nonblind and also coupled with concept detection. Another possible extension is to mix and select di erent localization heuristics for di erent concepts. Acknowledgments. This work is supported in part by a grant from the French Research Agency ANR (Agence Nationale de la Recherche) under the MLVIS project. 18. Juergen Gall and Victor Lempitsky, \Class-speci c hough forests for object detection," in Decision Forests for Computer Vision and Medical Image Analysis, pp. 143{157. Springer, 2013. 19. Xuming He and Stephen Gould, \An exemplar-based crf for multi-instance object segmentation," in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014, pp. 296{303. 20. Stan Z Li, Markov random eld modeling in Image Analysis (was: Markov random eld modeling in computer vision)., 2011. 21. ImageNET webpage. http://image-net.org/about-publication.