CNRS TELECOM ParisTech at ImageCLEF 2016 Scalable Concept Image Annotation Task: Overcoming the Scarcity of Training Data

Abstract

We introduce our participation at the ImageCLEF 2016 scalable concept detection and localization task. As in ImageCLEF 2015, this edition focuses on generating not only annotations (concept detection) but also localizing concepts into a large image collection. In our runs, we focus mainly on concept detection; our solution is purely visual and based on deep features combined with standard linear support vector machines (SVMs) built on top of well enriched training sets. Starting from loosely labeled training sets, we propose an algorithm that learns the statistical dependencies between concepts and allows us to enrich the labels of these training sets, resulting into more e ective SVMs for image annotation.

Keywords

• label enrichment
• SVMs
• deep learning
• image annotation

-

Automatic image annotation is one of the major challenges in computer vision and machine learning. It consists in learning intricate relationships between keywords (a.k.a concepts/labels/categories) and training images, in order to assign list of keywords to newly observed visual contents (see for instance [1{5]). These concepts may either correspond to well de ned physical entities (pedestrians, cars, etc.) or to high level, ne-grained notions resulting from the interaction of many entities into scenes (parties, ghts, etc.). In both cases, image annotation is challenging due to the perplexity when assigning concepts to scenes especially when the number of possible concepts is taken from a large vocabulary, when training data are scarce and also when analyzing highly semantic and variable content.

Early image annotation techniques are content-based (e.g. [6{9]). They model straightforward \concept-image" relationships and learn how to assign concepts to new images; they rst describe image observations using visual features1, treat each concept as an independent class, and then train the corresponding 1 either handcrafted such as color, texture, etc. or learned such as deep features [ 10 ] concept-speci c classi er to identify (separately) images belonging to that concept using a variety of machine learning and inference techniques, either generative or discriminative [11{16, 9, 17{24, 10, 25{36]. Extensions of these methods achieve structured output predictions [ 37, 38 ] by modeling not only \conceptimage" relationships, but also \concept-concept" dependencies [39{42]. Indeed, concepts in image annotation are usually interdependent, i.e., the presence of one concept may tell us something about the presence of another one; for instance the presence of the concept \sea" usually implies the presence of other concepts such as \sky" or \sand". Hence modeling the statistical dependencies between concepts (both for training and inference) is crucial and this is usually achieved with graphical models and markov/conditional random elds [ 16 ]. Relationships between concepts can also be modeled by extracting (hand-crafted or learned) mid-level characteristics which are common to di erent concepts. This has recently received a particular attention in the context of deep networks and transfer learning [ 43, 10, 44 ]. However, the lack of labeled data may severely limit the usability of these methods and requires solutions in order to learn from few shots. Hence, learning from common data and characteristics is valuable in order to overcome the scarcity of training data especially when handling image annotation problems with a large number of concepts.

In this paper, we describe the participation of \CNRS-TELECOM ParisTech" at the ImageCLEF 2016 Scalable Concept Image Annotation Task [ 45, 46 ]. Our solution focuses mainly on concept detection; it combines e ective deep features with SVM classi ers. As training data are scarce, we propose a solution that enriches the labels of these training data. This solution is based on measuring the statistical correlation between concepts in the training set and makes it possible to propagate labels to larger training sets. Note that our solution does not require the use of the meta-data associated to training and test data; indeed it is purely visual. In spite of this, the proposed runs are competitive. 2

Our Concept Prediction and Localization at a Glance

Our concept detection and localization results are obtained according to the two following steps: i) Holistic concept detection: this step is achieved using global (holistic) visual features. For that purpose, we train \one versus all" SVMs for each concept, in order to detect whether that concept exists in a given test image (see extra details in Section 4.2). ii) Blind concept localization: concept localization is achieved blindly, i.e., without observing the content of a given test image. In contrast to our last year participation [ 47 ], we did not investigate heuristics for concept localization and we use the whole image dimensions as bounding boxes; this turns out to be su cient for many concepts as discussed in Section 4.2. So in this participation, Fig. 1. (Top) Sample of pictures taken from the ImageCLEF2016 database (dev set). (Bottom) Sample of external pictures collected from the web; the leftmost picture belongs to the category \beach", while the middle and rightmost pictures belong to the categories \anchor" and \apple" respectively. It is clear that these pictures can also be assigned to categories (\sea", \sand", \cloud"), (\boat"), (\tree"), etc. as these concepts are highly correlated with the concepts \beach", \anchor" and \apple" respectively. So these pictures can be reused to train the classi ers of these concepts. we focus on the rst step only (i.e., Holistic concept detection) and mainly issues about enriching training datasets. 3

Training Datasets and Label Enrichment

Besides training data provided in ImageCLEF 2016, we collected automatically an external training set using the \googlebot-image" crawler. This external set consists in 42,272 images belonging to the 251 concepts of ImageCLEF. No post processing of these images was achieved (the whole content is used in order to train our SVM models and without localizing the concepts in images). Figs. 1, 2 show a sample of those images as well as the distribution of the number of images per concept. We also use the 2,000 images of the dev set provided by the ImageCLEF 2016 organizers as an internal training set in order to train and tune the parameters of our SVM models. All images are described using the coe cients of the FC7 layer of the pre-trained VGG network in [ 48 ].

Label enrichment As some concepts are rare, we use the 2,000 images of the dev set in order to enrich the labels of all the training set. The idea is to transfer the knowledge about the co-occurrence of some labels using a simple principle: given two concepts c and c0, if c, c0 are highly correlated, then the presence of one of these two concepts in a given training image implies the presence of the other concept. 00 00 50 100 Concepts 150 200 250 50 here N is the size of the dev set and K is the number of concepts (N = 2; 000, K = 251 in practice) and Y 2 RN K is a matrix whose entry Yic = 1 i the concept c is present into image Ii and Yic = 0 otherwise. As external images are collected using \individual keywords" as queries, they have a single label per image and cannot be used to learn these co-occurrences. In contrast, dev set images have multiple labels and are used instead. Hence, labels in the external set are enriched as follows 8c; c0 2 f1; : : : ; Kg, 8i 2 fN + 1; : : : ; N + N 0g (N 0 = 42; 272), if Yic = 1 and C(c0jc) then Yic0 1 (see Section 4 about the tuning of ).

Fig (1, bottom) shows an example of this label enrichment process, where the presence of concept \apple" implies the presence of the concept \tree". Note that this enrichment process could also be achieved as a post processing step (i.e., after image annotation), however due to shortage of time, this issue has not been investigated. 4

ImageCLEF 2016 Evaluation

The targeted task is, again, concept detection and localization: given a picture, the goal is to predict which concepts (classes) are present into that picture and the coordinates of the bounding boxes surrounding these concepts. 4.1

ImageCLEF 2016 Collection

A very large amount of images was gathered by the organizers, and using associated web pages, tags and meta-data were also provided. This set includes more than 500k images with only 2k images with known ground truth (i.e., labels and bounding boxes are given). These images belong to 251 concepts (see example in Fig. 1). In our runs, each image is again described with a visual feature vector corresponding to the FC7 layer of the VGG pretrained network. Note that the parameters of this network are not ne-tuned on training data and concepts. 4.2

Submitted Runs

All our submitted runs (discussed below) are based on SVM training. For each concept, we trained \one-versus-all" SVM classi ers; we use many random folds (taken from training data) for multiple SVM training and we use these SVMs in order to predict the concepts on the test set. We repeat this training process, for each concept, through di erent random folds from the training set and we take the average scores of the underlying SVM classi ers. This makes classi cation results less sensitive to the sampling of the training set. Given a test image x, a concept c is declared as present into x i fc(x) > , here fc(x) = L1 1fg`(x)>0g and g`() is an SVM classi er trained on a random fold of positive and negative data (in practice L = 10; see also Tab. 1 for the setting of ).

Our ten submitted runs correspond to the combination of ve dataset enrichment strategies (see columns of Tab. 1 and section 3) and two datasets used for SVM training (external and ImageCLEF16/dev set). For all the submitted runs, performances are evaluated, by the organizers, using a variant of the Jaccard measure; the latter is de ned as the intersection over union of bounding boxes provided in the submitted runs and those in the ground truth. Mean average precision (MAP) measures { based on di erent percentages of bounding box overlaps { are given for each concept and also averaged through di erent concepts (see our results in Tables 2, 3, 4, 5, 6). Details about these measures can be found in the ImageCLEF 2016 website2. In contrast to our last year participation, we do not address the issue of bounding box (BB) generation; our bounding boxes cover the whole areas of the test images. We expect further improvement of performances if we consider the BB generation heuristics used last year (as already shown in [ 47 ]).

From all these tables, we observe the following issues: { For all the runs shown in table 2, we observe that combining external data with the ImageCLEF16 dev set provides a clear gain compared to the use of external data only; this may be explained by the fact that the ImageCLEF16 dev set has (possibly) a similar distribution compared to ImageCLEF16 test set, and this makes it possible to adapt training parameters (mainly the SVM weights) to the conditions of the test data. In contrast, the use of external data only does not allow to adapt these SVM parameters appropriately. 2 http://www.imageclef.org/2016/annotation#Results { The same behavior also occurs when considering the performance with 50% overlap (see tables 5, 6). We also notice that concepts which are usually centered in pictures (such as \motorcycle", \kitchen", \shirt") are relatively well localized using our simple blind localization. Other di cult concepts (such as \cat" in table 6) get substantial improvement. It is also clear that better concept detection implies better localization results (see again tables 3, 4 vs. tables 5, 6). { From all these results it is clear that the run \TAB.1.5.res" is better than the run \TAB.1.3.res" as the former is more conservative (i.e., threshold is relatively high) while the latter is less conservative and bene ts from larger training sets. Finally, gures 3, 4 show the concepts for which we obtained the best results among di erent participants in ImageCLEF16. 5

Conclusion

We discussed in this paper, our participation at the ImageCLEF 2016 Scalable Concept Image Annotation Task. In our runs, concept detection is based on deep 3 The correlation factor of a concept is de ned as F (c) = PcK0=1 1fC(c0jc) g.

Our best run

Other best run anchorbootbowblucketbullectandleclockcrossdolldoorforkhkoeoykboarldibrarymirproarintingpenpenciplianpoicturepillowrackramprodrusgusntigclkassesswordwatchwheeleyenefawcsepmapaegrazineletteprotatroibbboannananut Fig. 3. This

gure shows the concepts for which we outperform other participants' runs (blue bars: our best performances, red bars: other participants' best performances on these concepts). These performances correspond to 0 % overlap.

Our best run

Other best run 50 45 40 35 30 25 20 15 10 5 0 anchor blanket bowl bullet candle farmhospital piano ramp rod rug swordewspaper letter salad hill n Fig. 4. This

gure shows the concepts for which we outperform other participants' runs (blue bars: our best performances, red bars: other participants' best performances on these concepts). These performances correspond to 50 % overlap. features combined with linear SVMs trained on well enriched datasets. The enrichment process is based on measuring the co-occurrence of concepts and this makes it possible to reuse training images across correlated concepts. Observed results show that i) the enrichment process has a positive impact on performances especially for concepts with high correlations with others, and ii) the use of both external and provided ImageCLEF16 dev set enhances performances compared to the use of external data only; indeed, in spite of being relatively small, the provided dev set makes it possible to adapt the parameters of our SVM models to the distribution of dev and test data.

A future possible extension, of this work, is to make the enrichment process label dependent, i.e., how to mix and select di erent enrichment strategies for di erent concepts. Another possible extension is to achieve late label enrichment, as a post processing step, by augmenting annotation results on the test set using the same label enrichment strategy.

Acknowledgments. This work was supported in part by a grant from the Research Agency ANR (Agence Nationale de la Recherche) under the MLVIS project ANR-11-BS02-0017.

References

1. A. Torralba , K.P. Murphy , and W.T. Freeman, \ Sharing visual features for multiclass and multiview object detection," IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) vol 25 , issue 5, 2007 .

2. F. Kang , R. Jin , and R. Sukthankar , \ Correlated label propagation with application to multi-label learning," in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. IEEE , 2006 , vol. 2 , pp. 1719 { 1726 .

3. J. Li and J. Wang , \ Real-time computerized annotation of pictures," Pattern Analysis and Machine Intelligence , IEEE Transactions on, vol. 30 , no. 6 , pp. 985 { 1002 , 2008 .

4. V. Lavrenko , R. Manmatha , and J. Jeon , \ A model for learning the semantics of pictures," In: Proc. of NIPS , 2004 .

5. S. Moran and V. Lavrenko , \ A sparse kernel relevance model for automatic image annotation," International Journal of Multimedia Information Retrieval , vol. 3 , no. 4 , pp. 209 { 229 , 2014 .

6. C. Carson , M. Thomas, S. Belongie , J.M. Hellerstein , and J. Malik , \ Blobworld: A system for region-based image indexing and retrieval," In Third International Conference on Visual Information Systems , pp. 509 { 516 , 1999 .

7. J. Zhang , M. Marszalek , S. Lazebnik , and C. Schmid , \ Local features and kernels for classi cation of texture and object categories: A comprehensive study," In Proceedings of the Beyond Patches workshop, in conjunction with CVPR2006 , 2006 .

8. J. Jeon , V. Lavrenko , and R. Manmatha , \ Automatic image annotation and retrieval using cross-media relevance models," In: Proc. of ACM SIGIR , pp. 119 - 126 , 2003 .

9. G. Carneiro , A.B. Chan , P.J. Moreno , and N. Vasconcelos , \ Supervised learning of semantic classes for image annotation and retrieval," Pattern Analysis and Machine Intelligence , IEEE Transactions on, vol. 29 , no. 3 , pp. 394 { 410 , 2007 .

10. A. Krizhevsky , I. Sutskever , and G. Hinton, \ Imagenet classi cation with deep convolutional neural networks," in Advances in Neural Information Processing Systems 25 , 2012 , pp. 1106 { 1114 .

11. K. Barnard , P. Duygululu , D. Forsyth , D. Blei , and M. Jordan , \ Matching words and pictures," The Journal of Machine Learning Research , 2003 .

12. H. Sahbi, \ Network-dependent image annotation based on explicit contextdependent kernel maps," in Pattern Recognition (ICPR ), 2014 22nd International Conference on. IEEE, 2014 , pp. 625 { 630 .

13. D.M. Blei and M.I. Jordan , \ Modeling annotated data," in Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval , New York, NY, USA, 2003 , SIGIR ' 03 , pp. 127 { 134 , ACM.

14. O. Yakhnenko and V. Honavar , \ Annotating images and image objects using a hierarchical dirichlet process model," in Proceedings of the 9th International Workshop on Multimedia Data Mining: held in conjunction with the ACM SIGKDD 2008 . ACM, 2008 , pp. 1 { 7 .

15. J. Li and J. Z. Wang , \ Automatic linguistic indexing of pictures by a statistical modeling approach," IEEE Trans. on PAMI , vol. 25 , no. 9 , pp. 1075 { 1088 , 2003 .

16. X. He , R.S. Zimel , and M.A. Carreira , \ Multiscale conditional random elds for image labeling," In CVPR , 2004 .

17. F. Monay and D. GaticaPerez, \ Plsa-based image autoannotation: Constraining the latent space," in Proc. of ACM International Conference on Multimedia , 2004 .

18. Y. Wang and S. Gong , \ Translating topics to words for image annotation," In: Proc. of ACM CIKM , 2007 .

19. E. Chang , K. Goh , G. Sychay, and G. Wu, \ Cbsa: content-based soft annotation for multimodal image retrieval using bayes point machines," Circuits and Systems for Video Technology , IEEE Transactions on, vol. 13 , no. 1 , pp. 26 { 38 , 2003 .

20. H. Zhang , A.C. Berg , M. Maire , and J. Malik , \ Svm-knn: Discriminative nearest neighbor classi cation for visual category recognition," in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. IEEE , 2006 , vol. 2 , pp. 2126 { 2136 .

21. M. Guillaumin , T. Mensink , J. Verbeek , and C. Schmid , \Tagprop: Discriminative metric learning in nearest neighbor models for image auto-annotation," in Computer Vision , 2009 IEEE 12th International Conference on. IEEE , 2009 , pp. 309 { 316 .

22. A. Makadia , V. Pavlovic , and S. Kumar , \ A new baseline for image annotation," in Computer Vision{ECCV 2008 , pp. 316 { 329 . Springer, 2008 .

23. T. Mei, Y. Wang , X.S. Hua , S. Gong , and S. Li , \ Coherent image annotation by learning semantic distance," in Computer Vision and Pattern Recognition, 2008 . CVPR 2008 . IEEE Conference on . IEEE, 2008 , pp. 1 { 8 .

24. D. Grangier and S. Bengio , \ A discriminative kernel-based approach to rank images from text queries," Pattern Analysis and Machine Intelligence , IEEE Transactions on, vol. 30 , no. 8 , pp. 1371 { 1384 , 2008 .

25. C. Cusano , G. Ciocca, and R. Schettini , \ Image annotation using svm," in Electronic Imaging 2004 . International Society for Optics and Photonics , 2003 , pp. 330 { 338 .

26. Y. Gao , J. Fan , X. Xue , and R. Jain , \ Automatic image annotation by incorporating feature hierarchy and boosting to scale up svm classi ers," in Proc. of ACM MULTIMEDIA , 2006 .

27. H. Sahbi , J.Y. Audibert , and R. Keriven , \ Context-dependent kernels for object classi cation," IEEE transactions on pattern analysis and machine intelligence , vol. 33 , no. 4 , pp. 699 { 708 , 2011 .

28. H. Sahbi, \ Cnrs-telecom paristech at imageclef 2013 scalable concept image annotation task: Winning annotations with context dependent svms ., " in CLEF (Working Notes) , 2013 .

29. P. Vo and H. Sahbi , \ Transductive kernel map learning and its application to image annotation," in BMVC , 2012 , pp. 1 { 12 .

30. H. Sahbi and X. Li , \ Context based support vector machines for interconnected image annotation (the saburo tsuji best regular paper award)," in In the Asian Conference on Computer Vision (ACCV), 2010 .

31. M. Jiu and H. Sahbi , \ Deep kernel map networks for image annotation," in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2016 , pp. 1571 { 1575 .

32. M. Jiu and H. Sahbi , \ Laplacian deep kernel learning for image annotation," in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2016 , pp. 1551 { 1555 .

33. H. Sahbi, \ Imageclef annotation with explicit context-aware kernel maps," International Journal of Multimedia Information Retrieval , vol. 4 , no. 2 , pp. 113 { 128 , 2015 .

34. P. Vo and H. Sahbi , \ Transductive inference & kernel design for object class segmentation," in 2012 19th IEEE International Conference on Image Processing. IEEE , 2012 , pp. 2173 { 2176 .

35. M. Jiu and H. Sahbi , \ Semi supervised deep kernel design for image annotation," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2015 , pp. 1156 { 1160 .

36. H. Sahbi , J.Y. Audibert , J. Rabarisoa , and R. Keriven , \ Context-dependent kernel design for object matching and recognition," in Computer Vision and Pattern Recognition, CVPR . IEEE, 2008 , pp. 1 { 8 .

37. B. Taskar , V. Chatalbashev , D. Koller , and C. Guestrin , \ Learning structured prediction models: A large margin approach," in Proceedings of the 22nd international conference on Machine learning. ACM , 2005 , pp. 896 { 903 .

38. I. Tsochantaridis, T. Joachims , T. Hofmann , and Y. Altun , \ Large margin methods for structured and interdependent output variables," in Journal of Machine Learning Research , 2005 , pp. 1453 { 1484 .

39. S. Nowozin and C.H. Lampert , \ Structured learning and prediction in computer vision," Foundations and Trends R in Computer Graphics and Vision , vol. 6 , no. 3 { 4 , pp. 185 { 365 , 2011 .

40. L. Bertelli , T. Yu , D. Vu , and B. Gokturk , \ Kernelized structural svm learning for supervised object segmentation," in Computer Vision and Pattern Recognition (CVPR ), 2011 IEEE Conference on. IEEE , 2011 , pp. 2153 { 2160 .

41. P. Vo and H. Sahbi , \ Modeling label dependencies in kernel learning for image annotation," in 2014 IEEE International Conference on Image Processing (ICIP) . IEEE, 2014 , pp. 5886 { 5890 .

42. P. Vo and H. Sahbi , \ Contextual kernel map learning for scene transduction," in Image Processing (ICIP) , 2015 IEEE International Conference on. IEEE , 2015 , pp. 3797 { 3801 .

43. P. Duygulu , K. Barnard , J.F.G. deFreitas, and D. Forsyth , \ Object recognition as machine translation: Learning a lexicon for a xed image vocabulary," In: Heyden , A. , Sparr , G. , Nielsen , M. , Johansen , P. (eds.) ECCV 2002 . LNCS , vol. 2353 , pp. 97 - 112 . Springer, Heidelberg, 2002 .

44. S.J. Pan and Q. Yang , \ A survey on transfer learning," Knowledge and Data Engineering, IEEE Transactions on , vol. 22 , no. 10 , pp. 1345 { 1359 , 2010 .

45. A. Gilbert , L. Piras , J. Wang , F. Yan , A. Ramisa , E. Dellandrea, R. Gaizauskas , M. Villegas , and K. Mikolajczyk , \ Overview of the ImageCLEF 2016 Scalable Concept Image Annotation Task," in CLEF2016 Working Notes, Evora, Portugal, September 5-8 2016 , CEUR Workshop Proceedings, CEUR-WS.org <http://ceurws.org>.

46. M. Villegas , H. Muller, A. Garc a Seco de Herrera, R. Schaer , S. Bromuri , A. Gilbert , L. Piras , J. Wang , F. Yan , A. Ramisa , E. Dellandrea, R. Gaizauskas , K. Mikolajczyk , J. Puigcerver , A.H. Toselli , J.A. Snchez , and E. Vidal, \ General Overview of ImageCLEF at the CLEF 2016 Labs," in ' ', Lecture Notes in Computer Science . Springer International Publishing, 2016 .

47. H. Sahbi, \ Cnrs telecom paristech at imageclef 2015 scalable concept image annotation task: Concept detection with blind localization proposals," in CLEF 2015 Evaluation Labs and Workshop, Online Working Notes, 2015 .

48. K. Chat eld, K. Simonyan , A. Vedaldi , and A. Zisserman , \ Return of the devil in the details: Delving deep into convolutional nets," in British Machine Vision Conference , 2014 .