BUAA-iCC at ImageCLEF 2015 Scalable Concept Image Annotation Challenge

Abstract

In this working note, we mainly focus on the image annotation subtask of ImageCLEF 2015 challenge that BUAA-iCC research group participated. For this task, we ¯rstly explore textual similarity information between each test sample and prede¯ned concept. Subsequently, two di®erent kinds of semantic information are extracted from visual images: visual tags using generic object recognition classi¯ers and visual tags relevant to human being related concepts. For the former information, the visual tags are predicted by using deep convolutional neural network (CNN) and a set of support vector machines trained on ImageNet, and ¯nally transferred to textual information. For the latter visual information, human related concepts are extracted via face and facial attribute detection, and ¯nally transferred to similarity information by using manually designed mapping rules, in order to enhance the performance of annotating human related concepts. Meanwhile, a late fusion strategy is developed to incorporate aforementioned various kinds of similarity information. Results validate that the combination of the textual and visual similarity information and the adopted late fusion strategy could yield signi¯cantly better performance.

Keywords

• Textual similarity information
• visual similarity information
• deep convolutional neural network (CNN)
• face and facial attribute detection
• late fusion
• ImageCLEF

-

For the ImageCLEF 2015 Scalable Concept Image Annotation Challenge [ 1, 2 ], we aim to develop a scalable image annotation approach, which could also yield high performance.

As shown in Fig.1, our proposed framework mainly consists of three components: exploration of textual similarity information between each testing sample and prede¯ned concept, visual similarity information extraction and fusion of various kinds of similarity information.

For textual similarity information, we directly utilize the path[ 4 ] semantic distance that is based on WordNet[ 5 ] ontology to construct textual similarity matrix SimM at text, of which the element in row i, column j indicates the similarity score between the i ¡ th sample and the j ¡ th prede¯ned concept.

For visual similarity information, two di®erent kinds of semantical information from visual images are ¯rstly extracted: visual tags using generic objection recognition classi¯er and human-being related visual tags. As to the former one, a deep convolutional neural network is trained to extract discriminative features, and a set of support vector machine are trained on on ImageNet [ 6 ], based on which each visual image from ImageCLEF 2015 testing data are tagged by using object categories with top 5 probabilities. Through this way, a given visual image is then transferred into textual semantic information, based on which one kind of visual similarity matrices SimM at vis could be calculated by following the same method for constructing SimM at text. As to the latter one, we use existing face and facial attribute detectors [ 17 ] to obtain the following information from visual images: the number of faces, facial age and facial gender, in order to enhance the image annotation of human related concepts such as 'female-child ', 'man' and 'woman'. Subsequently, these face and facial attribute detection results are transferred into another kind of visual similarity matrix SimM at f ace via manually designed mapping rules.

Each kind of similarity information could individually yield an image annotation result. However, in order to further enhance the performance of the proposed annotation method, a late fusion strategy is adopt to learn the optimal weight coe±cient using develop set. Results demonstrate that the incorporation of the proposed textual and visual similarity information could boost the performance, and the late fusion strategy could further enhance the annotation accuracy compared with trivial fusion schemes.

The remainder of this working notes is organized as follows. Section 2 describes the details of the textual/visual similarity information extraction, and late fusion strategy of fusion multiple modal similarity information. In Section 3, we summarize the implementation details of the submitted runs, and demonstrate the experimental results together with the corresponding experimental analysis. Finally, in section 4, we draw the conclusion of this working note. 2

The proposed Framework

In this section, we will elaborate the details of each components. We ¯rstly describe the textual similarity information exploration in Section 2.1, followed by the visual similarity information exploration via deep convolutional neural networks in Section 2.2. In section 2.3, we present the visual similarity information exploration by using face and facial attribute detection, and ¯nally detail the late fusion strategy of fusing multiple sources of similarity information. 2.1

Textual Similarity Information Exploration

As provided by ImageCLEF 2015, each sample is composed by a visual image and corresponding web textual xml ¯le. Textual similarity is therefore an important information for image annotation. In our submission, the textual similarity is explored by the strategy elaborated in Algorithm 1.

Speci¯cally, given the textual description of the textual xml ¯les of all 500000 500000 and the prede¯ned 251 concepts fDj gj2=511, our goal is test samples fWigi=1 to calculate the similarity score sij between the i ¡ th sample Wi to the j ¡ th concept Dj , which ¯nally consists of the similarity matrix SimM at text with SimM at text(i; j) = sij . sij is calculated as the following:

Ni Nj sij = X X dist(wi;k; dj;m);

k=1 m=1 where Wi = fwi;kgkN=i 1 and Dj = fdj;mgNmj=1.

In our implementation, we follow the method depicted in [ 3 ] and utilize path[ 4 ] distance that is based on the WordNet ontology[ 5 ] to measure the semantic similarity dist(wi;k; dj;m) between two synsets wi;k and dj;m: dist(wi;k; dj;m) = dpath(wi;k; dj;m) = 1 1 + spl(wi;k; dj;m) ; where spl(wi;k; dj;m) returns the distance of the shortest path linking the two synsets (if one exists).

Finally, we normalized the similarity matrix SimM at text to [ 0,1 ] as the following:

SimM at text(i; j) = (SimM at text(i; j) ¡ vmin)=(vmax ¡ vmin); where vmax = maxi;j SimM at text(i; j) and vmin = mini;j SimM at text(i; j). Algorithm 1 (Textual Similarity Computation) Input: textual xml ¯le fWigi5=001000 of the complete samples; textual description fDjgj2=511 of prede¯ned concepts Output: A similarity matrix SimMat text 2 R500000£251, of which the element in the i ¡ th row and j ¡ th column is the similarity score between the textual ¯le of each sample and the prede¯ned concept Steps: 1. Preprocess fWigi5=001000 and fDjgj2=511 by using a stop-words ¯lter. 2. For each textual description of concept i = 1 to 50000

For each textual xml ¯le of sample j = 1 to 251

SimMat text(i; j) = PkN=i 1 PNmj=1 dist(wi;k; dj;m), end end where Wi = fwi;kgkN=i 1 and Dj = fdj;mgNmj=1, and dist(wi;k; dj;m) is the semantic similarity distance de¯ned by WordNet.

3. Calculate the maximal value vmax and minimal value vmin of SimMat text, and normalize SimMat text: SimMat text(i; j) = (SimMat text(i; j) ¡ vmin)=(vmax ¡ vmin). 2.2

Visual Similarity Exploration Based on Objection Recognition

Via Deep Convolutional Neural Network

In the past few years, signi¯cant progress in generic visual object recognition has been achieved, by virtue of the availability of large scale datasets such as ImageNet [ 6 ] and advances in recognition algorithms such as deep learning [ 7, 8 ]. Current visual recognition systems based on deep learning are capable of recognizing thousands of object categories with promising accuracy. For instance, by using the deep convolutional neural network (CNN), He et al. [ 9 ] has reduced the visual recognition error rate on ImageNet 2012 dataset to 4.94%, which has amazingly surpassed human-level performance (with error rate 5.1% for comparison).

It is therefore reasonable to adopt deep CNN trained on ImageNet to help automatically annotate a visual image with a list of terms representing concepts depicted in the image. In our proposed framework, we follow the similar way as depicted in [ 11, 12 ] for visual objection recognition using deep CNN. Speci¯cally, we use the 1,571,576 ImageNet images in the 1,372 synsets as our training set, and the 500,000 images in the image annotation task of imageCLEF 2015 as our test set. For images with di®erent sizes, we uniformly wrapped all training and testing images into 256x256. For each image in both sets, we extracted activation of a pre-trained CNN model as its feature. The model is a reference implementation of the structure proposed in Krizhevsky et al. [ 7 ] with minor modi¯cations, and is made publicly available through the Ca®e project [ 13 ].

Once the feature is extracted for both training and test sets, 1,372 binary classi¯ers are trained and applied using LIBSVM [ 14 ], which give probability estimates for the test images. For each image, the 1,372 classi¯ers are then ranked in order of their probability estimates. In order to reliably capture the semantic information contained in the test image, we only choose the categories with top 5 probabilities, through which visual images are ¯nally transferred into tex500000. We therefore could construct visual similarity information tual tags fTigi=1 SimM at vis 2 R500000£251 via the same way as the textual xml ¯le, i.e., by using Algorithm 1 in Section 2.1. 2.3

Visual Similarity Exploration Via Face and Facial Attribute

Detection

Human being is one of the most frequently occurred objects in visual images, of which face is one of the most representive and critical biometrics. Recent years have seen the substantial progress in face detection, face recognition together with facial attribute recognition. For example, on the largest unconstrained face dataset Label Face in the Wild (LFW) [ 15 ], the most state-of-the-art approach [ 16 ] has archived 99.63% accuracy using deep learning. Many open access cloud planform for face recognition have also merged, such as Face++ [ 17 ], which could provide free API services for face and facial attributes (such as age, gender, race and etc.) detection.

In our framework, we assume that the face and facial attribute detection could explore useful semantic information from visual images. Considering its promising performance [ 18, 19 ] and open access, we adopt Face++ as a tool for face and facial attribute detection, and ¯nally utilize the following three results to enhance performance of visual image automatic annotation: 1). F Anum face: number of face detected. 2). F Aage: age of each detected face. 3). F Agender: gender of each detected face.

It should be noted that F Anum face 2 N; F Aage 2 N; F Agender 2 f'female', 'male'g, where F Anum face and F Aage are numeric variables, and could not be directly used for image annotation. Here, we manually design a mapping from F Anum face, F Aage, F Agender to the similarity score between each visual image and the prede¯ned 251 concepts.

Speci¯cally, we ¯rstly select a concept subset C14 containing 14 concepts related to human being from the complete 251 concept set CAll: C14 = f 'arm', 'eye', 'face', 'female-child ', 'foot ', 'hair ', 'head ', 'leg ', 'male', 'man', 'mouth', 'neck ', 'nose', 'woman' g. For the i¡th input visual image, we obtain F Anum face, F Aageand F Agender using Face++. Subsequently, we calculate the 251 dimensional vector si describing the similarities between the visual image and CAll as the following: if CAll(j) 2= C14, s(i; j) = 0; if CAll(j) 2 C14, s(i; j) is evaluated according to F Anum face, F Aage, F Agender and the mapping rules described in concept Call(j) is 'men', s(i; j) = 1, if F A(niu)m face > 0 and F A(gie)nder =0 male0, and s(i; j) = 0 otherwise. It is worth noting that di®erent concepts in C14 are assigned di®erent values according to their prior tightness to facial attributes. Considering that f 'eye', 'face', 'female-child ', 'male', 'man', 'mouth', 'nose', 'woman' g could be determined by the face and facial attribute with high con¯dence, they are assigned the highest similarity score 1 once the condition is satis¯ed as described in Tabel 1. f 'arm', 'hair ', 'head ', 'leg ', 'neck ' g are assigned similarity score 0.8, since they are less closely related to facial attributes. f 'foot ' g is evaluated 0.6, which is further less relevant to facial attributes.

For each visual images of the 500000 testing samples, we could calculate a 251 dimensional similarity score vector, which ¯nally consists the similarity score matrix SimM at f ace 2 R500000£251.

Until now, we have obtained three di®erent kinds of similarity matrices: SimM at text, SimM at vis, SimM at f ace, each of which could yield an individual image annotation result. For instance, we could sort each row of SimM at text, and select the concepts with top k similarity scores as the ¯nal annotations. However, SimM at text, SimM at vis, SimM at f ace are three di®erent sources of similarity information between each sample and concept. It could be expected that better performance could be archived by fusing these three similarity matrices.

In our submission, we adopt similar late fusion scheme proposed by [ 3 ]. Generally, given K di®erent similarity matrix fSimM at igiK=1, the overall similarity K matrix SimM at f inal is a weighted sum of fSimM at igi=1 as follows: SimM at f inal =

K X wi ¤ SimM at i; i=1 where PK

i=1 wi = 1 and wi ¸ 0.

In our implementation, the optimal weights fwigiK=1 are determined by using the Selective Weighted Late Fusion (SWLF) algorithm [ 3 ] on the develop set with 1980 annotated samples.

After obtained the ¯nal similarity matrix SimM at f inal, we could assign the label to each sample. In this submission, we mainly adopt two di®erent schemes:

Annotation scheme 1 : we select concepts with top N similarity scores as the ¯nal annotations;

Annotation scheme 2 : we select concepts, of which the similarity score is greater than the given threshold T , as the ¯nal annotations.

The optimal N and T could be chosen by maximizing the Mean Average Precision (MAP) on the develop set. 3 3.1

BUAA-iCC Runs and Experimental Results

Description of Submissions

We submitted total ten runs, which are di®er from similarity information fused, fusion strategies, and annotation schemes. The brief description of each submission are summarized as follows:

IRIP-iCC 01 : SimMat ¯nal=SimMat vis; adopt Annotation scheme 1, where N = 6 is chosen by maximizing MAP on develop set.

IRIP-iCC 02 : SimMat ¯nal=SimMat text; adopt Annotation scheme 1, where N = 6 is chosen by maximizing MAP on develop set.

IRIP-iCC 03 : SimMat ¯nal=SimMat text+SimMat vis+SimMat face; adopt Annotation scheme 1, where N = 6 is chosen by maximizing MAP on develop set.

IRIP-iCC 04 : SimMat ¯nal=SimMat text+SimMat vis+SimMat face; adopt Annotation scheme 1, where N = 7 is chosen manually.

IRIP-iCC 05 : SimMat ¯nal=SimMat text+SimMat vis+SimMat face; adopt Annotation scheme 1, where N = 251 is chosen manually.

IRIP-iCC 06 : SimMat ¯nal=w*(SimMat text+SimMat vis)+(1¡w)*SimMat face, where w = 0:6 is chosen by SWLF; adopt Annotation scheme 2, where T = 0:5 is chosen manually.

IRIP-iCC 07 : SimMat ¯nal=w*(SimMat text+SimMat vis)+(1¡w)*SimMat face, where w = 0:6 is chosen by SWLF; adopt Annotation scheme 1, where N = 6 is chosen by maximizing MAP on develop set.

IRIP-iCC 08 : SimMat ¯nal=w*(SimMat text+SimMat vis)+(1¡w)*SimMat face, where w = 0:6 is chosen by SWLF; adopt Annotation scheme 1, where N = 7 is chosen manually.

IRIP-iCC 09 : SimMat ¯nal=w*(SimMat text+SimMat vis)+(1¡w)*SimMat face, where w = 0:6 is chosen by SWLF; adopt Annotation scheme 2, where T = 0:6 is chosen by maximizing MAP on develop set.

IRIP-iCC 10 : SimMat ¯nal=w*(SimMat text+SimMat vis)+(1¡w)*SimMat face, where w = 0:6 is chosen by SWLF; adopt Annotation scheme 2, where T = 0:4 is chosen manually. 3.2

Results of Submitted runs

As described in [ 1, 2 ], the annotation accuracy together with the localization precision are evaluated by Mean Average Precision (MAP) with ® overlap. For instance, mAP 0 overlap stands for MAP without considering localization overlap, and mAP 0:5 overlap stands for MAP with o.5 localization overlap. In our submission, we mainly focus on evaluate the performance of the proposed framework on annotation accuracy. So we mainly analyze the experimental results of mAP 0 overlap. For mAP 0:5 overlap, we simply use the objectness detector proposed in [ 20 ] for concept localization.

The experimental results are shown in Table 2. From submissions IRIPiCC 01, IRIP-iCC 02 and IRIP-iCC 03, we can see that visual similarity by using CNN performs better than textual similarity information extracted from xml ¯le, and fusion of visual similarity information and textual similarity information could signi¯cantly boost the performance of each single similarity information. It also could be seen that IRIP-iCC 06 and IRIP-iCC 09 yield top 1 and top 2 mAPs, respectively, indicating that the weight w via late fusion strategy using SWLF could enhance the performance. The mAP of IRIP-iCC 09 is about 2% higher than IRIP-iCC 06, which could verify that the performance could be further boosted by choosing optimal threshold T via develop set. 4

Conclusion

In this paper, we described the participation of BUAA-iCC at ImageCLEF 2015 Scalable Concept Image Annotation Challenge. We proposed a novel image annotation framework by fusing textual similarity information, and visual similarity information explored by deep convolutional neural network (CNN), face and facial attribute detection.

Experimental results reveals that the visual similarity information extracted by deep CNN, face and facial attribute detection could enhance the performance of the textual similarity information extracted from xml ¯les. The similarity information fusion strategy using selective weighted late fusion could signi¯cantly boosts the performance. The annotation scheme by selecting concepts with similarity score larger than an automatically determined threshold, which maximizes the MAP of samples from develop set, yields better performance than other annotation schemes.

Acknowledgement

This work was supported in part by the HongKong, Macao and Taiwan Science & Technology Cooperation Program of China under the grant L2015TGA9004, the Fundamental Research Funds for the Central University in UIBE under grant 14QD21.

References

1. Villegas , M. , MuÄller , H., Gilbert , A. , Piras , L. , Wang , J. , Mikolajczyk , K. , Herrera , A. , Bromri , S. , Amin , M. , Mohammed , M. , Acar , B. , Uskudarli , S. , Marvasti , N. , Aldana , and J., Garc¶³a., M.: General Overview of ImageCLEF at CLEF2015 Labs . Lecture Notes in Computer Science, Springer International Publishing,( 2015 ).

2. Gilbert , A. , Piras , L. , Wang , J. , and et al.: Overview of the ImageCLEF 2015 Scalable Image Annotation, Localization and Sentence Generation task . In: CEUR Workshop Proceedings ( 2015 ).

3. Liu , N. , Dellandrea , E. , Chen , L. , Zhu , C. , Zhang , Y. , Bichot , C. E. , Bras , S. , and Tellez , B. : Multimodal recognition of visual concepts using histograms of textual concepts and selective weighted late fusion scheme . Computer Vision and Image Understanding, 117 ( 5 ), ( 2013 ) 493 - 512 .

4. Budanitsky , A. , and Hirst , G.: Semantic distance in WordNet: An experimental, application-oriented evaluation of ¯ve measures . In Workshop on WordNet and Other Lexical Resources (Vol. 2) ( 2001 ).

5. Miller , G. A. : WordNet: a lexical database for English . Communications of the ACM , 38 ( 11 ), ( 1995 ) 39 - 41 .

6. Deng , J. , Dong , W. , Socher , R. , Li , L. J. , Li , K. , and Fei-Fei , L. : Imagenet: A largescale hierarchical image database . In Computer Vision and Pattern Recognition , 2009 . CVPR 2009 . IEEE Conference on . IEEE ( 2009 ) 248 - 255 .

7. Krizhevsky , A. , Sutskever , I. , and Hinton , G. E.: Imagenet classi¯cation with deep convolutional neural networks . In Advances in neural information processing systems ( 2012 ) 1097 - 1105 .

8. Szegedy , C. , Liu , W. , Jia , Y. , Sermanet , P. , Reed , S. , Anguelov , D. , Erhan , D. , Vanhoucke , V. , Rabinovich , A. : Going deeper with convolutions . arXiv preprint arXiv:1409.4842 ( 2014 ).

9. He , K. , Zhang , X. , Ren , S. , and Sun , J.: Delving deep into recti¯ers: Surpassing human-level performance on imagenet classi¯cation . arXiv preprint arXiv:1502 . 01852 ( 2015 ).

10. Russakovsky , O. , Deng , J. , Su , H. , Krause , J. , Satheesh , S. , Ma, S. , Huang , Z. , Karpathy A. , Khosla , A. , Bernstein , M. , Berg , A.C. , and Fei-Fei , L. : ImageNet Large Scale Visual Recognition Challenge (ILSVRC) . arXiv:1409.0575 ( 2014 ).

11. Donahue , J. , Jia , Y. , Vinyals , O. , Ho®man, J ., Zhang, N., Tzeng , E. , and Darrell , T.L : Decaf: A deep convolutional activation feature for generic visual recognition . arXiv preprint arXiv:1310.1531 ( 2013 ).

12. Wang , J. K. , Yan , F. , Aker , A. , and Gaizauskas , R.: A Poodle or a Dog? Evaluating Automatic Image Annotation Using Human Descriptions at Di®erent Levels of Granularity . V&L Net ( 2014 ).

13. Jia , Y.: Ca®e: An open source convolutional architecture for fast feature embedding . http://ca®e. berkeleyvision. org ( 2013 ).

14. Chang , C. C. , and Lin , C. J.: LIBSVM: a library for support vector machines . ACM Transactions on Intelligent Systems and Technology (TIST) , 2 ( 3 ) ( 2011 ) 27 .

15. Huang , G. B. , Ramesh , M. , Berg , T. , and Learned-Miller , E: Labeled faces in the wild: A database for studying face recognition in unconstrained environments (Vol. 1 , No. 2 , p. 3 ) . Technical Report 07-49 , University of Massachusetts, Amherst ( 2007 ).

16. Schro®, F. , Kalenichenko , D. , and Philbin , J.: Facenet: A uni¯ed embedding for face recognition and clustering . In: CVPR. ( 2015 ),

17. Megvii Inc .: Face++ Research Toolkit. http: //www.faceplusplus.com, December 2013 .

18. Zhou , E. , Cao , Z. , and Yin , Q. : Naive-Deep Face Recognition: Touching the Limit of LFW Benchmark or Not? . arXiv preprint arXiv:1501.04690 ( 2015 ).

19. Fan , H. , Cao , Z. , Jiang , Y. , Yin , Q. , and Doudou , C. : Learning deep face representation . arXiv preprint arXiv:1403.2802 ( 2014 ).

20. Alexe , B. , Deselaers , T. , and Ferrari , V.: Measuring the objectness of image windows . Pattern Analysis and Machine Intelligence , IEEE Transactions on, 34 ( 11 ), 2189 - 2202 ( 2012 ).