This paper presents the participation of the BUAA AUDR group at ImageCLEF 2012 in the Photo Annotation and Retrieval task. We selected Flickr photos as data set to perform visual concept detection, annotation and retrieval. In this task, we had proposed multi-modal approaches that considered visual information and Flickr user tag information. We presented our visual-based and tag-based photo annotation methods, and also proposed Annotation Refining Algorithm (ARA), which attempted to make use of the relation between concepts to improve the annotation result. It was our first time to participate the Photo Annotation and Retrieval task. We submitted 3 runs totally and the purely visual submission using the global visual features and BoW features get better performance.
This paper presents the first participation of the BUAA AUDR group at ImageCLEF photo annotation and retrieval task.
ImageCLEF 2012 includes four types of tasks: medical image retrieval, photo annotation, plant identification and robot vision. In the photo annotation task, the aim is to analyze a collection of Flickr photos in terms of their visual or textual features in order to detect the presence of one or more concepts. The detected concepts can then be used for the purpose of automatically annotating the images or for retrieving the best matching images to a given concept-oriented query. This task provides 15000 images for training and requires the annotation of 10000 images in the provided test corpus according to the 94 pre-defined categories.
We proposed multi-modal approaches that considered visual information and Flickr user tag information. We presented our visual-based and tag-based photo annotation methods, and also attempted to make use of the relation between concepts to improve the annotation result.
The remainder of this paper is organized as follows. In section 2 we describe our approaches in detail. And our submitted runs are discussed in section 3. Then we conclude in section 4. 2
For the visual concept annotation task, we proposed multi-modal approaches that considered visual information and Flickr user tag information. We presented our visual-based and tag-based photo annotation methods in this section.
Local visual features 1.Extract SIFT 2.Generate
codebook 3.quantize
BoW features
Train/test images -images -tags Global visual features - Extract RGB -Extract FCTH -Extract CEDD
Textual features 1.Filter tag 2.Coutnt tag frequency 3.geterate text vector Global visual features
Textual features
Recently, the Bag-of-Words (BoW) [
Soft assignment was used to form the feature vector. We used the mapping from
[
Since metadata of images were provided for annotation task, using text-based method in image annotation became possible. Compared with visual features of images, metadata was usually more semantic, so it could be used for identification of many abstract concepts, which was difficult with visual features.
For the experiment presented in this paper, a tag-based image annotation method, which makes use of custom tags in image metadata in determination of existence of specific concepts, was implemented and used besides visual-feature-based annotation methods. It is a rather simple algorithm, and its progress is presented below: 1．Extract unique tags from the training dataset. Since tags are attached by users, they don't have any common rules, so even tags that share the same meaning may appear different in the same dataset. Therefore, the amount of tags could be intolerably large, and it would be difficult to generate the codebook.
2．Select tags with high frequency to form a small tag set. In this experiment, totally 2400 most frequently used tags were selected from the extracted tag set. 3．Each image is given a tag-based feature vector.
4．Train SVM classifiers for each concept with tag-based feature vectors of images.
Through the above steps, SVM classifiers are obtained, and they could be used to identify the concepts in annotation task just like any other kind of classifier. 2.3
Annotation Refining Algorithm Annotation refining algorithm (ARA) works after all previous annotation process. Its input is a complete annotation which used to be directly submitted. That is to say, Annotation refining is an extra process after common annotation process, and it is used to improve the annotation result.
It could be observed that each concept is not totally individual, e.g. “night” is usually along with “moon” and hardly with “sun”. Therefore, if each concept in a picture is identified separately, the information provided by their relations could be neglected, so the annotation refining algorithm presented here attempts to make use of this usually ignored information to improve the annotation result.
In short, the process of annotation refining is simply minimization of an evaluation function. Moreover, in order to learn parameters in this evaluation function in specific dataset, ARA needed a training process. The training process and evaluation function form the main part of ARA.
The ARA training process is a series of solution of a Linear Programming (LP) problem, which is
x j bTj x a j . (2.3)
In equation (2.3), x is a vector with the jth element being xj, which means the possibility of the existence of the jth concept in the given picture. In this experiment, xj is the score of the jth concept in annotations_raw dataset. For training, x is given by the training dataset. bj and aj are parameters that need to be learned. They are simply the parameters in general LP problems. By solving this problem in the whole training dataset, a pair of parameters could be obtained for each concept.
The minimization problem needed to be solved in ARA can be presented as below: f (x) aT (BT I)x xT (B I)(BT I)x xT x x 'T x min s.t.
xi [1,1], i 1, 2, 1 2 , n In the above problem, a and B are learned parameters in training stage. a
1
a a 2 , B b1, b2 , an , bn 1 2 (2.4) (2.5) x’ is the input annotation result. α is tuning parameter used to balance the importance of previous annotation process and refining, i.e. the refining result would be closer to x’ with a bigger eα, and vice versa.
This is a convex optimization problem and has only one solution. Refined annotation result could be obtained as the solution x*. 3
We submitted three different runs. The purely visual submission (BUAA_AUDR_1 in Table 1) was adopted using the global visual features and BoW features. For each concept, we selected separately the best classifier from a set of classifiers by MAP values obtained on 5-fold cross-validation on the training data. BUAA_AUDR_1 obtained the rank 52 on 80 submissions, with a MAP value of 0.142. This value was 0.08 lower than the median value for these runs, 0.228.
The purely text submission (BUAA_AUDR_2) obtained the rank 76. The result was poor, but it got a better result than the visual method for the valid set which we adopted to validate these methods. Maybe the tag-based photo annotation method was not robust.
BUAA_AUDR_3 was adopted as a multi-modal approaches with ARA. However, it didn't obtain better results than BUAA_AUDR_1. For two reasons: 1. We used a linear integration which might lead to a worse result than purely visual submission for that our purely text submission had a poor result and was not robust.
2. ARA had a shortcoming that poorly recognized concepts could spoil classification rates of better performing classes. This article describes the approaches and results of BUAA AUDR group at ImageCLEF 2012 photo annotation task. We submitted 3 runs totally and their results were not competitive among all the submitted runs. As this is our first time to participate in this task, we will investigate our methods to find its weak and improve its performance.