Inner Mongolia University have participated the Visual concept detection, annotation, and retrieval using Flickr photos task of ImageCLEF for the first time in 2012. We have conducted experiments and submitted results for both the Concept Annotation and the Conceptbased Retrieval subtasks. This paper describes the methods we have adopted and the analysis of the results for the two subtasks. We focus our attention mainly on the user's tag since we believe that user annotation provides strong semantic information which can be used to accurately determine the presence or absence of each concept and the relevance level between the images and queries. For the Concept Annotation subtask, we use only a simple statistical method that scores the confidence of the presence of each concept by the maximum conditional probability of the concept between the different given tags. For the Concept-based Retrieval task, we adopted the language modeling approach which has been widely used in text information retrieval field. Official evaluations show that the performance of our method is competitive. We rank in the middle of the pack for the Concept Annotation subtask with the best run's MiAP equal 0.2441. For the Concept-based Retrieval subtask, we rank at the top with the best run's MnAP equal 0.0933. Beside the main submissions, we also submit two visual runs, although no very good, with the MiAP for Concept Annotation is 0.0819 and the MnAP for Concept Retrieval is 0.0045. As a whole, the results confirm that although the methods we have adopted are simple, the performances we have achieved are satisfied.
In this paper we will describe the approaches we have adopted to accomplish the “Visual concept detection, annotation, and retrieval using Flickr photos” task for ImageCLEF 2012. We have participated both the Concept Annotation subtask and the Concept-based Retrieval subtask and submitted 15 runs in total. The official evaluation shows that we rank in the middle pack in the Concept Annotation subtask and get the highest rank in the Concept-based Retrieval subtask[1]. We base our methods mainly on User Tags, both for the Concept Annotation and for the Concept-based Retrieval subtask. The main reason for choosing User Tag as feature is that we believe that users generally annotate an image with the words that have strong relationships to its meaning. This semantic information can then be used to determine the contents of the image accurately. In all, we used statistical methods to address both the two subtasks. For the annotation subtask, we use the training set to estimate a conditional probability distribution and use the probability of the most supportive tag as the confidence of the presence of a concept in an image. For the retrieval subtask, we construct language models for the tags of each image and take the probability of the query being generated by the tags model as ranking score. Beside the main methods, we also test some visual feature based method.
The rest of the paper is organized as follows: in section 2 we firstly discuss the method we have used to accomplish the Concept Annotation subtask, including the experiments and the results achieved. Then in section 3, we similarly discuss the method for Concept Retrieval, again, experiments and result analysis are included. Finally, we conclude our work and shed lights on the future work in section 4 and 5 respectively. 2 2.1
We have adopted a very simple statistical method for the Concept-based
Annotation subtask here. That is: we just calculate the conditional probability of
the presence of the concept given the tags of the image and then taking the
maximum probability among the different tags as the confidence. This can be
shown by (
Conf idence(C) = max P (C|T ag) ,
T ag
(
The flowchart of our Tag-based Concept Annotation system is illustrated in Fig. 1.
We use the training set which has been released by CLEF organization to estimate the conditional probability distribution. Since the number of tags for each image is relatively small (6.34 on average according to our statistics), we think that it’s reasonable to expand them first[2]. We use the Official INEX 09 collection[3] which contains about 2640000 articles to perform User Tag expansion. 1 More specifically, we first use the tags for each image as query and the INEX 1 http://www.mpi-inf.mpg.de/departments/d5/software/inex/
Training Set Conditional Probability Estimation Conditional Probability Distribution
Tag1
Tag2 P(C|Tag1)
P(C|Tag2) Lookup the Conditional
Probability
MAX Confidence of Concept C
Tagn P(C|Tagn) corpora as document collection to perform retrieval and then extend the tags by using the top 15 words among the returned top 200 documents. We test our annotation approach on the officially released testing set, which contains 10000 images in total. The collections we used for our Tag-basd annotation approach are listed in Tab. 1.
We use Indri[4] in the Lemur Toolkit2 to perform retrieval. Indri is a search engine which has been widely used in the Information Retrieval field. We will mention it again in the Concept-based Retrieval section (section 3). The parameters set in Indri for the Tag-based Annotation are listed in Tab. 2.
The experimental results are summarized in Tab. 3. In Tab. 3,Max CondProb denotes Maximum Conditional Probability; WE Train Max CondtProb means Maximum Conditional Probability with tag expansion for Training Set by Wiki INEX09; WE Train Test Max CondProb is the Maximum Conditional Probability method with tag expansion for both training and testing set by Wiki INEX 09; WE Train Test Norm means Maximum Conditional Probability with tag expansion for both training and testing set by Wiki INEX 09 and scores are normalized to the range of 0 to 1.
Comparing the first two results Max CondProb and WE Train Max CondProb in Tab. 3, we can see that it really reaps the benefit of the Wiki Expansion greatly (more than 1.0 percent increase for the MiAP matric).
The huge drop from the performance of WE Train Max CondProb to that of WE Train Test Max CondProb indicates that it’s not reasonable to expand the tags for the images to be annotated. We can explain this as that the expanded tags are more likely to drift away to the concepts that are not in fact exist in the given image.
Intuition tells us that every image should contain some concepts in general, otherwise it will not be chosen for sharing. This means that even though the absolute probability of a concept may be low, we should still have great confidence that the concept is present if it has a relatively higher probability than the other concepts. This assumption can be confirmed by comparing WE Train Test Max CondProb to the run with the probability value normalized across the concepts (WE Train Test Norm Max CondProb). The great improvement in performance shows that normalization plays an important role in transforming the conditional probability to the confidence of the concept. 2.2
Visual Feature based Method for Annotation Extraction of Visual Features We extracted three features that are mostly considered in the literatures we found (i.e., Color Histograms, Fuzzy-Color-andTexture-Histogram (FCTH) and Bag-of-Visual Words).
Color Histograms are among the most basic approaches and are widely used in image retrieval. The color space is partitioned and for each partition the pixels with their color within this range are counted, resulting in a representation of the relative frequencies of the colors. We use the RGB color space for the histograms[5]. And we use the Jensen-Shannon divergence (JSD) as shown in Tij = t(xi, xj) =
xiT xj xiT xi + xjT xj − xiT xj .
We extracted the SIFT local features from harris-laplace region of interest
detection. Each of these features is represented as a Bag-of-Visual Word. The
visual words vocabulary is generated by adopting the K-means clustering
algorithm on the features of the training set, which is implemented in the LIRE
Toolkit3[7]. In our experiment, we take 10000 as the size of the Visual Word
Vocabulary and adopt the same dJSD in (
The three features described above are then used independently for k-NN
classifier.
(
′ M " dJSD(H, H ) = X m=1
Hm log 2Hm ′
′ + Hm log Hm + Hm
′
2Hm ′
Hm + Hm
#
,
(
Fuzzy Color and Texture Histogram (FCTH) is appropriate for accurately
retrieving images even in distortion cases such as deformations, noise and
smoothing. FCTH is a low level descriptor that contains both quantized histogram color
and texture information[6]. For the measurement of the distance of this feature
between the images, we use Tanimoto coefficient as shown by (
Simf (C, Ti) =
Pp∈P (distf (Ti, p) + ε)−1 Pn∈N (distf (Ti, n) + ε)−1 + ε where P and N are the k-nearest positive and negative images for each concept and satisfy |Q| + |N | = k and ε is a small positive number to avoid division by zero.
However, the experiment results on the training set are not very good, so we submitted only one visual feature based run (Bag-of-Visual Words).
Tab. 4 lists the official evaluation result of our visual submission, which again confirms that by now our visual based method is not good.
(
Language modeling is a formal probabilistic framework that has been widely
used in the text retrieval field. The language modeling approach to text retrieval
is to model the idea that a document is a good match to a query if the document
model is likely to generate the query[9]. Formally, we want to estimate a model
Md for each document d and rank the documents for a query Q according to the
probability of Q being generated by Md. When further assuming beg-of-words
modeling, we get (
m log p (Q|Md) = X log p (Qi|Md) .
i=1
(
Observed that, in the official image collection, if an image is relevant to a
query, its tags annotated by the user are more likely to occur in the query, we
guess that the Concept Retrieval task can be addressed by building language
models for the image tags and retrieving the images by ranking them according
to the likelihood of generating the query (
m log p (Q|MImageT ags) = X log p (Qi|MImageT ags) .
i=1
(
To test our assumption, we use the Indri Search Engine in the Lemur Toolkit which we have mentioned in section 2 to perform retrieval. Indri is a search engine that implements the language modeling approach under the Bayes Inference Network Framework [10]. The Tag models are smoothed by the Dirichlet Smoothing[11] method. For each topic, we retrieve 1000 images. A total of 418 stop words from the standard InQuery[12] stoplist are removed from the queries. Pseudo Relevance Feedback (PRF)[9] is adopted for both Tag Expansion and Concept Retrieval with fbTerms=15 and fbDocs=200. The detailed parameters set in Indri for Tag-based Retrieval are listed in Tab. 5.
In Tab. 6, we list the collections used to perform our Tag-based retrieval experiments and illustrate the flowchart of our system in Fig. 2.
In our experiments, we have tested different methods to form the queries for language modeling based retrieval and the results we have achieved are listed in Tab. 7 and Tab. 8.
In Tab. 7, Title means that the queries are formulated by just using the terms in the <title> field of the topic file queries.xml which has been officially released in the test set. Image(all 3) denotes that the queries are constructed by using all the three images’ tags, i.e., the <image> fields in the topic file; Image(first not null) refers to the query construction method that only the first image’s tags in the topic file are used. If the first image’s tag set is empty or the tags cannot return any result, use the next one’s tags, and so forth. Title&Image(all 3) means that the queries are constructed by combining the title portion and tags associated with all the three images in the topic file.
Since the number of tags associated with each image is relatively small (6 on average according to our statistics), we think it would be helpful to extend the tags for the images to perform retrieval. We perform the tag expansion similarly as in section 2 and the results are listed in Tab. 8, where runs with the prefix WE denote the Wiki expanded version of the corresponding runs.
Fig. 3 illustrates the comparision of the results in Tab. 7 with that in Tab. 8. We can see that tag expansion really improve the performance greatly (with only one exception that for Image(all 3), which decrease about 3 percent on MnAP matric). We explain this as follows: the tags for the images may not be so accurate as that in the title field. So the expanded terms may drift away seriously. But if the title terms is also present, the expanding precess will be directed by them and the performance will increase finally as shown by the results of Title&Image(all 3) V.S. WE Title&Image(all 3).
From Tab. 7 and Tab. 8, we can see that using the title filed as query perform better than using the tags of the images in the image fields, both for the original tags and for the Wiki Expanded ones. This means that the title field in the topic has more strong descriptive ability than the tags of the images. We can also see that when combining the title part with the image part to form the queries, the retrieval performance can always be greatly improved. This phenomenon is in
Wiki Expansion accordance with our intuition that the more information provided, the better we can determine the user’s Information Need.
We also have performed experiment to test the combination method at result
level, which is denoted by WE Combine Title&Image(all 3) at Result Level. To
do this, we first generate the ranked Image Lists (WE Title and WE Image(all 3))
by the two methods independently. After normalizing the score for each Image
Im to the same scale (
Comparing the result to that at query level (WE Title&Image(all 3)), we can see that they achieve similar performance.
0.1 0.09 0.08 0.07 0.06 P nA0.05 M 0.04 0.03 0.02 0.01 0
Runs without Wiki Expansion
Runs with Wiki Expansion Title
Image(all_3)
Image(first_not_null)
Title&Image(all_3)
Extraction of Visual Features For visual feature extraction, we take the
same method as that for visual concept annotation. So we omit their description
here to reduce the lenght of the paper. Please refer to section 2 for more detail.
Classification We use the weighted distance from the query images in the topic
file to the image being considered as the ranking score, which can be shown by
(
Tab. 9 lists the result of our visual based retrieval submission, which shows that by now our method does not achieve good performance either. 4
In this paper, we described the experiments we have performed for the “Visual concept detection, annotation, and retrieval using Flickr photos” task in detail. BoV Retrieval 0.0045 0.0030 0.0064 We based our methods mainly on the Tags annotated by the users since we believe that there is strong relationship between the user’s tag and the presence of a concept for annotation and between the user’s tag and the query for retrieval. Official evaluation show that we achieved satisfied results for the User Tag based methods. Beside the main submission, we also perform some initial visual feature based experiments. However, the results we can achieved by now are not very good. 5
Since this is the first time we participated the ImageCLEF task, we just did some initial work. More detailed experiments should be performed in the future. For example, for the Concept Annotation subtask, we didn’t consider the relationships between different concepts. In reality, there are correlations between different concepts, such as the probability of the presence of timeofday day is usually low given the presence of timeofday night whereas the probability of view outdoor will be high given the presence of the concept flora tree. For the Concept-based Retrieval subtask, we just applied the traditional Language Modeling approach to the tag modeling application but did not take the specific characteristic of image tags into consideration, like that the terms in the documents in text retrieval are sufficient in general whereas the amount of tags for each image are relatively small, even after being expanded. More refined modification should be made to address these problems. We will do all these works in the future.
Acknowledgments. We would like to thank Prof. Jian-Yun Nie of Universit´e de Montr´eal for his invaluable suggestions.