The challenges of searching the increasingly large collections of digital images which are appearing in many places mean that automated annotation of images is becoming an important technology. We describe our participation in the ImageCLEF 2010 Visual Concept Detection and Annotation Task. Our approach used only the textual features (Flickr user tags and EXIF information) provided with the images to perform automatic annotation. Our method explores the use of a combination of techniques to address the annotation problem. Our results indicate that the techniques works reasonably given the limitations inherent in using only textual data for this task. We identify the drawbacks of our approach and how these might be addressed and optimized in further work.
The exponential increase in the number of images available on the World Wide Web has led to a great interest in the topic of Automatic Image Annotation (AIA) to support applications such as effective search of online image collections. This paper describes details of our participation in the ImageCLEF 2010 Photo Annotation task which aims to explore methods for automatic annotation of large photo collections. The task involves assigning 93 concepts to images from the MIR Flickr 25,000 image dataset. The training and test sets consist of 8,000 and 10,000 images respectively. The Flickr images in each collection include user assigned tags and EXIF data for the photos where they are present. Automatic image annotation can broadly be classified into three approaches: visual, textual and hybrid models. In our work for ImageCLEF 2010 we concentrated only on use of text metadata for this task.
We submitted one run for the annotation task. The focus of our work was to attempt to exploit different methods to derive more text information from available resources to do the automatic annotation. In this our participation in the task we extracted features from the training set; used document expansion to enrich the existing text information resources; and extracted identified additional features. This paper is organized as follows: Section 2 describes our indexing and retrieval methods, Section 3 gives our experimental results and finally Section 4 concludes the paper.
Attempting to annotate images based on the available text information poses a significant challenge. Images are provided with tags of varying quality and scope manually assigned by users and with standard EXIF information. Investigation revealed of the provided Flickr dataset revealed that some images do not in fact have any user tags at all. In our experiments, we investigated approaches to making use of the limited information which was available to capture more features from both the training set and test set to assist with the annotation. These methods included document expansion and feature extraction which are introduced in the following subsections. The stages of processing and annotation are summarised in Figure 1.
The limitations of the text descriptions provided with images can lead to significant
problems for reliable processing of the images in applications such as search tools and
classifiers. Particular problems can arise due to mismatch between the manually
assigned tags when comparing images and when attempting to identify images
relevant to user queries in retrieval application, and due to the general inadequacy of
the tags assigned by users. In our approach to this task we sought to enrich text
information about images by using a process of document expansion [
We used document expansion to expand the image metadata, but also the concepts which are to be used to annotate the images. Each concept usually consists of only 1 or 2 words. Thus it is hard to reliably match concepts to image metadata. Thus it is interesting to try to expand concepts to include words related to the concept or which describe the concept. We thus hoped that after this expansion, concepts could be more reliably matched to image metadata. To perform concept expansion each of the concepts was treated as a query and again applied to external DBpedia information resource. Selected expansion terms were then added to the concept.
Our document expansion method uses the Okapi feedback method [
While this approach has the potential to assign good concept annotations for images which have manual tags to seed the expansion process, it does not work well for images which do not have manual tags as a starting point for expansion. In order to be able to annotate these images another method is required.
The annotation scheme has been setup in such a way to make it easy to extend it with
new keywords without having to go through all images again [
From the training set, some general concepts can be found. They cover a number of proper subtopics, see Table 1. We used a simple greedy algorithm method to assign an affiliation relation selection. The algorithm operates as follows:
Greedy Algorithm 1 (affiliation): 1. for each concept ci (0 <= i <= 92), count how many times it appears in image collection, Nci 2. when concept ci appears, count how many times another concept cj (0 <= j <=92, j!=i) appears, Ncj 3. compute Pij = Ncj / Nci 4. if Pij >= 0.97, then we assume, concept cj is the subtopic of concept ci
The value 0.97 was selected empirically for this collection. According to this relationship, if any subtopic is annotated in one photo, then its corresponding general topic will be annotated in the same photo.
In addition to the affiliation, an opposite relationship was also identified, see examples in Table 2. Similar to the affiliation relation method, a greedy algorithm was used to identify these relations. The algorithm operates as follows: Greedy Algorithm 2 (opposite relationship): 1. for each concept ci (0 <= i <= 92), count how many times ci appears in the image collection, Ncia 2. when ci occurs, count how many times another concept cj (0<= j <=92, j!=i) does not occur, Ncjn 3. compute Pij = Ncjn / Ncia 4. for each pair of concepts ci and cj, compute
Pji = Ncin/ Ncja 5. if Pij >= 0.7 & Pji >= 0.7, we assume concept ci and cj are an opposite pair of concepts Ncia = number of times concept ci appears in the image collection Ncjn = the number of times concept cj does not appear in the image collection when ci appears
This relationship means that if one concept occurred in a photo, its opposite concept is unlikely to have occurred in the same photo. The value 0.7 was again chosen empirically. In this experiment, only two of these opposite pairs were found (the pair with ‘*’ mark in Table 2). How to find more opposite pairs is another challenge for our future work in this kind of task.
For concept classification, assignment of each concept was treated as an individual classification task. Thus for each concept, we consider that there is an annotated image collection. We find the common features of all images in this collection from their EXIF information file. Then this common feature is used to annotate this concept on test set.
EXIF metadata represents a number of properties and settings of the digital
camera at the time of taking picture [
Camera itself: brand… Camera settings: exposure, aperture, focal length, ISO speed… Image settings: orientation, resolution, compression…
Time and Date
Because not all of these fields are present in every EXIF file and the time restrictions to perform this task. We did not use EXIF collection effectively. We only extracted the Date and Time properties from EXIF metadata. Pictures which were taken at times between 08.00 and 17.00 were annotated as the day time concept and other times are assumed to be associated with a night concept. Further features could be extracted if more time were available to analyze this EXIF metadata. Further study of EXIF metadata is also planned in future work for this task. 2.3
To calculate the concept assignments the features need to combined to produce a final result. Following the application of document expansion, we get a binary result matrix A. All other feature functions are then applied on this matrix. The final combination result is calculated using the following equation:
Final result = (A + Rb + Rc) ⊗ D where: A represents the document expansion binary result matrix; + represents application of the following method on the previous matrix; Rb is the affiliation relation method; Rc is the opposite relation method; ⊗ is exlusive or symbol; D is the binary result matrix achieved by the EXIF metadata method.
We made only one submission for this task. This used all the methods introduced above in combination to annotate the test dataset. The official result of this run is reported in Table 3.
For this task, 64 runs were submitted in total, only two runs chose to use the textbased approach (our submission and another from the MLKD group). Based on the reported MAP measure, these two runs got very close results, and were ranked at positions 42 (MLKD group) and 45 (our run) out of 64 submitted runs, respectively. The best run used a hybrid approach.
For each concept, the EER (Equal Error Rate) and AUC (Area Under Curve) were calculated. The results of each concept are shown in Figure 2 (the x axis indicates the 93 concepts; the y axis indicates the Accuracy Rate). From the figure we can see that the results of our experiment are variable, it can be noted that some concepts are not detected at all. One of the main reasons underlying poor results is that the text resource available for some concepts is not sufficient for this task. In particular, some images do not have tags and EXIF files at all. This is obviously a big problem when using a text only based approach to doing the annotation task. Another issue is that both the EER and AUC evaluation methods require confidence scores of each annotated concept. However, our method cannot provide this score information.
We have presented and analysed our submission to the ImageCLEF 2010 Photo Annotation Task and compared our results to those of other participants. Although the text-based approach only achieves moderate and inconsistent results, it has potential to be improved further. In this experiment we used document expansion to enhance image text metadata. In future work we plan to explore used of other external information resources for this task. Some images do not have tags and EXIF information and thus cannot be annotated at all. How to identify more features and information from this limited resource is a big challenge for text-based approach. All of these problems define our future work. This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (CNGL) at Dublin City University.