This paper describes Southampton's submissions to the 2009 ImageCLEF photo annotation task. For the task we used an annotation system based on the idea of constructing semantic spaces, which was developed previously at Southampton. To represent the image content, we used a combination of di erent SIFT and Colour-SIFT features detected using the di erence-of-Gaussian and MSER techniques. These features were converted into a visual term representation by applying vector quantisation using a codebook learnt from a hierarchical k-means clustering. In terms of EER and AUC, the annotator performs reasonably well, however, it struggles when evaluated using the hierarchical measure proposed for the task, due to the way the annotation con dences are thresholded.
As with many automatic annotation approaches, the methodology applied to this task involved extracting feature vectors for each of the images, and then feeding the features of a training set, together with annotations to a machine learning system. The machine learning system attempts to learn low-level relationships between all of the features and annotations. Once the training phase is complete, features from un-annotated images can be fed into the system to use the learnt relations to get predictions of annotations. 2.1
The images were represented by vectors of visual-term occurrences [
The machine-learning component is based on a linear-algebraic semantic space [
The use of the cosine similarity measure gives each possible annotation a score between -1 and 1; however, these scores are not themselves all that informative. A higher score does mean more con dence in an annotation, but only when considered against all the other annotations. For the purposes of evaluating the annotator using the EER and AUC measures this is not too much of a problem | we can just scale the scores to the 0..1 range (i.e add one and divide by two). However, as will be discussed in the next section, the hierarchical scoring measure [10] thresholds the annotation con dences at 0.5 to produce a binary indication of present/notpresent. Unfortunately, for the semantic space this poses a big problem as the position of the threshold should ideally be set di erently for each image, based on con dences of all the predicted annotations. 3
We submitted three di erent runs to the task organisers. The rst run was trained on the raw
annotations. The second included a partial expansion of the annotation hierarchy [
Raw annotations Partial hierarchical expansion
Full hierarchical expansion The two runs that included the hierarchical information did not perform as well (based on average EER and AUC) as the one based on the raw annotations. Looking at the EER scores for each annotation term, the hierarchical methods were consistently worse performing. For these reasons the results for these runs will not be discussed further.
The EER and AUC scores are summarised in Table 2. Our scores are better than the averages of the other participants, however, they are still a fair way o of the top scores. Using a semantic space approach for annotation, from past experience, we expect that there will be a large diversity in the performance for di erent terms. This is due to di erences in the amount of training data for each term, and also the amount of visual diversity that might be associated with a term. For example, visually speci c annotation terms require less training data than diverse ones. Table 3 shows the top- and bottom-most 5 annotation terms. It is interesting that the worst performing terms are those that are rather general and unspeci c. The top performing terms all have very speci c visual representations.
Table 4 shows the results of our annotator using the hierarchical measure [10]. Unfortunately, because this scoring measure performs a binary thresholding operation on the con dence scores,
Sunset-Sunrise Landscape-Nature
Night
Sea
Mountains Aesthetic-Impression
Overall-Quality Neutral-Illumination
Sports Fancy
EER 0.232 0.234 0.237 0.243 0.249 0.416 0.436 0.466 0.470 0.478 the performance of our technique measures at the lower end of the spectrum of results from the di erent participants. As previously discussed, the semantic space annotation approach doesn't really permit the global setting of such a threshold. The feature extraction phase was performed in parallel (4 images being processed at once) on a quad core machine (Intel Core 2 Quad @ 2.66Ghz, 8G ram, Redhat Enterprise 5.3). The time for image processing varied based on both the size of the image, and the image content. Timings for a typical image from the training set are shown in Table 5.
Training the semantic space took approximately 1 hour on a dual quad core 2.8GHz Xeon workstation running Mac OS X (the semantic space code is single threaded, so only uses a single core). We would estimate that no more than 1G of ram was used during the semantic space training phase. Projecting all the test image in bulk took under 2 minutes, and it took about 5 minutes to generate annotations for all the 13000 images; so, in general, it took less than .05s to get from a list of visual terms to the suggested annotations for a single image. Implementation. The semantic-space software is written in C and makes use of Doug Rohde's SVDLIBC 1 for e ciently performing the large sparse SVD. The feature detector and descriptor software is written in C and C++. The image processing components were driven by a standard UNIX make le, which enabled easy parallelisation. 4
For this task we applied an older technique for automatically annotating images using a semantic space. The performance of the technique in terms of EER and AUC is fairly competitive, however, the technique does not mesh well with the hierarchical scoring measure proposed for this task. The semantic space technique is reasonably computationally e cient; the most time is spent processing
Di erence-of-Gaussian detection + SIFT extraction
MSER detection
SIFT extraction on MSER Colour-SIFT extraction on MSER
Vector quantisation
Estimated total
Time 1.8s/image 0.1s/image 2.7s/image 1.0s/image <0.1s per set of extracted features 5.9s/image the images to extract features. In our experiments, the use of the hierarchy did not lead to any improvement in the annotation quality.
The authors wish to thank the European Union, which supported this work under the Seventh Framework project LivingKnowledge (IST-FP7-231126) and the LiveMemories project, graciously funded by the Autonomous Province of Trento (Italy).