This paper describes the diversity enabled retrieval system constructed at Southampton for the ImageCLEFphoto 2009 task. The retrieval system used Terrier as the underlying textual indexing and retrieval system, and combined it with a technique for re-ranking the results by maximising the visual dissimilarity of retrieved images. The results show that our visual re-ranking methods does indeed work at increasing the diversity in the top results, however, at the same time it causes a slight drop in precision. The text-based approach designed for handling the `part 1 topics' of the task is also shown to perform very well.
The 2009 ImageCLEF photo retrieval task [
In the 2009 ImageCLEF photo retrieval task, Southampton's baseline system used standard text retrieval techniques for the part 2 topics. The baseline handling of the part 1 topics augmented the standard text search with multiple sub-queries (one per cluster) followed by a merge phase in order to build a complete ranking for the topic. On top of the baseline system we developed a reranking procedure for the results lists that leveraged visual features extracted from the images and attempted to re-order the list such that the rst images in the list were highly visually dissimilar.
The overall methodology for tackling the task involved building a baseline retrieval system using only the textual captions, and then augmenting the search results generated by the baseline system with information extracted from the actual content of the images in order to promote a diverse spectrum of di erent images near the top of the ranked search results. Each of the di erent aspects of this methodology is described below. 2.1
The Terrier text retrieval system [
The caption tokeniser was con gured to tokenise all of the text in the input record after the
document identi er. The tokeniser took any non-alphanumeric character, or run of non-alphanumeric
characters, as being a token separator. Each token was converted to lowercase, and tokens with
more than four digits or three consecutive instances of the same letter were rejected. Tokens
matching the standard Terrier list of stopwords were also discarded. For experimentation, we build two
separated indices; one with the Porter Stemmer [
For retrieval, Terrier was con gured to use the standard TF-IDF weighting model (based on
Robertson's de nition of TF and the standard Sparck-Jones IDF de nition [
Part 1 topics. This year, we only considered the cluster titles for building our search. Because the cluster titles also contained the terms used in the overall topic title, for the purposes of reporting it is assumed that that the topic title is used as well. For each of the topics, a three stage process was used to generate the results: 1. Convert each \clusterTitle" into a Terrier query such that all words with a `-' must not appear anywhere in the captions of the returned images, and all the remaining words must appear in the captions of the result documents. 2. Query the index using each of the generated queries from the cluster titles in turn and store the results. 3. Merge the results lists in a round-robin fashion, ignoring the scores assigned by Terrier (i.e. just using ranked position). The top ranked results from each of the sub-queries will come rst, followed by the second most relevant images, and so on. Duplicate images (i.e. those retrieved from more than one of the searches) are also ltered, so only the higher ranking example is retained. At this point we also gave each image an arbitrary score based on its position in the sub-query search; in our implementation all of the top-ranked images from the sub-queries scored 4000 and the second ranking images scored 3999, etc.
Part 2 topics. The second set of queries was processed in a much simpler manner; basically, results were generated by feeding the title into a Terrier query (marking all terms in the title eld as required, as in the part 1 topics). No attempt to improving diversity by further analysis of the textual information was made for these topics. 2.2
We hypothesise that the use of image features could give a large boost to the diversity of a result set. In particular, in our approach we developed a technique that re-ranks a list of search results by maximising the visual dissimilarity of the top-ranking images. 2.2.1
The image feature used in the submitted experiments was a visual-term representation based
on quantised SIFT features extracted from a multiscale di erence-of-Gaussian pyramid [
In order to try and improve the diversity of the result set, we propose a technique that incorporates visual information into the ranking. The proposed algorithm works by maximising the distance between the set of already re-ranked images, R, and an image from the ranked list of images retrieved from the text-based search, I. The distance between the feature-vectors of a set of images, R, and the feature-vector from a single image, fq, is calculated using Equation 1. The function d(:; :) in Equation 1 can be any distance function that compares two vectors; in this work we chose to use the Euclidean distance.
D(fq; R) = Y d(fq; fr) fr2R (1)
Algorithm 1 shows the steps taken to re-order the results from a text-based search using image feature-vectors. The output is a list containing all of the input images, but in a di erent order. Note that the algorithm treats the rst input image as a special key, and that that image will also appear in the rank 1 position in the output.
Algorithm 1: Re-ranking by maximising visual dissimilarity.
input : A ranked list of n images, I = I1:::In, from the text-based search output: A re-ranked list of n images, R = R1:::Rn begin
Construct an empty list R Add the rst element of I, I1, to R, and remove it from I while I is not empty do
Find Ix from I such that D(Ix; R) (from Equation 1) is maximised
Add Ix to R and remove it from I end
return R Application to Part 1 Topics. In our experiments, we applied the visual re-ranking procedure individually to each of the results sets formed from the sub-searches created from the cluster titles. The re-ranked sub-result-sets were then merged as with the text-based search for part 1 topics. Application to Part 2 Topics. Re-ranking of the results was simply a matter of applying the algorithm to the result set formed through the part 2 topics text search. 3
We submitted four runs to the task organisers. The run parameters were con gured by controlling the application of the Porter Stemmer in the text indexing and retrieval stage, and applying, or not applying, the visual re-ranking. The run titles and their con gurations can be seen in Table 1.
Diversity Re-ranking Con guration
None Visual Features
Terrier Con guration
Porter Stemmer No Stemming
SOTON1 T CT TXT SOTON2 T CT TXT
SOTON1 T CT TXT IMG SOTON2 T CT TXT IMG Run Name
Mean Median
Min
Max
SOTON1 T CT TXT SOTON1 T CT TXT IMG
SOTON2 T CT TXT SOTON2 T CT TXT IMG
Run Name
Mean Median
Min
Max
SOTON1 T CT TXT SOTON1 T CT TXT IMG
SOTON2 T CT TXT SOTON2 T CT TXT IMG
MAP 0.294 0.330 0.003 0.506 0.372 0.332 0.379 0.339 MAP 0.297 0.347 0.001 0.513 0.361 0.322 0.371 0.333
The multiple sub-query and merge approach for the part 1 topics clearly works very well. The approach taken for the part 2 topics su ers from a lack of precision in the retrieved result sets from the text retrieval.
The visual re-ranking algorithm described in section 1 has been shown to work as planned through the increased cluster recall scores it is able to produce; however, at the same time it causes a drop in F-measure because precision at the top-end of the result list also drops. One possible remedy to this problem would be to improve the precision of baseline text retrieval system so that fewer irrelevant images get passed into the re-ranking algorithm. Another possible approach would be to incorporate the retrieval score of the text-retrieval phase into the re-ranking so that images that were predicted to be more relevant still appear higher in the nal result list.
Turning o the Porter stemmer gives a small boost in performance. We need to do some more analysis of the results, however, we hypothesise that the reasons are attributable to the problem of stemming named entities as described earlier. A future modi cation to the textual indexing and query processors might be to incorporate natural language processing (NLP) techniques to automatically detect named entities and not stem them, whilst still using stemming for other words.
In the experiments described in this paper we only used a single form of visual feature. It would be interesting to repeat the experiments in the future using a broader spectrum of visual features, and to also look at combining various features. There are also possibilities for using the sample images that were provided as part of the topic speci cation to help drive the search using both content-based techniques, and perhaps query expansion or automatic relevance feedback using information in the captions belonging to those images.
The authors wish to thank the European Union, which supported this work under the Seventh Framework project LivingKnowledge (IST-FP7-231126) and the LiveMemories project, graciously funded by the Autonomous Province of Trento (Italy).