The 2013 MediaEval Retrieving Diverse Social Images Task looked to tackling the problem of search result diversi cation of Flickr results sets formed from queries about geographic places and landmarks. In this paper we describe our approach of using a min-max similarity diversi er coupled with pre- lters and a reranker. We also demonstrate a number of novel features for measuring similarity to use in the diversi cation step.
The diversi cation of search results is increasingly
becoming an important topic in the area of information
retrieval. The 2013 MediaEval Retrieving Diverse Social
Images Task [
Our motivation for this task was to build on the diversi
cation techniques we developed in ImageCLEF'09 [
In terms of overall approach, after a number of experiments, we settled on the work ow illustrated in Figure 1. In order to improve precision, we applied lters to the input results list to remove images unlikely to be relevant, and for the runs that allowed use of the text and metadata, we reranked the results before applying diversi cation. To diversify the results, after testing a number of techniques (i.e. clustering followed by round-robin selection), we reverted to a Min-Max diversi cation technique as it gave the best results on the development dataset with the features we used.
Brie y, the Min-Max technique takes as input a similarity matrix and a pivot image, and uses this to build a result list. The pivot is taken as the rst image in the result list. The second image is chosen as the one that has the minimum similarity to the pivot. The remaining images are chosen such that they have the maximum dissimilarity to all of the previously chosen images. Similarity of an image from a set of images can be computed via a number of functions
Pre-Filter
Binary Predicate Faces, Text, Blur GeoLocation (Geo), Number of Views (Views), Description Length (DLen) Reranker Lucene-based proximity search
Greedy Min-Max Similarity Diversifier Diversifier CN-CORR (CN), SIFT-LSH (SIFT),
TimeUser (TU), MonthDelta (MD)
over the piecewise similarities of the image to each element
of the set; sum, product and max are typical choices. On
the development set, we found max worked best. The
implementation of our methodology was realised in Java using
OpenIMAJ1 [
We removed many of the images which weren't relevant using some of our pre- lters before diversifying them. Images which contained frontal or side-views of faces in focus were discarded and we also got rid of the blurred out-of-focus ones. Images were further checked for the amount of text they contained and those with high percentage were thrown away. Those images which had been geotagged more than 8 km away from their actual location were removed. We found that images without any views were usually not relevant and so only took into consideration those which had more than 2 views. Similarly we discovered that images with very large descriptions tend to be irrelevant and hence ltered out those whose descriptions were over 2000 characters long. 2.2
The original results lists provided in the task were retrieved by searching Flickr with a given monument name. The exact search implementation used by Flickr is unknown, but it is likely to be a variant of the vector space model with stemming. A better, more precise, ranking of the results can be achieved by performing a phrase or proximity search in which the results are scored higher if the query terms occur in close proximity in the metadata. To apply proximitybased reranking, we indexed the title, description and tags elds of each image in the ltered results list with Lucene, and performed the following query: (TITLE:"monument "~20 OR TAGS:"monument "~20 OR DESCRIPTION:"monument "~20) 1http://openimaj.org 2http://lucene.apache.org
TU
MD 0.6994 0.8158 CN
X X
X
X
X X
X X X
X OR (TITLE:"monument ")^0.5 (TAGS:"monument ")^0.1
In order to use the Min-Max diversi er, a similarity matrix is required. At the beginning of the task we spent some time analysing the data, and looking at features which could be sensibly used to compute similarity. One particular problem we noticed was that many of the images had the same description and tags, even though they were visually diverse. This means that standard techniques for diversi cation based on the text are unlikely to work well in many cases, and would in all likelihood end up being similar to just diversifying based on the users that took the photo. With this in mind, we started to explore other features that could work better.
Color Naming Histogram (CN). The provided CN
histogram features [
Time User (TU). Images taken by the same user within a short time period are likely to be similar. A similarity matrix was constructed with the following constraints: pairs of images taken a less than a minute apart had similarity 1; images more than 3.25 mins apart had 0 similarity. Between 1 and 3.25 minutes the similarity falls o logarithmically. Month Delta (MD). Similar to the TU feature, images have increasing similarity with closer month of year.
Three runs were submitted; their con guration with respect to the methodology and features described in Section 2 is illustrated in Tables 1a and 1b. Where multiple features were used, the similarity matrices were just averaged to create a single matrix. Two major points can be noted from the results. Firstly, using textual and visual features outperforms the use of either modality alone with our techniques. It is also clear that the reranking stage massively helps improve precision. Secondly, the high variability in results across the experts and crowdworkers indicates that the task is actually rather subjective; it is particularly interesting that when compared against the crowdworker groundtruths our cluster recall scores are almost double, perhaps indicating that the experts tended to over-segment the result sets. 4.
In this work we've explored di erent features for search result diversi cation, and also explored how relevance can be maximised by pre- ltering and re-ranking prior to the diversi cation step. The results indicate that our re-ranking step gives a good increase in precision. The combination of features from multiple modalities leads to a modest increase in diversity. In the future we intend to investigate whether automatically generated classi cations from the visual features (indoor/outdoor, etc) can be leveraged to increase diversity.
The described work was funded by the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreements 270239 (ARCOMEM), and 287863 (TrendMiner).
1 2 3