=Paper=
{{Paper
|id=None
|storemode=property
|title=Experiments in Diversifying Flickr Result Sets
|pdfUrl=https://ceur-ws.org/Vol-1043/mediaeval2013_submission_18.pdf
|volume=Vol-1043
|dblpUrl=https://dblp.org/rec/conf/mediaeval/JainHSPDDL13
}}
==Experiments in Diversifying Flickr Result Sets==
https://ceur-ws.org/Vol-1043/mediaeval2013_submission_18.pdf
Experiments in Diversifying Flickr Result Sets
Neha Jain Jonathon Hare Sina Samangooei
nj1g12@ecs.soton.ac.uk jsh2@ecs.soton.ac.uk ss@ecs.soton.ac.uk
John Preston Jamie Davies David Dupplaw
jlp1g11@ecs.soton.ac.uk jagd1g11@ecs.soton.ac.uk dpd@ecs.soton.ac.uk
Electronics and Computer Science, University of Southampton, United Kingdom
Binary Predicate
ABSTRACT Input Results List
Pre-Filter
Faces, Text, Blur
GeoLocation (Geo), Number of Views
The 2013 MediaEval Retrieving Diverse Social Images Task (Views), Description Length (DLen)
looked to tackling the problem of search result diversifica-
Reranker Lucene-based proximity search
tion of Flickr results sets formed from queries about geo-
graphic places and landmarks. In this paper we describe Greedy Min-Max Similarity Diversifier
our approach of using a min-max similarity diversifier cou- Diversifier CN-CORR (CN), SIFT-LSH (SIFT),
Output Results List TimeUser (TU), MonthDelta (MD)
pled with pre-filters and a reranker. We also demonstrate a
number of novel features for measuring similarity to use in Figure 1: Overall workflow for diversify results
the diversification step.
over the piecewise similarities of the image to each element
1. INTRODUCTION AND MOTIVATION of the set; sum, product and max are typical choices. On
The diversification of search results is increasingly be- the development set, we found max worked best. The im-
coming an important topic in the area of information re- plementation of our methodology was realised in Java using
trieval. The 2013 MediaEval Retrieving Diverse Social Im- OpenIMAJ1 [2] and Lucene2 .
ages Task [4] aimed to foster new multimodal approaches to
the diversification of result sets from social photo retrieval. 2.1 Pre-Filters
Our motivation for this task was to build on the diversifi-
We removed many of the images which weren’t relevant
cation techniques we developed in ImageCLEF’09 [1] by in-
using some of our pre-filters before diversifying them. Im-
corporating truly multimodal data. We were also motivated
ages which contained frontal or side-views of faces in focus
to explore how the precision of the search results could be
were discarded and we also got rid of the blurred out-of-focus
improved by filtering and re-ranking prior to the diversifica-
ones. Images were further checked for the amount of text
tion step, thus minimising the loss in precision usually seen
they contained and those with high percentage were thrown
when diversification is applied.
away. Those images which had been geotagged more than
8 km away from their actual location were removed. We
2. METHODOLOGY found that images without any views were usually not rel-
In terms of overall approach, after a number of experi- evant and so only took into consideration those which had
ments, we settled on the workflow illustrated in Figure 1. more than 2 views. Similarly we discovered that images with
In order to improve precision, we applied filters to the input very large descriptions tend to be irrelevant and hence fil-
results list to remove images unlikely to be relevant, and tered out those whose descriptions were over 2000 characters
for the runs that allowed use of the text and metadata, we long.
reranked the results before applying diversification. To di-
versify the results, after testing a number of techniques (i.e. 2.2 Reranker
clustering followed by round-robin selection), we reverted to The original results lists provided in the task were re-
a Min-Max diversification technique as it gave the best re- trieved by searching Flickr with a given monument name.
sults on the development dataset with the features we used. The exact search implementation used by Flickr is unknown,
Briefly, the Min-Max technique takes as input a similarity but it is likely to be a variant of the vector space model with
matrix and a pivot image, and uses this to build a result list. stemming. A better, more precise, ranking of the results can
The pivot is taken as the first image in the result list. The be achieved by performing a phrase or proximity search in
second image is chosen as the one that has the minimum which the results are scored higher if the query terms occur
similarity to the pivot. The remaining images are chosen in close proximity in the metadata. To apply proximity-
such that they have the maximum dissimilarity to all of the based reranking, we indexed the title, description and tags
previously chosen images. Similarity of an image from a fields of each image in the filtered results list with Lucene,
set of images can be computed via a number of functions and performed the following query: (TITLE:"monument "~20
OR TAGS:"monument "~20 OR DESCRIPTION:"monument "~20)
1
Copyright is held by the author/owner(s). http://openimaj.org
2
MediaEval 2013 Workshop, October 18-19, 2013, Barcelona, Spain http://lucene.apache.org
Table 1: Run configuration
(a) Pre-filters applied in each of the runs. (b) Reranker and features in each of the runs.
Visual Meta/Textual Visual Meta/Textual
Run Face Blur Text Geo Views DLen Run Reranker CN SIFT TU MD
1 X X X 1 X
2 X X X 2 X X X
3 X X X X X 3 X X X X
Table 2: Official Results. Crowdworker evaluation was performed on a subset of 50 locations from the testset (346 locations).
Expert Crowdworker
ALL GT1 GT2 GT3
Run P@10 CR@10 F1@10 P@10 CR@10 F1@10 CR@10 F1@10 CR@10 F1@10
1 0.6994 0.4081 0.4926 0.6612 0.8174 0.7043 0.7858 0.688 0.6398 0.6197
2 0.8231 0.4306 0.5397 0.7694 0.8124 0.7689 0.7474 0.7276 0.6745 0.6944
3 0.8158 0.4398 0.5455 0.7714 0.8184 0.7734 0.7486 0.7263 0.668 0.6906
OR (TITLE:"monument ")^0.5 (TAGS:"monument ")^0.1 across the experts and crowdworkers indicates that the task
is actually rather subjective; it is particularly interesting
2.3 Similarity Matrices that when compared against the crowdworker groundtruths
In order to use the Min-Max diversifier, a similarity ma- our cluster recall scores are almost double, perhaps indicat-
trix is required. At the beginning of the task we spent ing that the experts tended to over-segment the result sets.
some time analysing the data, and looking at features which
could be sensibly used to compute similarity. One particular 4. CONCLUSIONS
problem we noticed was that many of the images had the In this work we’ve explored different features for search re-
same description and tags, even though they were visually sult diversification, and also explored how relevance can be
diverse. This means that standard techniques for diversifi- maximised by pre-filtering and re-ranking prior to the diver-
cation based on the text are unlikely to work well in many sification step. The results indicate that our re-ranking step
cases, and would in all likelihood end up being similar to just gives a good increase in precision. The combination of fea-
diversifying based on the users that took the photo. With tures from multiple modalities leads to a modest increase in
this in mind, we started to explore other features that could diversity. In the future we intend to investigate whether au-
work better. tomatically generated classifications from the visual features
Color Naming Histogram (CN). The provided CN his- (indoor/outdoor, etc) can be leveraged to increase diversity.
togram features [4], were used to create a similarity matrix
by using correlation to measure the pairwise similarity.
Scale-Invariant Feature Transform (SIFT). SIFT fea-
5. ACKNOWLEDGMENTS
tures from the images were extracted and hashed using an The described work was funded by the European Union
LSH scheme [3]. A sparse binary similarity matrix was cre- Seventh Framework Programme (FP7/2007-2013) under grant
ated from these, by setting a similarity of 1 to pairs of images agreements 270239 (ARCOMEM), and 287863 (TrendMiner).
in which there was a hash collision.
Time User (TU). Images taken by the same user within 6. ADDITIONAL AUTHORS
a short time period are likely to be similar. A similarity Additional author: Paul Lewis (phl@ecs.soton.ac.uk)
matrix was constructed with the following constraints: pairs
of images taken a less than a minute apart had similarity 1; 7. REFERENCES
images more than 3.25 mins apart had 0 similarity. Between [1] J. Hare, D. Dupplaw, and P. Lewis.
1 and 3.25 minutes the similarity falls off logarithmically. IAM@ImageCLEFphoto 2009: Experiments on
Month Delta (MD). Similar to the TU feature, images Maximising Diversity using Image Features. In CLEF
have increasing similarity with closer month of year. 2009 Workshop, pages 42–42, September 2009.
[2] J. S. Hare, S. Samangooei, and D. P. Dupplaw.
3. EXPERIMENTS AND RESULTS OpenIMAJ and ImageTerrier: Java libraries and tools
Three runs were submitted; their configuration with re- for scalable multimedia analysis and indexing of
spect to the methodology and features described in Section 2 images. In ACM MM’11, pages 691–694. ACM, 2011.
is illustrated in Tables 1a and 1b. Where multiple features [3] J. S. Hare, S. Samangooei, D. P. Dupplaw, and P. H.
were used, the similarity matrices were just averaged to cre- Lewis. Twitter’s visual pulse. In ICMR’13, pages
ate a single matrix. Two major points can be noted from 297–298, New York, NY, USA, 2013. ACM.
the results. Firstly, using textual and visual features outper- [4] B. Ionescu, M. Menéndez, H. Müller, and A. Popescu.
forms the use of either modality alone with our techniques. Retrieving diverse social images at mediaeval 2013:
It is also clear that the reranking stage massively helps im- Objectives, dataset and evaluation. In MediaEval 2013
prove precision. Secondly, the high variability in results Workshop, Barcelona, Spain, October 18-19 2013.