This paper provides an overview of the Retrieving Diverse Social Images task that is organized as part of the MediaEval 2016 Benchmarking Initiative for Multimedia Evaluation. The task addresses the problem of result diversi cation in the context of social photo retrieval where images, metadata, text information, user tagging pro les and content and text models are available for processing. We present the task challenges, the proposed data set and ground truth, the required participant runs and the evaluation metrics.
An efficient image retrieval system should be able to present
results that are both relevant and that are covering different
aspects, i.e., diversity, of the query. By diversifying the
pool of possible results, one can increase the likelihood of
providing the user with the information needed. Relevance
was more thoroughly studied in existing literature than
diversi cation [
The 2016 Retrieving Diverse Social Images task is a
followup of the 2015 edition [
The task is built around the use case of a general ad-hoc image retrieval system, which provides the user with diverse representations of the queries (see for instance Google Image This task is partly supported by the Vienna Science and Technology Fund (WWTF) through project ICT12-010.
Search1). Participants are required, given a ranked list of query-related photos retrieved from Flickr2, to re ne the results by providing a set of images that are at the same time relevant to the query and to provide a diversi ed summary of it. Compared to the previous editions, this year's task includes complex and general-purpose multi-concept queries.
The requirements of the task are to re ne these results by providing a ranked list of up to 50 photos that are both relevant and diverse representations of the query, according to the following de nitions: Relevance: a photo is considered to be relevant for the query if it is a common photo representation of the query topics (all at once). Bad quality photos (e.g., severely blurred, out of focus, etc.) are not considered relevant in this scenario; Diversity: a set of photos is considered to be diverse if it depicts different visual characteristics of the query topics and subtopics with a certain degree of complementarity, i.e., most of the perceived visual information is different from one photo to another.
To carry out the re nement and diversi cation tasks, participants may use the social metadata associated with the images, the visual characteristics of the images, information related to user tagging credibility (an estimation of the global quality of tag-image content relationships for a user's contributions) or external resources (e.g., the Internet). 3.
The 2016 data consists of a development set (devset )
containing 70 queries (20,757 Flickr photos | including 35
multi-topic queries related to events and states associated
with locations from the 2015 dataset [
queries (19,017 Flickr photos).
Each query is provided with the following information: query text formulation (the actual query formulation used on Flickr to retrieve all the data), a ranked list of up to 300 photos in jpeg format retrieved from Flickr using Flickr's default \relevance" algorithm (all photos are Creative Commons licensed allowing redistribution4), an xml le containing metadata from Flickr for all the retrieved photos (e.g., photo title, photo description, photo id, tags, Creative Common license type, the url link of the photo location from Flickr, the photo owner's name, user id, the number of times the photo has been displayed, etc), and ground truth for both relevance and diversity.
Apart from the metadata, to facilitate participation from
various communities, we also provide the following content
descriptors:
- convolutional neural network based descriptors | generic
CNN based on the reference convolutional neural network
(CNN) model provided along with the Caffe framework5
(this model is learned with the 1,000 ImageNet classes used
during the ImageNet challenge); and an adapted CNN based
on a CNN model obtained with an identical architecture to
that of the Caffe reference model. Adaptation is done only
for the 2015 location-based multi-topic queries (35 queries
from the devset), i.e., the model is learned with 1,000 tourist
points of interest classes of which the images were
automatically collected from the Web [
4.
Both relevance and diversity annotations were carried out by expert annotators. For relevance, annotators were asked to label each photo (one at a time) as being relevant (value 1), non-relevant (0) or with \don't know" (-1). For devset, 9 annotators were involved, for credibilityset 9 and for testset 8. The data was partitioned among annotators such that in the end each image has been marked by 3 different annotators. The nal relevance ground truth was determined after a lenient majority voting scheme. For diversity, only the photos that were judged as relevant in the previous step were considered. For each query, annotators were provided with a thumbnail list of all relevant photos. After getting familiar with their contents, they were asked to re-group the photos into clusters with similar visual appearance (up to 25). Devset and testset were annotated by 5 persons, each of them annotating distinct parts of the data (leading to only one annotation). An additional annotator acted as a master annotator and reviewed once more the nal annotations. 5.
Participants were allowed to submit up to 5 runs. The rst 3 are required runs: run1 | automated using visual information only; run2 | automated using text information only; and run3 | automated using text-visual fused without other resources than provided by the organizers. The last 2 runs are general runs: run4 and run5 | everything allowed, e.g., human-based or hybrid human-machine approaches, including using data from external sources (e.g., Internet). For generating run1 to run3 participants are allowed to use only information that can be extracted from the provided data (e.g., provided descriptors, descriptors of their own, etc). 6.
Performance is assessed for both diversity and relevance. The following metrics are computed: Cluster Recall at X (CR@X) | a measure that assesses how many different clusters from the ground truth are represented among the top X results (only relevant images are considered), Precision at X (P@X) | measures the number of relevant photos among the top X results and F1-measure at X (F1@X) | the harmonic mean of the previous two. Various cut off points are to be considered, i.e., X=5, 10, 20, 30, 40, 50. Official ranking metric is the F1@20 which gives equal importance to diversity (via CR@20) and relevance (via P@20). This metric simulates the content of a single page of a typical Web image search engine and re ects user behavior, i.e., inspecting the rst page of results with priority. 7.
The 2016 Retrieving Diverse Social Images task provides participants with a comparative and collaborative evaluation framework for social image retrieval techniques with explicit focus on result diversi cation. This year in particular, the task explores the diversi cation in the context of a challenging, ad-hoc image retrieval system, which should be able to tackle complex and general-purpose multi-concept queries. Details on the methods and results of each individual participant team can be found in the working note papers of the MediaEval 2016 workshop proceedings.