This paper provides an overview of the Retrieving Diverse Social Images task that is organized as part of the MediaEval 2015 Benchmarking Initiative for Multimedia Evaluation. The task addresses the problem of result diversi cation and user annotation credibility estimation in the context of social photo retrieval. We present the task challenges, the proposed data set and ground truth, the required participant runs and the evaluation metrics.
An efficient image retrieval system should be able to present
results that are both relevant and that are covering different
aspects, i.e., diverse, of the query. Relevance was more
thoroughly studied in existing literature than diversi
cation [
The 2015 Retrieving Diverse Social Images task is a
followup of last years' editions [
The task is built around a tourist use case where a person tries to nd more information about a place she is potenThis work is supported by the European Science Foundation, activity on \Evaluating Information Access Systems". yThis work is supported by the CHIST-ERA FP7 MUCKE - Multimodal User Credibility and Knowledge Extraction project (http://ifs.tuwien.ac.at/ mucke/). zThis work has been funded by the Ministry of European Funds through the Financial Agreement POSDRU 187/1.5/S/155420. tially visiting. Before deciding whether this location suits her needs, the person is interested in getting a more complete and diversi ed visual description of the place.
Participants are required to develop algorithms to automatically re ne a list of images that has been returned by Flickr in response to a query. Compared to the previous editions, this year's task includes not only single-topic queries (i.e., formulations such as the name of a location), but also multi-concept queries related to events and states associated with locations. The requirements of the task are to re ne these results by providing a ranked list of up to 50 photos that are both relevant and diverse representations of the query, according to the following de nitions: Relevance: a photo is considered to be relevant if it is a common photo representation of all query concepts at once. This includes sub-locations (e.g., subsuming indoor/outdoor, close up), temporal information (e.g., historical shots, times of day), typical actors/objects (e.g., people who frequent the location, vehicles), genesis information (e.g., images showing how something got the way it is), and image style information (e.g., creative views). Low quality photos (e.g., severely blurred, out of focus, etc) as well as photos with people as the main subject (e.g., a big picture of me in front of the monument) are not considered relevant in this scenario; Diversity: a set of photos is considered to be diverse if it depicts different visual characteristics of the target concepts, e.g., sub-locations, temporal information, typical actors/objects, genesis and style information, etc with a certain degree of complementarity, i.e., most of the perceived visual information is different from one photo to another.
To carry out the re nement and diversi cation tasks, participants may use social metadata associated with the images, the visual characteristics of the images, information related to user tagging credibility (an estimation of the global quality of tag-image content relationships for a user's contributions) or external resources (e.g., Internet). 3.
The 2015 data consists of a development set (devset )
containing 153 location queries (45,375 Flickr photos | the
2014 dataset [
Each query is provided with the following information: query text formulation (used to retrieve the data), GPS coordinates (latitude and longitude in degrees | only for single-topic location queries), a link to a Wikipedia webpage (only when available), up to 5 representative photos from Wikipedia (only for single-topic location queries), a ranked list of up to 300 photos retrieved from Flickr using Flickr's default \relevance" algorithm (all photos are Creative Commons licensed allowing redistribution, see http: //creativecommons.org/), and an xml le containing metadata from Flickr for all the retrieved photos (e.g., photo title, photo description, photo id, tags, Creative Common license type, number of posted comments, the url link of the photo location from Flickr, the photo owner's name, user id, the number of times the photo has been displayed, etc).
Apart from the metadata, to facilitate participation from
various communities, we also provide content descriptors:
general purpose visual descriptors (e.g., color, texture and
feature information) identical to the ones in 2014 [
Both relevance and diversity annotations were carried out by expert annotators with advanced knowledge of the location characteristics (mainly learned from last years' tasks and Internet sources). For relevance, annotators were asked to label each photo (one at a time) as being relevant (value 1), non-relevant (0) or with \don't know" (-1). For devset, 11 annotators were involved, for credibilityset 9 and for testset single-topic 7 and multi-topic 5. Each annotator annotated different parts of the data leading in the end to 3 different annotations for each photo. The nal relevance ground truth was determined after a lenient majority voting scheme. For diversity, only the photos that were judged as relevant in the previous step were considered. For each location, annotators were provided with a thumbnail list of all relevant photos. After getting familiar with their contents, they were asked to re-group the photos into clusters with similar visual appearance (up to 25). Devset and testset were annotated by 3 persons, each of them annotating distinct parts of the data (leading to only one annotation). An additional annotator acted as a master annotator and reviewed once more the nal annotations. 5.
Participants were allowed to submit up to 5 runs. The rst 3 are required runs: run1 | automated using visual information only; run2 | automated using text information only; and run3 | automated using text-visual fused without other resources than provided by the organizers. The last 2 runs are general runs: run4 | automated using user annotation credibility descriptors (either the ones provided by organizers or computed by the participants) and run5 | everything allowed, e.g., human-based or hybrid human-machine approaches, including using data from external sources (e.g., Internet). For generating run1 to run4 participants are allowed to use only information that can be extracted from the provided data (e.g., provided descriptors, descriptors of their own, etc). This includes also the Wikipedia webpages of the locations (via their links). 6.
Performance is assessed for both diversity and relevance. The following metrics are computed: Cluster Recall at X (CR@X) | a measure that assesses how many different clusters from the ground truth are represented among the top X results (only relevant images are considered), Precision at X (P@X) | measures the number of relevant photos among the top X results and F1-measure at X (F1@X) | the harmonic mean of the previous two. Various cut off points are to be considered, i.e., X=5, 10, 20, 30, 40, 50. Official ranking metric is the F1@20 which gives equal importance to diversity (via CR@20) and relevance (via P@20). This metric simulates the content of a single page of a typical Web image search engine and re ects user behavior, i.e., inspecting the rst page of results with priority. 7.
The 2015 Retrieving Diverse Social Images task provides participants with a comparative and collaborative evaluation framework for social image retrieval techniques with explicit focus on result diversi cation. This year in particular, the task explores also the diversi cation of multi-concept queries. Details on the methods and results of each individual participant team can be found in the working note papers of the MediaEval 2015 workshop proceedings.