In this paper, we describe our approach for the MediaEval 2015 Retrieving Diverse Social Images Task. The proposed approach exploits available user-generated textual descriptions and the visual content of the images in a combination with common, unsupervised clustering techniques in order to increase the diversi cation of retrieval results. Preliminary experiments indicate that the approach generalizes well for di erent datasets and achieves comparable results for singleand multi-topic queries.
Manual assessment of the relevance of publicly available
images to a particular query is not feasible due to the
immense amount of data captured and shared daily on social
media platforms. As a result, the automated optimization
of image retrieval results gains constantly in importance.
Next to relevance, the aspect of diversi cation of retrieval
results plays a crucial role in order to reduce the redundancy
in the retrieved images and, thus, to increase the e ciency
in overviewing the underlying data. The MediaEval 2015
Retrieving Diverse Social Images Task [
Previous work in this context shows a broad range of
possible approaches. The original Flickr ranking is commonly
improved by a direct comparison with the corresponding
Wikipedia images [
In this paper, we address relevance re-ranking by means of a similarity score to a reference set of images. This reference set is given by Wikipedia images (if available) or by the top ranked images provided by Flickr. To increase diversi cation, we employ a hierarchical clustering algorithm and compare the performance of recently-introduced powerful visual features with text-based approaches, which are well-established in the context of web mining and retrieval. 2.
We employ a multi-stage work ow for the retrieval of diverse social images, which passes the following steps: 1) data preprocessing, 2) relevance reranking, and 3) image clustering and nal image selection.
In the rst step, data preprocessing, we lter potentially
irrelevant images, i.e., images with humans as the main
subjects and images that are captured far away from the topic of
interest. We employ the OpenCV1 face detector and remove
images with faces of area exceeding 5% of the total image
area. Additionally, if GPS data is available, we measure the
distance between the topic of interest and the
corresponding images and remove those with a Harvesine distance [
The aim of the second stage, relevance reranking, is to
improve the original Flickr rating. Since the provided
Wikipedia images are per de nition representative [
In the third step, image clustering, we aim at nding
groups of similar images which can be used to diversify the
nal image results. For the visual-based runs, preliminary
experiments with the provided visual descriptors [
{ CNN CNN
{ CNN
{ CNN CNN { { { TF-IDF TF-IDF LDA LDA
{ CNN CNN CNN TF-IDF TF-IDF LDA LDA CNN
For the text-based runs we consider two approaches. First, we perform topic modeling on the textual descriptions of each image (title and tags) using Latent Dirichlet Allocation (LDA) and the MALLET Toolbox2 and extract T topics for the employed dataset. For each image, we estimate the likelihoods l1 and l2 of the rst- and second-best matching topics. If the di erence of the likelihoods is larger than a threshold (l2=l1 < ) the most likely topic (l1) is assigned to the photo otherwise no topic is assigned. We set T = 50 and = 0:8 for all experiments.
The second text-based approach considers the well-established term frequency-inverse document frequency (TF-IDF). We compute the TF-IDF vector for each image using the complete textual description (title, tags, and descriptions). The textual descriptions are rst preprocessed to increase their expressiveness, i.e., we remove potential occurrences of the corresponding user name, web links, and stopwords and we additionally stem all remaining terms. Furthermore, we account for images with missing textual descriptions. In such a case, we search for timely closest image with a description which is either captured within a prede ned radius (10 meter in our experiments) or by the same user within a prede ned short time span (e.g., 5 minutes). In the following, we cluster the resulting TF-IDF vectors using again the AHC method, whereas the similarity between the TF-IDF vectors is measured using the cosine similarity. The selection of the nal image set follows the Round-Robin approach as described for the visual-based approach.
Table 1 presents a selection of our preliminary experiments on the development dataset. The results show that the prerpocessing step (face and GPS lter) only marginally improves the performance for the top 20 retrieved images in comparison to the Flickr baseline results. Nevertheless, 95% of the rejected images are irrelevant with respect to the underlying search query. Experiments with the text-based runs show only minor di erences in the performance of the TF-IDF and the LDA-based methods. While the achieved precision (P@20) is comparable to those of the Flickr baseline, the cluster recall (CR@20) improves notably, e.g. from 0:34 to 0:46 using the TF-IDF approach. For the visual2http://mallet.cs.umass.edu.
Run 1 (V) 2 (T) 3 (V,T) 5 (V,T) based runs, the consideration of the relevance reranking step using the CNN features demonstrates a signi cant increase in the relevance (P@20-score of 0:94). However, the drop in the clustering recall indicates an increase of redundancy in the retrieved images as a side-e ect. Overall, the bestperforming text-based and visual-based runs are comparable in terms of F1@20 with the computational costs for the textbased runs being signi cantly lower. The multimodal runs additionally slightly improve both the clustering recall and the F1-scores by approximately 1%. Surprisingly, the consideration of the reranking step in a combination with the text-based image clustering and selection cannot compensate for the drop in the clustering recall.
Following our preliminary experiments we submitted four runs corresponding to the best con guration for the respective modality (see Table 2). Table 3 summarizes the results of the o cial runs on the test dataset. In opposite to the development data, which contains the retrieval results of single-topic queries only, the test data di erentiates between single- (e.g., Niagara Falls) and multi-topics queries (e.g., Academy awards in Hollywood ). Overall, there is no signi cant di erence in the performance for the two subsets. While the (predominantly) visual-driven runs (runs 1 and 3) show a slight decrease in the clustering recall for the multitopic queries, the text-driven runs (runs 2 and 4) indicate the opposing trend. Furthermore, in contrast to the results on the development data, the test runs show notable di erence between the performance of the text- and the visualbased runs. This reveals the better generalization ability of the visual-based runs to di erent datasets. Overall, the best performance in terms of F1-score of 0:55 is achieved by the visual-based run which additionally considers the face and GPS lters to reject irrelevant images (run 3). 4.
In this paper we investigated both text- and visual-driven approaches for the diversi cation of Flickr image retrieval results. The achieved performances indicate that the visualbased approach copes well with di erent data and varying query types. Overall, the relevance ranking shows promising results in terms of precision. However, the diversi cation increases only slowly by means of clustering recall. Our future work will exploit the potential of combining features of different modalities in the clustering process, e.g. by means of a late fusion approach.
This work has been partly funded by the Vienna Science and Technology Fund (WWTF) through project ICT12-010.