In this paper, we attempt to tackle the MediaEval 2013 Retrieving Diverse Social Images challenge, which is a lter and re nement problem on a Flickr-based ranked set of social images. We developed three di erent approaches, using visual data, textual data and a combination thereof, respectively. Hierarchical clustering on highly relevant images, combined with a greedy approach to complement the ranking, forms the basis of our approach.
In this paper, we describe our approach for tackling the
MediaEval 2013 Retrieving Diverse Social Images Task [
We suggest a cluster-based approach for the visual run and a semantic similarity-based approach for the textual run. The third run focuses on hierarchical clustering of relevant images and represents a combination of the purely visual and textual techniques.
We propose a hierarchical clustering-based approach for
the ranking of images in accordance with their relevance
and diversity for a speci c location. This method builds on
the approach provided in [
The set of m images is hierarchically clustered to produce k clusters. Similarity between two images xi and xj (represented by a CN3x3 and LBP3x3 feature vector) is measured using a Gaussian kernel: s(xi; xj) = exp jjxi 2 xjjj22 For each of the k clusters produced, we calculate a Synthetic Representative Image (SRI). The SRI acts as a representative image for all the images within a particular cluster. The SRI for a set of images is calculated by taking the mean of all the corresponding feature vectors of these images. Intra-cluster ranking of all of the m images is produced by calculating their Euclidean distance to the SRI. The image with rank 1 (topmost rank) is closest to the SRI of the cluster.
Subsequently, we rank the k di erent clusters, again by calculating the distance of the cluster SRIs to a general SRI value, which is the mean over all cluster SRIs. Finally, n images are selected by iterating over the ranked clusters and taking the topmost ranked image within each cluster. 3.
The textual run makes use of information derived from
tags and other textual metadata. This approach aims at
diversifying the results by reranking the images retrieved from
Flickr using textual relevance and semantic similarity. Our
solution is based on [
We implemented a greedy approach that optimizes an estimation of the ADP measurement. Let denote an ordering of the images, and let (i) be the image at the position of rank i (a lower number indicates an image with a higher rank). With the top i 1 documents established, we can derive that the ith image should be decided as follows: (i) = arg max x2D S
Rel(x) i
Div(x)(C + Div(x)) ; where
S = f (1); (2); : : : ; (i
1)g ; i 1 C = X Rel( (k))Div( (k));
k=1 with Rel(x) and Div(x) denoting the estimated relevance and diversity of the image, respectively. 3.1
We estimate the relevance of an image by making use of textual metadata (number of views, number of comments) (2) (3) (4)
Development Set (comparison with Flickr) + GPS 43.6 (2.4) 44.2 (2.9) 79.2 (-6.8) 81.6 (-4.4)
- GPS 48.4 (2.0) 51.6 (5.2) 71.6 (2.8) 67.2 (-1.6) and social tags. This data was provided together with three textual features: TF{IDF, Social TF{IDF and a probabilistic feature type. We suggest a linear combination of this normalized data.
Rel(x) = tagsx + viewsx + commentsx; (5) where viewsx and commentsx denote normalized number of views and number of comments, respectively and tagsx =
Pt2Tx a probt;x + b t dft;x + c st dft;x jTxj ; (6) with Tx denoting the set of tags linked to image x. We found that the precision of image orderings, which are purely based on relevance information, can be maximized by setting parameters a and to zero and increasing the impact of the parameters linked to 'tag' and 'views' information. 3.2
Diversity of an image to a ranking is de ned as the minimal di erence with the other images in that ranking: Div(x) =
min i2f1;:::;ng f 1 s(x; (i))g (7)
We use a semantic similarity measure based on Google distance to assess the di erence between two images. The average of the summation of the similarities between the di erent tags of both images gives us a value that describes the semantic similarity between both images.
To estimate the relevance of an image, we use the textualbased method (cf. Section 3.1). Similarity between images is based on visual image features and a Gaussian kernel method to measure the di erence between two image vectors (cf. Equation 1). In order to provide both a relevant and diverse ordering, we make use of hierarchical clustering techniques. We, again, try to re ne a set of m images retrieved from Flickr to a ranking of size n.
First, l images are selected with the highest estimated relevance. l is an arbitrary number that depicts a subset of the nal ranking. The larger l becomes, the more the focus will shift from relevance to diversity.
Next, these l images are hierarchically clustered based on a distance matrix calculated with the above described visual similarity. Per cluster, the most relevant item is selected and added to the nal ranking.
Depending on the number of remaining places of the rst l spots in the nal ranking, images are greedily added based on a gain score. This score is higher for images that maximize the diversity and relevance related to the current ranking.
When the rst l spots in the ranking are lled, the algorithm runs from the start until all n spots are taken. 83.0 77.7 83.6 72.5 65.7 69.4 78.7 72.1 77.4 72.5 65.7 69.4 69.9 63.5 66.1 72.5 65.7 69.4 5.
In Table 1, we can see the results of our algorithms (three runs), compared to the regular Flickr ranking and the results on the test set. We notice that the combined method creates the most precise and diverse rankings based on both precision and cluster recall measures (averages 42.1% cluster recall on test set). Analysis of the results on the development set indicate the importance of nding the right balance between relevance and diversity. We also observe the similarity of the visual run and the textual run in terms of e ectiveness. Table 2 lists the results on 50 locations of the test set evaluated with crowd-sourcing ground truths.
We observe that a method clustering images focused on high relevance outperforms all others. This method uses textual data for the relevance estimation and visual features for the diversity assessment.