The Mediaeval 2013 Retrieving Diverse Social Image Task addresses the challenge of improving both relevance and diversity of photos in a retrieval task on Flickr. We propose a clustering based technique that exploits both textual and visual information. We introduce a k-Nearest Neighbor (kNN) inspired re-ranking algorithm that is applied before clustering to clean the dataset. After the clustering step, we exploit social cues to rank clusters by social relevance. From those ranked clusters images are retrieved according to their distance to cluster centroids.
Photo sharing is a frequent activity on social media
platforms, such as Flickr and Facebook. Since there is no quality
control of the photos and of their annotations, retrieved
images for a given query are usually not su ciently relevant.
Even when the relevance criteria is met, it is still di cult to
obtain diversi ed representations [
Considering these challenges, the MediaEval 2013
Retrieving Diverse Social Images Task addresses the problem of
diversi cation of social image search results (see [
In the following, we summarize our e orts in solving the challenges mentioned above through exploiting both textual and visual features, as well as social cues. 2.1
We exploit visual features in order to overcome the sparsity of textual annotations since there are only few tags for each image. We use the Histogram of Gradients (HOG) in our experiments. In addition, we extracted GIST and bags of visual words (BOVW) based on dense SIFTs that proved to be e cient in large scale image retrieval. Dense SIFT descriptors are extracted using a codebook of size 1024. A spatial pyramid model with 2 levels is used and the resulting feature size is 8192. HOG, GIST and BOVW features capture di erent low-level characteristics of images and they can be combined to have more comprehensive visual representations. Since these features can be combined in di erent kind of runs, all features were L1-normalized in order for each of feature to have the same contribution, regardless of their size. 2.2
We exploit a classical TF-IDF weighting scheme to model textual information associated to points of interests (POIs). Di erent re-weighting schemes were tested and the best results were obtained when we took the square root of TF-IDF scores. The dimension of the model is equal to the number of unique tags associated to each POI. Given the limited number of images per POI (up to 150), the dimension is usually in the range of hundreds. Similar to visual features, L1-normalization is applied to textual features. 3.
The initial result set of Flickr is noisy and we introduce a k-NN inspired approach that exploits visual and social cues to rerank results to reduce this noise. We considered all the images of the POI as a positive set and constructed a negative set of the same size by sampling images of other POIs from the collection. Then we compared the GIST features of each image to all other images' features from positive and negative sets. The resulting top ve most visually similar images are retained to be considered in the next steps. The top ve neighbors from both the positive and the negative sets of each image that depict a POI are considered to produce a reranked results list according to social cues. For each image we counted the number of di erent users that contributed to the top ve neighbors and the number of top ve neighbors that belong to the positive set. Finally the average distance to the rst ve positive neighbors of the target image is considered. These criteria were cascaded to break ties. Images are then ranked by cascading the three scores described and we used the top 70%, 80% and 90% of the reranked images. These images are given as an input for the clustering process. The best results are obtained on the devset with 70% of the initial list retained and this threshold is retained for clustering. 4.
k-means++ [
Clusters are not all born equal and we need to be able to
rank them by probability of relevance of contained images.
Inspired by [
For each POI, we retain only the top 10 clusters obtained with the cluster ranking procedure and then diversify images by choosing one image from each cluster by descending similarity to the cluster centroid.
To address the diversi ed social image retrieval problem, participants are asked to submit di erent types of runs. We submitted four runs that were produced by using different types of features and their combinations on the same dataset. The submitted four runs are described below: Run1 - visual is based on visual features. We concatenate HOG and GIST features described in Section 2.1. Run2 - textual is produced using only textual features described in Section 2.2. Run3 - textual-visual is produced using a combination of textual features described in Section 2.2 and GIST features in Section 2.1. Visual and textual features are concatenated and to produce feature vectors. Linear weighting is used with 0.7 and 0.3 weights that are given to visual and textual features respectively. These weights were empirically chosen by testing di erent combinations on the devset. Run5 - everything allowed is similar to Run 3 - textualvisual, the only modi cation being the replacement of HOG features by BOVW features.
The results in Table 1 are based on expert evaluation on 346 testset locations. Table 2 shows the average results of crowd sourcing evaluation carried out only on a subset of 50 locations from the testset via the CrowdFlower platform. Relevance ground truth is based on a majority voting scheme on the annotations and the diversity ground truth is determined by the same three annotators. Both tables show that there are only small di erences between the four runs. We were surprised to see how well the textual run (Run2) performed compared to visual and multimedia runs since we had expected visual diversi cation to work better for POIs, which usually have a limited number of visual aspects. The performance drop from Run3 to Run5, due to the replacement of HOG features with BOVW features also came as a surprise since the latter usually work well for retrieval processes over visually diversi ed datasets. 7.
In this paper we propose a clustering based technique using textual and visual features. To re-rank the images before clustering we introduced a kNN inspired technique. This technique helped us to spot strongly connected images of a topic. We further apply cluster ranking method based on social cues to increase the diversity and the relevancy. Table 1 and Table 2 show that there are no big di erences between di erent runs. Performance of Run2 - textual and Run5 everything allowed is a surprise for us. Since BOVW representation generally work well for retrieval processes. We would expect from the visual representations to improve the results more than the textual features. 8.
This research was supported by the MUCKE project funded within the FP7 CHIST-ERA scheme and also Scienti c and Technical Research Council of Turkey (TUBITAK) under grant number 112E174.