In this paper we attempt to solve the Retrieving Diverse Social Images task by proposing an enhanced version of the method in [2] and studying the influence of its parameters in achieving high retrieval result diversification and relevance. Image search results diversification, visual and textual descriptors.
The 2013 Retrieving Diverse Social Images Task [
Re-ranking techniques are the closest to our approach. Re-ranking
attempts to re-order the initial retrieval results by taking advantage
of the visual content and the additional information, such as textual
data. Many approaches have been proposed in the literature, from
methods that revaluate relational facts about the entities by
estimating a model parameter, to methods proposing functions to optimize
a diversity criterion or methods selecting representative images for
a local group in the set that cover as many distinct groups as
possible and that incorporate an arbitrary pre-specified ranking as prior
knowledge [
Our method, as presented in the sequel, selects from a given set of N retrieved images a small set of F images that are relevant and diverse representations of the query. First, it ranks the images in terms of representativeness using the similarity to the rest of the set. Then, all images are clustered and a small number of diverse images coming from different clusters are selected. Finally, a diversity rank is given by means of the dissimilarity to the rest of the This research was partially supported by the CUbRIK project, grant agreement n287704, FP7/2007-2013. selected images. A mediation between the two ranks guarantees the representativeness and diversity in images: step 1: each image in the initial set is described using different combinations of descriptors. Further, in order to assess image similarity, we compute the Euclidean distance between the corresponding feature arrays and then construct a Synthetic Representative Image Feature (SRI) by averaging all distances. step 2: a N-dimensional array is obtained by computing for each image the average of the Euclidean distances to the rest of the images. The value of SRI is subtracted from the new array which is further sorted in ascending order. The position of each value in the sorted array will be the new rank in terms of representativeness for the corresponding image. step 3: all re-ranked images are clustered in M clusters using a k-means approach. step 4: for each cluster a SRIj value is computed and a new reranking is performed. From each cluster, a small equal number of best ranked images are selected to totally sum F best representative images. step 5: another array is obtained by computing for all F images previously selected the average of the Euclidean distances to the rest F 1 images. The new array is sorted in descending order and the position of each value in the sorted array will be the new rank in terms of diversity for the corresponding image. step 6: the average between the representativeness and diversity ranks is computed, resulting another array which is sorted in ascending order. Images are, thus, arranged and returned according to their final position in the sorted array. 4.
The performance of our approach is influenced by a series of
parameters: the descriptors and the number M of clusters to be built
from all the images. We will first calibrate the method by
experimenting on the development dataset using the provided visual and
textual descriptors (i.e., color histograms, Histogram of Oriented
Gradients, color moments, Locally Binary Patterns, MPEG-7 color
structure descriptor, run-length matrix statistics and spatial pyramid
representation of these descriptors, textual models [
For the development dataset of 50 locations several tests were performed by varying the parameters of the method as previously mentioned. Thus, different visual and textual descriptors combinations were tested while the number M of clusters was independently modified to 10 and 20. Figure1 presents the results obtained for the 1textual models are represented through histograms. 0.7 0.65 0.6 0.55 0.5 0.45 0.4 0.35
Prob&soctifdf&tifdf HOG Prob&CSD Soctifdf&tifdf Flyckr initial Prob&soctifdf Random run CN3x3 CM3x3 0.8 0.7 0.6 0.5 0.4 0.3 official evaluation metrics (cluster recall at 10 - CR@10, precision at 10 - P @10 and the harmonic mean of CR@10 and P @10 - F 1 measure@10) when M equals 10. For space reasons, we didn’t graphically include the results obtained for M set to 20, since the overall results are less accurate. As Figure 1 a) depicts, the combination of all textual descriptors (the probabilistic model plus TF-IDF weighting and Social TF-IDF weighting) returns the best results among all visual and/or textual combinations in terms of the main evaluation metric (CR@10). That one is closely followed by the solely HOG visual descriptors and the combination between a textual (the probabilistic model) and a visual descriptor (CSD). The rest of descriptors’ combination depicted in Figure 1 a) also return close results to the top 3 combinations.
On the other hand, when taking in consideration both CR@10 and P @10, thus evaluating F 1 measure@10, the results were also very close between the top combinations in terms of CR@10.
Textual descriptors perform better because they explicate better, when chosen carefully, the content and the details of the images than the visual descriptors that depict them in a simplified way. 4.2
Following the previous experiments, we submitted four official runs computed as following: run1 - visual information only (using HOG descriptor), run2 - textual information only (using all provided textual descriptor, i.e., probabilistic model, term frequencyinverse document frequency (TF-IDF) weighting and social TFIDF weighting), run3 - textual and visual fused information (using probabilistic model and CSD descriptors) and run5 - everything allowed (using CM3x3 descriptor).
Average results obtained in the official runs on the testing dataset are displayed in Table 1. The ground truth returned by the crowd is reported averagely. The overall results obtained using the expert annotation are close to the ones obtained using the crowd annotation in terms of precision. Instead, the evaluation on the crowd generated ground truth lead to significantly higher cluster recall.
For the expert annotation, the best results in terms of the main evaluation metrics (CR@10) are achieved using a combination of all provided textual descriptors, thus only textual information. Moreover, the same combination offers the best results when considering both CR@10 and P @10, thus evaluating F 1 measure@10.
For the crowd annotation, the best results for CR@10 are achieved for the combination between a visual and a textual descriptor. In terms of F 1 measure@10, the general run obtained using CM3x3 run tre run2 exp run3 run1 run5
P
CR
F 1 0,6901 0,6684 0,7371 0,6878 0,7163 0,6796 0,7143 0,6889 0,6813 0,7254 0,6898 0,7255 0,6929 0,7327 0,3631 0,3498 0,3742 0,7281 0,7407 0,7514 0,7322 0,5533 0,5444 0,5614 0,8594 0,8583 0,8653 0,8606 0,4582 0,438 0,4726 0,6676 0,6941 0,6675 0,6942 0,5915 0,5795 0,6067 0,7393 0,7641 0,744 0,77 visual descriptor returned the best results. 5.
We have presented a method for refining a set of noisy images retrieved from the web in terms of representativeness and diversity. Based on an extensive evaluation, our method proves to achieve great potential that overcome the initial retrieval using a broad range of visual and textual descriptors, leading to a precision up to 0:7371.