LAPI @ Retrieving Diverse Social Images Task 2013: Qualitative Photo Retrieval using Multimedia Content Anca-Livia Radu1,2∗, Bogdan Boteanu1 , Oana Pleş1 , Bogdan Ionescu1,2 1 The Image Processing and Analysis Laboratory, University “Politehnica” of Bucharest, Romania 2 Department of Information Engineering and Computer Science, University of Trento, Italy ancalivia.radu@unitn.it, bionescu@imag.pub.ro ABSTRACT selected images. A mediation between the two ranks guarantees In this paper we attempt to solve the Retrieving Diverse Social Ima- the representativeness and diversity in images: ges task by proposing an enhanced version of the method in [2] and step 1: each image in the initial set is described using different studying the influence of its parameters in achieving high retrieval combinations of descriptors. Further, in order to assess image sim- result diversification and relevance. ilarity, we compute the Euclidean distance between the correspond- ing feature arrays and then construct a Synthetic Representative Im- Keywords age Feature (SRI) by averaging all distances. step 2: a N-dimensional array is obtained by computing for each Image search results diversification, visual and textual descriptors. image the average of the Euclidean distances to the rest of the im- ages. The value of SRI is subtracted from the new array which is 1. INTRODUCTION further sorted in ascending order. The position of each value in the The 2013 Retrieving Diverse Social Images Task [1] challenged sorted array will be the new rank in terms of representativeness for participants to develop algorithms for selecting a small subset of the corresponding image. representative and diverse images that correctly and completely step 3: all re-ranked images are clustered in M clusters using a summarize a query. Participants were provided with a development k-means approach. dataset containing 50 locations and a testing dataset containing 346 step 4: for each cluster a SRIj value is computed and a new re- locations. The images for both data sets were retrieved from Flickr ranking is performed. From each cluster, a small equal number of using the name of the location as query and also using the name of best ranked images are selected to totally sum F best representative the location and the GPS coordinates [1]. We dealt with the task by images. developing a computer vision and linguistic processing algorithm step 5: another array is obtained by computing for all F images that only employs visual and/or textual descriptors [2]. previously selected the average of the Euclidean distances to the rest F − 1 images. The new array is sorted in descending order and 2. PREVIOUS WORK the position of each value in the sorted array will be the new rank Re-ranking techniques are the closest to our approach. Re-ranking in terms of diversity for the corresponding image. attempts to re-order the initial retrieval results by taking advantage step 6: the average between the representativeness and diversity of the visual content and the additional information, such as textual ranks is computed, resulting another array which is sorted in as- data. Many approaches have been proposed in the literature, from cending order. Images are, thus, arranged and returned according methods that revaluate relational facts about the entities by estimat- to their final position in the sorted array. ing a model parameter, to methods proposing functions to optimize a diversity criterion or methods selecting representative images for 4. EXPERIMENTAL RESULTS a local group in the set that cover as many distinct groups as possi- ble and that incorporate an arbitrary pre-specified ranking as prior The performance of our approach is influenced by a series of pa- knowledge [3] [4] [5]. rameters: the descriptors and the number M of clusters to be built from all the images. We will first calibrate the method by experi- menting on the development dataset using the provided visual and 3. OUR APPROACH textual descriptors (i.e., color histograms, Histogram of Oriented Our method, as presented in the sequel, selects from a given set Gradients, color moments, Locally Binary Patterns, MPEG-7 color of N retrieved images a small set of F images that are relevant and structure descriptor, run-length matrix statistics and spatial pyramid diverse representations of the query. First, it ranks the images in representation of these descriptors, textual models [1]1 ) . Then, we terms of representativeness using the similarity to the rest of the report the official results obtained on the testset. set. Then, all images are clustered and a small number of diverse images coming from different clusters are selected. Finally, a di- 4.1 Results on devset versity rank is given by means of the dissimilarity to the rest of the For the development dataset of 50 locations several tests were ∗ performed by varying the parameters of the method as previously This research was partially supported by the CUbRIK project, grant agreement n287704, FP7/2007-2013. mentioned. Thus, different visual and textual descriptors combina- tions were tested while the number M of clusters was independently modified to 10 and 20. Figure1 presents the results obtained for the Copyright is held by the author/owner(s). 1 MediaEval 2013 Workshop, October 18-19, 2013, Barcelona, Spain textual models are represented through histograms. 0.82 P CR 0.75 F1-measure 0.8 0.8 0.7 0.78 0.7 0.76 0.65 0.6 0.74 0.6 0.72 Prob&soctifdf&tifdf 0.55 HOG 0.5 0.7 Prob&CSD 0.5 Soctifdf&tifdf 0.68 Flyckr initial 0.4 Prob&soctifdf 0.66 0.45 Random run 0.64 0.4 CN3x3 0.3 CM3x3 0.62 0.35 @5 @10 @20 @30 @40 @5 @10 @20 @30 @40 @5 @10 @20 @30 @40 Figure 1: Official evaluation metrics for M = 10 (experiments on devset). official evaluation metrics (cluster recall at 10 - CR@10, preci- Table 1: Official evaluation results (∗ - evaluation on a selection of sion at 10 - P @10 and the harmonic mean of CR@10 and P @10 50 locations) - F 1 − measure@10) when M equals 10. For space reasons, we didn’t graphically include the results obtained for M set to 20, run P CR F1 since the overall results are less accurate. As Figure 1 a) depicts, @10 @20 @10 @20 @10 @20 the combination of all textual descriptors (the probabilistic model run1 0,6901 0,6889 0,3631 0,5533 0,4582 0,5915 plus TF-IDF weighting and Social TF-IDF weighting) returns the expert run2 0,717 0,7111 0,3774 0,5734 0,4736 0,6078 best results among all visual and/or textual combinations in terms run3 0,6684 0,6813 0,3498 0,5444 0,438 0,5795 of the main evaluation metric (CR@10). That one is closely fol- lowed by the solely HOG visual descriptors and the combination run5 0,7371 0,7254 0,3742 0,5614 0,4726 0,6067 between a textual (the probabilistic model) and a visual descriptor run1 0,6878 0,6898 0,7281 0,8594 0,6676 0,7393 crowd∗ (CSD). The rest of descriptors’ combination depicted in Figure 1 a) run2 0,7163 0,7255 0,7407 0,8583 0,6941 0,7641 also return close results to the top 3 combinations. run3 0,6796 0,6929 0,7514 0,8653 0,6675 0,744 On the other hand, when taking in consideration both CR@10 run5 0,7143 0,7327 0,7322 0,8606 0,6942 0,77 and P @10, thus evaluating F 1 − measure@10, the results were also very close between the top combinations in terms of CR@10. Textual descriptors perform better because they explicate better, visual descriptor returned the best results. when chosen carefully, the content and the details of the images than the visual descriptors that depict them in a simplified way. 5. CONCLUSIONS We have presented a method for refining a set of noisy images 4.2 Official runs retrieved from the web in terms of representativeness and diversity. Following the previous experiments, we submitted four official Based on an extensive evaluation, our method proves to achieve runs computed as following: run1 - visual information only (using great potential that overcome the initial retrieval using a broad range HOG descriptor), run2 - textual information only (using all pro- of visual and textual descriptors, leading to a precision up to 0.7371. vided textual descriptor, i.e., probabilistic model, term frequency- inverse document frequency (TF-IDF) weighting and social TF- 6. REFERENCES IDF weighting), run3 - textual and visual fused information (using [1] B. Ionescu, M. Menéndez, H. Müller, A. Popescu, probabilistic model and CSD descriptors) and run5 - everything al- "Retrieving Diverse Social Images at MediaEval 2013: lowed (using CM3x3 descriptor). Objectives, Dataset and Evaluation", MediaEval 2013 Average results obtained in the official runs on the testing dataset Workshop, October 18-19, Barcelona, Spain, 2013. are displayed in Table 1. The ground truth returned by the crowd is reported averagely. The overall results obtained using the expert [2] A.-L. Radu, J. Stöttinger, B. Ionescu, M. Menéndez, F. annotation are close to the ones obtained using the crowd anno- Giunchiglia, "Representativeness and diversity in photos via tation in terms of precision. Instead, the evaluation on the crowd crowd-sourced media analysis". AMR, 2012. generated ground truth lead to significantly higher cluster recall. [3] B. Taneva, M. Kacimi, G. Weikum, "Gathering and ranking For the expert annotation, the best results in terms of the main photos of named entities with high precision, high recall, and evaluation metrics (CR@10) are achieved using a combination of diversity". Int. Conf. on Web Search and Data, 2010. all provided textual descriptors, thus only textual information. More- [4] T. Deselaers, T. Gass, P. Dreuw, H. Ney, "Jointly optimising over, the same combination offers the best results when considering relevance and diversity in image retrieval". ACM Int. Conf. both CR@10 and P @10, thus evaluating F 1 − measure@10. on Image and Video Retrieval, 2009. For the crowd annotation, the best results for CR@10 are achieved [5] X. Zhu, A. Goldberg, J. V. Gael, D. Andrzejewski, for the combination between a visual and a textual descriptor. In "Improving Diversity in Ranking using Absorbing Random terms of F 1−measure@10, the general run obtained using CM3x3 Walks". Int. Conf. HLT-NAACL, 2007.