USEMP: Finding Diverse Images at MediaEval 2015

Abstract

We describe the participation of the USEMP team in the Retrieving Diverse Social Images Task of MediaEval 2015. Our runs are produced based on a supervised diversi cation method that jointly optimizes relevance and diversity. All runs are automated and use only resources given by the task organizers. Our best results in terms of the o cial ranking metric on the one-topic part of the test set came by the runs that combine visual and textual information while the textual-only run performed better on the multi-topic part.

1. INTRODUCTION

The Retrieving Diverse Social Images task of MediaEval 2015 [ 7 ] deals with the problem of result diversi cation in social image retrieval. This year there are two notable di erences with previous editions: (1) a larger development set is available and (2) \multi-concept" queries which are mainly related to events rather than speci c places or landmarks were introduced.

We deal with the task using supervised Maximal Marginal Relevance (sMMR) [ 10 ], a re ned version of the supervised diversi cation method that we developed in [ 12 ]. sMMR and earlier versions of the method are discussed in Section 2. Section 3 gives further details about our methodology and describes modi cations compared to [ 10 ]. Section 4 provides descriptions of the employed features and Section 5 describes the submitted runs. Finally, Section 6 presents and discusses the experimental results.

PREVIOUS WORK

In the 2013 edition of the task, the SocialSensor team developed a supervised diversi cation method [ 3 ] and applied it for producing the visual-only run that achieved the best performance among runs of this type. Similarly to previous diversi cation methods [ 2, 5 ], that method greedily optimized a utility function that jointly accounts for relevance and diversity. The main di erence compared to earlier approaches was the replacement of the unsupervised de nition of relevance with a task-speci c de nition that is learned directly from the ground truth. More speci cally, instead of computing an image's relevance score by measuring its similarity to a reference image (e.g., the Wikipedia image of a query topic), the approach exploited the available relevance annotations (for the images of the development set) to train a classi er which was then used to assign relevance scores to images of unseen queries. Although that classi er was not adapted to any particular query, it could accurately estimate relevance by capturing its task-speci c notion.

In 2014, the SocialSensor team re ned their approach [ 12 ], topping the scoreboard in several categories: best visualonly run, third textual run and best visual+textual run that was also ranked second overall, slightly surpassed by a run that used specialized lters (face/blur detectors) and user credibility information [ 4 ]. The main addition compared to [ 3 ] was that a di erent relevance classi er was trained for each query, using the query's Wikipedia images as additional positive examples. These examples were assigned a larger weight to have increased in uence on the learned model. Thus the query-speci c notion of relevance was captured in addition to the task-speci c notion captured in [ 3 ].

The approach was evaluated in [ 10 ] in more detail identifying a link between relevance detection accuracy and diversi cation performance. Furthermore, a multimodal ensemble classi er, called Multi-Modal Stacking (MMS), was proposed for combining di erent types of features for relevance detection in a principled manner. Due to the addition of this multimodal scheme and to the use of state-of-the-art convolutional neural network features for relevance detection, [ 10 ] managed to achieve a 5.7% relative increase over the best result obtained in the 2014 [ 4 ] edition of the task. 3.

METHOD

Given the e ectiveness of sMMR in previous editions of the task, we opted for applying it in this year's task as well. In particular, we applied the sMMRaq variant: the relevance detection model for each query was trained using relevant and irrelevant examples from other queries, combined with representative examples of the query itself (in the form of either the corresponding Wikipedia images or the Wikipedia page). For multi-topic queries, which were not accompanied by representative Wikipedia images, the visual relevance models were trained using only examples from other queries (sMMRa variant).

To further improve the relevance detection models compared to [ 10 ], we performed careful tuning of two parameters: a) the number no of examples from other queries employed by each model and b) the ratio r = nnoe de ned as the number of examples of this query1 divided by the 1ne is modi ed by repeating each representative example nnee times, where n e is the actual number of representative examples. number of examples from other queries, by performing a grid search over the values no = f1K; 5K; 10K; 20Kg and r = f0:0; 0:1; : : : ; 1:0g. Model selection was based on area under ROC (AUC), computed using a modi ed leave-one(query)-out cross-validation procedure. For each query of the development set, no examples were randomly selected from other queries and combined (if possible) with ne representative (Wikipedia) examples of that query to build a model that was evaluated on the remaining (Flickr) examples of that query. The per-query AUC scores were then averaged to obtain a single estimate. As in [ 10 ], an L2-regularized Logistic Regression classi er was used [ 6 ] with appropriate tuning of the c parameter. Besides the parameters of the relevance model, we also tuned the w and N parameters of the sMMR approach, so as to maximize F1@20 on the development set, as done in [ 12 ]. 4.

FEATURES

VLAD: VLAD+CSURF [ 11 ] vectors were computed from a 128-dimensional visual vocabulary and projected to 128 dimensions with PCA and whitening. Both the visual vocabulary and the PCA projection matrix are learned using the images of the development set.

CNN : Convolutional neural network features were adapted for the tourism use case using 1,000 Points Of Interest (POIs) instead of ImageNet classes. These features were computed by ne-tuning the VGG model proposed by [ 9 ]. Approximately 1,200 images were collected for each POI and fed directly to Ca e [ 8 ] for training. This change of training classes was inspired by recent domain adaptation work presented in [ 1 ] which shows that the feature transfer is more e cient when the training classes are conceptually close to the target dataset. The features are constituted by the outputs of the fc7 layer and include 4; 096 dimensions.

BOW : To generate textual features, we transformed each query and each Flickr image into a text document. For queries, we used a parsed version of the corresponding Wikipedia page and for Flickr images we used a concatenation of the words in their titles, descriptions and tags. Bag-of-words features (BOW) were then computed for each document using all terms that appear at least twice in the collection to form the dictionary, and word frequencies as term weights. This led to an 80K-dimensional representation.

META: The following one-dimensional features were also computed from textual metadata and used as additional features in the meta input space of the MMS algorithm: distance from POI (only for one-topic queries) and Flickr rank.

RUNS

Run 1: CNN features were used for relevance and VLAD features for diversity. no was set to 20K in both instantiations and r was set to 0:5 for the one-topic instantiation. fw = 0:55; N = 170g and fw = 0:00; N = 120g were used for the one-topic and the multi-topic instantiation respectively.

Run 2: BOW features were used for both relevance and diversity. The following parameters were used: no = 20K, r = 0:4, N = 80 and w = 0:8.

Run 3: A di erent instantiation was used for each part of the collection. MMS was used to combine the outputs of relevance detection models built using CNN and BOW features with one-dimensional META features, and VLAD features were used for diversity in both instantiations. fw = 0:50; N = 220g and fw = 0:55; N = 170g were used for the one-topic and the multi-topic instantiation respectively.

Run 5: This is a variation of run 3 where we use the same relevance detection models but also tune the M parameter of the approach in addition to N and w, as done in [ 12 ]. This resulted into setting fw = 0:4; N = 210; M = 2g for the one-topic instantiation and fw = 0:4; N = 300; M = 5g for the mutli-topic instantiation. 6.

RESULTS AND DISCUSSION

Table 1 shows the performance of the submitted runs on each part of the test collection and estimates of their performance obtained from the development set. The best overall performance on the test set is obtained with run 2. We observe that the performance is much better on the one-topic part. This was expected given the fact that model and parameter tuning was performed on the development set which did not contain examples of multi-topic queries. The best performance in terms of F1@20 on the one-topic part is obtained by runs 3 and 5, followed by run 1 and then run 2. We see that although being slightly over-optimistic ( 5% on average) our F1@20 estimates for the one-topic part are strongly correlated with the nal results and are indicative of the relative run strength.

On the multi-topic part, the best performance is obtained by run 2, followed by run 1 and then runs 3 and 5. The superiority of run 2 over 1 on this part of the collection is attributed to the fact that representative examples of multitopic queries were available only in textual form. Comparing the nal results on this part with our estimates we see very poor correlation2. This suggests that this part of the collection has signi cantly di erent characteristics from the development set and that performing model selection and parameter tuning on the development set was not helpful. We expect that better results could have been achieved on the multi-topic part, provided that the development set contained queries of this type. 7.

ACKNOWLEDGEMENTS

This work is supported by the USEMP FP7 project, partially funded by the EC under contract number 611596. 2Nevertheless, these estimates could serve as an indication of the performance of a system that has no access to the Wikipedia images of the one-topic queries.

References

[1] A. Babenko , A. Slesarev , A. Chigorin , and V. Lempitsky . Neural codes for image retrieval . In ECCV , 2014 .

[2] J. Carbonell and J. Goldstein . The use of mmr, diversity-based reranking for reordering documents and producing summaries . In ACM SIGIR , 1998 .

[3] D. Corney , C. Martin , A . Goker, E. Spyromitros-Xiou s , S. Papadopoulos, Y. Kompatsiaris , L. Aiello , and B. Thomee . Socialsensor: Finding diverse images at mediaeval 2013 . In MediaEval, 2013 .

[4] D .-T. Dang-Nguyen, L. Piras , G. Giacinto, G. Boato, and F. De Natale. Retrieval of diverse images by pre- ltering and hierarchical clustering . In MediaEval , 2014 .

[5] T. Deselaers , T. Gass , P. Dreuw , and H. Ney . Jointly optimising relevance and diversity in image retrieval . In ACM CIVR , 2009 .

[6] R.-E. Fan , K.-W. Chang , C.-J. Hsieh , X.-R. Wang , and C.-J. Lin . LIBLINEAR: A library for large linear classi cation . Journal of Machine Learning Research , 9 : 1871 { 1874 , 2008 .

[7] B. Ionescu , A. G^nsca, B. Boteanu , A. Popescu , M. Lupu , and H. Mu ller. Retrieving diverse social images at MediaEval 2015: Challenge, dataset and evaluation . In MediaEval, 2015 .

[8] Y. Jia , E. Shelhamer , J. Donahue , S. Karayev , J. Long , R. Girshick , S. Guadarrama , and T. Darrell. Ca e: Convolutional architecture for fast feature embedding . arXiv preprint arXiv:1408.5093 , 2014 .

[9] K. Simonyan and A. Zisserman . Very deep convolutional networks for large-scale image recognition . In International Conference on Learning Representations , 2015 .

[10] E. Spyromitros-Xiou s , A. Ginsca , A. Popescu , S. Papadopoulos , Y. Kompatsiaris , and I. Vlahavas . Improving diversity in image search via supervised relevance scoring . In International Conference on Multimedia Retrieval (ICMR) , 2015 .

[11] E. Spyromitros-Xiou s , S. Papadopoulos, I. Kompatsiaris, G. Tsoumakas, and I. Vlahavas. A comprehensive study over vlad and product quantization in large-scale image retrieval . IEEE Transactions on Multimedia , 2014 .

[12] E. Spyromitros-Xiou s , S. Papadopoulos, Y. Kompatsiaris , and I. Vlahavas . Socialsensor: Finding diverse images at mediaeval 2014 . In MediaEval, 2014 .