This paper presents the MultiBrasil team experience in the Retrieving Diverse Social Images Task at MediaEval 2017. The teams were required to develop a diversification approach for social image retrieval, enhanced with visual summarization. Our proposal for relevance improvement relies on text and credibility-based reranking and rank aggregation. For diversification, we use diversity-oriented reranking and also propose a clustering-based query-adaptive diversity promotion approach. We applied Genetic Programming and Genetic Algorithm-based approaches for combining textual, visual, and user credibility information.
Beyond relevance, for many search tasks the coverage of diferent
query aspects/intents in the retrieved set has great impact on
fuliflling user needs [
However, tackling the balance between relevance and diversity
is still a great challenge. The Retrieving Diverse Social Images Task
2017 [
The proposed approach consists in improving the original Flickr ranking for relevance-based filtering followed by a diversification step with diversity-oriented reranking or query-adaptive clusteringbased summarization. By improving the original ranking and keeping only the top-ranked items, we intend to construct a more relevant subset, which may reduce data noise for the subsequent diversity promotion step. In turn, for diversification, we have evaluated two approaches: (i) relevance-diversity balancing via reranking; and (ii) representative image selection via query-adaptive clustering metric learning. 2.1
For improving the original ranking of the images from each topic, we explored textual and credibility-based ranking. For text ranking, we used the original topic terms as query and for each flickr image obtained for that topic, the title, description, and tag data were concatenated before preprocessing (see Section 3.2). For credibilitybased ranking, the user credibility scores were used as relevance of her uploaded images (see Section 3.3).
Additionally, for the aggregation of multiple rankings, we applied
the Genetic Programming approach from GPAgg [
As a relevance-based filtering, from the final aggregated list, the top-ranked images were selected as input for the diversification step. We evaluated multiple cutof points with best results achieved by using only the 200-top images (run 1) and the 50-top images (runs 2 to 5). Considering the diversification approaches, keeping more than 50 images degraded the final ranking by pushing more non-relevant images to be reranked and consequently also degrading diversity. Since we did not visually reranked the images for run 1, a deeper reranking had to be considered to allow better diversification, which in turn may negatively impact final relevance. 2.2
For diversification, we tested two methods. First, a reranking method
following the traditional Maximal Marginal Relevance [
Alternatively, our query-adaptive clustering method seeks to construct a more suitable clustering structure based on an evolutionary weight adjustment metric learning method guided by intrinsic clustering fitness evaluation. Here, we used a Genetic Algorithm (GA) for per-query feature weight learning. For combination compatibility, min-max normalization was applied for all distance matrices.
As an unsupervised optimization criteria, we evaluated the clustering quality of the discovered functions by clustering the 50top images using agglomerative hierarchical clustering (averagelinkage). For metric learning and final diversification, we used 25 clusters. The clusters were ranked according to their sizes in descending order and intra-cluster sorting was applied using the images original ranking positions. The final ranking was constructed in a round-robin fashion from the final clusters. 3 3.1
We evaluated only the provided visual descriptors for run 1. Additionally, the combination of each visual feature with the best text similarity measures was evaluated for the remaining runs. For the diversification step, the features were combined by averaging the respective similarity scores. 3.2
For text-only and multimodal reranking (runs 2 to 5), the text-based
scores were computed as the similarity between the text vectors
associated with the images and the original query terms. As text
preprocessing, we applied stopwords removal1 and stemming [
The user credibility scores were individually used for ranking. In this step, we first evaluated the ranking quality of each score individually and finally aggregated the ranking for: bulkProportion, meanImageTagClarity, meanTagRank, meanTagsPerPhoto, meanTitleWordCounts, photoCount, uniqueTags and uploadFrequency. Additionally, we have also created a ranking considering a linear combination of such scores with the weights empirically adjusted, which here we name linearCred. 4
We submitted 5 runs. In Table 1, GPAgg (in run 4) is the GP-based rank aggregation of BM25, Jaccard, DICE, and linearCred rankings. For all runs, we have used only the data provided by the task, and the parameters and features used were chosen according to the best results yielded from development set. 5
Regarding the test set, the text-only run achieved superior efectiveness than the visual-only run, which is a direct consequence of the textual reranking and filtering of the input list. The visual-only run handled more non-relevant images, which impacted the final relevance and diversity.
In general, all multimodal runs (3, 4, and 5) achieved similar efectiveness, with run 3 being slightly superior. Nevertheless, considering F1@20 for the test set, although run 3 outperformed run 5, in a per-query analysis, we noticed that run 5 outperforms for ∼40% of the topics. Moreover, the absolute diference between runs 3 and 5 was 0.0714 in terms of F1@20. Furthermore, even though runs 3 and 4 rely on the same diversification method, the average F1@20 diference was 0.0571, with run 4 outperforming for roughly 43% of the topics. Hence, we highlight the opportunity for further improvement with per-query adaptiveness, for instance, by selecting the most suitable diversification model or even dynamically combining them. 6
In our experiments, we have combined traditional reranking and clustering-based diversification methods along with ranking fusion and per-query adaptive feature fusion for clustering. Even though traditional methods slightly outperformed our more complex proposals, we found the results to be satisfactory. In this case, the small training corpus is considered a challenging factor for the learning strategies. Moreover, our results have shown the importance of improving the original ranking for allowing better results, both in terms of relevance and diversity. We have also shown that properly selecting the most suitable diversification approach or integrating alternative methods may lead to further improvements.
The authors thank FAPESP, FAPEMIG, CAPES, CNPq, and FAPESB (grant #4098/2016) for partially funding this work.