This paper presents the RECOD team experience in the Retrieving Diverse Social Images Task at MediaEval 2016. The teams were required to develop a diversi cation approach for social photo retrieval. Our proposal is based on re-ranking, rank aggregation, and diversity promotion, allowing employment of textual and visual information apart or fused.
The relevance-diversity trade-o is an important problem
associated with several search scenarios. Promoting
diversity in retrieval results has been shown to positively impact
the user search experience specially for ambiguous,
underspeci ed, and visual summarization queries [
The Retrieving Diverse Social Images Task 2016 [
Our proposal follows the general work ow presented in
Figure 1. The rst step, Re-ranking, ranks the original list
provided by Flickr according to a text-based descriptor. The
Fusion step employs Genetic Programming (GP) [
The next sections provide a more detailed description of our approach.
For visual similarity, besides the provided features, we
also extracted: (i) two general-purpose global descriptors
(BIC [
For text-only and multimodal runs, we used the cosine [
For improving the original list ranking, several textual measures (Section 2.1) were employed for re-ranking. The text-based scores were computed as the similarity between the text vectors associated with the query topic and the image associated text vectors. For visual only run, the reranking step was skipped.
For feature fusion, re-ranked lists were combined using
the GP approach from [
After re-ranking and aggregation steps, the improved
relevance-based lists were submitted to explicit diversi
ca1CEDD, FCTH, OpponentHistogram, JointHistogram,
AutoColorCorrelogram, ColorLayout, EdgeHistogram,
Gabor, JCD, JpegCoe cientHistogram, ScalableColor,
SimpleColorHistogram, Tamura, LuminanceLayout, and PHOG.
Available at: http://www.lire-project.net/ (As of Sep.
2016).
tion. Visual and textual descriptors were employed
(Section 2.1). We evaluated ve methods: clustering-based
(k-Medoids, agglomerative and Birch [
Agglomerative and Birch methods achieved signi cantly superior results on the development set, thus they were used in the submitted runs.
In the clustering step, for agglomerative method, centroid and average link linkage methods were employed using ColorLayout for distance computing. Forty clusters were considered in our approach. For Birch method, a maximum of 51 entries per node was admitted, with a distance threshold of 0.3, and also considering cluster re ning. The representative images were selected in a round robin fashion from the nal clusters. 2.4
It is important to observe that re-ranking and GP fusion are optional steps at the work ow presented in Figure 1. Depending on run requirements or on the desired experiment goals, one or both can be skipped. For example, for run 1, which is visual only, neither of those steps were employed.
Furthermore, the diversi cation step can employ textual or visual information alone or together. It allows adherence to the single modality requirement of runs 1 and 2. Section 3 will present details of each run con guration. 3.
We submitted ve runs.
Run 1 { (required) visual information only. No re-ranking and no GP-fusion were employed. Diversi cation provided by Agglomerative method, using average link method with ColorLayout as visual feature and grouping images into 40 clusters.
Run 2 { (required) text information only. Re-ranking considering cosine similarity was employed. Diversi cation provided by Birch method, using 51 as maximum entries per node with distance threshold of 0.3.
Run 3 { (required) text-visual fused. Re-ranking considering cosine similarity was employed. Diversi cation provided by Birch method, using 51 as maximum entries per node with distance threshold of 0.19;
Run 4 { (optional) general run. GP-fusion rank aggregation employed (Figure 2 - a). Diversi cation provided by Agglomerative method, using average link method with ColorLayout as visual feature and grouping images into 40 clusters;
Run 5 { (optional) general run. GP-fusion rank aggregation employed (Figure 2 - b). Diversi cation provided by Agglomerative method, using centroid linkage method with ColorLayout as visual feature and grouping images into 40 clusters.
The diversi cation methods, parameters, features, and textual similarities used were selected according to the best results on the development set.
Table 1 presents the results for the ve runs for the development and test sets. The best results (F1@20) on the development set were achieved on run 2, followed by runs 1 and 3, in which textual information was used. Runs 4 and 5, which employed GP-fusion rank aggregation, have shown CombMAX( BordaCount( CombMAX( CombMNZ(COSINE, BM25), COSINE_ME ), CombMAX( CombMNZ(DICE, DICE), COSINE )), CombMIN( RRF( CombMNZ(DICE, TFIDF), COSINE_ME ), CombMAX( MRA(TFIDF, BM25Orig), COSINE ))) (a)
MRA( CombMNZ( CombMAX( CombSUM(BM25, COSINE), RRF(JACCARD, COSINE)), BM25), MRA(TFIDF, COSINE_ME)) (b) the worst results on this set. However, on the test set, they presented the best results.
As we can observe, in most cases, the re-ranking over the Flickr initial ranking improved the overall results, even when employing only textual features for this task.
Considering test set results of runs 4 and 5 over 1 and 2, we can also notice that the retrieval process illustrated in Figure 1 can bene t from fusing visual and textual information. We believe that textual and visual information have a complementary nature for an e ective image retrieval process. Textual information introduces the notion of context around retrieval, but ignores the image itself by not inspecting its content. On the other hand, content-based image retrieval lacks context. By initial retrieval and re-ranking based on textual information and introducing visual features at the diversi cation step this complementary nature is explored and provided the best results. 5.
For relevance and diversity maximization, we proposed re-ranking strategies and the combination of multiple features with a rank fusion method. These improved ranked lists were used as input for a clustering-based summarization method. Our experiments suggest that aggregation of multiple re-ranked lists and fusion of visual and textual information can improve retrieval e ectiveness. It is interesting to observe the recurrent selection of the cosine measure on both GP aggregation individuals.
We thank the support of CAPES, CNPq, and FAPESP.