GIBIS at MediaEval 2017: Predicting Media Interestingness Task

Abstract

This paper describes the GIBIS team experience in the Predicting Media Interestingness Task at MediaEval 2017. In this task, the teams were required to develop an approach to predict whether images or videos are interesting or not. Our proposal relies on late fusion with rank aggregation methods for combining ranking models learned with diferent features and by diferent learning-to-rank algorithms.

INTRODUCTION

In this paper, we explore the use of rank aggregation methods for predicting the interestingness of images and videos. For that, content-based representations for images and videos are obtained by diferent features, which are used to train diferent learningto-rank algorithms, creating rankers capable of predicting the interestingness degree of images and videos. Then, the information provided by diferent pairs of feature-ranker are combined by rank aggregation methods, yielding more efective predictions [ 3 ].

This work is developed in the context of the MediaEval 2017 Predicting Media Interestingness Task, whose goal is to automatically select the most interesting frames or portions of videos according to a common viewer by using features derived from audio-visual content or associated textual information. Details about data, task, and evaluation are described in [ 7 ].

PROPOSED APPROACH

The start point for our proposal is the work of Almeida [ 1 ], where motion features were extracted from videos and then used to train four diferent ranking models, which were combined with a majority voting strategy [ 13 ]. The key idea exploited in the Almeida’s work was the use of multiple learning-to-rank algorithms, and their combination was pointed out as promising.

Here, we extend the work of Almeida [ 1 ] by exploring rank aggregation methods for combining ranking models learned with diferent features and by diferent learning-to-rank algorithms.

Features

Images. For the image subtask, we used only the pre-computed features provided by the task organizers [ 7 ]. Five low-level features were considered: Dense SIFT, Histogram of Gradients (HoG), Local Binary Patterns (LBP), GIST, and Color Histogram. Also, two deep learning features were used and they refer to Convolutional Neural Network (CNN) features extracted from the last layers (i.e., fc7 and prob) of the pre-trained AlexNet model [ 11 ].

Videos. For the video subtask, we used nine pre-computed features provided by the task organizers [ 7 ]. One of them represents audio information: Mel-Frequency Cepstral Coeficients (MFCC). Seven features are the same used for images and encode visual content: ifve low-level features (Dense SIFT, HoG, LBP, GIST, and Color Histogram) and two deep learning features (CNN-fc7 and CNN-prob). These eight features are frame-based representations [ 11 ]. To obtain a single video representation, we built a Bag-of-Features (BoF) [ 4 ] model for each feature. In the BoF framework, visual words [ 15 ] are obtained by quantizing a feature space according to a pre-learned dictionary. Thus, a video is represented as a normalized frequency histogram of visual words associated with each feature. In this work, we construct a codebook of 4000 visual words using a random selection. In addition, we considered three video-based representations. One of them is also a pre-computed feature provided by task organizers, denoted C3D [ 16 ]. The two others refer to additional visual features we extracted from videos: Histogram of Motion Patterns (HMP) [ 2 ] and Bag-of-Attributes (BoA) [ 8 ]. 2.2

Learning-to-Rank Algorithms

Each of the above features was used as input to train four diferent learning-to-rank algorithms, which are the same used in [ 1 ]. The ifrst three are based on pairwise comparisons: Ranking SVM [ 12 ], RankNet [ 5 ], and RankBoost [ 10 ]. The latter approach considers lists of objects by using ListNet [ 6 ].

The SVMr ank package1 [ 12 ] was used for running Ranking SVM. The RankLib package2 was used for running RankNet, RankBoost, and ListNet. Ranking SVM was configured with a linear kernel. The others were configured with their default parameter settings. 2.3

Rank Aggregation Models

Let C = {o1, o2, . . . , on } be a collection of n objects (i.e., images or videos). Let R = {r1, r2, . . . , rm } be a set of m feature-ranker pairs. Let ρj (i) be the interestingness degree assigned by the featureranker pair rj ∈ R to the object oi ∈ C. Based on the score ρj , a ranked list τj can be computed. The ranked list τj can be defined as a permutation of the collection C, which contains the most interesting objects according to the feature-ranker pair rj . A permutation τj is a bijection from the set C onto the set [n] = {1, 2, . . . , n}. For a permutation τj , we interpret τj (i) as the position (or rank) of the object oi in the ranked list τj . We can say that, if oi is ranked before ok in the ranked list τj , that is, τj (i) < τj (k), then ρj (i) ≤ ρj (k) [ 3 ].

Given the diferent scores ρj and their respective ranked lists τj computed by distinct pairs rj ∈ R, a rank aggregation method aims to compute a fused score F (i) to each object oi [ 3 ]. In this work, we used three diferent methods based on score and rank information: m (1) Borda Method [ 17 ]: F (i) = Í τj (i), j=0 1https://www.cs.cornell.edu/people/tj/svm_light/svm_rank.html (As of August 2017) 2https://sourceforge.net/p/lemur/wiki/RankLib/ (As of August 2017) m (2) Multiplicative Approach [ 14 ]: F (i) = Î (1 + ρj (i)), j=1 m (3) Weighted Sum Model [ 9 ]: F (i) = Í (τj (i) × ρj (i)). j=1

3 EXPERIMENTS & RESULTS

Five diferent runs were submitted for each subtask configured as shown in Table 13. For both subtasks, the first run is the best feature-ranker pair in isolation and the others refer to the fusion of the top performing feature-ranker pairs with rank aggregation methods. All the evaluated approaches were calibrated through a 3-fold cross validation on the development data.

The development data is composed of 7,396 videos from 78 movie trailers. For the image subtask, the middle keyframe of each video was extracted, forming a dataset with 7,396 images. Each of the features (Section 2.1) was used as input to train each of the learningto-rank algorithms (Section 2.2). In this way, we obtained 28 featureranker pairs (i.e., 7 features × 4 rankers) for the image subtask and 44 feature-ranker pairs (i.e., 11 features × 4 rankers) for the video subtask. Next, each of the feature-ranker pairs was used to predict the interestingness degree of test images and videos. Finally, the prediction scores of the top performing feature-ranker pairs in isolation were combined using rank aggregation methods (Section 2.3), producing fused prediction scores.

To assess the efectiveness of each approach, we computed the Mean Average Precision (MAP). For that, we transformed prediction scores into binary decisions using the strategy proposed in [ 1 ]. First, the prediction scores associated with images and videos of a same movie trailer were normalized using a z-score normalization. Then, an empirical threshold of 0.7 was applied to the normalized prediction scores, producing binary decisions.

Table 2 presents MAP scores obtained for each run on the development data. For both subtasks, the fusion of the top performing feature-ranker pairs (runs 2 to 5) performed better than the best feature-ranker pair in isolation (run 1). The only exception was the run 2 of the video subtask, which was a required run for the task where the use of audio features (i.e., MFCC) was mandatory. All 3The run 1 of the image subtask and the run 2 of the video subtask were the required runs for the task, while the other runs were optional.

J. Almeida and R. Savii the machine-learned rankers using MFCC achieved poor results. By analyzing the confidence intervals, it can be noticed that the results achieved by the rank aggregation methods seem promising.

4 CONCLUSIONS

Our approach has explored rank aggregation methods for combining feature-ranker pairs. Obtained results demonstrate that the proposed approach is promising. Future work includes the investigation of a smarter strategy for selecting the pairs to be combined.

ACKNOWLEDGMENTS

We thank the São Paulo Research Foundation - FAPESP (grant 2016/ 06441-7) and the Brazilian National Council for Scientific and Technological Development - CNPq (grant 423228/2016-1) for funding. This work has also benefited from the support of the Association for the Advancement of Afective Computing (AAAC) and the ACM Special Interest Group on Information Retrieval (SIGIR).

References

[1] J. Almeida . 2016 . UNIFESP at MediaEval 2016: Predicting Media Interestingness Task . In Proc. of the MediaEval 2016 Workshop . http: //ceur-ws. org/ Vol- 1739 /MediaEval_2016_paper_28.pdf

[2] J. Almeida , N. J. Leite , and R. S. Torres . 2011 . Comparison of Video Sequences with Histograms of Motion Patterns . In IEEE Intl. Conf. Image Processing (ICIP'11) . 3673 - 3676 .

[3] J. Almeida , L. P. Valem , and D. C. G. Pedronette . 2017 . A Rank Aggregation Framework for Video Interestingness Prediction . In Intl. Conf. Image Analysis and Processing (ICIAP'17) . 1 - 11 .

[4] Y.-L. Boureau , F. Bach , Y. LeCun, and J. Ponce . 2010 . Learning MidLevel Features for Recognition . In IEEE Intl. Conf. Computer Vision and Pattern Recognition (CVPR'10) . 2559 - 2566 .

[5] C. J. C. Burges , T. Shaked , E. Renshaw , A. Lazier , M. Deeds , N. Hamilton , and G. N. Hullender . 2005 . Learning to rank using gradient descent . In Intl. Conf. Machine Learning (ICML'05) . 89 - 96 .

[6] Z. Cao , T. Qin , T-Y. Liu, M-F. Tsai , and H. Li . 2007 . Learning to rank: from pairwise approach to listwise approach . In Intl. Conf. Machine Learning (ICML'07) . 129 - 136 .

[7] C-H. Demarty , M. Sjöberg , B. Ionescu , T-T. Do , M. Gygli , and N. Q. K. Duong . 2017 . MediaEval 2017 Predicting Media Interestingness Task . In Proc. of the MediaEval 2017 Workshop . Dublin, Ireland.

[8] L. A. Duarte , O. A. B. Penatti , and J. Almeida . 2016 . Bag of Attributes for Video Event Retrieval . CoRR abs/1607 .05208 ( 2016 ). http://arxiv. org/abs/1607.05208

[9] P. C. Fishburn . 1967 . Additive Utilities with Incomplete Product Set: Applications to Priorities and Assignments . Operations Research Society of America (ORSA).

[10] Y. Freund , R. D. Iyer , R. E. Schapire , and Y Singer . 2003 . An Eficient Boosting Algorithm for Combining Preferences . Journal of Machine Learning Research 4 ( 2003 ), 933 - 969 .

[11] Y-G. Jiang , Q. Dai , T. Mei , Y. Rui , and S-F. Chang . 2015 . Super Fast Event Recognition in Internet Videos . IEEE Transactions on Multimedia 17 , 8 ( 2015 ), 1174 - 1186 .

[12] T. Joachims . 2006 . Training linear SVMs in linear time . In ACM SIGKDD Intl. Conf. Knowledge Discovery and Data Mining (ACM SIGKDD'06) . 217 - 226 .

[13] L. Lam and C. Y. Suen . 1997 . Application of majority voting to pattern recognition: an analysis of its behavior and performance . IEEE Trans. Systems, Man, and Cybernetics , Part A 27 , 5 ( 1997 ), 553 - 568 .

[14] D. C. G. Pedronette and R. S. Torres . 2013 . Image Re-Ranking and Rank Aggregation based on Similarity of Ranked Lists . Pattern Recognition 46 , 8 ( 2013 ), 2350 - 2360 .

[15] J. Sivic and A. Zisserman . 2003 . Video Google: A Text Retrieval Approach to Object Matching in Videos . In IEEE Intl. Conf. Computer Vision (ICCV'03) . 1470 - 1477 .

[16] D. Tran , L. Bourdev , R. Fergus , L. Torresani , and M. Paluri . 2015 . Learning Spatiotemporal Features with 3D Convolutional Networks . In IEEE Intl. Conf. Computer Vision (ICCV'15) . 4489 - 4497 .

[17] H. P. Young . 1974 . An axiomatization of Borda's rule . Journal of Economic Theory 9 , 1 ( 1974 ), 43 - 52 .