Predicting the interestingness of images or videos can greatly improve people's satisfaction in many applications, such as video retrieval and recommendations. In this paper, we present our methods in the 2017 Predicting Media Interestingness Task. We propose deep ranking model based on aural and visual modalities which simulates the human annotation procedures for more reliable interestingness prediction.
The interestingness prediction task [
We propose an interestingness prediction model based on aural and visual modalities and deep ranking model to calculate interestingness score by the given images or video clips.
Aural Features We extract 39-dim Mel-Frequency Cepstral
Coeficients (MFCCs) features from each video segment and
create their bag of words features with 128 codewords
denoting the responding segment. L1-norm is used to get the
probability distributions on the codebook for each video.
Visual Features We utilize oficially provided features
including Alex fc7, Alex prob, ColorHist, DenseSIFT, GIST,
HOG and LBP. Additionally, we consider 2048-dim
framelevel features from the penultimate layer of InceptionV3
network, which is trained on 1.2 million images of ImageNet
challenge dataset[
2.2.1 Ranking Loss. Suppose we have a set of video segments pairs P sampled from the original video segment pool. In P , each pair contains a segment pi with higher interestingness and a segment ni with lower score. If function f denotes the output of branches, we can get a score pair as follows: (f (pi), f (ni)), ∀(pi, ni) ∈ P (1)
We set the margin in the loss as 1 by default according
to [
n minimization : X max(1 − f (pi) + f (ni), 0)
i=1 2.2.2 Pairwise Generation. Input data is an essential factor of the training stage. We try to use diferent strategies to sample data pairs as inputs. Diferent principles impact the training process and result in big diference. Let x and y denote float numbers in the range of 0 to 1. Our four strategies can be presented as follows: (2) (3) (4) (5) (6) f (pi) − f (ni) > x f (pi) − f (ni) < y x < f (pi) − f (ni) < y f (pi) − f (ni) < x or f (pi) − f (ni) > y
Our basic empirical parameter of sampling is to set the distance of ground truth interestingness labels of the two videos in the same pair as 0.55. At first, big distances and small distances are both taken into account but does not show significant performance. We suppose that the network cannot learn much from two pretty similar videos and huge gap between the two videos, which are the reasons that the network result in worse results. 3 3.1
There are 7396 images or video clips in each subtask. We use video with id from 0 to 61 as the local training set, 62 to 69 as local validation set and 70 to 77 as local testing set.
We utilize the Support Vector Regression (SVR) and Random Forest Regression (RF) as our baseline models as the comparison with the deep ranking models. For SVM, RBF kernel is applied and the cost is searched from 22 to 210. And for Random Forest, the number of trees is searched from 100 to 1000 with step 100 and the depth of the tree is searched from 2 to 16. 3.2
In both subtasks, we consider diferent prediction models and features. The results are shown in Figure 2 and Figure 3 respectively.
In image subtask, we can find the pairwise ranking model shows greater performance generally and the deep neural network features are distinctive for interestingness prediction. It is not surprising that deep neural network displays its state-of-art capability.
In video subtask, we test the MFCC BoAW feature on local testing set and get a MAP of 0.151. Early fusion is applied over various visual features and MFCC BoAW. The results are generally consistent with the conclusions of image subtask. In our ranking model, the greatest MAP on local testing set is 0.210, which overpasses the other results. While the fusion boosts the performance not very much for each visual feature. We suppose it is due to its low dimensionality.
Given the experiments results on local testing set, we pick the winners of various models and features, namely pairwise ranking model and InceptionV3 feature, as our final choice for submissions. The oficial results for both subtasks are shown in Table 1. We utilize two types of input in the experiments of InceptionV3 feature, which are original images and normalized images. Normalized images are scaled into 0 to 1 for each pixel. As the results shown, the InceptionV3 feature from original image performs a little better than the normalized one on oficial testing set.
As the results show, image interestingness prediction is generally accurate than video subtask. We think that it is easier to fetch distinctive features from static images than from videos. Firstly, audio displays completely diferent cues with images and the fusion of the two modalities may present brand new interestingness. Secondly, dynamic properties like changes of scene make videos more informative, so that we cannot capture the interestingness precisely only by the static images inside a video.
After investigating the testing set, we find out some interesting phenomena. For image subtask, images containing varied scenes can be ranked precisely, but a series of images Shuai Wang et al. We develop an interestingness prediction system based on pairwise ranking. Comparing with basic regression models, we notice the efectiveness of ranking model and the InceptionV3 feature is distinctive for interestingness prediction task. In training process, optimizing the data pair sampling strategy is always considered a fundamental and essential point. In the future, we will also use more temporal cues to guarantee that the information within the internal frames of the same video is not wasted.
This work was supported by National Key Research and Development Plan under Grant No.2016YFB1001202.