In this paper, we present our approach and investigation on the MedialEval 2017 Predicting Media Interestingness Task. We used most of the visual and audiotory features provided. The standard kernel fusion technique was applied to combine features and we used the ranking support vector machine to learn the classification model. No extra data was introduced to train the model. Oficial results, as well as our investigation on the task data is provided at the end.
MediaEval 2017 Predicting Media Interestingness [
In this section, we first describe the features we employed and then present our classification method.
We used features from diferent modalities. All features were provided by the task organizers.
Visual Features We used nearly all features provided, including
Color histogram in HSV space, GIST [
Audio Features We used the provided Mel-frequency Cepstral Coeficients (MFCC) features. An MFCC descriptor (60 dimensions) is computed over every 32ms temporal window with 16ms shift. The first and second derivatives of the cepstral vectors are also included in the MFCC descriptors.
For the image prediction task, we vectorized each feature per frame. For the video prediction task, we took the mean of raw feature values of all frames in the segment. Given the original feature ft,n for n-th frame in t -th segment, we compute the summarized feature for the segment t as xt =
N 1 Õ N n=1 ft,n where N denotes the total number of frames in the segment.
We used prob (fc8) layer to incorporate semantic information of the training data that can be extracted from the deep neural network. 2.2
We applied the standard kernel fusion approach: we compute a kernel for each type of features, and combine the kernels either by additions or multiplications. We used the multiplication within the same modality and we used the addition across the diferent modalities. For the image prediction subtask, we used the following combination of kernels:
K1 = Kchist · Kдist ,
K2 = Kdhist · Khoд · Klbp ,
Kall = K1 + K2 + Kpr ob .
The rational behind this choice was to consider features with global
histograms and features using the spatial pyramids [
Kall = K1 + K2 + Kpr ob + Kc3d + Kmf cc .
Since C3D and MFCC features model temporal aspect of input, we consider them as diferent modalities from the visual features. For the kernel choice, we used RBF kernel with the median of training data for the hyper-parameter choice.
For the classification model, we used the ranking support vector
machine. We used SV Mr ank [
The oficial metric for evaluation is the mean average precision at 10 (MAP@10) computed over all videos, and over the top 10 best ranked images/video segments. First, we present diferent kernel combinations we tested on the development set. Table 1 describes the diferent kernel fusion formula we used in the experiments. We report both MAP and MAP@10 results in Table 2. As one can see, (1) (2) (3) (4) (5) there is no significant diferences among kernel fusion choices. We used 50-50 split, i.e. 39 movies each for train and test splits of the development set.
We also report both MAP and MAP@10 results on the testset in Table 3 provided by the task organizers. As described in the previous section, we used the visual feature combination, Eq. 4 for the image prediction task, and we used the multi-modal combination, Eq. 5 for the video prediction task. SV Mr ank takes the ranking information as the label of input data and generates pairwise constraints. All provided ranking information in the development set was used for training the SV Mr ank model, with image snapshots and video segments in each movie grouped together.
As it can be seen, in both image and video subtasks, the system shows low performance. This is not surprising given the very simple nature of the approach we applied to the task. What was not expected is that the video prediction result is much better (although still not reaching the level of good performance) than the image prediction result, which was not observable in the development set. This is interesting because we used the same set of features for image and video prediction subtasks, and the only diferences are the two additional features modeling the temporal aspect of data (C3D, MFCC). We believe this reiterates a known understanding on the task: we must somehow incorporate temporal information to improve video interestingness prediction.
One of the major challenges in video interestingness prediction is to fill the semantic gap. Initially, we intended to fill this gap by capturing expected emotional status of viewers and connect it to the notion of interestingness. Table 4 shows our categorization
MAP@10
of the most interesting segments in each movie clip we gathered during the progress. As it can be seen, many of the categories are closely related to key emotional states that modern and existing afect prediction methods can predict. This is particularly true for violence, horror, and joy which consist in large proportion of the most interesting video segments. On the other hand, there are many other video segments for which one cannot readily identify the root of interest stimuli. These typically require a higher level of understanding of the context. The best example is the third movie in the Others category which requires fusion of all modalities plus reading of a sentence shown on the image frame.
In the future, we hope to challenge the media interestingness prediction problem in this direction. Maybe the most promising approach at this point is to understand human activities and link them to emotions and the interestingness.
This work was supported in part by The College of New Jersey under Support Of Scholarly Activity (SOSA) 2017-2019 grant.