This paper presents the work done at Technicolor regarding the MediaEval 2016 Predicting Media Interestingness Task, which aims at predicting the interestingness of individual images and video segments extracted from Hollywood movies. We participated in both the image and video subtasks.
The MediaEval 2016 Predicting Media Interestingness Task
aims at predicting the level of interestingness of multimedia
content, i.e., frames and/or video excerpts from
Hollywoodtype movies. The task is divided in two subtasks depending
on the type of content, i.e., images or video segments. A
complete description of the task can be found in [
For the image subtask, Technicolor's contribution is double: a Support Vector Machine (SVM)-based system is compared with several deep neural network (DNN) structures: Multi-layer perceptrons (MLP), Residual networks (ResNet), Highway networks. For the video subtask, we compare different systems built on DNN including both the existing Long Short Term Memory (LSTM) with ResNet blocks, and the proposed architecture named Circular State-Passing Recurrent Neural Network (CSP-RNN).
The paper is divided in two main parts corresponding to the two subtasks. Before this, section 2 gives insight on the features used. In each subtask's section, we present the systems built, then give some details on the derived runs. The results for the two subtasks are discussed in Section 5.
For both subtasks, input features for the visual modality
are the CNN features extracted from the fc7 layer of the
pre-trained Ca eNet model [
For the audio modality, 60 Mel Frequency Cepstral Coefcients (MFCCs) concatenated with their rst and second derivatives were extracted from windows of size 80 ms centered around each frame, resulting in an audio feature vector of size 180 per video frame. The mean value of the feature vector over the whole dataset is then subtracted for the normalization purpose.
For the image subtask, the philosophy was to experiment using several DNN structures and to compare with a SVMbased baseline. For all system types and both subtasks (image and video), the best parameter con gurations were chosen, by splitting the development set (either MediaEval data or external data) into some training (80%) and validation sets (20%). As the MediaEval development dataset is not very large, we proceeded to a nal training on the whole development set, when building the nal models, except for the CSP-based runs, for which this nal training was omitted.
SVM-based system We tested SVM with di erent kernels: linear, RBF and polynomial and with di erent parameter settings on the development dataset. We observed that the validation accuracy varies from one run to another (even for the same parameter setting) as the training samples change due to the random partition into the training and validation sets. This may suggest that the dataset is not large enough. Also, because of the class imbalance, the validation accuracy used for training makes it di cult to choose the best SVM con guration during grid search, that targets the optimization of the o cial MAP metric.
MLP-based system Several variations of network
structures have been tested, with di erent number of layers, layer
sizes, activation functions and topologies among which
simple MLP, residual [
Run #1: SVM-based SVM in Python Scikit-learn package1 with RBF kernel, gamma = 1=n f eatures, c = 0:1 and default parameter settings elsewhere, is used. Upsampling strategy to enlarge interesting samples by factor of 11 is used.
Run #2: MLP-based A simple structure with 2 layers of sizes (1000, 2) (cf. section 3) was selected for its performance on the Flickr dataset. Best performances with this structure were obtained with a learning rate of 0.1, decay rate of 0.1, ReLu activation function, Adadelta optimizing method and a maximum of 10 epochs. Resampling with probability of 0.6 for the interesting class gives the best MAP value on the MediaEval development set. 4.
Di erent DNN structures capable of handling the temporal evolution of the data were tested with variation of size and depth. We also investigated the performances of di erent modalities separately vs. in a multimodal approach.
Di erent simple RNN and LSTM [
obtained while validating on the Jiang dataset corresponds to gure 1, with a multimodal processing part composed of a residual block built upon 2 LSTM layers of size 436. After re-training on the MediaEval dataset, upsampling with a factor of 9 was applied to the input samples.
Figure 2 illustrates the philosophy of this new structure, which can be seen as a generalization of the traditional RNN, in which at a time instance t, N samples of the input sequence go through N recurrent nodes arranged in a circular structure (allowing to take into account both the past and the future over a temporal window of size N ) to produce N internal outputs. These outputs are then combined to form a nal output at t. This architecture targets a better modeling 1http://scikit-learn.org/stable/modules/generated/ sklearn.svm.SVC
0.1656
The obtained results are reported in terms of MAP in Table 1, with some baseline values computed from a random ranking of the test data. At least on the image subtask our systems perform signi cantly higher than the baseline. For the video subtask, MAP values are lower and we may wonder whether these performances come in part from the di culty of the task itself or the dataset which contains signi cant number of very short shots that were certainly di cult to annotate.
For the image subtask, we observed that simple SVM systems perform similarly (development set) and even slightly better (test set) than more sophisticated DNNs, leading to the conclusion that the size of the dataset was probably not large enough for DNN training. This is also supported by our test on the external Flickr dataset containing 200,000 images for which DNN reached more than 80% accuracy.
For the video subtask, several conclusions may be drawn. First, multimodality seems to bring bene t to the task (this was con rmed by some additional but not submitted runs). Second, the new CSP structure seems be able to capture the temporal evolution of the videos better than classic RNN and more sophisticated LSTM-ResNet structures, and this independently of the monomodal branches which were the same in both cases. This very rst results support the need for further testing of this new structure in the future.