In this paper, we present our approaches for the Mediaeval Emotional Impact of Movies Task. We extract features from multiple modalities including audio, image and motion modalities. SVR and Random Forest are used as our regression models and late fusion is applied to fuse di erent modalities. Experimental results show that the multimodal late fusion is bene cial to predict global a ects and continuous arousal and using CNN features can further boost the performance. But for continuous valence prediction the acoustic features are superior to other features.
The 2016 Emotion Impact of Movies Task [
Statistical Acoustic Features: Statistical acoustic
features are proved to be very e ective in speech emotion
recognition. We use the open-source toolkit OpenSMILE [
MFCC-based Features: The Mel-Frequency Cepstral
Coe cients (MFCCs) [
In the continuous subtask, the audio features are extracted with the window of 10s and shift of 1s to cover more context. 2.2
Hand-crafted Visual Features: We extract the HueSaturation Histogram (hsh) to describe the color information and the Dense SIFT (DSIFT) features to represent the visual appearance information. For hsh features, we quantize the hue to 30 levels and the saturation to 32 levels. For DSIFT features, we use Fisher Vector encoding to construct the video-level features. Then kernel PCA is utilized to reduce the dimensionality into 4096.
DCNN Features: To explore the performance of
different pre-trained CNN models, we extract multiple layers
from di erent CNN models including inception-v3 [
To exploit the temporal information in the video, we
extract the improved Dense Trajectory (iDT) [
The challenge also provides baseline features [
In the global subtask, there are 9,800 video clips from 160 movies in the development set. We randomly select 6093, 1761 and 1946 videos as our local training, validation and testing sets respectively. Video clips from the same movies are kept in the same set. In the continuous subtask, the 30 movies in the development set are also divided into 3 parts with 24 for training, 3 for validation and 3 for testing.
We train SVR and Random Forest for each kind of features and use grid search to select the best hyper-parameters. For SVR, we explore linear and RBF kernels and tune the cost from 2 5 to 212 and the epsilon-tube from 0.1 to 0.4. For Random Forest, the number of trees and the depth of trees are tuned from 100 to 1000 and from 3 to 20 respectively. We apply late fusion to fuse di erent features by training a second-layer model (linear SVR) with input of the best predictions for each kind of features using the local validation set. We use Sequential Backward Selection algorithm to nd the best subset of feature types for late fusion.
In the global subtask, we use the mean standard error (MSE) as evaluation metric. Figure 1 presents MSE of the di erent features for arousal prediction. The audio modality performs the best. Since the baseline feature contains multimodal cues, it achieves the second best performance following our mfccBoW feature. The run1 is the late fusion of all the audio features, baseline and iDT features. In the run2 system, besides the features used in run1, c3d_fc6, vgg16_fc7 and vgg19_fc6 features are also used in late fusion. The arousal prediction performance is signi cantly improved by the multimodal late fusion.
The MSE of di erent features for global valence prediction is shown in Figure 2. The image modality features especially the CNN features are better than other modalities for valence prediction. The run1 system consists of baseline, IS10, mfccBoW, mfccFv and hsh. The run2 system also uses c3d_fc6, c3d_prob, vgg16_fc6, vgg16_fc7 and the features in run1. Although the late fusion performance does not outperform the unimodal performance with CNN vgg16 features on our local testing set, it might be more robust than using one single feature. 3.3
In the continuous subtask, we use the Pearson Correlation Coe cient (PCC) for performance evaluation instead of MSE. Because the labels in the continuous subtask have closer temporal connections than those in the global subtask and thus the shape of the prediction curve is more important. Since the testing set is relative small and the performance is quite unstable in the evaluation, we average the performance of the validation and testing set. Figure 3 shows the PCC results of di erent features. The mfccFV feature performs the best in both arousal and valence prediction. The settings for the submitted three runs are as follows. In run1, we apply late fusion over mfccFV, IS09 and IS10 for arousal and use the mfccFV SVR for valence. In run2, mfccFV, IS09, IS10 and inc_fc are late fused for arousal and mfccFV and IS09 are late fused for valence. The run3 late fuses mfccFV, IS09 and inc_fc for arousal and use mfccFV Random Forest for valence. In our experiment, late fusion is bene cial for the arousal prediction but not for valence prediction. 3.4
In Table 1, we list our results on the challenge testing set. For the global subtask, comparing between run1 and run2, fusing CNN features can greatly improve the arousal and valence prediction performance. For the continuous subtask, the fusion of image and audio cues improves the arousal prediction performance. But for valence prediction, the mfccFV feature alone achieves the best results. 4.
In this paper, we present the multimodal approach to predict global and continuous a ects. The best result on the global subtask is achieved by the late fusion of audio, image and motion modalities. However, for the continuous subtask, the mfccFV feature signi cantly outperforms other features and bene ts little from late fusion on valence prediction. In the future work, we will explore more features for continuous a ects prediction and use LSTMs to model the temporal structure of the videos.
This work is supported by National Key Research and Development Plan under Grant No. 2016YFB1001202.