In this paper we describe our team's approach to MediaEval 2016 Challenge \Emotional Impact of Movies". Except for the baseline features, we extract audio features and image features from video clips. We deploy Convolutional Neural Network (CNN) to extract image features and use OpenSMILE toolbox to extract audio ones. We also study multi-scale approach at di erent levels aiming at the continuous prediction task, using Long-short Term Memory (LSTM) and Bi-directional Long-short Term Memory (BLSTM) models. Fusion methods are also considered and discussed in this paper. The evaluation results show our approaches' e ectiveness.
2.1.2
2.1
Support Vector Regression (SVR) models with RBF kernel were trained for valence and arousal separately. We've 1. INTRODUCTION tried both early fusion and late fusion for audio- and visual
The MediaEval 2016 Challenge \Emotional Impact of Movies" features, which will be elaborated in Section 3.
consists of two subtasks: Global emotion prediction of a
short video clip (around 10 seconds) and continuous emo- 2.2 Subtask 2: continuous emotion prediction
tion prediction of a complete movie. LIRIS-ACCEDE [
For audio features, we used a set of features provided by
INTERSPEECH 2013 Computational Paralinguistics
Challenge [
Except for the baseline features provided by the organizers, there are two types of features used in our experiments, which are audio features and image features. Audio features only utilize the audio wave les extracted from video les, and image features only utilize the static frames extracted from videos.
As to the audio features, we use the extended Geneva
Minimalistic Acoustic Parameter Set (eGeMAPS), which
consists of 88 features and has been used in many emotion
recognition tasks for their potential and theoretical
significance [
The image features were extracted by using a ne-tuned Convolutional Neural Network (CNN). We adopt a 19-layer 2.2.2
We applied Long-Short Term Memory (LSTM) [
In our experiments, two types of models with three layers were used. Type 1 has three LSTM layers and type 2 is the same except the middle layer is BLSTM. The dimensions of the two hidden layers are as listed in Table 1. 2.2.3
Similar to [
We divided the whole dataset into three parts: 70% for training, 20% for validation and 10% as the test set for fusion and post-processing.
Finally, we applied a post-processing with a sliding triangular lter to smooth the nal results. In our experiments, the lter window size is 9. 3.
In this section, we will describe our methods and experiments in more detail and show the results. 3.1
We've submitted three runs for global prediction task in total, listed below:
Run 1: (Baseline + eGeMAPS + CNN) features + SVR + early fusion
Run 2: (Baseline + eGeMAPS) features + SVR + early fusion
Run 3: (Baseline + eGeMAPS + CNN) features + SVR + late fusion
In detail, CNN features in Run 1 and Run 3 are compressed using PCA algorithm, which is 512 dimensions for arousal and 128 dimensions for valence. These dimensions are decided upon the results of 5-fold cross-validation on training set. Besides, the weight of late fusion in Run 3 is also determined on validation.
From Table 2 we can see that, the best run of valence is Run 3 while the best of arousal is Run 1, which are late fusion and early fusion respectively. Notice that runs using CNN features performs better on arousal than those who don't, indicating that image features may contain more information about emotion's polarity than audio ones. Besides, it is worth mentioning that the arousal's Pearson r of Run 2 is the highest among all runs, implying that higher relevance may lead to higher MSE loss to some content. 3.2
In order to select the best model for each scale and fusion, we designed a series of experiments. We have ve di erent scales, two feature sets and two types of models. For each possible combination, we trained 3 trials with randomized initial weights. Therefore, there are total 60 (5 2 2) experiments.
Run 1: Only audio features were used. The sequence length of LSTM was 16 for valence while 64 for arousal. Runs Run 1 Run 2 Run 3 Run 4 Run 5 The model was type 2 for valence and type 1 for arousal.
Run 2: Audio and video feature vectors were concatenate as a multi-modality feature vector. The scale was 16 for valence and 128 for arousal. The model was type 2 for valence and type 1 for arousal.
Run 3: We used the same features as Run 1. Multi-scale models were trained and fused by using simple average to generate the nal results.
Run 4: We used the same features as Run 2. Multi-scale models were trained and fused by using simple average to generate the nal results.
Run 5: Same as Run 3 except the fusion in which we weighted those models' results with di erent weights, i.e. 0.4, 0.3, 0.2 and 0.1 from the low loss to high loss, respectively. 4.
In this paper, we illustrate our approach to the MediaEval 2016 Challenge \Emotional Impact of Movies" task. As to global emotion prediction subtask, combining the features learnt from video by using CNN enhances the regression performance of arousal with early fusion as well as the performance of valence with late fusion.
As to continuous prediction, the best result obtained in this paper is Run 3 for valence and Run 2 for arousal. Fusion by multi-scale has a good performance for valence. For arousal, the Run 3 is better than Run 1 but Run 4 is worse than Run 2, so fusion could not always make it better. 5.
This work was partially supported by the 863 Program of China (2015AA016305), the National Natural Science Foundation of China (61171116, 61433018) and the Major Project of the National Social Science Foundation of China (13&ZD189).