This paper describes our working approach for the Emotional Impact of Movies task of MediaEval 2016. There are 2 sub-tasks set to make a ective predictions, based on Arousal and Valence values, on video clips. Sub-task 1 requires global emotion prediction. Here a framework is developed using Deep Auto-Encoders, a feature variation algorithm and a Deep network. For sub-task 2, a set of audio features are extracted for continuous emotion prediction. Both sub-tasks are approached as a regression problem evaluated by Mean Squared Error and Pearson Correlation Coe cient.
The 'Emotional Impact of Movies Task' comprises two
sub-tasks with the goal of creating a system that
automatically predicts the emotional impact on video contents, in
terms of Arousal and Valence, which in a 2-D scale can be
used to describe emotions. Sub-task 1 - Global emotion
prediction, predicting a score on induced Valence
(negativepositive) and induced Arousal (calm-excited) for the whole
clip; Sub-task 2 - Continuous emotion prediction,
predicting a score of induced Arousal and Valence continuously for
each 1s-segment of the video. The development dataset used
in both task is the LIRIS-ACCEDE dataset [
The framework is primarily based on visual cues, with
the use of Deep Learning to bene t from the large sample
video dataset. The content of the videos have many di
erent scenes making the emotion detection process
challenging. To tackle this, a 3 stage framework has been designed.
The rst stage of the framework is a Deep Auto-Encoder,
which can be understood in [
There are two Deep Auto-Encoders trained, one based on
MSE loss and the other on Euclidean loss. Using a similar
architecture of Fig. 1 found in [
The FDHH algorithm, based on the idea of Motion
History Histogram (MHH) [
The nal stage of the framework is the regression model, in which 2 are utilized. First being a Deep Network trained on the FDHH features using MSE and Euclidean Regression loss function, and the other treats this trained Deep Network as a Pre-Trained feature extractor, and applies Partial Least Squares (PLS) on the features to predict the Arousal and Valence values.
The Deep Network consists of 9 Conv layers, 1 Pooling
layer, 8 ReLu Activation Layers and a Loss Layer at the
end. Training is done using Support Gradient Descent until
100 epochs, with the weights initialized using the Xavier
method [
The Pre-Trained features are extracted from the 100th epoch network, which are concatenated with Audio descriptors that are mentioned in Section 2.2.1, however they are based on a whole video clip samples rather than 1s segments. These features are concatenated, Rank Normalized between 0 and 1, and then PLS regression is used. 2.2
2.2.1
The Audio descriptors are extracted from openSMILE
software [
A total of 3 regressive models are trained on the audio descriptors. These are mentioned in the following:
Run 1 Linear Regression + Gaussian smoothing (LR+Gs): After obtaining the predicted label from the regression stage, a smoothing operation is performed, using Gaussian ltering with a window size of 10. The smoothing window is carefully selected, in order to retain the pattern of the labels whilst increasing the performance. It is required for removing the high frequency noise irrelevant to the a ective dimensions.
Run 2 Partial Least Square (PLS): PLS is a
statistical algorithm that bears some relation to principal
components regression. Previous EmotiW 2015 employed PLS in
the systems, which gave better results than the baseline [
Run 3 Least Square Boosting + Moving
Average smoothing (LSB + MAs): LSB is regression model
trained with gradient boosting [
5 di erent runs were made based on the framework Subtask 1, and 3 di erent runs for Sub-task 2, which are:
Run 1 & Run 2 Deep Audio PLS: These runs utilize a trained Auto-Encoder with a Euclidean loss function (EUC Loss) and MSE loss function (MSE Loss), followed by FDHH feature extraction. Finally trained Deep Networks, with Euclidean and MSE loss respectively, are used as a PreTrained Feature Extractors. These features are fused with Audio features and a PLS regression model is trained.
Run 3 Audio + PLS: This run is based on using just the openSMILE Audio descriptors, along with a PLS regression
On the o cial test results, each of the sub-task were evaluated using Mean Squared Error (MSE) and Pearson Correlation Coe cient (PCC). For Sub-Task 1, Table 1, the results show a strong performance for Run 5, using a trained Deep Network with Euclidean loss and no Audio descriptors. It is closely matched by Run 4, the identical con guration using MSE loss to train the Deep Networks. The Audio descriptors have shown the weakest performance of them all, with the possibility of increasing the errors of Runs 1 and 2, as they use Audio fusion. In terms of Training loss functions (EUC VS MSE), when comparing Runs 1 VS 2 and Runs 4 VS 5, there is a performance boost for Euclidean loss in most cases, but only marginal. For Sub-Task 2, Table 2, PLS gives lowest MSE but LR+Gs gives highest results on PCC. However all algorithms perform unacceptably bad on Valence, a situation that requires further investigation. 5.
In this working notes paper, we proposed a di erent framework for each sub-task. The frameworks are composed on feature extraction using deep learning, FDHH for capturing Feature variations across the Deep Features at frame level, and nally audio descriptors taken from the speech signal. Several machine learning algorithms were also implemented as a regression model. The o cial test results show that features proposed by the framework are informative, give good results in terms of MSE and PCC in sub-task 1 and good results in terms of MSE in sub-task 2. The future work will focus on the dynamic relationship of the emotion data.