This work reports the methodology that CERTH-ITI team developed so as to recognize the emotional impact that movies have to its viewers in terms of valence/arousal and fear. More Specifically, deep convolutional neural newtworks and several machine learning techniques are utilized to extract visual features and classify them based on the predicted model, while audio features are also taken into account in the fear scenario, leading to highly accurate recognition rates.
Emotion based content have a large number of applications,
including emotion-based personalized content delivery[
Emotional Impact of Movies Task is a challenge of MediaEval interval to record human extracted groundtruth data. The provided
2018 that comprises of two subtasks: (a) Valence/Arousal prediction visual features are then concatenated into one vector representation
and (b) Fear prediction from movies. The Task provides a great so as to have a common and fixed representation scheme throughout
amount of movies video, their visual and audio features and also diferent video samples.
their annotations[
In this work, CERTH-ITI introduces its algorithms for valence/arousal approximation (Run 3). PCA is also deployed on our final visual
and fear recognition subtasks, which include the deployment of features vectors so as to reduce their dimensionality and keep only
deep learning and other classification schemes to recognize the de- the most discriminant principal components (in our case the first
sired outcome. More specifically, a 3-layer neural network(NN) and 2000) to represent all features (Run 4).
a simple linear regression model are deployed, with and without A Neural Network (NN) framework has also been deployed so as
PCA, so as to predict the correct emotion in the valence-arousal to fulfil the valence/arousal recognition substask. For that purposes,
subtask, while a pre-trained V GG16 model [
For the fear recognition subtask, we initially keyframe extraction
every one second, as we perform in valance subtask. The frames
annotated as "fear" were significantly less than the "no-fear" class
and, therefore, in order to balance our dataset we used data
augmentation techniques. Firstly, we downloaded from Flickr about 10, 000
images with tag "fear" and we also download emotion images 1 and
kept those which are annotated as "fear". In order to further increase
the number of fear frames, we additionally use data augmentation
techniques on the provided annotated frames. We randomly rotate
and translate pictures vertically or horizontally and we randomly
apply shearing transformations, randomly zooming inside pictures,
lfipping half of the images horizontally and filling in newly created
pixels which can appear after a rotation or a width/height shift.
Finally, we reduce the set of no-fear frames. After these, we had
about 23,000 "fear" and 30,000 tagged as "no fear" images to train
our model. We used transfer learning to gain information from a
large scale dataset and also trained our model in a very realistic
and eficient time. The architecture that we chose to represent our
features is the V GG16 pre-trained on Places2 dataset [
Additionally, in order to exploit auditory information, we
developed a classification method applied on audio features already
extracted from the challenge committee using openSmile toolbox [
Results from visual and audio analysis were submitted both separately, as diferent runs, and in combination by taking the post probabilities of visual and auditory classifications and setting a threshold of 0.7 on their average probability. The overall block diagram of this approach is depicted in Figure 1. 3
We have submitted 4 runs for valence/arousal prediction and their results are introduced in Table 1. In the experiments two evaluation measures are used: (a) Mean Square Error (MSE) and (b) Pearson Correlation Coeficient (r). We observe that the N N approach that we describe in the previous section has the best performance amongst all the others. Furthermore, it is worth mentioning that the linear regression model produces some extremely high scores, probably because the original feature vectors weren’t neither discriminative nor adequate enough to create the regression model. However, PCA projection to lower dimensional space, with higher discriminative power show to solve this problem as it reduces the
redundant noise and keep the most important features. Moreover, there is a "NaN" score for the Pearson measure in the arousal prediction scores, because we accidentally set the training value stable and so our model predicts the same score for all frames, but this score does not characterize our model, since it does not appear in any other prediction within the valence/arousal prediction sub-task.
We have also submitted 4 runs for fear prediction subtask and their results are also presented in Table 1 and are evaluated in terms of Intersection over Union (IoU). From Table 1 we see that, the best performance for the fear recognition subtask is Run 1, using all predicted scores of the pre-trained VGG16 model. In addition, our intuition to remove isolated predicted frames (Run 2), as they are not associated with any duration, did not perform better than Run 1, hence we miss significant information (video frames that invoke fear). 4
Eval 2018 Challenge "Emotional Impact of Movies" task. The results in valence/arousal prediction subtask shows that according to MSE, the best result obtained in Run 3 for both valence and arousal, while regarding to Pearson Correlation Coeficient, Run 1 has the best performance for arousal and the second best performance for valence. The Pearson correlation is able to measure linear correlations between two or more variables. However, the MSE is obtained by a sum of squared deviations between predicted and ground-truth values, no matter if they are linearly correlated or not.
The results of the fear prediction subtask shows that the inclusion of audio features failed to enhance the classification performance, as expected. This could be due to several reasons, with the prominent one to be the incapability of performing data augmentation on audio features such as in the case of visual analysis. Both the aforementioned reason and the large inequality between the two classes, which led us discard many "no-fear" annotations, in order to balance the training set, resulted on a very limited training set. These drawbacks could be overcome by using classification methods able to handle unbalanced training sets, such as penalized models, or by enriching the training set with external annotated datasets and by exploring more eficient fusion methods, such as performing classification on fused audiovisual features, instead of a posterior combining separate classification results.
Emotional Impact of Movies Task