A framework is proposed to predict the emotional impact of movies by using the audio, action, object and scene features. First, four state-of-the-art features are extracted from four pre-trained convolutional neural networks to depict video contents, and an early fusion strategy is used to combine vectors of these features. Then, the linear support vector regression or linear support vector machine is employed to separately learn afective models or fear models, and the strategy of cross-validation is utilized to select training parameters. Finally, the Gaussian blur function is used to smooth scores of video segments. The experiments show that the combination of these features obtains promising results.
The 2018 emotional impact of movies task consists of two
subtasks, including the valence-arousal prediction and the
fear prediction. A brief introduction about this challenge has
been given in [
The selection of features is crucial to emotional analysis.
Intuitively, the audio, action, object and scene features can
influence emotions. Therefore, vectors of four
state-of-theart features are calculated in this framework. Then, the
afective models or fear models are learned by using linear
support vector regression (SVR) or linear support vector
machine (SVM) [
Video
Audio Action Object Scene 2.1 To depict a video, four features are separately extracted from four pre-trained Convolutional Neural Networks (CNNs), including audio, action, object and scene features.
2.1.1 Audio Feature. The audio signals are important
information that describes emotions. VGGish [
2.1.2 Action Feature. The actions in the video can
inlfuence viewer’s emotions. The two-stream Convolutional
Networks (ConvNet) [
2.1.3 Object Feature. The objects in the video may afect
emotions of the viewer. The Squeeze-and-Excitation
Network (SENet) [
2.1.4 Scene Feature. The scenes of the video afect the
emotions of the audience. The Places365 dataset is a large
dataset for scene classification [
To combine vectors of these features, we utilize the early fusion strategy because of its simplicity and eficiency. As shown in Fig. 1, we directly connect vectors of these features for each sample.
For diferent subtasks, the linear SVR and the linear SVM are used to learn the emotional models, separately. The number of positive samples is less than that of the negative samples in the fear subtask. To solve this problem, we weight positive and negative samples in an inverse manner. The regularization parameter is set by the strategy of crossvalidation. The LIBLINEAR toolbox4 is used to implement the L2-regularized L2-loss SVM and SVR.
After obtaining the scores of video segments, we use the function of Gaussian blur to smooth these scores. Let the score vector of a video be . Then, the Gaussian blur function is defined as
Gaussianblur( ) = ⊗ , where ⊗ is the convolution operator, is the specified Gaussian kernel. In experiments, we set the size of Gaussian kernel to 11 for the valence-arousal subtask and 5 for the fear subtask. 3
In order to evaluate the aforementioned features described in
Section 2.1, the features provided by the task organizers are
selected as the baseline features. As required in the task, we
submit five runs for each of the two subtasks. Table 1 shows
the features used in these runs.
2https://github.com/hujie-frank/SENet
3https://github.com/CSAILVision/places365
4https://www.csie.ntu.edu.tw/∼ cjlin/libsvmtools/multicore-liblinear
learning algorithm, SVR is employed in the valence-arousal
subtask, and SVM is used in the fear subtask. The Mean
Square Error (MSE) and Pearson Correlation
Coeficient (PCC) are reported for the valence-arousal subtask, and
the Intersection over Union (IoU) of time intervals is
considered as the evaluation metric for the fear subtask [
As shown in Table 2, Run 2 obtains the best result in the valence-arousal subtask. This suggests that the combination of audio feature and scene feature is suficient to predict valence-arousal values. In the fear subtask, Run 4 achieves the top performance as shown in Table 3. This demonstrates that the combination of audio, scene and action features is enough to describe fear, and that the method using more features does not necessarily lead to better experimental results. By comparing the results of Run 2 and Run 3 in Table 2 and Table 3, the usage of the object feature improves the performance in the fear subtask, but it decreases the performance in the valence-arousal subtasks. This may be due to the reason that some objects can cause people’s fears, such as blood, guns, etc. In Table 3, Run 4 obtains better performances than Run 3. This partly demonstrates that actions are more likely to cause fear than objects. 4
In this work, we propose a framework to predict the emotional impact of movies. Vectors of four features are calculated by using four pre-trained convolutional neural networks. The afective models or fear models are separately learned by using SVR or SVM, and the function of Gaussian blur is utilized to smooth the temporal scores. Experimental results show that the combination of audio feature and scene feature is enough in the valence-arousal subtask, and that additional action feature improve the performance in the fear subtask. The 2018 Emotional Impact of Movies Task