In this paper, we describe our approach for the Emotional Impact of Movies Task at the MediaEval 2018 Challenge. Specifically, we employ features extracted from ResNet-50 from image frames. Then, a fully connected neural network is used for learning the prediction models. Later, we applied the Window Sliding Technique for post-processing the results. The experimental results show the efectiveness of our approach.
Analysing the emotional impact of a video clip to viewers can be
utilized to enhance or control psychological efects of media to
people [
The MediaEval 2018 Emotional Impact of Movies Task consists
of two subtasks. The first subtask is to predict the score of induced
valence and induced arousal every second along movies. The other
is fear prediction, but we have not worked on it. Both subtasks
are evaluated by Mean Squared Error and Pearson’s Correlation
Coeficient. The dataset used for both is the LIRIS-ACCEDE [
There are various sources of information that can be exploited to predict the emotional impact of a movie clip. Although visual content is an essential source to infer viewers’ emotion, audio and text are also potential components for this task. Frame-based and sequence-based approaches can be applied to analyse video frames to evaluate emotional impact.
In our method, we follow the frame-based approach to predict video emotional impact. From the training dataset, we extract deep features of each frame and train two models to predict valence and arousal properties of a video frame. Then we apply the two trained models to evaluate each frame in the test set independently. Finally, we employ the sliding window technique to smooth the ifnal results.
In this section, we will describe how we approach the valencearousal prediction subtasks. The proposed method contains four stages: frame extraction, features extraction, prediction models and post-processing methods. Our system pipeline is shown in Figure 1 below. 2.1
Firstly, we extracted one frame per second of all movies on the training and test set. For frame extraction, we use fmpeg the framework and the extract command provided by the organizers to extract frames. 2.2
For the frame extraction, we use pre-trained 50-layer Residual
Network (ResNet-50) [
We apportion the training set provided by the organizer into training and validation sets with a ratio of 80:20. An overview of the prediction models is shown in Figure 2.
We employ 2-layer fully connected neural network to learn the emotional models. The models take 2048-dim features vectors extracted from ResNet-50 as input. We experimented with varying the number of the nodes for the first and the second layer with 128, 256, 512 and training epochs as 10, 15, 20 . We use Root Mean Square Propagation (RMSProp) with the learning rate 10−4. All prediction models are trained separately for valence and arousal. 2.4
After get the valence/arousal results, we applied the Average Window Sliding Technique to smooth out the random noise. We tested the window size of the algorithm with 3, 5, 7. 3
In this section, we will describe in detail the experimental specification, five runs that we have summitted for the valence-arousal subtask and the result. 3.1
The experiments are processed on Google Compute Engine with 2 vCPU, 7.5 GB RAM and Nvidia Tesla K80 GPU. The average times for extracting 93406 frames on the training set about 1 hour, 40 minutes for extracting features by ResNet and 3 minutes for training each models. 3.2
We tested all trained models with the validation set. After we got the results of all models on the validation set, we sorted descending by the mean square error and selected from Top-1 to Top-4 model to submit. The details of each run are listed below.
All runs take ResNet-50 features as input. From Run 2 to Run 5, the results take the Window Sliding Technique with the window size = 7.
• Run 1: For both valence and arousal, 2-layer fully connected neural network with 128 nodes on the first layer, 512 nodes on the second layer trained on 20 epochs. • Run 2: The same models with Run 1 but we also take the Window Sliding Technique with the window size = 7 to smooth out the random noise. • Run 3: For valence, 2-layer fully connected neural network with 256 nodes on the first layer, 512 nodes on the second layer trained on 10 epochs. For arousal, 2-layer fully connected neural network with 512 nodes on the first layer, 512 nodes on the second layer trained on 15 epochs. • Run 4: For valence, 2-layer fully connected neural network with 256 nodes on the first layer, 512 nodes on the second layer trained on 15 epochs. For arousal, 2-layer fully connected neural network with 128 nodes on the first layer, 512 nodes on the second layer trained on 10 epochs. • Run 5: For valence, 2-layer fully connected neural network with 512 nodes on the first layer, 512 nodes on the second layer trained on 10 epochs. For arousal, 2-layer fully connected neural network with 512 nodes on the first layer, 512 nodes on the second layer trained on 10 epochs. 3.3
We propose a simple method to evaluate the emotional impact, i.e. valence and arousal properties, of a video frame. We study several settings of classification modules with 1 to 2 fully connected layers and diferent numbers of nodes in each layer to select an appropriate model for each property. Experimental results demonstrate that although our method is simple, it achieves promising results for this task. This is the initial step to develop better method to utilize temporal information of frame sequences, and other media types, such as audio and text components.
We would like to express our appreciation to Multimedia Communications Laboratory, University of Information Technology, VNU-HCM, Vietnam, and Software Engineering Laboratory, University of Science, VNU-HCM, Vietnam.