This paper presents our approaches for the MediaEval Emotional Impact of Movies Task. We employed features from image frames and audio signal. We use support vector regression for the learning and prediction. In addition, we introduce a new feature using exponential decay of the initially predicted emotion labels. The motivation behind this is to computationally model lingering efect. Experimental results and future direction are also discussed.
MediaEval 2017 Emotional Impact of Movie Task [
First, we describe multimodal features we employed. Next, we introduce our new feature based on prediction label to model lingering efect of induced emotions. Lastly, we describe our hierarchical regression framework for emotion prediction.
We employed all standard set of visual and audio features provided
by the MediaEval task organizers. For image frame-based features,
we used Auto Color Correlogram, Color and Edge Directivity
Descriptor, Color Layout, Edge Histogram, Fuzzy Color and Texture
Histogram, Gabor, Joint descriptor joining CEDD and FCTH in one
histogram, Scalable Color, Tamura, Local Binary Patterns, fc6 layer
of VGG16 network [
For auditory features, we employed the audio features provided.
In the provided description, there should be 1,582 features which
result from a base of 34 low-level descriptors, with 34 corresponding
diferential coeficients, and 21 functionals applied to each of these
68 contours, thus 1,428 features (21 × (34 + 34)) in total. Out of
remaining 154 features, 152 features were computed by applying
19 additional functionals to the 4 pitch-based low-level descriptors
and their 4 diferential coeficient contours ( 19 × (4 + 4)). Last
two features are additional statistical features, the number of pitch
onsets and the total duration of the input. These features were
computed every ten seconds segments sliding over the whole movie
with a shift of 5 seconds. All these features were computed by
openSmile toolbox [
In addition to the provided features, we introduced additional
feature, using the ground truth labels of emotional levels. The
motivation behind this new feature, is to computationally model the
gradually amplifying or decaying emotional flow, what is typically
referred as lingering emotion. Traditional and even state-of-the-art
afect prediction systems focus on predicting induced emotions as
a spike noise detection model, regardless of whether they modeled
the temporal aspect of afect or not. On the other hand, lingering
emotions do not directly induced by the stimuli, rather, they are
generated from the emotional change, i.e. response, already existing a
priori. In short, we argue that what is called climax of a movie is not
only a consequence of short segment stimuli, but also amplified (or
degraded) by the emotional state change itself across the segments.
A similar idea was utilized to predict media interestingness [
To model this lingering emotion, we use emotion level label values. For each segment t = 1..T where T denotes the total number of segments, assume that we are given emotion level label yt . So, at each time segment t , we have (1) (2) (3) (4) (5) x1, x2, · · · , xt , y1, y2, · · · , yt , where xt denotes the vectorized visual / audio features and yt denotes either the ground truth or predicted emotion level (it could be valence, arousal, or fear label). Then, we can define the lingering feature lt as an exponential decay function of labels as l(t −w ) = y(t −w ) . .
. l(t −w +1) = (1 − α ) · l(t −w ) + α · y(t −w +1)
ls = (1 − α ) · ls−1 + α · ys where s = (t − w), ..., t , and w denotes the lingering window size. Parameter α is the decay factor. Intuitively, we take weighted accumulated emotions over time, and consider it to model the lingering efect. In training phase, we can utilize the ground truth emotion. In testing phase, we can devise a two-step, hierarchical regression model to obtain the emotion level feature values. We will describe this model in the next section.
ρ MSE
ρ Accuracy
MSE
ρ MSE
ρ Accuracy Precision
Recall
F1
There are considerations why we think this can be a reasonable
model for the lingering efect. First, with the exponential decay
function, we can consider both smoothness and also decaying of
emotional change over time. Second, one can view this as simplified
version of traditional temporal models, e.g., Hidden Markov Models
(HMM), where we fix the transition probability. If we can obtain
large number of emotion labels, one may try to learn HMM-based
features instead, as in [
To combine features, we utilized standard multiple kernel learning
approach [
Kvis = Kacc · Kcedd · · · Kf c6 Kall = Kvis + Kaud + Klin (6) (7) where each K · denotes the kernel computed using the features. We used Radial Basis Function (RBF) kernel with median of training data as the hyperparameter.
Once the combined kernel is computed, we can use it as feature
vectors. For the regression model, we used linear Support Vector
Regression (SVR). We used MATLAB’s fitrsvm function for this.
One important aspect of our approach is that we use emotion
prediction labels to compute the lingering features. Since we do not
have ground truth labels for testset, we design a two-step,
hierarchical regression framework. In this framework, we need to train two
SVR models in the training phase. One model (Model A) is trained
with the kernel computed using training data, but the kernel is only
combines visual and auditory features. The other model (Model
B) is trained with the kernel computed using all modalities. In the
testing phase, we first perform an initial emotion prediction on the
test data using Model A. Then, we compute the lingering feature
using the predicted afect labels. Note that this is computationally
not expensive since the lingering feature itself is easy to compute
and the labels of all training data is only 1 dimensional vector.
Finally, we perform final emotion prediction on the test data using
Model B. We applied regression framework for all subtasks. For
the fear subtask, we first rescaled the output into [
For the measure, we used Mean Squared Error (MSE) and Person’s correlation coeficient ( ρ) for the valence and arousal subtasks, and accuracy, precision, recall, and F1 score for the fear subtask. We used α = 0.5 for all experiments.
In the Devset, shown in Table 1, one can see that there is no significant benefit in using lingering feature in this case. We used 50-50 split to obtain the result but the readers should take this result with a grain of salt (particularly, accuracy of fear) since we did not make strict data split based on the information which frame belongs to which video, to obtain this result.
In the Testset, shown in Table 2, the oficial results are more interesting. It is obvious that the lingering feature does not help (actually hinders) the valence prediction. On the other hand, for the arousal and fear, lingering feature seems to make positive contribution to the prediction although the overall MSE and accuracy sacrificed a little. It is also notable that the similar tendency could be observed in the Devset in Table 1. One intuitive explain here would be following: what we are modeling with lingering feature, is how the prior, recent emotional change might afect or induce the new emotion. In case of arousal (either active or passive to the stimuli) or fear (feeling horror, anxiety or not), this happens often. On the other hand, valence (positive or negative) is rather dificult to capture with a fixed window size of the linger feature. Moreover, changing from positive to negative emotional state, or vice versa, requires more contextual (or semantic) information of stimuli to understand why that change has happened. 4
In this paper, we introduced a new feature modeling lingering effect and presented a hierarchical regression framework to predict emotions. We found promising applications of the new features in arousal and fear prediction, with limitations in valence prediction. In the future, it would be interesting to investigate how one can more robustly capture this lingering efect with in-depth understanding of the feature’s impact on the valence prediction.
This work was supported in part by The College of New Jersey under Support Of Scholarly Activity (SOSA) 2017-2019 grant. Emotional Impact of Movies Task