We present our methods for the MediaEval 2018 Emotional Impact of Movies Task to predict the expected valence and arousal continuously in movies. Our approach leverages image, audio, and face based features computed using pre-trained neural networks. These features were computed over time and modeled using a gated recurrent unit (GRU) based network followed by a mixture of experts model to compute multiclass predictions. We smoothed these predictions using a Butterworth filter for our final result.
The Emotional Impact of Movies Task [
This task is recurring with multiple submissions every
year [
We approached the valence and arousal prediction as a multivariate regression problem. Our objective is to minimize the multi-label sigmoid cross-entropy loss and this could allow the model to use potential relationships between the two dimensions for regression. We first used pre-trained networks to extract features. To model the temporal aspects of the data, the methods we evaluated included long short-term memory (LSTM), gated recurrent unit (GRU) and temporal convolutional network (TCN). Multiple modalities were fused 1This work completed during Jennifer’s internship at Google. Copyright held by the owner/author(s).
MediaEval’18, 29-31 October 2018, Sophia Antipolis, France
with late fusion, and valence along with arousal was predicted
jointly. Our method is implemented using TensorFlow and
we used the Adam optimizer [
We extracted image, audio and face features from each frame
of the movies. Our image features (Inception-Image) were
from the Inception network [
To model the temporal dynamics of the emotion in the videos,
we used recurrent neural networks. In particular, we used
LSTMs [
The input from each modality (image, audio, or face) is fed
into separate recurrent models. The output we use from each
recurrent model is its hidden state which contains information
on previous data seen by the model. We use the hidden state
corresponding to the final timestamp in the input sequence.
The state vectors for each modality are concatenated into a
single vector and fed into a context gate [
To prevent overfitting, we regularized our models using L2
regularization, dropout and batch normalization. Finally, a
low pass filter is applied on the predictions to smooth the
prediction outputs. In LIRIS-ACCEDE, the measured
emotion data vary smoothly in time but our regression outputs
contain high frequency signals. To smooth our outputs, we
tested weighted moving average filters and low-pass filters
(Butterworth filter [
We optimized the hyperparameters of our models to have the best performance on the validation set, which consists of 13 movies from the development set. We then trained our models on the entire development set to run inference on the test set. Our setup used a batch size of 512.
Through evaluating our recurrent models, we found that Inception-Image+AudioSet features had better performance in terms of MSE and PCC compared to VGG16+openSMILE features. In some cases, the recurrent model would predict near the mean for both valence and arousal while using VGG16+openSMILE. This may be because the features did not have enough information for the models to discriminate between diferent values of valence and arousal. We also found a significant increase in performance when we added the Inception-Face features, which may point to salient information captured in connection with the expected emotions in the videos.
The sequence-to-one recurrent models worked best with longer input sequences of 60 seconds versus those of 10 or 30 seconds. This may be because the invoked emotion is afected by longer lasting scenes. Our recurrent models also performed better on the validation set than the TCN and the GRU models had similar performance to the LSTMs. We used GRUs for our implementation because GRUs are computationally simpler than LSTMs. Since we have a small dataset, we wanted to reduce model complexity to prevent underfitting. Our temporal model architecture ranged from 32 to 256 units and 1 to 2 layers, optimized for each of the modalities. For post processing, the low-pass Butterworth iflter worked better than the moving average filter. This is likely because the Butterworth filter is designed to have a frequency response as flat as possible (with no ripples) in the pass-band. Fluctuations of the magnitude response within the passband may decrease the accuracy of our regression output.
In Table 1 we list the performance of our best models that were submitted to the task. Each of the 5 runs is defined as follows where we used Inception-Image, AudioSet, and Inception-Face as the features and a GRU with mixture-ofexperts for regression.
(1) No dropout or batch normalization. (2) Regularized with dropout and batch normalization.
Trained on approximately 70% of the data. (3) Regularized with dropout and batch normalization. (4) Regularized with dropout and batch normalization, diferent initialization and epoch.
(5) Average over all runs.
We see that creating an ensemble from our models by averaging over the runs has the lowest MSE (Run 5). This
Arousal
MSE PCC Run 1 Run 2 Run 3 Run 4 Run 5 is likely because by averaging, we decrease the variance of predictions and thus overall, the mean is closer to the ground truth labels. While averaging improves MSE, it does not improve correlation. Our model with the best correlation is from Run 4.
We note that using batch normalization during inference increases the variance of our predictions. This is because we are using the batch statistics instead of the population statistics from the train set to normalize the batches. Our validation results (with repeated runs) as well as test results show that using batch normalization in this way improves predictions for valence, but not as much for arousal. This is most likely because the statistics of the test set for valence is diferent from train set while the test set statistics for arousal may be closer to the train set statistics. One explanation could be the small size of the dataset so that the statistics of the train set does not generalize well to the test set. 4
We found that precomputed features modeling image, audio, and face in concert with GRUs provided the optimal performance in predicting the expected valence and arousal in movies for this task. Based on our test set metrics, ensemble methods such as bagging could be useful for this task.
We found some evidence that recurrent models performed better than TCN. However since we only evaluated the encoder-decoder TCN more investigation will be necessary for a broader conclusion.
The pre-computed features we used to model image, audio, and face information showed better performance when compared with the VGG16+openSMILE baseline. A future direction could be to train the network in an end-to-end manner to better capture the frame level data, with the caveat that we may need a much larger training dataset.