In this paper, we describe our approach to subtask 2 of the Emotional Impact of Movies task from the Mediaeval 2018 Challenge. We compared the performances of LSTM ensembles to single LSTM models for predicting the fear-inducing seconds in movies. We also compared the performance of an LSTM model trained on audio feature data to the performance of an LSTM model trained on the outputs of a pretrained VGG16 model. Ultimately, we found that a single LSTM trained on VGG16 outputs achieved the highest F1 score on the test set.
The Mediaeval emotional impact of movies task contains two
subtasks: 1) valence and arousal score prediction, and 2) fear
prediction. More information can be found in [
This task is a sequel to the 2017 emotional impact of movies task,
so there is a sizable body of related work from last year. Support
vector regression algorithms were used in [
To address subtask 2, fear prediction, we trained three Long
ShortTerm Memory (LSTM) ensembles and two single-layer LSTM
models. LSTMs are known for their effectiveness at modeling
timeseries data and capturing long-term dependencies in this type of
data [
We set aside the last 12 movies from DevSet part 2 as our
crossvalidation set so we could compare LSTM model architecture
variations. A simple single-layer LSTM with batch normalization
[
To train our models, we used the pre-extracted audio features
and VGG16 fc6 layer visual features from the Liris-Accede dataset.
The audio features were extracted using the openSmile toolbox, and
the fc6 features were extracted with the Matlab neural networks
toolbox [
The fc6 and audio feature data were compiled into matrices and padded so that the max number of timesteps for each movie was 6262 seconds. To reduce memory requirements, we chose a window size of 101 seconds to slide over the time series data with no seconds of overlap. To create labels, we converted the fear annotations into one-hot vectors for each movie. Each element in a movie’s one-hot vector represented one second: fear-inducing seconds were ones while non-fear-inducing seconds were zeros. Finally, the training data for our models were handpicked so that about 20% of all timesteps fed into the model were fear-inducing. 4
We submitted the following five runs:
Run 1: Ensemble of LSTMs + fc6 and Audio features Run 2: Ensemble of LSTMs + Audio features Run 3: Ensemble of LSTMs + fc6 Run 4: Single-layer LSTM + fc6 Run 5: Single-layer LSTM + Audio Each ensemble consisted of four LSTM models trained on different subsets of DevSet part 1 and DevSet part 2 data. For the first run, two single-layer LSTM models were trained using fc6 feature data, and two single-layer LSTMs were trained using audio feature data. For run 2, all four LSTM models were trained using fc6 data. Three of these models were single-layer LSTMs and one was a single layer LSTM with a 1D convolutional layer attached. For run 3, all four LSTM models (again three of which were single-layer LSTMs and one of which contained a 1D convolutional layer) were trained using audio feature data. The results from these runs are listed in Table 1. Run 1 Run 2 Run 3 Run 4 Run 5
Intersection_ Over_Union 0.06496 0.07507 0.08742 0.11992 0.09874 The predictions of the individual models in each ensemble were averaged together to produce the ensemble’s final output. On the whole, the ensembles performed worse than the single LSTM models. This could be due in part to the fact that predictions in the ensembles were joined via a simple average function, not a weighted average function. Results might have improved if the models with higher F1 scores were given greater influence over the final decision of the ensemble.
Between the two individual models in runs 4 and 5, the single layer LSTM trained on fc6 data (run 4) performed the best. This may suggest that visual features were more relevant to predicting fear-inducing segments than audio features. 5
In this paper, we described our approach to addressing the fear prediction subtask using LSTM models. We compared the performances of LSTM ensembles to single LSTMs trained on either fc6 or audio feature data.
There are a number of interesting future research avenues to explore and ways to improve what has been shared here. First, it may be beneficial to decrease the length of the sliding windows from 101 seconds and introduce a greater amount of overlap between them. Because of our design choice in this paper, we faced a class imbalance issue in our training data, which was heavily skewed towards non-fear inducing seconds. Decreasing the window size would make it easier to handpick a set of training data with a higher ratio of fear-inducing seconds to non-fear inducing seconds. Training the LSTM models on a balanced dataset may improve their performances.
Another potentially helpful strategy for dealing with the bias in the fear prediction dataset is to weight the cost function so the models are penalized more heavily for predicting zeros (non-fear inducing seconds) when they should be predicting ones (fear inducing seconds).
In terms of feature data, AlexNet and GoogleLeNet outputs could
be promising to work with in the future. LSTMs trained on this
feature data have achieved notable results for action recognition in
videos [
This work was supported in part by the AI Grant (now the Pioneer Fund).