IM-JAIC at MediaEval 2018 Emotional Impact of Movies Task Chloe Loughridge1, Julia Moseyko2 1 Punahou School, Honolulu, Hawaii, United States 2 Massachusetts Institute of Technology, Cambridge, Massachusetts, United States ABSTRACT We set aside the last 12 movies from DevSet part 2 as our cross- In this paper, we describe our approach to subtask 2 of the validation set so we could compare LSTM model architecture Emotional Impact of Movies task from the Mediaeval 2018 variations. A simple single-layer LSTM with batch normalization [5] trained on the fc6 feature data from DevSet 1 achieved the best Challenge. We compared the performances of LSTM ensembles to results on this cross-validation set. The LSTM model with a 1D single LSTM models for predicting the fear-inducing seconds in temporal convolutional layer (trained on the same feature data movies. We also compared the performance of an LSTM model from DevSet 1) performed slightly worse, but still achieved non- trained on audio feature data to the performance of an LSTM model zero F1 scores. We ultimately used both model architectures in trained on the outputs of a pretrained VGG16 model. Ultimately, our ensembles, though we included more of the single-layer we found that a single LSTM trained on VGG16 outputs achieved LSTMs in each ensemble. the highest F1 score on the test set. 3.3 Preprocessing the feature data To train our models, we used the pre-extracted audio features 1 INTRODUCTION and VGG16 fc6 layer visual features from the Liris-Accede dataset. The Mediaeval emotional impact of movies task contains two The audio features were extracted using the openSmile toolbox, and subtasks: 1) valence and arousal score prediction, and 2) fear the fc6 features were extracted with the Matlab neural networks prediction. More information can be found in [1]. In this paper we toolbox [1]. These features have been the most useful in past papers describe our work on subtask 2, fear prediction. [2, 7]. To test whether the same applied for our data, we trained multiple LSTM models on the visual features provided in the Liris 2 RELATED WORK Accede dataset (i.e., the fc6 feature data and the other visual This task is a sequel to the 2017 emotional impact of movies task, features extracted using the LIRE library) [1]. Each LSTM was trained on 4 movies and tested on 3 movies from DevSet part 1. In so there is a sizable body of related work from last year. Support this testing, only the models trained on audio features and fc6 visual vector regression algorithms were used in [2] to predict fear. A features produced nonzero F1 scores. random forest algorithm was used in [7] to predict fear with considerable success. In the task of video action classification, The fc6 and audio feature data were compiled into matrices and LSTMs have achieved notable results when trained on the outputs padded so that the max number of timesteps for each movie was of the AlexNet model and the GoogleLeNet model [3]. We aim to 6262 seconds. To reduce memory requirements, we chose a build on the previous work done for subtask 2 by implementing window size of 101 seconds to slide over the time series data with LSTM models for fear prediction. no seconds of overlap. To create labels, we converted the fear annotations into one-hot vectors for each movie. Each element in a 3 APPROACH movie’s one-hot vector represented one second: fear-inducing seconds were ones while non-fear-inducing seconds were zeros. 3.1 Overview Finally, the training data for our models were handpicked so that To address subtask 2, fear prediction, we trained three Long Short- about 20% of all timesteps fed into the model were fear-inducing. Term Memory (LSTM) ensembles and two single-layer LSTM models. LSTMs are known for their effectiveness at modeling time- 4 RESULTS AND ANALYSIS series data and capturing long-term dependencies in this type of data [4]. We submitted the following five runs: 3.2 LSTM Model Architecture Run 1: Ensemble of LSTMs + fc6 and Audio features Run 2: Ensemble of LSTMs + Audio features Run 3: Ensemble of LSTMs + fc6 Run 4: Single-layer LSTM + fc6 Copyright is held by the owner/author(s). Run 5: Single-layer LSTM + Audio MediaEval’18, 29-31 October 2018, Sophia Antipolis, France MediaEval’18, 29-31 October 2018, Sophia Antipolis, France M. Larson et al. Each ensemble consisted of four LSTM models trained on different Another potentially helpful strategy for dealing with the bias in subsets of DevSet part 1 and DevSet part 2 data. For the first run, the fear prediction dataset is to weight the cost function so the two single-layer LSTM models were trained using fc6 feature data, models are penalized more heavily for predicting zeros (non-fear and two single-layer LSTMs were trained using audio feature data. inducing seconds) when they should be predicting ones (fear For run 2, all four LSTM models were trained using fc6 data. Three inducing seconds). of these models were single-layer LSTMs and one was a single In terms of feature data, AlexNet and GoogleLeNet outputs could layer LSTM with a 1D convolutional layer attached. For run 3, all be promising to work with in the future. LSTMs trained on this four LSTM models (again three of which were single-layer LSTMs feature data have achieved notable results for action recognition in and one of which contained a 1D convolutional layer) were trained videos [3], a task that seems related to fear prediction in movies. using audio feature data. The results from these runs are listed in Table 1. Finally, the Phased LSTM [6] is a relatively recent model architecture innovation that could improve the accuracy scores of LSTMs when it comes to predicting irregular events in long Table 1: Results from the Fear Subtask sequences. Given the infrequency of fear-inducing seconds in the training data, adopting a Phased LSTM architecture could be Runs Intersection_ promising. Over_Union ACKNOWLEDGMENTS Run 1 0.06496 This work was supported in part by the AI Grant (now the Pioneer Run 2 0.07507 Fund). Run 3 0.08742 Run 4 0.11992 0.09874 REFERENCES Run 5 [1] E. Dellandrea, Martijn Huigsloot, L. Chen, Y. Baveye and M. Sjoberg, The MediaEval 2018 Emotional Impact of Movies Task, In MediaEval 2018 Workshop, Sophia Antipolis, France, 29-31 October 2018. The predictions of the individual models in each ensemble were [2] Yang Liu, Zhonglei Gu, and Tobey H. Ko. 2017. HKBU at averaged together to produce the ensemble’s final output. On the MediaEval 2017 Emotional Impact of Movies Task. In whole, the ensembles performed worse than the single LSTM Proceedings of MediaEval 2017 Workshop. Dublin, Ireland. models. This could be due in part to the fact that predictions in the [3] Ng, J., Hausknecht, M., Vijayanarasimhan, S., Rajat Monga, ensembles were joined via a simple average function, not a O. and Toderici, G. 2015. Beyond Short Snippets: Deep weighted average function. Results might have improved if the Networks for Video Classification. arXiv: 1503.08909. models with higher F1 scores were given greater influence over Retrieved from https://arxiv.org/abs/1503.08909 the final decision of the ensemble. [4] Hochreiter, S. and Schmidhuber, J., 1997. LONG SHORT- TERM MEMORY. Neural Computation, 9(8), pp.1735-1780. Between the two individual models in runs 4 and 5, the single [5] Sergey Ioffe, Christian Szegedy. 2015. Batch Normalization: layer LSTM trained on fc6 data (run 4) performed the best. This Accelerating Deep Network Training by Reducing Internal may suggest that visual features were more relevant to predicting Covariate Shift. arXiv: 1502.03167. Retrieved from fear-inducing segments than audio features. https://arxiv.org/abs/1502.03167 [6] Daniel Neil, Michael Pfeiffer, Shih-Chii Liu. 2016. Phased 5 CONCLUSIONS LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences. arXiv: 1610.09513. Retrieved from In this paper, we described our approach to addressing the fear https://arxiv.org/abs/1610.09513 prediction subtask using LSTM models. We compared the [7] Zitong Jin, Yuqi Yao, Ye Ma, and Mingxing Xu. 2017. performances of LSTM ensembles to single LSTMs trained on THUHCSI in MediaEval 2017 Emotional Impact of Movies either fc6 or audio feature data. Task. In Proceedings of MediaEval 2017 Workshop. Dublin, Ireland. There are a number of interesting future research avenues to explore and ways to improve what has been shared here. First, it may be beneficial to decrease the length of the sliding windows from 101 seconds and introduce a greater amount of overlap between them. Because of our design choice in this paper, we faced a class imbalance issue in our training data, which was heavily skewed towards non-fear inducing seconds. Decreasing the window size would make it easier to handpick a set of training data with a higher ratio of fear-inducing seconds to non-fear inducing seconds. Training the LSTM models on a balanced dataset may improve their performances.