Linear Models for Video Memorability Prediction Using Visual and Semantic Features Rohit Gupta, Kush Motwani Conduent Labs, India rohit.gupta@conduent.com,kush.motwani@conduent.com ABSTRACT and high level visual features, deep learning based action recogni- Memorability is defined as the quality or state of being easy to tion representations (C3D-Preds), and image and video captions remember. In the context of videos today, with billions of hours of for memorability prediction. The major findings on memorability user generated video content on online platforms like social media, from these papers are that models using captions give the best prediction of a cognitive measure like memorability has many po- individual results, and how features learned using deep learning tential applications. We investigate the use of various visual and improve those results. Additionally, researchers have found that semantic features in building models for video memorability pre- high level semantic features learned by CNNs trained for image diction. Along with the features provided as part of the Predicting classification achieve state of the art performance on a variety of Media Memorability Task at MediaEval 2018, we utilize generic computer vision tasks [8]. descriptors extracted from the feature map of Convolutional Neu- ral Networks. We are able to discover intrinsic semantic factors 3 APPROACH affecting memorability based on our analysis of models that use video captions. Finally, we are able to build an ensemble of models 3.1 Models that outperforms models built using a single feature type. Since most of the features provided are very high dimensional and the number of videos is of the same order of magnitude as the 1 INTRODUCTION dimensionality of the features, high variance and over-fitting are a major potential concern in this task. As a result we preferred In this work, we investigate the use of various visual and semantic simple, linear, highly regularized models, namely: features to predict video memorability, and conduct an extensive analysis over the chosen features, to develop a robust video memo- (1) LASSO (L1) regularized Logistic Regression [10] rability predictor. Among the features provided by challenge orga- (2) Linear Support Vector Regression [3] nizers [1] we train models over the HMP, LBP and ColorHistogram (3) ElasticNet (L1 and L2 Regularized Linear Regression) [4] visual features and InceptionV3-Predictions & C3D-Predictions semantic features. In addition, we train models over the video cap- For each set of features we tried each of these 3 models and select tions provided, and over features extracted from the last convolu- the best one. Values for various model hyper-parameters controlling tion layers of Convolutional Neural Networks trained for image the extent of regularization (such as C, alpha and L1-Ratio) were classification [6] [5] [8] applied to frames of the video. The models picked by using grid search over the dev set. are evaluated using Spearman’s rank correlation as the metric. Our In order to improve prediction accuracy, we build ensemble key findings and contributions are as follows: models using some of our best models. We use a simple weighted (1) Models for short term memorability perform much better averaging technique, where we blend the outputs of the best single than models for long term memorability. models developed previously. Weights for ensemble averaging are (2) Models based on InceptionV3-Preds, LBP and ColorHis- also picked using grid search over the dev set. togram do not work well and are outperformed by those based on C3D-Preds and HMP. 3.2 Features and Data Pre-Processing (3) Models based on the video captions provided outperform models trained on all above mentioned visual features. Video level features like HMP and C3D-Preds are used as-is, while (4) Analysis of models trained on video captions allows us to frame level features such as ColorHistogram and LBP are concate- identify semantic factors affecting video memorability. nated across frames. We pre-process the text captions into Bag-of- (5) We find that models based on high level representations Words features using CountVectorizer from Scikit-Learn. We use learned by CNNs trained on image classification tasks out- word unigrams and bigrams, remove English stopwords and pick perform both models based on visual features and captions. the optimal vocabulary size by cross-validation. ResNet and DenseNet features are extracted for the 1st, 56th, 2 RELATED WORK 112th frames of the video from the penultimate layer of ResNet50 and DenseNet121 models respectively. These features are then av- Work on video memorability has recently begun to generate a lot of eraged across frames and L2-Normalized. This results in a 1024 and interest, and recent works [2] [9] investigate the use of various low 2048 dimensional feature vector for DenseNet and ResNet respec- tively. Copyright held by the owner/author(s). The memorability scores is pre-processed in 2 different ways MediaEval’18, 29-31 October 2018, Sophia Antipolis, France depending on the model: MediaEval’18, 29-31 October 2018, Sophia Antipolis, France R. Gupta et al. • Z-score Normalization for Regressors: carried out in or- woman 9.4 der to make the scores scale-invariant, this results in a eating 8.5 significant increase in model accuracy. putting 8.3 • Sampling for Logistic Regression Classifier: We train the lying 7.9 Logistic Regression Classifier using binary labels sampled from a binomial distribution parameterized by the memo- girl 7.5 rability scores. This is motivated by the fact that the mem- selfie 6.9 orability score is the percentage of subjects who can recall relaxing 6.5 the contents of the video [2]. jellyfish 6.4 cat 6.2 4 RESULTS super slow 5.9 super 5.9 Tables 1 and 2 give an overall summary of our experimental results. Results for the best model for each feature are presented. For HMP, american 5.8 LBP and ColorHistogram, Lasso Logistic Regression is the best potrait 5.8 model, while ElasticNet is the best model for the other features. For pregnant 5.6 ensembling, we discover the optimal weights are as follows: couple 5.6 (1) Ensemble1: 0.25*Caption score + 0.75*ResNet score Figure 1: Terms for the most positive coefficients (2) Ensemble2: 0.1*HMP score + 0.1*Caption score + 0.8*ResNet score Table 1: Long Term Memorability Scores mountains −19.6 Model Validation Test sunset −14.6 Spearman Pearson MSE trees −12.5 C3D-Preds 0.153 mountain −12.2 HMP 0.136 clouds −11.6 InceptionV3-Preds 0.092 river −11.1 LBP 0.098 aerialof −9.9 Color Histogram 0.048 Captions 0.208 0.213 0.228 0.0194 lake −9.8 ResNet 0.261 0.234 0.259 0.0190 forest −9.2 DenseNet 0.259 0.245 0.269 0.0189 kenya −8.8 Ensemble1 0.235 0.253 0.278 0.0188 landscape −8.7 Ensemble2 0.257 0.244 0.268 0.0189 valley −8.4 leaves −8.2 sun −8 Table 2: Short Term Memorability Scores Figure 2: Terms for the most negative coefficients Model Validation Test Spearman Pearson MSE C3D-Preds 0.311 HMP 0.281 that the highest negative coefficients are all dominated by terms InceptionV3-Preds 0.150 related to natural scenery; whereas the highest positive coefficients LBP 0.254 are dominated by terms related to people and indoor actions & Color Histogram 0.105 objects. Contrary to intuition, videos of nature are not memorable, Captions 0.438 0.402 0.408 0.00560 a result that has also been observed in the context of image memo- ResNet 0.501 0.488 0.524 0.00487 rability. [7] DenseNet 0.491 0.473 0.504 0.00500 Like in past work on video memorability, our caption based Ensemble1 0.484 0.497 0.530 0.00488 models give superior performance than the C3D features, and low- Ensemble2 0.508 0.495 0.528 0.00489 level visual features. In contrast to past work however, we also examine the use of features extracted from the penultimate layer of CNNs and observe that models trained on these features outperform 5 ANALYSIS AND DISCUSSION the models based on captions. Coupled with the fact that models To analyze our interpretable captions based model we look at vo- trained over InceptionV3 predictions give poor results, we infer that cabulary terms corresponding to the most positive (Figure 1) and the representation learned by CNNs capture additional semantic negative (Figure 2) coefficients of the caption based models (aver- information relevant to predicting memorability beyond simply the aged over 100 models using different validation splits). This reveals category the image belongs to. Predicting Media Memorability Task MediaEval’18, 29-31 October 2018, Sophia Antipolis, France REFERENCES [1] Romain Cohendet, Claire-Hélène Demarty, Ngoc Q.K. Duong, Mats Sjöberg, Bogdan Ionescu, and Thanh-Toan Do. 2018. MediaEval 2018: Predicting Media Memorability Task. In Working Notes Proceedings of the MediaEval 2018 Workshop. [2] Romain Cohendet, Karthik Yadati, Ngoc Q.K. Duong, and Claire- Hélène Demarty. 2018. Annotating, Understanding, and Predicting Long-term Video Memorability. In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval. ACM, 178–186. [3] Rong-En Fan, Pai-Hsuen Chen, and Chih-Jen Lin. 2005. Working Set Selection Using Second Order Information for Training Support Vector Machines. Journal of Machine Learning Research 6, Dec (2005), 1889–1918. [4] Jerome Friedman, Trevor Hastie, and Rob Tibshirani. 2010. Regular- ization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software 33, 1 (2010), 1. [5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778. [6] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely Connected Convolutional Networks.. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [7] Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, and Aude Oliva. 2014. What Makes a Photograph Memorable? IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 7 (2014), 1469–1482. [8] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. CNN Features Off-the-Shelf: An Astounding Baseline for Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. [9] Sumit Shekhar, Dhruv Singal, and Harvineet Singh. 2017. Show and Recall: Learning What Makes Videos Memorable. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2730– 2739. [10] Guo-Xun Yuan, Chia-Hua Ho, and Chih-Jen Lin. 2012. An Improved GLMNET for L1-regularized Logistic Regression. Journal of Machine Learning Research 13, Jun (2012), 1999–2030.