Memorability is defined as the quality or state of being easy to remember. In the context of videos today, with billions of hours of user generated video content on online platforms like social media, prediction of a cognitive measure like memorability has many potential applications. We investigate the use of various visual and semantic features in building models for video memorability prediction. Along with the features provided as part of the Predicting Media Memorability Task at MediaEval 2018, we utilize generic descriptors extracted from the feature map of Convolutional Neural Networks. We are able to discover intrinsic semantic factors affecting memorability based on our analysis of models that use video captions. Finally, we are able to build an ensemble of models that outperforms models built using a single feature type.
Copyright held by the owner/author(s).
In this work, we investigate the use of various visual and semantic features to predict video memorability, and conduct an extensive analysis over the chosen features, to develop a robust video memorability predictor. Among the features provided by challenge organizers [1] we train models over the HMP, LBP and ColorHistogram visual features and InceptionV3-Predictions & C3D-Predictions semantic features. In addition, we train models over the video captions provided, and over features extracted from the last convolution layers of Convolutional Neural Networks trained for image classification [6] [5] [8] applied to frames of the video. The models are evaluated using Spearman's rank correlation as the metric. Our key findings and contributions are as follows:
(1) Models for short term memorability perform much better than models for long term memorability.
Since most of the features provided are very high dimensional and the number of videos is of the same order of magnitude as the dimensionality of the features, high variance and over-fitting are a major potential concern in this task. As a result we preferred simple, linear, highly regularized models, namely:
(1) LASSO (L1) regularized Logistic Regression [10] (2) Linear Support Vector Regression [3] (3) ElasticNet (L1 and L2 Regularized Linear Regression) [4] For each set of features we tried each of these 3 models and select the best one. Values for various model hyper-parameters controlling the extent of regularization (such as C, alpha and L1-Ratio) were picked by using grid search over the dev set.
In order to improve prediction accuracy, we build ensemble models using some of our best models. We use a simple weighted averaging technique, where we blend the outputs of the best single models developed previously. Weights for ensemble averaging are also picked using grid search over the dev set.
Video level features like HMP and C3D-Preds are used as-is, while frame level features such as ColorHistogram and LBP are concatenated across frames. We pre-process the text captions into Bag-of-Words features using CountVectorizer from Scikit-Learn. We use word unigrams and bigrams, remove English stopwords and pick the optimal vocabulary size by cross-validation.
ResNet and DenseNet features are extracted for the 1st, 56th, 112th frames of the video from the penultimate layer of ResNet50 and DenseNet121 models respectively. These features are then averaged across frames and L2-Normalized. This results in a 1024 and 2048 dimensional feature vector for DenseNet and ResNet respectively.
The memorability scores is pre-processed in 2 different ways depending on the model:
• Z-score Normalization for Regressors: carried out in order to make the scores scale-invariant, this results in a significant increase in model accuracy. • Sampling for Logistic Regression Classifier: We train the Logistic Regression Classifier using binary labels sampled from a binomial distribution parameterized by the memorability scores. This is motivated by the fact that the memorability score is the percentage of subjects who can recall the contents of the video [2].
Tables 1 and 2
To analyze our interpretable captions based model we look at vocabulary terms corresponding to the most positive (Figure 1) and negative (Figure 2) coefficients of the caption based models (averaged over 100 models using different validation splits Like in past work on video memorability, our caption based models give superior performance than the C3D features, and lowlevel visual features. In contrast to past work however, we also examine the use of features extracted from the penultimate layer of CNNs and observe that models trained on these features outperform the models based on captions. Coupled with the fact that models trained over InceptionV3 predictions give poor results, we infer that the representation learned by CNNs capture additional semantic information relevant to predicting memorability beyond simply the category the image belongs to.