=Paper=
{{Paper
|id=Vol-2283/MediaEval_18_paper_31
|storemode=property
|title=Linear Models for Video Memorability Prediction Using Visual and Semantic Features
|pdfUrl=https://ceur-ws.org/Vol-2283/MediaEval_18_paper_31.pdf
|volume=Vol-2283
|authors=Rohit Gupta,Kush Motwani
|dblpUrl=https://dblp.org/rec/conf/mediaeval/GuptaM18
}}
==Linear Models for Video Memorability Prediction Using Visual and Semantic Features==
https://ceur-ws.org/Vol-2283/MediaEval_18_paper_31.pdf
Linear Models for Video Memorability Prediction Using Visual
and Semantic Features
Rohit Gupta, Kush Motwani
Conduent Labs, India
rohit.gupta@conduent.com,kush.motwani@conduent.com
ABSTRACT and high level visual features, deep learning based action recogni-
Memorability is defined as the quality or state of being easy to tion representations (C3D-Preds), and image and video captions
remember. In the context of videos today, with billions of hours of for memorability prediction. The major findings on memorability
user generated video content on online platforms like social media, from these papers are that models using captions give the best
prediction of a cognitive measure like memorability has many po- individual results, and how features learned using deep learning
tential applications. We investigate the use of various visual and improve those results. Additionally, researchers have found that
semantic features in building models for video memorability pre- high level semantic features learned by CNNs trained for image
diction. Along with the features provided as part of the Predicting classification achieve state of the art performance on a variety of
Media Memorability Task at MediaEval 2018, we utilize generic computer vision tasks [8].
descriptors extracted from the feature map of Convolutional Neu-
ral Networks. We are able to discover intrinsic semantic factors 3 APPROACH
affecting memorability based on our analysis of models that use
video captions. Finally, we are able to build an ensemble of models 3.1 Models
that outperforms models built using a single feature type. Since most of the features provided are very high dimensional and
the number of videos is of the same order of magnitude as the
1 INTRODUCTION dimensionality of the features, high variance and over-fitting are
a major potential concern in this task. As a result we preferred
In this work, we investigate the use of various visual and semantic
simple, linear, highly regularized models, namely:
features to predict video memorability, and conduct an extensive
analysis over the chosen features, to develop a robust video memo- (1) LASSO (L1) regularized Logistic Regression [10]
rability predictor. Among the features provided by challenge orga- (2) Linear Support Vector Regression [3]
nizers [1] we train models over the HMP, LBP and ColorHistogram (3) ElasticNet (L1 and L2 Regularized Linear Regression) [4]
visual features and InceptionV3-Predictions & C3D-Predictions
semantic features. In addition, we train models over the video cap- For each set of features we tried each of these 3 models and select
tions provided, and over features extracted from the last convolu- the best one. Values for various model hyper-parameters controlling
tion layers of Convolutional Neural Networks trained for image the extent of regularization (such as C, alpha and L1-Ratio) were
classification [6] [5] [8] applied to frames of the video. The models picked by using grid search over the dev set.
are evaluated using Spearman’s rank correlation as the metric. Our In order to improve prediction accuracy, we build ensemble
key findings and contributions are as follows: models using some of our best models. We use a simple weighted
(1) Models for short term memorability perform much better averaging technique, where we blend the outputs of the best single
than models for long term memorability. models developed previously. Weights for ensemble averaging are
(2) Models based on InceptionV3-Preds, LBP and ColorHis- also picked using grid search over the dev set.
togram do not work well and are outperformed by those
based on C3D-Preds and HMP.
3.2 Features and Data Pre-Processing
(3) Models based on the video captions provided outperform
models trained on all above mentioned visual features. Video level features like HMP and C3D-Preds are used as-is, while
(4) Analysis of models trained on video captions allows us to frame level features such as ColorHistogram and LBP are concate-
identify semantic factors affecting video memorability. nated across frames. We pre-process the text captions into Bag-of-
(5) We find that models based on high level representations Words features using CountVectorizer from Scikit-Learn. We use
learned by CNNs trained on image classification tasks out- word unigrams and bigrams, remove English stopwords and pick
perform both models based on visual features and captions. the optimal vocabulary size by cross-validation.
ResNet and DenseNet features are extracted for the 1st, 56th,
2 RELATED WORK 112th frames of the video from the penultimate layer of ResNet50
and DenseNet121 models respectively. These features are then av-
Work on video memorability has recently begun to generate a lot of
eraged across frames and L2-Normalized. This results in a 1024 and
interest, and recent works [2] [9] investigate the use of various low
2048 dimensional feature vector for DenseNet and ResNet respec-
tively.
Copyright held by the owner/author(s). The memorability scores is pre-processed in 2 different ways
MediaEval’18, 29-31 October 2018, Sophia Antipolis, France
depending on the model:
MediaEval’18, 29-31 October 2018, Sophia Antipolis, France R. Gupta et al.
• Z-score Normalization for Regressors: carried out in or-
woman 9.4
der to make the scores scale-invariant, this results in a
eating 8.5
significant increase in model accuracy.
putting 8.3
• Sampling for Logistic Regression Classifier: We train the
lying 7.9
Logistic Regression Classifier using binary labels sampled
from a binomial distribution parameterized by the memo- girl 7.5
rability scores. This is motivated by the fact that the mem- selfie 6.9
orability score is the percentage of subjects who can recall relaxing 6.5
the contents of the video [2]. jellyfish 6.4
cat 6.2
4 RESULTS super slow 5.9
super 5.9
Tables 1 and 2 give an overall summary of our experimental results.
Results for the best model for each feature are presented. For HMP, american 5.8
LBP and ColorHistogram, Lasso Logistic Regression is the best potrait 5.8
model, while ElasticNet is the best model for the other features. For pregnant 5.6
ensembling, we discover the optimal weights are as follows: couple 5.6
(1) Ensemble1: 0.25*Caption score + 0.75*ResNet score
Figure 1: Terms for the most positive coefficients
(2) Ensemble2: 0.1*HMP score + 0.1*Caption score + 0.8*ResNet
score
Table 1: Long Term Memorability Scores
mountains −19.6
Model Validation Test
sunset −14.6
Spearman Pearson MSE
trees −12.5
C3D-Preds 0.153
mountain −12.2
HMP 0.136
clouds −11.6
InceptionV3-Preds 0.092
river −11.1
LBP 0.098
aerialof −9.9
Color Histogram 0.048
Captions 0.208 0.213 0.228 0.0194 lake −9.8
ResNet 0.261 0.234 0.259 0.0190 forest −9.2
DenseNet 0.259 0.245 0.269 0.0189 kenya −8.8
Ensemble1 0.235 0.253 0.278 0.0188 landscape −8.7
Ensemble2 0.257 0.244 0.268 0.0189 valley −8.4
leaves −8.2
sun −8
Table 2: Short Term Memorability Scores Figure 2: Terms for the most negative coefficients
Model Validation Test
Spearman Pearson MSE
C3D-Preds 0.311
HMP 0.281
that the highest negative coefficients are all dominated by terms
InceptionV3-Preds 0.150
related to natural scenery; whereas the highest positive coefficients
LBP 0.254
are dominated by terms related to people and indoor actions &
Color Histogram 0.105
objects. Contrary to intuition, videos of nature are not memorable,
Captions 0.438 0.402 0.408 0.00560
a result that has also been observed in the context of image memo-
ResNet 0.501 0.488 0.524 0.00487
rability. [7]
DenseNet 0.491 0.473 0.504 0.00500
Like in past work on video memorability, our caption based
Ensemble1 0.484 0.497 0.530 0.00488
models give superior performance than the C3D features, and low-
Ensemble2 0.508 0.495 0.528 0.00489
level visual features. In contrast to past work however, we also
examine the use of features extracted from the penultimate layer of
CNNs and observe that models trained on these features outperform
5 ANALYSIS AND DISCUSSION the models based on captions. Coupled with the fact that models
To analyze our interpretable captions based model we look at vo- trained over InceptionV3 predictions give poor results, we infer that
cabulary terms corresponding to the most positive (Figure 1) and the representation learned by CNNs capture additional semantic
negative (Figure 2) coefficients of the caption based models (aver- information relevant to predicting memorability beyond simply the
aged over 100 models using different validation splits). This reveals category the image belongs to.
Predicting Media Memorability Task MediaEval’18, 29-31 October 2018, Sophia Antipolis, France
REFERENCES
[1] Romain Cohendet, Claire-Hélène Demarty, Ngoc Q.K. Duong, Mats
Sjöberg, Bogdan Ionescu, and Thanh-Toan Do. 2018. MediaEval 2018:
Predicting Media Memorability Task. In Working Notes Proceedings of
the MediaEval 2018 Workshop.
[2] Romain Cohendet, Karthik Yadati, Ngoc Q.K. Duong, and Claire-
Hélène Demarty. 2018. Annotating, Understanding, and Predicting
Long-term Video Memorability. In Proceedings of the 2018 ACM on
International Conference on Multimedia Retrieval. ACM, 178–186.
[3] Rong-En Fan, Pai-Hsuen Chen, and Chih-Jen Lin. 2005. Working
Set Selection Using Second Order Information for Training Support
Vector Machines. Journal of Machine Learning Research 6, Dec (2005),
1889–1918.
[4] Jerome Friedman, Trevor Hastie, and Rob Tibshirani. 2010. Regular-
ization Paths for Generalized Linear Models via Coordinate Descent.
Journal of Statistical Software 33, 1 (2010), 1.
[5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep
Residual Learning for Image Recognition. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition. 770–778.
[6] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q.
Weinberger. 2017. Densely Connected Convolutional Networks.. In
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition.
[7] Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, and Aude
Oliva. 2014. What Makes a Photograph Memorable? IEEE Transactions
on Pattern Analysis and Machine Intelligence 36, 7 (2014), 1469–1482.
[8] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan
Carlsson. 2014. CNN Features Off-the-Shelf: An Astounding Baseline
for Recognition. In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) Workshops.
[9] Sumit Shekhar, Dhruv Singal, and Harvineet Singh. 2017. Show and
Recall: Learning What Makes Videos Memorable. In Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition. 2730–
2739.
[10] Guo-Xun Yuan, Chia-Hua Ho, and Chih-Jen Lin. 2012. An Improved
GLMNET for L1-regularized Logistic Regression. Journal of Machine
Learning Research 13, Jun (2012), 1999–2030.