=Paper=
{{Paper
|id=Vol-2670/MediaEval_19_paper_26
|storemode=property
|title=Combining Textual and Visual Modeling for Predicting Media
Memorability
|pdfUrl=https://ceur-ws.org/Vol-2670/MediaEval_19_paper_26.pdf
|volume=Vol-2670
|authors=Alison Reboud,Ismail
Harrando,Jorma Laaksonen,Danny
Francis,Raphaël Troncy,Héctor Laria
Mantecón
|dblpUrl=https://dblp.org/rec/conf/mediaeval/ReboudHLFTM19
}}
==Combining Textual and Visual Modeling for Predicting Media
Memorability==
https://ceur-ws.org/Vol-2670/MediaEval_19_paper_26.pdf
Combining Textual and Visual Modeling
for Predicting Media Memorability
Alison Reboud* , Ismail Harrando* , Jorma Laaksonen+ ,
Danny Francis * , Raphaël Troncy* , Héctor Laria Mantecón+
* EURECOM, Sophia Antipolis, France
+ Aalto University, Espoo, Finland
{alison.reboud,ismail.harrando,danny.francis,raphael.troncy}@eurecom.fr
{jorma.laaksonen,hector.lariamantecon}@aalto.fi
ABSTRACT of SUN-397 [15] concept features. The image and concept features
This paper describes a multimodal approach proposed by the MeMAD were extracted from the middle frames of the videos. The hidden
team for the MediaEval 2019 “Predicting Media memorability” task. layer uses ReLU activations and dropout during the training phase,
Our best approach is a weighted average method combining pre- while the output unit is sigmoidal. We trained separate models
dictions made separately from visual and textual representations of for the short and long term predictions with the Adam optimizer.
videos. In particular, we augmented the provided textual descrip- The number of training epochs was selected with 10-fold cross-
tions with automatically generated deep captions. For long term validation with 6000 training and 2000 testing samples.
memorability, we obtained better scores using the short term pre- CaptionsA. Our first captioning model uses the DeepCaption
dictions rather than the long term ones. Our best model achieves software1 and is quite similar to the best-performing model of
Spearman scores of 0.522 and 0.277 respectively for the short and the PicSOM Group of Aalto University’s submissions in TRECVID
long term predictions tasks. 2018 VTT task [13]. The model was trained with COCO [10] and
TGIF [9] datasets using the concatenation of ResNet-152 and ResNet-
101 [6] features as the image encoding. The embed size of the LSTM
1 INTRODUCTION network [7] was 256 and its hidden state size 512. The training used
Considering video memorability as a useful tool for digital content cross-entropy loss.
retrieval as well as for sorting and recommending an ever growing CaptionsB. Our second model has been trained on the TGIF [9]
number of videos, the Predicting Media Memorability Task aims and MSR-VTT [16] datasets. First, 30 frames have been extracted
at fostering the research in the field by asking its participants to for each video of these datasets. Then, these frames have been pro-
automatically predict both a short and long term memorability cessed by a ResNet-152 [6] that had been pretrained on ImageNet-
score for a given set of annotated videos. The full description for 1000: we keep local features after the last convolutional layer of the
this task is provided in [2]. Last year’s best approaches for both ResNet-152 to obtain features maps of dimensions 7x7x2048. At that
the long term[5] and short term tasks [14] indicated that high point, videos have been converted into 30x7x7x2048-dimensional
level representations extracted from deep convolutional models tensors. A model based on the L-STAP method [4] has been trained
performed the best in terms of visual features. Furthermore, the best on MSR-VTT and TGIF: all videos from TGIF, and training and
long term model [5] was a weighted average method including Bag- testing videos from MSR-VTT have been used for training, and
of-Words features extracted from the provided captions. Following validation has been performed throughout training with the usual
this approach, we created multimodal weighted average models validation set of MSR-VTT, containing 497 videos. Cross-entropy
with visual deep features and textual features extracted from both has been used as the training loss function. The L-STAP method has
the provided video titles, as well as from automatically generated been used to pool frame-level local embeddings together to obtain
deep captions. 7x7x1024-dimensional tensors: each video is eventually represented
by 7x7 local embeddings of dimension 1024. These have been used
2 APPROACH to generate captions as in [4].
VisualEmbeddings. The local embeddings used for CaptionsB
2.1 Visual Approaches
have also been used to derive global video embeddings, by averaging
VisualScore. Our visual-only memorability prediction scores are the mentioned 7x7 local feature embeddings. These global video
based on using a feed-forward neural network with visual features embeddings have then been fed to a model of two hidden layers,
in the input, one hidden layer of 430 units and one unit in the output the first one and the second one having respectively 100 and 50
layer. The best performance was obtained with 6938-dimensional units, and ReLU activation function. The number of training epochs
features consisting of the concatenation of I3D [1] video features, is 200 with an early stopping monitor.
ResNet-152 and ResNet-101 [6] image features and two versions
Copyright 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution
1 https://github.com/aalto-cbir/DeepCaption
4.0 International (CC BY 4.0).
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France A. Reboud et al.
2.2 Textual Approaches Table 1: Results on test set for short term memorability
Through initial experiments and from last year’s results on this
task, the descriptive titles provided with each video prove to be Method Spearman Pearson MSE
an important modality for predicting the memorability scores. In Textual 0.441 0.464 0.01
order to build on this observation, we generate captions for each VisualScore 0.495 0.543 0
video using the two visual models described above (CaptionsA and WA1 0.512 0.552 0
CaptionsB). While the generated captions are not always accurate, WA2 0.522 0.559 0
they seem to noticeably help the model disambiguate some titles WA3 0.520 0.557 0
and use some of the vocabulary already seen on the training set
(e.g. the title contains words such as couple" or "cat" while the
generated caption would say "a man and a woman" or "an animal",
respectively, which are more common words in the training set Table 2: Results on test set for long term memorability
and thus help the model generalize better on inference time). The
models described in this section use a concatenation of the original Method Spearman Pearson MSE
provided title and the generated captions as their input.
Textual 0.239 0.25 0.03
Multiple techniques for generating a numerical score from this
VisualScore 0.268 0.289 0.03
input sequence were considered (in ascending order of their perfor-
WA2 0.277 0.296 0.03
mance on cross-validation).
WA3 0.275 0.295 0.03
Recurrent Neural Network. We use an LSTM [7] to go through
WA3lt 0.260 0.285 0.02
the GloVe embeddings [12] of the input and predict the scores at the
last token. This model performed consistently the worst, probably
due to the length of the input sequence at times, and the empirical
observation that word order doesn’t seem to matter for this task.
the mean predictions from the combinations of the four models
Convolutional Neural Network. We use the same model as
trained on 6000 videos (VisualScore6k). For the Long Term task, all
[8] except for a regression head instead of a classifier trained on
models except from the WA3lt exclusively use short-term scores.
top of the CNN, and GloVe embeddings as input. This model leaks
less information thanks to max-pooling, and performs much better • WA1 = 0.5Textual+0.5VisualScore
than its recurrent counterpart. • WA2 = 0.25Textual+0.25VisualEmb+0.5VisualScore8k
Self-attention. Similar to the previous methods, we feed our • WA3 = 0.25Textual+0.25VisualEmb+0.5VisualScore6k
input text to a self-attentive bi-LSTM [11] to generate a sentence • WA3lt = WA3 with long-term scores
embedding that we use to predict the memorability scores. This
We observe that the weighted average method which was trained
model performs on par with the CNN method.
on the whole training set and included our two visual approaches
BERT. We used a pre-trained BERT model [3] to generate a
and our textual approach works the best for short term predictions.
sentence embedding for the input by max-pooling the last hidden
For long term prediction, one of the key observations to make is
states and reducing their dimension through PCA (from 768 to 250).
that WA3lt got the second worst results. This is consistent with our
This model performs better than the previous ones but it is more
early observation that short-term scores for long-term evaluation
computationally demanding.
yields substantially better results.
Bag of Words. We vectorize the input string by counting the
number of instances of each token (and frequent n-grams) after
removing the stop words and the least frequent tokens. The score 4 DISCUSSION AND OUTLOOK
is predicted by training a linear model on the counts vector. This This paper describes a multimodal weighted average method out-
simple model performs the best on our cross-validation, which can performing the best results of the Predicting Media Memorability
be justified by the lack of linguistic or grammatical structure in the Task 2018. One of the key contribution of this paper is to have
titles and generated captions that would justify the use of a more demonstrated that using deep captions helped improving the pre-
sophisticated model. dictions. We also conclude that, quite surprisingly, a simple n-gram
For all the models considered, the addition of the generated frequency count was more efficient at modelling memorability than
captions improves the prediction score on the validation set consid- more sophisticated textual models. Finally, the fact that long term
erably. It also should be noted that the use of short-term scores for memorability was better predicted using short term predictions
long-term evaluation yields substantially better results throughout indicates that we failed at capturing the memorability decay of a
all of our experiments. scene from a few minutes to a few days. In the future, we would
like to focus more on this aspect of the task.
3 RESULTS AND ANALYSIS
During the evaluation process, we created four test folds of 2000 ACKNOWLEDGEMENTS
videos and therefore four models trained on 6000 videos. For the This work has been partially supported by the European Union’s
VisualScore approach, we decided to use predictions from a model Horizon 2020 research and innovation programme via the project
trained on the entire set of 8000 videos (VisualScore8k), as well as MeMAD (GA 780069).
Media Memorability MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
REFERENCES
[1] João Carreira and Andrew Zisserman. 2017. Quo Vadis, Action Recog-
nition? A New Model and the Kinetics Dataset. In IEEE Conference on
Computer Vision and Pattern Recognition (CVPR). 4724–4733.
[2] Mihai Gabriel Constantin, Bogdan Ionescu, Claire-Helene Demarty,
Ngoc Q. K. Duong, Xavier Alameda-Pineda, and Mats Sjoberg. 2019.
The Predicting Media Memorability Task at MediaEval 2019. Proc.
MediaEval workshop (2019).
[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
2018. BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding. CoRR abs/1810.04805 (2018). http://arxiv.
org/abs/1810.04805
[4] Huet B. Francis, D. 2019. L-STAP : Learned Spatio-Temporal Adaptive
Pooling for Video Captioning. In First International Workshop on AI
for Smart TV Content Production (AI4TV).
[5] Rohit Gupta and Kush Motwani. 2018. Linear Models for Video Memo-
rability Prediction Using Visual and Semantic Features. In MediaEval.
[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep
Residual Learning for Image Recognition. In IEEE Conference on Com-
puter Vision and Pattern Recognition (CVPR). 770–778.
[7] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term
memory. Neural computation 9, 8 (1997), 1735–1780.
[8] Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classi-
fication. Conference on Empirical Methods in Natural Language Pro-
cessing (EMNLP) (2014).
[9] Yuncheng Li, Yale Song, Liangliang Cao, Joel R. Tetreault, Larry Gold-
berg, Alejandro Jaimes, and Jiebo Luo. 2016. TGIF: A New Dataset
and Benchmark on Animated GIF Description. CoRR abs/1604.02748
(2016). http://arxiv.org/abs/1604.02748
[10] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Per-
ona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Mi-
crosoft COCO: Common Objects in Context. In European Conference
on Computer Vision (ECCV).
[11] Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing
Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A Structured Self-
attentive Sentence Embedding. CoRR abs/1703.03130 (2017). http:
//arxiv.org/abs/1703.03130
[12] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.
Glove: Global vectors for word representation. In In EMNLP.
[13] Mats Sjöberg, Hamed R. Tavakoli, Zhicun Xu, Héctor Laria Mantecón,
and Jorma Laaksonen. 2018. PicSOM Experiments in TRECVID 2018.
In Proceedings of the TRECVID 2018 Workshop. Gaithersburg, MD, USA.
[14] Duy-Tue Tran-Van, Le-Vu Tran, and Minh-Triet Tran. 2018. Predicting
Media Memorability Using Deep Features and Recurrent Network. In
MediaEval.
[15] Jianxiong Xiao, J. Hays, K.A. Ehinger, A. Oliva, and A. Torralba. 2010.
SUN database: Large-scale scene recognition from abbey to zoo. In
IEEE Computer Vision and Pattern Recognition (CVPR). 3485–3492.
[16] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. MSR-VTT: A large
video description dataset for bridging video and language. In IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). 5288–
5296.