This paper describes the GIBIS team experience in the Predicting Media Memorability Task at MediaEval 2018. In this task, we were required to develop an approach to predict a score reflecting whether videos are memorable or not, considering short-term memorability and long-term memorability. Our proposal relies on diferent learning strategies: for long-term memorability, we adopted k-NN regressors trained on hand-crafted motion features; and for shortterm memorability, we trained deep learning models.
The Predicting Media Memorability task is part of the MediaEval
2018 Benchmarking Initiative for Multimedia Evaluation. The goal
of this task is to automatically predict a memorability score for
a video reflecting its probability to be remembered. For this, it is
provided a dataset composed of 10,000 short, soundless videos split
into 8,000 videos for the development set and 2,000 videos for the
test set. Also, pre-computed visual features are provided to facilitate
participation. For more details about this task, please, refer to [
In this paper, we explore two main approaches: (1) for long-term
memorability, an ensemble of ten KNR (k-Nearest Neighbor
Regressor) or SVR (Support Vector Regression) [
The proposed approach exploits diferent strategies for long-term
and short-term memorability. The former relies on hand-crafted
motion features extracted with HMP whereas the latter uses
datadriven features learned with C3D. One limitation of C3D is its
capacity to capture subtle but long-term motion dynamics, as it
requires to break a video into small clips. Unlike C3D, HMP captures
motion dynamics of a video as a whole, and not just parts.
Our proposed approach for the long-term memorability subtask
consists of using the pre-computed HMP feature [
HMP encodes an entire video into a single histogram
representing its overall motion dynamics. From this, we can consider the
HMP vector as a hash identifying each video as a point in a
highdimensional space. This idea of space is the foundation for the use
of the KNR and SVR algorithms [
For reproducibility, the KNR and SVR implementations used
comes from the scikit-learn python package [
For short-term memorability, we use a deep learning model based
on the C3D architecture [
The implemented C3D model follows a 3D convolution and 3D max pooling architecture1 and it outputs a fully connected layer with 2048 neurons. This is the first of three fully connected layers that feed a multi-head output for regression and classification. Figure 1 shows the network architecture of our first experiment.
In our second experiment, we used the same C3D model, however,
we consider a two-stream network. For this, a second C3D model
receives as input the optical flow [
For both networks, the input data are normalized to real values
in the range [
For training, a diferent loss function was used for each head: mean squared logarithmic error for the regression head and categorical crossentropy for the classification head. Then, a weighted sum of these individual losses with weights 1.0 and 0.7, respectively, was computed as the final loss to be minimized by a RMSProp optimizer with a learning rate of 0.0015. 3
We submit three diferent runs configured as shown in Table 1. We calibrated the long-term memorability subtask through 10-fold cross-validation on the development data and use a holdout method with 10% of the development data for validation to calibrate the short-term memorability subtask. The evaluation metrics are: Spearman’s rank correlation, Pearson correlation coeficient, and MSE (Mean Squared Error). The former is the oficial metric for the task.
Table 2 presents the results for the development and test sets considering the long-term memorability subtask. In the development set, we tested diferent regression models: SVR with RBF kernel and KNR. Also, the values experimented for the parameter k of KNR were 5, 20, and 30. Notice that KNR performs better than SVR for Spearman and Pearson metrics and the best results were achieved by KNR with k = 20. Therefore, we submit one run for the long-term memorability subtask considering our best result on the 2https://docs.opencv.org/3.4/d7/d8b/tutorial_py_lucas_kanade.html development set. From it, we achieved a Spearman value of 0.11845 for the test set.
Table 2: Long-term memorability results.
Dev. Set Test Set
Approach HMP + SVR kernel RBF HMP + 5-NN regressor HMP + 20-NN regressor HMP + 30-NN regressor HMP + 20-NN regressor
It is important to notice that HMP and C3D have an important diference: HMP captures motion dynamics of a video as whole whereas C3D is limited to a short window of fixed duration. An intention of future work is to analyze if features encoding long-term motion dynamics, like HMP or RNN (Recurrent Neural Network), are better for predicting video memorability than those capturing short-term motion dynamics, like C3D or ORB.
We can think about some reasons for the failure of our deep learning models. First, we constrain a full length video to a sequence of 16 consecutive frames. Smarter strategies to capture the temporal structure of a video, like RNN with LSTM (Long-Short Term Memory), could led to improvements. Second, we trained our deep neural networks from scratch. As the training set is rather small, data augmentation could be used to improve the results.
Another promising direction is to combine diferent features. For short memorability, we fused optical flow and video data. Would it improve results if we fuse video (visual data) and captions (textual data) provided for the task? Or other visual features, like HMP?
We thank the São Paulo Research Foundation - FAPESP (grant 2016/06441-7), the Brazilian National Council for Scientific and Technological Development - CNPq (grants 423228/2016-1 and 313122/2017-2) and the Brazilian Federal Agency for Coordination for the Improvement of Higher Education Personnel - CAPES (grant 1703269) for funding. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.