In the Predicting Media Memorability Task at the MediaEval Challenge 2018, our team proposes an approach that uses deep visual features and recurrent network to predict videos' memorability. Features are extracted from CNN for a number of frames in each video. We forward these through a LSTM network to model the structure of the video and predict its memorability score. Our method achieves a correlation score of 0.484 on short-term task and 0.257 on long-term task in the oficial test set.
The Predicting Media Memorability task’s main objective is to
automatically predict a score which indicates how memorable a
video will be [
In this paper, we examine the sequential structure of videos with LSTM. We take advantage of deep convolutional neural networks to get image features as our main source of data for predicting video memorability. In our approach, there are two main stages: (i) extract image features through multiple frames of a video, (ii) predict its memorability score.
In the first stage, we sample 8 frames from each video. These
frames are then fed into a pretrained Inception-v3 convolution
network [
The task of predicting image memorability (IM) has made significant
progress since the release of MIT’s large-scale image memorability
dataset and their MemNet [
On the other hand, several attempts have been made in
annotating and predicting video memorability (VM) [
In our work, we explore the efect of videos’ sequential aspect on
memorability by using LSTM on visual features. To our knowledge,
LSTM based approach in VM has only been tried in [
Feature extraction: In order to resolve the temporal factor, instead
of using C3D [
We decide to use pre-trained Inception-v3 Convolutional Neural
Network [
Predicting memorability: We consider several approaches
regarding image and video memorability. In our attempts at adapting
IM to VM, we simply use only the middle frame of each video and
train two models with them as input data. We implemented a simple
model which consists of a CNN for feature extraction and 2 fully
connected (FC) layers for computing output score. We also retrain
the model in [
Videos’ captioning features are also tested by using provided
captions from the dataset. These captions accurately represent
the videos in terms of semantics. Moreover, all videos are short
and mostly single scene. Therefore, we use only 1 caption per
video instead of generating each for every frame. A vector of 300
dimensions is extracted from each video’s caption, which has been
preprocessed, using the pretrained word2vec model [
Furthermore, we propose to use a LSTM model to predict VM score using features extracted above (figure 1). Each extracted feature vector of every frame of a video is an input of a time step in our LSTM model. At the last step, a dense layer takes a 1024-dimension output vector of LSTM model and calculates the memorability score of that video.
For the short-term task, three out of five submitted runs are the
results of our proposed method with three diferent configurations
Duy-Tue, Le-Vu, Minh-Triet
(512, 1024, and 2048 hidden units). The remaining two are outputs
of the retrained AMNet in [
In this section, we evaluate our LSTM model on the task’s dataset. We present our quantitative results as well as some insight that we learned from this dataset.
Evaluation: Since we do not have the ground truth of the ofifcial test set, in order to compare these methods, we divide the development set into 3 parts: 6,000 videos for training, 1,000 videos for validating, and 1,000 videos for testing. Table 1 shows the results of diferent methods that we tested with our 1,000 test videos as well as the task’s oficial test set.
With our approach of using sequential visual features of videos with LSTM, the model with 1024 hidden units achieves the best score of ρ = 0.501 on 1,000 test videos mentioned above and ρ = 0.484 on the oficial test set for the short-term task; while for the longterm task, the model which was trained on short-term labels and validated on long-term labels gets ρ = 0.261 and ρ = 0.257 respectively. Meanwhile, if we use only long-term labels, our method gets ρ = 0.214 on the oficial test set.
In order to prevent overfitting while training, we apply a dropout rate of 0.5 on LSTM layer. We found that this rate gives the best results among 3 dropout rates of 0.25, 0.5, 0.75. The model also starts overfitting as it reaches its peak at around ρ = 0.50 − 0.52 and ρ = 0.24 − 0.26 on the validation subset of the short-term and long-term tasks respectively.
Discussion: The dataset from [
On the contrary, we think predicting long-term memorability on this dataset requires more in-depth research. For all of our tried methods, the results are always better when training/validating with short-term labels. Long-term labels seem to confuse the model which leads to worse performance. One possible reason that can cause the inconsistency in this particular dataset is that there exists multiple similar videos with opposite scores about or of specific objects.
As in figure 3, both videos are almost identical in terms of visual features such as color, angle, and actor. These videos might cause participants to make mistake when deciding whether they had watched it or not. Hence, their long-term labels give opposite results. 5
In our approach, we focus on the temporal aspect of videos by using their frames in a LSTM recurrent network. We have not tried using a combination of features in the process, hence, we will try using multiple aspects of a video to measure its performance.
Acknowledgments: We would like to thank SE-AI Lab, VNUHCMUS for their precious support.