Predicting the memorability of videos has great values in various applications including content recommendation, advertisement design and so on, which can bring convenience to people in everyday life, and profit to companies. In this paper, we present our methods in the 2018 Predicting Media Memorability Task. We explored some deeply-learned visual features and textual features in regression models to predict the memorability of videos.
The MediaEval 2018 Predicting Media Memorability Task [
In general, we utilize a regressor to predict the memorability score of each video and consider late fusion to combine diferent features. Two kinds of fusion strategies are utilized, namely score average and second-layer regression.
Our system framework is shown in Figure 1. We firstly run regressions to get videos’ memorability scores using diferent single features. In order to fuse multiple features, two strategies are considered and shown in Figure 1. For the score average strategy, we average the scores of diferent types of features from the same video, and the obtained score is the final memorability score of this video. For the second-layer regression, we concatenate the scores of diferent features from the same video as second-layer features, and the obtained second-layer features are fed into a second-layer regressor, which will predict the final memorability scores.
The videos are soundless, so we focus on visual and textual features, especially some high-level and semantic features.
The captions of videos are short with only a few words. We argue that people may be impressed by some particular objects or their combinations. The meanings of each word should be embedded into the representations of sentences for the memorability prediction.
A pre-trained word embedding contains a large amount of
semantic information, contributing to encoding the meaning of sentences.
We try the word embedding GloVe [
For visual features, we consider some deeply-learned
representations and aesthetic descriptors as our features, including C3D [
Additionally, we add some label information. If we are solving the long-term task, we firstly train a model with short-term labels and then use this model to predict the short-term scores of the test set. Then we transform the short labels of both train and test sets into 10 dimensional one-hot vectors. The 10 buckets of a one-hot vector denote the range between 0 to 1 with step 0.1. If a label is in the range of a bucket, e.g. the label is 0.56 and it is in the range of 0.5 to 0.6, we set the value of this bucket as 1 and the rest of buckets are set to 0. And then we add them to the end of each text feature. For short-term task, we map the long-term labels into one-hot vectors and use them in the same way mentioned above. 3.1
The development set and the test set contain 8000 and 2000 videos respectively. We firstly rank the videos by their memorability scores and sample videos with a constant step value of 4. Finally we split the development set into 2 parts, namely 6000 videos in train set and 2000 videos in local test set.
We simply consider two types of regressors, namely Support
Vector Regression (SVR) and Random Forest Regression (RFR). The
parameters were determined by grid searches. The Penalty
parameter C in SVR is searched from 0.125 to 32. The parameters of
n_estimators and max_depth are searched in the range [100, 1000]
with step 100 and [
Table 1 and Table 2 show the results on the test set, m and s means the score average and the second-layer regression strategy respectively. all means fusing all features, namely visual, textual and labels. visual denotes fusing visual representations and text is the fusion of all word embedding features. required means using average strategy and not using the label information. We used average strategy in required runs because average strategy performed generally better than second-layer regression on the local test set. We can notice that the required runs in long-term and short-term task both have the best performances. Label information helps little on local test set and does not work in oficial test set. We consider that maybe mapping labels into one-hot vectors is not a proper way to fully utilize the label information and it is worthy to find a proper representation format of the labels or a fusion method with other features.
We pick out a number of videos for analysis and we find that some of them depict close-ups of objects or regions, while some of them show overall scenes such as natural landscape, stories of some characters.
We draw 3 conclusions after viewing these videos and their labels.
(1) The videos with low short-term labels usually have low long-term labels. (2) The videos with high short-term labels and low long-term labels usually depict some close-ups. (3) There are few numbers of videos with low short-term labels and high long-term labels. These videos generally have open and wide scenes.
We find that it is dificult to predict the memorability of the videos in the second and third situations.
In sum, we consider that if a video is memorable in a long term, it is also memorable in a short term generally. Conversely, videos with high short-term labels cannot determine the long-term memorability. 4
In conclusion, we explored visual and textual representations for videos and built a regression model which can calculate a memorability score for a given video. The results show that textual representations perform better than visual features. In the future, we will focus on the visual semantic representations and object detection related works to find more interesting methods to predict memorability of videos. And how to use label information is another interesting point to be explored.
This work is supported by National Key Research and Development Plan under Grant No. 2016YFB1001202. This work is partially supported by National Natural Science Foundation of China (Grant No. 61772535). visual-m The results of each single feature for long-term and short-term memorability prediction are printed in Figure 2 and Figure 3 respectively. As shown in Figure 2 and Figure 3, textual representations are on the same level and textual features perform better than visual representations. We think that the captions contain more clear