Memorability of videos has great values in diferent applications such as education system, advertising design and media recommendation. Memorability automatic prediction can make people's daily life more convenient, and bring companies profit. In this paper, we present our approaches in The Predicting Media Memorability Task at MediaEval 2019 . We explored some visual, textual and artificially designed concept related features in regression models to predict the memorability of videos.
The MediaEval 2019 Predicting Media Memorability Task [
Generally, we concentrate on visual features extracted from videos and textual features drew from given textual metadata. Among visual features, we consider both the visual information in a frame and the temporal factors between successive frames. In addition, we use deep network to extract high-level and semantic feature representation. Based on each individual extracted feature, we then do feature normalization. Further, we perform feature fusion to get better performance. Finally, we consider two simple but eficient regressors called Support Vector Regression (SVR) and Random Forest Regression (RFR) to get final memorability scores.
In addition to the eight video special features provided by the
oficial benchmark, we try to extract other new features that may be
related to video memorability. We try to extract high-level
representation of videos with DenseNet[
The origin title of each video summarizes inclusion objects
and events briefly. We try some popular word embedding
models to get textual features from these captions, including GloVe[
We find that when people watch videos, they do not pay equal
attention to each region in the scene, but first focus on a certain
area, which may change over time. And we learn from Baveye
et al.[
Specifically, we fine-tune AMNet on the dataset of MediaEval 2019, training the long-term and short-term sub-tasks separately. Considering that AMNet is designed for still images, we extract 11 frames at a uniform time interval for each video as input. As for prediction, we take the median memorability score of 11 frames as the final result. As in figure 1, we can visually observe that the output attention maps of video frames are closely related to the highly memorable visual contents in the picture. 2.3
Generally, people have a preference for paying attention to diferent
concepts. According to [
To find out whether the idea makes sense on our data, we counted the number of entities belonging to each concept. Then we take the average of the memorability scores of the videos corresponding to these concepts. The result is shown in Figure 2, showing that the preference on concepts also afects the memorability of the video to some extent.
Then, with the help of GloVe word vector pre-trained on Common Crawl data, we calculate the distance between entities and the above 7 concepts. For each entity, we can get a distance vector with 7 elements which can represent the correlation between the entity and each concept. For each caption, we take the average of the distance vectors of all the entities the caption contains, so that we get a feature vector. Then we apply a random forest regression on the feature vectors. The Spearman score on long-term memorability prediction is 0.11. This result, based solely on the textual manual features shows that the concepts of entities in videos is meaningful for predicting memorability of the videos.
We made further exploration in this direction. [
We split the develop set into two parts, namely the training set and the validation set. We train and test on these two sets and determine the final methods according to the performances on validation set, ifnally the models are trained on the whole develop set and predict on the oficial test set. The results on validation set and oficial test set are shown in Table 3 and Table 2 respectively.
In Table 1, Table 3 and Table 2, "Base1" means the early fusion of DenseNet169, GloVe and C3D features, while "Base2" additionally includes ConceptNet. The "Base1" and "Base2" are the best early fusion strategies on validation set. The ’AM’ is AMNet scores mentioned above and ’Dist’ denotes the scores from concept distances. The plus sign means late fusion and we apply a set of weights on them empirically, which is "Base * 0.9 + AM * 0.1" and "Base * 0.6 + Dist * 0.4"
Based on our previous experience, the deep CNN features and caption embedding features are the most efective in the memorability prediction task, such as DenseNet169 and GloVe word embeddings in our experiments. In addition, we also consider some other features to study whether there are some complementary points and pick out two combinations as "Base1" and "Base2". It’s easy to remember familiar things for us, so we consider there are a fuzzy and a clear way to represent these things. AMNet can automatically pay attention to a object or an area that may attract us, and this is like a fuzzy representation, because it does not show the concept directly. The clear way is the concept distances which depict the distance map of the current video. The late fusion of these two methods and the "base" boost the performaces slightly. We suppose that the "base" namely CNN features and caption embeddings are stable, and maybe the caption embeddings have already included some information about these concepts, so the improvement of results is not very obvious. 5
In conclusion, we design a model that uses visual and textual representations to predict the memorability scores of given videos. The results show that deep CNN and caption word embeddings are efective and the attention information from AMNet and semantic distance extracted from captions can boost the performance slightly. In the future, we will focus on the concept representation and semantic representations. Also the interaction of long term and short term ground-truth is a interesting point to be explored.
This work was supported by National Key Research and Development Plan under Grant No. 2016YFB1001202, Research Foundation of Beijing Municipal Science Technology Commission under Grant No. Z181100008918002 and National Natural Science Foundation of China under Grant No.61772535.