In the context of videos today, with billions of hours of user generated video content on online platforms like social media, prediction of a cognitive measure like memorability has many potential applications including content recommendation, advertisement design and so on, which can bring convenience to people in everyday life and profit to companies. This paper describes our approach designed for the MediaEval 2020 Predicting Media Memorability Task. Our approach uses the preextracted features and the features extracted through a deep mutillevel encoding network along with provided textual description to predict a probability-like memorability score of videos. We use the same set of features for predicting both short-term and longterm media memorability respectively.
The Predicting Media Memorability Task [
People have an ability to remember and recall photos and
videos with a surprising amount of detail. Interestingly, not all
content is stored and recalled equally well. Media Memorability
has attracted research interest recently in the area of Computer
Vision [
In this paper, we investigate the use of various visual and
semantic features to predict video memorability. Among the
features provided we train models over AlexNetFC7, HOG, LBP,
HSVHist, RGBHist features and C3D-Predictions semantic
feature. In addition, we train models over textual features which
were extracted from BERT feature extractor [
In this section, we discuss the task of Predicting Media Memorability using video, image and textual features. We were provided with the development set’s labels only. Statically we divided the development set into the training video data (800 videos), and 200 videos for the validation set, which each data sample is associated with its visual features and the corresponding memorability score. Our approach is to develop individual models per set of features provided or extracted and to then combine some features that seem to have a better performance. In addition, the short-term memorability and long-term memorability were modelled independently using the same set of features in our experiments. 2.1 We used image level features: Color Histogram in HSV and RGB space, AlexNetFC7, HOG, LBP, VGGFC7 and video-level feature: C3D. For the image-level features were extracted from 3 frames for each video, the 3 frames per each video represent the first, the middle and the last frame in the movie. C3D feature was used to represent spatio-temporal aspect of a video. We also employed two additional features:
Multi-level encoding feature. For a given video, we extract
uniformly a sequence of n frames with a pre-specified interval of
0.5 second. Per frame we extract deep features using a pertained
ImageNet CNN, as commonly used for video content analysis.
Consequently, the video is described by a sequence of feature
vectors {v1, v2,…,vn}, where vt indicates the deep feature vector
of the t-th frame. We utilize a multi-level encoding network [
Caption. Following a prior work [
Here, we describe our proposed model and provide the training
details. The development set is randomly divided into 8:2 split for
training and validation respectively. We simply consider three
types of regressors, namely Gradient Boosting Regression (GBR),
Random Forest Regression (RFR) and a mutil-layer perceptron
neural network (MLP neural network). The input feature of each
model was standardized. For the MLP neural network, we input
each set of features to the first input layer of it, two hidden layer
along with ReLU activation and a fully connected linear layer is
applied on it to obtain a single number representing the
memorability score in the range from 0 to 1. The model is trained
using Adam optimizer [
We trained different models for both short-term and longterm memorability scores. For each set of features we tried each of these 3 models on the validation dataset and results are presented in Table1. The results of each single feature for short-term and long-term memorability prediction on the validation dataset are shown in Table1. From Table 1, the multi-level encoding feature preforms better than other visual representations. We think that the network explicitly and progressively learns to represent global, local and temporal patterns in videos by jointly exploiting multi-level encodings. It means that the proposed method of feature representation is effective.
In addition, we think that the captions contain more clear descriptions about the elements in the videos. If a specific object is depicted by a word, the word embedding can describe the relations of this object and others in the whole environment. The visual features may contain some details of regions but not that intuitive. So we consider combining the two features to predict the final memorability score.
We used the simple concatenation to combine the mutil-level
encoding feature and the textual feature extracted utilizing BERT
feature extractor [
In this work, we explored visual and textual representations for videos and built a few regression models which can calculate a memorability score for a given video. From our experiments, it was clear that multi-level feature would improve the short-term and long-term video memorability prediction, but it seemed poor performance.
Due to the new dataset which is more complicated and with more actions happening. In future, we plan to conduct the Video Memorability experiment with improved features like Dense Optical Flow features, Action based features representing the sequence of actions in the video, and also aim to leverage the audio in the videos.
This work was supported in part by the Natural Science Foundation of China under Grant 61801441, in part by the Project of the Beijing National Research Center for Information Science and Technology of China under Grant BNR2019TD01022,in part by the Fundamental Research Funds for the Central Universities under Grant CUC19ZD003, Grant CUC2019B066, and in part by the State Key Laboratory of Media Convergence and Communication, Communication University of China.