In this paper, we present a multimodal approach to modelling media memorability for the "Predicting Media memorability" task at MediaEval 2018. Our approach uses video and image based features along with provided textual description to predict a probability-like memorability score for each of the seven second audioless video clips. We use the same set of features for predicting both short-term and long-term media memorability.
With the dramatic surge of visual media content on platforms like
Instagram, Flickr and YouTube, it is imperative that new methods
for curating, annotating and organizing this content be explored.
To this efect, non-traditional metrics for tagging media content
have been examined. Previous works have used metrics such as
aesthetics[
An important aspect of human cognition is the ability to
remember and recall photos and videos with a surprising amount of detail.
Interestingly, not all content is stored and recalled equally well
[
In this section, we outline our multimodal approach to model media memorability using video, image and text features. The visual features are inspired from diferent properties of images such as saliency and aesthetics. We assumed that memorability of the video is afected by properties of images comprising the video. We also hypothesized that captions provide additional cues for understanding semantics of videos. The short-term memorability and long-term memorability were modelled independently using the same set of features explained in this section. 2.1
We used image level features based on: color, saliency, aesthetics,
memorability and presence of human faces. Image features were
calculated on frame number 0, 56 and 112. Except for aesthetics, we
computed mean and standard deviation of these features across the
three frames to give feature representation for a video clip. C3D
features were used to represent spatiotemporal aspect of a video.
Color: Color and its distribution have a significant influence on
human cognition [
Saliency: [
Aesthetics: Aesthetics and human judgements of memorability are
highly correlated [
Face-based Feature: Using a state-of-the-art deep learning method
[
Image Memorability: We hypothesized that memorability of the
video is afected by the memorability of the images comprising
the video. We used MemNet proposed by [
To analyse the data with respect to captions, we divided the train data into 4 bins. Each bin consisted of captions corresponding to 4 quartiles in memorability annotation. We defined a metric ‘word relevance’ inspired from term frequency−inverse document frequency (Tf-Idf ) statistic typically used in Information Retrieval. We define word relevance for a word wi in a bin j as W Ri j . Let total number of bins be N, total number of words in a bin j be Wj , frequency of a word wi in a bin j be wi j , frequency of bins where the word wi appears be bwi , and frequency of word wi in other bins be wi jˆ.
W Ri j = wi j (1 + log N
Wj
1 ) bwi wi jˆ (1) We created a wordlist of all unique words from the video captions. After stemming and lemmatizing, we removed all stopwords from the list. Words with W R value above a threshold were shortlisted as candidate words and their frequency in captions was considered as a feature. We believe that higher value of W R quantifies the word’s association to a particular range of memorability. We hypothesized that W R increases with the relative higher frequency of a word in a particular bin with respect to its frequency in other bins. It was observed that words related to topics like food and toddlers fall in higher memorability range and generic words related to topics such as landscape and scenery fall in the lower memorability range.
We also used pre-trained GloVe embeddings1 of words to capture more information from textual description. We preprocessed the caption data by removing stopwords. We created a 100 dimensional word-embedding vector for each word. 3
We ranked videos by assigning a probability like score to each video clip, treating it as a regression problem. The annotations for short term and long term memorability were skewed towards higher values with mean of 0.86 (short term) and 0.78 (long term). All input features were normalized and the ground-truth was kept unwhitened so that the model captured the skewed output distribution. We divided the given dataset into train and validation sets in the ratio 3:1 such that the annotations in two sets have similar distribution. We explored diferent combinations of features for predicting memorability.
Experiment 1: Low level features namely, colorfulness, blur[
Experiment 2: We concatenated the face based feature to the 3D HSV. The resultant 130 dimension vector represents the color spread and facial information in an image. We passed this vector and features for C3D , aesthetics and saliency through dense fully connected layers indepedently. We ensembled these models using their normalized correlation values on validation as coeficients for weighted average.
Experiment 3: We used word embeddings to train diferent neural network architectures. CNN-LSTM and Bi-LSTM models give best correlation on training and validation data. We ensembled models 1https://nlp.stanford.edu/projects/glove/ trained on image features, video features and text embeddings using weighted average and SVR.
We used sigmoid activation in the last layer for all networks so as to restrict the output to the range of 0 to 1. ReLU activation was used for all other layers. We also fine-tuned the model over validation data before predicting on test data. Final predictions submitted for evaluation were based on models from experiments 2 and 3. 4
In experiment 1, the model performed poorly with a near zero Spearman’s rank correlation over the validation set. This shows that only low level image features are not suficient to understand media memorability. As part of the challenge we submitted results of five runs based on experiments 2 and 3. Table 1 lists correlations obtained from two best performing models. As per the evaluation on an unseen test set, best performing model gives Spearman’s and Pearson’s correlation for short-term memorability as 0.46 and 0.50 respectively. The correlations for long term memorability are 0.23 and 0.25 respectively. In the run with experiment 2, we obtained Spearman’s rank correlation of 0.39 and 0.17 for short term and long term memorability respectively. Our best submission based on experiment 3 gives an improvement of 7%.
As mentioned earlier, this model from experiment 3 used textual information along with visual features. The improvement shows that the words from captions are contributing to predicting memorability scores. We believe that additional textual features such as location cues, emotion cues may be useful in further improvements. Mean values of our predictions of short term and long term memorability over validation data are 0.83 and 0.78 respectively. The values are close to the mean values of their respective annotations in the training data. This shows that our model succeeds in capturing the skewed distribution of training data. 5
This paper presents the ensemble model by team AREA66 for predicting media memorability. We use visual features based on image and video along with textual features from given captions. The results show that better results are obtained by combining visual and textual features. On the other hand, only visual features give lowest values for prediction correlation. We also noticed that experiments with only low level image features give poor results. In future, we plan to explore efects of textual information on video memorability. Secondly, we aim to explore more sophisticated methods for utilizing low level image features to improve prediction performance.