This paper describes the approach developed to predict the shortterm and long-term video memorability at the 2018 MediaEval Predicting Media Memorability Task [1]. This approach utilizes the scene semantics derived from the titles of the videos using natural language processing (NLP) techniques and a recurrent neural network (RNN). Compared to using video-based features, this approach has a low computational cost for feature extraction. The performance of the semantic-based methods are compared with those of the aesthetic feature-based methods using support vector regression (ϵ-SVR) and artificial neural network (ANN) models, and the possibility of predicting the highly subjective media memorability with simple features is explored.
Knowledge of the memorability of a video has potential in
advertisement and content recommendation applications. Although highly
subjective, it has been shown that media memorability is
measurable and predictable. As with most other machine learning problems,
ifnding the most relevant features and the right model is the key
to the successful prediction of the media memorability. In [
Even though images are reportedly diferent from videos in many
aspects [
Compared to video-based features, the extraction of video
semantics from its title requires relatively low feature extraction cost.
Moreover, the authors in [
Run
Method
The main model corresponding to run 4 is a three-layer neural network with a recurrent layer; the structure of the model is depicted in Fig. 1. After importing the titles, punctuation and whitespace is removed. The texts are then tokenized to integer sequences with length equal to 20. Longer titles are truncated and short titles are padded with zeros. After the preprocessing, 80% of the training dataset is randomly chosen to train the model, and the remaining 20% is used for model evaluation.
The tokenized titles are fed to an embedding layer with the output dimension equal to 15. The embedding matrix is initialized following uniform distribution. No embedding regularizer is used. The semantics are extracted by adding a fully connected recurrent layer with 10 units after the embedding layer. The activation function for the recurrent layer is hyperbolic tangent. The layer uses a bias vector, which is initialized as zeros. Initializer for the kernel weight matrix used for the linear transformation of the inputs is chosen as “glorot uniform”. Initializer for the recurrent kernel weight matrix used for the linear transformation of the recurrent state is set as “orthogonal”. A 10-node fully connected dense layer follows using rectangular linear unit (ReLU) activation function. The kernel regularization function used is l1 −l2 regularization with λ1 = 0.001 and λ2 = 0.004. The initialization scheme is the same as that of the RNN layer. The last layer is a 2-node dense layer predicting the short-term and long-term memorability simultaneously, where a linear activation function is used. This model is trained using RMSprop optimizer against the mean absolute error (MAE). The model is trained 10 epochs with batch size equal to 20.
Similar to the model in [
Preprocessing RNN Titles
Punctuation removal
Vectorization
Embedding
Layer
RNN Layer (H.T.,10)
Dense
Layer (ReLU,10)
Dense
Layer (Linear,2)
Run 4 PCA 90% ߳ -SVR
Run 5 the error is set to be 0.1. The ϵ, which defines a tube within which no penalty is associated, is equal to 0.01. Radial basis functions are used as the kernel function. The above hyper parameters are obtained through a grid search cross-validation using the Spearman’s rank correlation as the scoring matrix. 2.2
Details of the models using precomputed aesthetic features [
The evaluation results show that the mean aesthetic features are more relevant to the video memorability. Thus run 3 is generated using ANN and mean aesthetic features as illustrated in Fig. 2. The ANN model consists of three dense layers, the first two layers are fully connected dense layers with 50 nodes, where ReLU activation function is used, and l2 regularization is applied. The regularization penalty constant is set to 0.001. Dropout rates for the first two layers are equal to 0.1 and 0.5, respectively. The output layer has two nodes and uses linear activation functions. Mean square error (MSE) is used as the loss function during the training process, where the validation data is randomly chosen from the training data within each epoch. 20 epochs are trained in total with the batch size equal to 32. From the returned evaluation results in Table. 1, the following conclusions can be observed: 1) The model using RNN and semantics is the best among all the five models. It confirms that the semantics of the videos are more relevant to both short and longterm memorability than aesthetic features. Especially for long-term memorability, the semantic based models outperform the aesthetic feature-based models unanimously. 2) Without the recurrent layer, the performance decreases. Thus it can be inferred that interaction between objects in a video has more impact on the video’s long-term and short-term memorability than knowing only the objects. 3) Even though there is certain correlation between short and long-term memorability as depicted in Fig. 3, results have shown that short-term memorability is more predictable than long-term ones since all models score higher in short-term than long-term memorability. As illustrated in Fig. 3, long-term scores range from 0.2 to 1 and exhibit higher variance than the short-term scores, which distribute from 0.4 to 1. Thus, one possible reason is that the long-term memorability is more subjective and depends more on individual’s memory.
It is observed that the SVR models using median and mean aesthetic features have close performance as run 4 in terms of shortterm memorability prediction. However, the long-term performance is far worse than run 4. Further investigations are needed to clarify this. Performance of run 3 is worse than that of run 2, even though both of them use mean aesthetic features. Possible reasons are over-fitting and the missing standardization procedure in run 4. In the future, ensemble methods are expected to further enhance the prediction accuracy.
Predicting media memorability task