Video memorability is one of the vital aspects of subjective multimedia perception and, as such, is closely and thoroughly studied in the computer vision literature. This paper presents the methods proposed by AIMultimediaLab for the generalization subtask of the 2023 edition of the Predicting Video Memorability task. We explore several methods for augmenting the training process for a video Vision Transformer network, aiming to increase the number of hard-to-predict samples in the training set in order to increase the robustness of the targeted AI model. Starting from our previous works, we analyze several visual features that define "hard-to-predict" samples, and based on these features, we augment the training data of our models to target those specific videos that pose problems for memorability prediction.
The prediction of video memorability is an essential aspect in the subjective analysis of
multimedia content, with the MediaEval Predicting Video Memorability1 series of benchmarking
tasks playing an important role in bringing attention in the computer vision community to
the study of this concept. While previous editions of this benchmarking task have focused on
memorability prediction in videos extracted, annotated and processed in similar conditions,
this edition [
As we will show throughout the paper, this work represents the continuation of some of our previous works on memorability, particularly those targeting the use of vision transformers in the prediction of subjective concepts, and a sample-based analysis of videos we defined as "hard-to-predict" from a memorability standpoint. We continue this work by applying training augmentation, particularly for the problematic videos for our vision transformer networks. The rest of the paper is structured as follows. Section 2 presents previous works our methods are based on. Following this, the methods employed by our team are presented in Section 3, while our results are presented in Section 4. Finally, the paper concludes with Section 5.
Our proposed method is built upon two of our previous works. The first one, published in the
previous edition of the MediaEval memorability task [
Starting from these two works, we select two segments as representatives for each video in the
training set, based on the Memorable Moments approach presented in the first work [
A general diagram of the training method we propose is presented in Figure 1. We propose
using the Memorable Moments selection scheme, as presented in [
We use the Memorable Moments selection scheme, as presented in [
In [
We use these four methods to determine which videos are more challenging to classify by AI models. As presented in the previous section, we will only keep two Memorable Moments video segments only for the videos deemed as "hard-to-predict". We will therefore compute the values for each of the four features (ground truth score, sharpness, contrast, and dynamism), and split the training set into four quartiles, according to the value of each feature, with the top quartile, 1, representing the videos that theoretically should be easier to predict according to each feature, and the bottom quartile, 4, representing those that would be hard to predict. The entire training set will thus be divided, for each feature , as follows: = 1, ∪ 2, ∪ 3, ∪ 4, . We will then keep two Memorable Moments segments only for the videos that belong to the bottom quartile, 4.
We apply these training augmentation schemes to a vision transformer deep neural network
that is based upon the ViViT architecture [
This network is then trained in five diferent setups. In the original setup, only one segment per video is fed into the network at training time. We consider this setup as the baseline for our approach. The following four setups will contain augmented samples, represented by one additional video segment for each video in the 4 quartile for each of the selected discriminative features.
Augmentation Method ViViT - baseline ViViT + GT score ViViT + sharpness ViViT + contrast ViViT + dynamism
We present the results in Table 1. Firstly, we analyze the results on the validation set of this task, which is composed of the Memento10k devset. While the un-augmented baseline system results are good enough with a Spearman’s Rank Correlation Coeficient (SRCC) of 0.651, two of the proposed augmentation methods outscore this for the validation experiments, even if by a small margin. These two methods are represented by the ground-truth score-based quartile augmentation with an SRCC value of 0.668 (ViViT + GT score), and the dynamism-based augmentation (ViViT + dynamism), with an SRCC value of 0.680. On the other hand, the two other methods score lower on the devset when compared with the baseline method.
Similar trends are noticeable when looking at the oficial testset results. The ViViT + GT score method has the highest performance, with an SRCC value of 0.382, closely followed by the ViViT + dynamism approach with 0.380. The baseline method scores 0.361, while the sharpness and contrast methods have even lower scores.
When comparing the direct prediction performance on the Memento10k devset with the oficial generalization scores on the VideoMem dataset, we notice a sharp decline in performance. This is an indication of the significant level of dificulty generalization tasks pose. On the other hand, we are pleased to report that at least two of the proposed feature-based augmentation methods scored better than the baseline method. The GT score-based method achieved a 5.81% increase over the baseline run, with similar performance from the dynamism-based method.
We presented a sample-based augmentation method for media memorability prediction, in a generalization setup, where the training and the testing data came from diferent datasets, which involved diferent video sources and annotators. Our best performing method augmented the samples at training time based on their ground truth scores. Concretely, videos that have ground truth memorability values close to the average were augmented, thus resulting in a schema that increases the number of hard-to-predict videos, allowing the AI model to learn more details about these videos.
Financial support provided under project AI4Media, a European Excellence Centre for Media, Society and Democracy, H2020 ICT-48-2020, grant #951911.