This paper summarises THAU-UPM's approach and results from the MediaEval 2023 Predicting Video Memorability task. Focused on the generalisation subtask, our work leverages a pre-trained Video Vision Transformer (ViViT), fine-tuned on memorability-related data, to model temporal and spatial relationships in videos. We propose novel, annotator-independent automatic segment selection methods grounded in visual saliency. These methods identify the most relevant video frames prior to conducting memorability score estimation. This selection process is implemented during both training and evaluation phases. Our study demonstrates the efectiveness of fine-tuning the ViViT model compared to a scratchtrained baseline, emphasising the importance of pre-training for predicting memorability. However, the model shows comparable sensitivity to both saliency-based and naive segment selection methods, suggesting that fine-tuning may harness similar benefits from various video segments. These results underscore the robustness of our approach but also signal the need for ongoing research.
Memorability is an aspect of human perception that has attracted the interest of researchers in psychology, neuroscience and computer science alike due to its relevance to areas as diverse as disease diagnosis, marketing and education. Taking advantage of the burgeoning advances in artificial intelligence architectures for media retrieval, classification and analysis as a proxy for modelling the connections between human senses and our understanding of the world through cognitive processes is particularly appealing, which explains the steady stream of work on the subject in recent years.
The MediaEval Predicting Video Memorability task, currently in its sixth edition [
To the best of our knowledge, most recent tackles on the Predicting Video Memorability task
rely on using image-level architectures to extract knowledge from a handful of frames and then
performing some sort of fusion strategy to obtain a single representation for the entire video,
using powerful image-only backbone models such as the Vision Transformer but neglecting
architectures that use video itself as input [
In the spirit of transfer learning and generalisation, we propose to fine-tune a Video Vision Transformer, pre-trained on a generic video classification task, on memorability related data. Furthermore, we evaluate diferent selection strategies, where video segments are fed into the model in both training and evaluation steps.
We hypothesise that the ViViT architecture has the potential to be a robust, data agnostic model for memorability prediction, and therefore perform well in the generalisation task scenario. With this in mind, our approach is based on incorporating generic knowledge into the training process using two complementary strategies: a) fine-tuning a pre-trained ViViT model instead of starting training from scratch, and b) proposing automatic segment selection methods that do not rely on annotator data.
The ViViT Transformer is an adaptation of the original Vision Transformer that is able to
process and model temporal relationships between frames as well as the spatial relationships
that appear in each image by including a three-dimensional Tubelet Embedding encoder before
the Transformer input. We start our training from the oficial ViViT checkpoint available at
huggingface1. Its training data, Kinetics 400 [
We train on a single 32-frame long segment extracted from each video from the training set,
using one of the segment selection methods that will be described next. The frame number
selection is imposed by the architecture of the model that we wish to fine-tune. In order to
compare our fine-tuning proposal, we train a baseline ViViT model from scratch, using the
implementation proposed in [
Using [
To test our approach, we compare it to two image-agnostic baseline methods: Uniform Sampling of frames from the entire clip, and extracting the frames from the Center Segment of the video.
As a preliminary study, we compare our fine-tuning approach with the from scratch baseline in order to analyse the efect of progressively unfreezing the weights of the Transformer encoders, starting from the one next to the regression head and going towards the input. We resort to the Uniform Sampling method for fine-tuning in this step. The results in term of Spearman Rank Correlation Coeficient (SRCC), the oficial metric for the task, are shown in Table 1, where we observe that our fine-tuning proposal significantly outperforms the baseline with just a single unfrozen encoder. This supports our idea that the ViViT model is greatly benefitted by a pre-training step in which general knowledge is acquired, and that it can translate this learnt relationships into the memorability problem. On the other hand, the fact that our best result comes from unfreezing all the model weights and letting it update as a whole leads us to think that the specific visual and semantic language related to the task still plays a crucial role in its solving, and therefore this aforementioned generic knowledge must be conditioned to it. This synergy between broad and specific expertise encourage us to use the fine-tuning approach for our runs, and to explore whether an automatic segment selection can enhance the adaptation process.
With this in mind, we show our final testing set results for our runs in Table 2, where we compare the diferent segment selection methods. We perceive that there is no significant diference between the saliency based methods and the naive approaches used for comparison, neither on the Memento10k developing set nor in the VideoMem test set, apart from a slight drop in performance when using the Spectral Residual method, indicating that the relationship between the spectral characteristics of an image and its memorability is somewhat weaker than a more nuanced approach. As can be seen in Figure 1, the Fine-Grained saliency maps are more detailed, in contrast with the less defined aspect of the Spectral Residual, which may influence on the selected segment. However, it seems that fine-tuned method benefits equally from segments across the whole video, independently of which part of it is used as input. Although we believe this is a sign of the robustness of our proposal, a more in-depth analysis of the relationship between image saliency and annotators response in terms of memorability could possibly further enhance the capabilities of this type of architecture.
In this paper we outline our contribution to the MediaEval 2023 Predicting Video Memorability task. We propose to leverage pre-trained Video Transformers in order to create robust memorability predictors that take sequences of frames as input. We also explore automatic segment selection methods based on saliency. Our results show that fine-tuning significantly outperforms training from scratch on our setup, but that the model is not specially sensible to automatic selection methods. We aim to deepen our exploration on the matter by developing advanced methods based on saliency and other perceptual features that output multiple candidate segments in order to broaden the training information, as well as evaluating the potential benefits of these methods on models trained from scratch.
We would like to thank M. Gabriel Constantin for his insights on his work, which have been greatly helpful for our research. I.M.-F.’s research was supported by the UPM (Programa Propio I+D+i). This work was funded by Project ASTOUND (101071191 — HORIZON-EIC-2021PATHFINDERCHALLENGES-01) of the European Commission and by the Spanish Ministry of Science and Innovation through the projects GOMINOLA (PID2020-118112RB-C22) and BeWord (PID2021-126061OB-C43), funded by MCIN/AEI/10.13039/501100011033 and by the European Union “NextGenerationEU/PRTR”