This paper describes the approach taken by the AI Multimedia Lab team for the MediaEval 2021 Predicting Media Memorability task. Our approach is based on a Vision Transformer-based learning method, which is optimized by filtering the training sets for the two proposed datasets. We attempt to train the methods we propose with video segments that are more representative for the videos they are part of. We test several types of filtering architectures, and submit and test the architectures that best performed in our preliminary studies.
Media Memorability has attracted the attention of researchers from
diferent domains for a long time. This included studies that revealed
that humans have an uncanny ability of memorizing large quantities
of images, going so far as to correctly encode details from those
images. Generally speaking, there is a certain discrepancy between
the study of image and video memorability, with more attention
given in the current literature to the former. In this context, the
MediaEval Predicting Media Memorability task [
Our proposed method for video memorability prediction relies on the use of Vision Transformer networks for feature extraction, a dense network ending for sample regression and an important frame filtering method that attempts to use the most memorable moments from the video samples in the training process. The rest of the paper is organized as follows: Section 2 presents the works most related to our proposed approach, while our method is presented in Section 3. Section 4 presents the results both in our training and development process and on the final testing set. Finally, the main conclusions are presented in Section 5.
Deep Neural Networks have come a long way in addressing many
machine learning problems, and for a long time, starting with the
success of AlexNet [
Recently, Vision Transformers shown their usefulness for image
processing, surpassing convolutional approaches in image
recognition tasks [
The general outline of our memorability prediction method is presented in Figure 1. We propose creating a three stage system. In the first stage, we theorize that not all frames might be valuable for memorability calculation and therefore propose a frame filtering method. Following this, we extract visual features by using a Vision Transformer architecture, and, in a final step, we perform regression with a dense MLP architecture.
Frame filtering. We base our frame filtering system on the assumption that not all frames are equal when trying to determine the properties of a larger video sequence. In our case, we propose using the annotations provided by the organizers for selecting the frames that may best characterize the video from a memorability standpoint. We call these frames "Memorable Moments", and while they may not represent the exact moment or the exact process of human memory retrieval, we theorize that they may represent a better approach than simply attempting to use the entire video for processing.
We test several setups for the frame filtering method as follows. First of all, we have to take into account the lag time between human memory recognition and button press. Therefore, given , a user’s response time in milliseconds from the start of the video, we subtract the following values: 500, 1000, 1500 milliseconds from the value in order to define the actual time of retrieval from memory. Of course we cap the resulting value at zero in case retrieval occurred very close to the start of the video. Furthermore, we take a variable number of frames, namely 15, 30, 60 from the resulting location and use them for analysis. We will compare our ifltering method (which we call 2) against a default method, where all the video is taken into consideration (called 1).
Visual Features. For visual feature extractors, we test two popular
Vision Transformer architectures, namely the DeiT [
Dense MLP. The final stage consists of classifying the chosen features extracted from the selected frames and outputting the final memorability score. This is done via a simple dense architecture with 3 hidden layers of size 1024, 512, and 256. 4
In the first stage of development, we test the setups proposed in the previous Section, by training on the Memento training set and testing on its development set. With regards to the frame filtering method, we find that a setup of 1000 milliseconds delay in response time and 30 frames analyzed are the best setups, though not by a significant margin. For the Transformer architecture, we select the DeiT architecture as the best performer in these preliminary tests, though again not by a large margin.
The final results computed on the testset are presented in Table 1. It is interesting to notice that, in five out of the six (1, 2) comparison pairs, the results were better for the variant of the system which employed filtered training, via Memorable Moments, while at times even being so with a significant margin.
For the prediction subtask (subtask 1) we find that results for the
Memento10K prediction are much better than the ones for TRECVid.
This may be a result of many factors, but one of them may be
represented by the lower number of video samples in the latter
dataset. Also, continuing the trend recorded at the previous version
of the Predicting Media Memorability task [
Finally, regarding the generalization subtask (subtask 2), we find a significant drop in performance when compared to subtask 1. This may be due to diferences in the types of movies in the dataset, but methods that reduce this issue must definitely be studied. 5
In this paper we present a media memorability prediction method that is based on the use of Vision Transformer architectures and frame filtering method we call Memorable Moments. Our experiments show good results for both these components and, for future developments we propose improving this framework by testing more feature extraction architectures, performing tests against convolutional architectures, predicting Memorable Moments on the testset, and testing this type of approach on other subjective multimedia concepts and properties.
This work was funded under project AI4Media “A European Excellence Centre for Media, Society and Democracy”, grant 951911, H2020 ICT-48-2020.