This paper describes AIMultimediaLab's approach and results achieved during the 2022 MediaEval Predicting Video Memorability task. The proposed approach represents a continuation of last year's work, using, updating and better analysing the concept of Memorable Moments. This is done by improving the scheme we use for selecting Memorable Moments, and allowing for the possibility that more than one video segment is representative for the entire video clip from a memorability standpoint. Furthermore, we propose studying a new architecture for processing the selected Memorable Moments, by implementing a variant of the popular ViViT architecture, that is more suited to analysing pure video content.
Media Memorability is one of the domains that lately gained considerable traction in the research
community, thanks to the need of novel and better methods of classifying the huge quantities
of data associated with social media and video content sharing platforms. While previous
work focused more on the prediction of image-based content, lately a significant push for
video-based processing can be noticed in the multimedia research environment. In this context,
the MediaEval 2022 Predicting Video Memorability task [
As we will show, this paper represents a continuation and an ongoing work on the study of Memorable Moments in particular, and of the way video segments can represent or can be interpreted as representative of an entire video in general. For this edition of the MediaEval Memorability task, we will only be participating to the first subtask. The rest of the paper is organized as follows: Section 2 presents our approach, while Section 3 presents and analyzes the results, and the paper concludes with Section 4.
Our proposed method represents a continuation of last year’s work [
Repeatable Spatio-Temporal Transformer MLP
Norm Multi-Head Attn
Norm Positional Embedding
Token Embedding
networks for extracting visual features from selected frames, and a frame selection method we
called Memorable Moments. We proposed and proven that having a frame selection method is
a positive addition to the overall performance, as this would spare the network from processing
frames that perhaps represent noise or are unimportant to the overall memorability score of
the video in itself. The first of the changes we propose in this paper is represented by the
replacement of the two architectures with an architecture dedicated to video processing, namely
the popular transformer-based ViViT [
As already stated, we use the ViViT architecture [
We vary several parameters of the network in order to search for a robust architecture that would best fit our experiments. First of all, we test several values for the number of parallel self-attention heads in each block, using the values 4, 8, 16, 32. Secondly, we vary the number of repeatable blocks, using the values 4, 8, 16, 32.
Regarding the Memorable Moments scheme, we propose several variations for deploying this frame filtering method. Theoretically, given a video clip that is composed of a set of frames denoted = [1, 2, ..., ], and a set of annotators = [1, 2, ... ], each annotator that will go watch the video will press a button whenever they recognise the target video. Given a delay in response time of 500 milliseconds, corresponding to approximately 15 frames, that we determined and used in the previous version of the Memorable Moments scheme, we can calculate the central frame that corresponds to each annotator’s moment of recognition.
Furthermore, we can allocate a score of 1 to each frame in a video that corresponds to an annotation, and can even extend that to the window of 15 frames that we chose at the begging of the experiments. Therefore if a central frame gets a score of 1 from annotator , the entire window composed of [− 7, ..., , ...+7] gets that score. Finally, summing up all the annotations and given the score for frame from annotator for a video we get the formula = ∑︀=1
In the next step we propose three methods for selecting the frames, called Single, Double and Multi. The Single method consists of basically selecting the highest value of and using a 15-frame window around it as the single representative segment of the video. The Double method consists of selecting the two highest values of , and using them as two central frames that generate two representative segments of the video. The final method, the Multi, uses a threshold value ∈ (0, 1). We select the highest value of , and all values that are higher than × , therefore getting a variable number of representative segments for each video. In cases of equality between values we choose to take the frame with the lowest value. Finally, we test several values for , namely 0.95, 0.90, 0.85, 0.75.
We conduct a set of preliminary experiments in the training phase, consisting of finding the optimal values for the ViViT size and for the parameter. The results of these experiments are presented in Table 1. Each variation of the method has only one variable parameter, the other ones being in default mode for the experiment. Also, when varying the number of heads and the number of repeats the Single method for Memorable Moments is applied. Considering these experiments are done on the development set, 7000 movies are used for training (the training-set) and 1500 for validation (the development-set).
8 16 32
Following this, we use the three values determined in the previous experiments, namely 8 multi-attention heads, 8 repeating blocks and an value of 0.85 and submit three systems for evaluation by the Memorability task organizers. These three systems are represented by the Single, Double and Multi variations of the Memorable Moments selection scheme. The results are presented in Table 2, where the best performing method is shown to be the Multi configuration, with a Spearman value of 0.665. Wihle we can observe a significant growth in performance even when comparing the Single with the Double methods, an even better performance is recorded by the Multi approach, with almost 2% growth over the Double approach.
We theorize at this moment that this type of performance was to be expected, as each additional Memorable Moments configuration progressively adds more videos as representative of the video clips in the collection, therefore creating more training data. We propose that it may be interesting to research this problem on a diferent dataset, that perhaps contains more actions. Our reason for proposing this is that, in the current Memento dataset the video clips have 3 seconds and generally the actions shown in the clips do not change. It is possible that in longer clips the changes in actions or angles may be more significant and having more representatives for each video may improve the results even more.
In this paper we presented our approach for the MediaEval 2022 Predicting Video Memorability, consisting of an updated frame selection method called Memorable Moments, that has the role of selecting one or more representatives from each video clip for processing and training, and a video vision transformer ViViT architecture. Results show that selecting more than one representative for each video improves overall performance.
Financial support provided under project AI4Media, a European Excellence Centre for Media,