This paper reports on the GTH-UPM team experience in the Predicting Media Memorability task at MediaEval 2020. Teams were requested to predict memorability scores at both short-term and long-term, understanding such score as a measure of whether a video was perdurable in a viewer's memory or not. Our proposed system relies on a late fusion of the scores predicted by three sequential models, each trained over a different modality: video captions, aural embeddings and visual optical flow-based vectors. Whereas single-modality models show a low or zero Spearman correlation coefficient value, their combination considerably boosts performance over development data up to 0.2 in the short-term memorability prediction subtask and 0.19 in the long-term subtask. However, performance over test data drops to 0.016 and -0.041, respectively.
Every video sample in the dataset presents the following sources of information: between 2 and 5 text captions that roughly describe the content of the video, the video audio signal and its visual frames. As stated before, multimodal systems are able to learn modality-wise
The improvement in computational capabilities is progressively allowing researchers to tackle problems long though to be out of reach due to the subjective nature of the phenomena involved. One good instance is memorability prediction. The seminal work of Isola et al. set the ground for later work on computational modelling of image memorability [11]. Since 2018 the Predicting Media Memorability Challenge, hosted within the MediaEval workshop, has pushed forward the extent of the original problem to encompass memorability prediction over multimedia sources of information [3,4]. In its current edition the goal of the task holds the same as previous years, yet video clips now cover a kind of material resembling short videos commonly found in social media. Further information can be found in the challenge description paper [7].
Several multimodal late fusion strategies have been proposed regarding the image and video memorability prediction problem [5]. Additionally, attention mechanisms have been successfully applied to problems in which data come naturally in a sequential form [16]. In particular, self-attention layers have been proved to boost performance when tackling the computational modelling of media memorability [6]. data representations, and combine their predictive power in order to make a final, unique memorability prediction. We hypothesize that a late fusion scheme will benefit from incorporating a selfattention mechanism that learns to focus on what it is particularly relevant on a given sample's prediction.
We propose a system based on the late fusion by a Support Vector Regressor (SVR) of the predictions made by three singlemodality models whose architecture is depicted in Figure 2. In all cases the biLSTM encoders have 75 units, with all the learners sharing the same architecture but trained independently. Prediction comes as the outcome of the last sigmoid layer. Learned layers suffer from a dropout rate fixed at 0.3. For every single-modality learner the training pipeline holds the same; batch size is set at 128, with initial learning rate 0.001 and Adam optimizer [12]. Figure 1 shows the general prediction pipeline from these models. Results shown in this paper are obtained following a 5-fold cross-validation procedure over the 1000 videos of the development data. Training is stopped after 5 epochs with no improvement over the Spearman correlation coefficient, computed over the fold's validation data. Experimental results are summarized in Table 1. Next we introduce in greater detail the feature extraction processing carried out for every modality.
We merge all the captions of a sample into a single one in a Bag-Of-Words fashion. Afterwards, we extract the lemma of every word in the text using NLTK's WordNet-based Lemmatizer [1,14]. Finally, the input of the text modality is made by the sequence of fasttext 300-dimensional word embeddings corresponding to every word in the sample's BOW-text [2]. At training time, random noise with 𝜇 = 0 and 𝜎 = 0.15 is added to the niput embeddings in order to improve learning robustness.
Based on previous experience, we hypothesize that event detectionoriented embeddings provide a robust basis to study multimedia perceptual variables such as attention or memorability [13]. Therefore we compute aural embeddings using the default VGGish configuration, which is pretrained on Audioset, a large audio event-detection database [8,9]. That way every video audio signal is defined by a sequence of 128-dimensional embeddings, each spanning 960 ms of audio and without overlap between them.
Videos in the dataset are no longer than a few seconds, characterized by an event happening quickly and conforming the most relevant Figure 1: Proposed video memorability prediction pipeline. The system is the same when dealing with both short-and longterm memorability scores, but single-modality learners are trained independently for every time interval and modality.
Test Set part of the clip. Because of that, videos are expected to display quick changes in pixel values between consecutive frames due to visual events taking place. In order to capture the degree of visual change along a clip, we compute optical feature maps for its frames, extracted at 3 FPS, using a LiteFlowNet model [10]. We further reduce optical flow features' dimensionality by projecting them into a 128-dimensional subspace computed by PCA [15]. A sample is represented by a temporally-sorted sequence of 128-dimensional features that retains most of the information regarding the optical flow features maps.
We independently train single-modality models from the features explained in the sections above. Thereafter, a memorability prediction is computed for every sample in the dataset. The combination of the three memorability scores is the input for a SVR that makes a final prediction that reflects the knowledge extracted from the different the modalities. As it can be seen from Table 1, individual learners are not able to fully characterize a video sample and learn the relationship with its memorability score. However, the ensemble of the three of them achieves a Spearman correlation coefficient value of 0.2 in the shortterm problem and 0.19 in the long-term one over development data. However, we notice that the performance on the test data significantly drops, achieving much lower scores on both subtasks.
Despite individual learners showing very low or even zero coefficient values, a SVR based on their posteriors seems to weakly capture the relationship between media content and its memorability score, with similar correlation values obtained at both shortterm and long-term subtasks. This might be partially caused by the limited amount of data available, which is likely to be dragging the learning process, and therefore making the SVR to learn the development dataset's score distribution. Prediction's distribution suggests that the system might be learning to approximate every sample to the mean memorability score, rather than exploiting the knowledge extracted from the computed features. Future work includes extending the amount of training data with similar datasets. It is also left for future studies to explore different data encodings, with special emphasis on smaller, more compact data representations that might better suited for cases where large datasets are not available.
The work leading to these results has been supported by the Spanish Ministry of Economy, Industry and Competitiveness through CAVIAR (MINECO, TEC2017-84593-C2-1-R) and AMIC (MINECO, TIN2017-85854-C4-4-R) projects (AEI/FEDER, UE). Ricardo Kleinlein's research was supported by the Spanish Ministry of Education (FPI grant PRE2018-083225).