In this paper, we address the video memorability prediction problem, which was a part of MediaEval 2022. This problem involves learning a model that maps the video content to a global memorability score. However, this value is still related to uncertain factors, including time and human subjectivity. Hence, we focus in this work mainly on modeling the subjective nature of the memorability experiments. Our approach relied on predicting the video memorability based on the maximum likelihood estimation of the Bernoulli distribution of the success variable of each video. To evaluate our approach, preliminary experiments were performed on the Memento10k dataset. First promising results were achieved by using a simple fusion of the visual and textual pre-trained features as input.
Understanding content memorability is a task that has gained psychologists’ interest for over a
decade [
Related works in video memorability prediction have mainly focused on extracting the
visual and temporal information from videos and the semantic information from textual data.
From the previous challenge, authors in [
In order to design a suitable model for memorability prediction, it is necessary to understand
what factors influence a video memorability score. In the Memento10K dataset [
Deep neural networks have shown remarkable success in many classification problems where the goal of the model is to find a function that directly maps the input space to an output value. However, in problems where subjectivity is present, the characterization of uncertainties is desired. In the memorability estimation experiment, several factors are involved, including the positions of the video the first and second time in the stream and the time spent since the first time looking at the video. For each video, these settings vary and lead to a diferent outcome, where this latter can be either a correct response or not.
Statistically, a single experiment for two possible outcomes is called a Bernoulli trial, where the probability of success is and failure is 1 − . Therefore, each video memorability may be regarded as binomial distribution with trials, where in each Bernoulli trial, the outcome variable may be a correct response or not.
We formulate the problem as the following: Let’s = {1, 2, · · · , } be a set of videos and = {1, 2, · · · , } the set of the corresponding memorability scores. For each video, a number of independent settings were performed to compute the global score, where each is characterized by a success or failure outcome. Hence, the memorability of a video can be modeled as a set of Bernoulli trials with the same parameter , where trials are independent of one another, ∼ (). The main quantity of interest is the proportion of subjects who respond favorably (the proportion of successful trials, which is the total number of successful trials divided by the total number of subjects).
Every video is represented by a single embedding . This embedding corresponds to
a simple concatenation of visual and textual features. The visual features were computed as
the mean of extracted pre-trained features from Resnet50 and Eficient-Net models on the
frame level (features provided with the dataset), while the textual features were obtained using
pre-trained features from the Bert transformer model [
The proposed approach consists, therefore, of designing a neural network that estimates , the probability of ’success’ (i.e., the proportion of observed successes) given the input features X. In order to estimate the parameter that is most likely to have given rise to the observation , we use as a loss function the log-likelihood of based on the observation with outputs with = 1 · · · . Hence, the goal of the model is to find * that minimizes ( | ) such that: ( | ) = − ∑︁ ( log + (1 − ) log (1 − )) =1 (1)
Preliminary analysis of the average memorability scores The histogram in figure 1b shows the distribution of the memorability scores on the training set. Indeed, the memorability score over the training examples follows a skewed and asymmetric distribution with a skew of − 0.529. The nature of this distribution makes learning the low values a dificult task. Therefore, a transformation should be applied to the memorability variable. We have applied the square transform to the output variable, reducing the distribution’s skewness (Figure 1b). Furthermore, we have used the inverse transform to the predictions not to impact the regression analysis results.
Preliminary analysis of the response time Figure 1a shows the distributions of the response time () for the examples where the correct variable is equal to 1 (success) and equal to 0 (failures). We notice that follows diferent distributions for the two cases. Therefore, this shows that the feature is highly correlated with the memorability outcome. Based on this observation, we investigated adding a regularization term to the loss function based on maximizing the correlation between and the predicted memorability as a simple approach. 0.00175 0.00150 0.00125 y it0.00100 s n e D0.00075 0.00050 0.00025 success cases 50
failure cases 40 ty30 i s n e D20 10 400 300 t n u oC200 100 400 350 300 t250 oun200 C 150 100 50 Obtained results on the validation set Table 1 shows obtained Mean Squared Error (MSE), Mean Absolute Error(MAE), Spearman and Pearson’s rank correlation coeficient values, and the coeficient of determination 2. We have compared the approach with using a neural network where the loss function corresponds to minimizing the MSE of the average memorability scores. These findings show that the proposed approach results in a better coeficient of determination (from 0.38 to 0.42). The Spearman and Person correlation coeficients have also been improved compared to using an MSE as a loss function. Obtained results on the test set In this section, we present the obtained results on the test set. We evaluated four independent runs to gain more insights about the approach. We started by evaluating the model using visual and textual features in the first run. We included the temporal dependencies between the frames in the second run using the LSTM model and visual information only. In the third run, we have added the proposed regularization term based on the response time (reg1). Finally, we added another simple regularizer to maximize the 2 between predicted and true values (reg2). The obtained results are shown in table 2.
Based on the obtained values, the best run corresponds to the model predictions using the proposed loss only (with a spearman correlation of 0.597). Although the performance has decreased compared to the values obtained on the validation set, but it still provides promising results. This can be improved by more investigation on the feature level as well as by considering more frames rather than using three frames. Furthermore, we observe that the added regularization terms could not result in a better performance. Therefore, we consider including the response time factor in a more suitable way in our future experiments.
In this paper, we investigated the subjectivity of the memorability experiments by considering each memorability score as a Bernoulli distribution with the parameter . We proposed a neural network that estimates p based on using the maximum likelihood as a loss function. The obtained performance on the validation set shows promising results. We consider this approach to be early work on the subjectivity modeling of video memorability. An interesting next step would be to examine the memorability distribution estimation by considering lags and video decays. Concerning the neural network, it might be beneficial to use more sophisticated approaches such as video transformers to capture the spatiotemporal relationships in the data.
Support for this research was provided by a grant from La Région Nouvelle Aquitaine (CPERFEDER P-2019-2022), in partnership with the EuropeanUnion (FEDER/ ERDF, European Regional Development Fund)