This paper outlines 6 approaches taken to computing video memorability, for the MediaEval Predicting Media Memorability Task. The approaches are based on video features, an end-to-end approach, saliency, aesthetics, neural feedback, and an ensemble of all approaches.
In our work we seek to explore theories from psychology and neuroaesthetics, which may guide predictors for memorability of visual media. Two caveats are that most of the ideas from neuroaesthetics come from perception of visual art or artificial experimental stimuli, rather than real life scenes so these ideas might not translate. The second caveat is that over and above the aesthetics of the video or keyframes, we cannot control for the semantic content or the emotional salience of the imagery for the viewer just as we cannot control for the viewer’s attention or concentration while initially viewing or subsequently trying to remember the video.
Our first principle is the idea that aesthetically pleasing features
are driven by Gestalt principles [
Our second principle, and in opposition to processing a coherent
whole, is that images that show distinctive figure/ground
arrangements may also capture attention thus promoting memorability. So,
another of Ramachandran’s laws of neuroaesthetics is “isolation” in
which a key visual feature has exaggerated importance and stands
out from the surrounding information [
Although these aesthetic features are intrinsic qualities in images that capture attention, it is less clear how they afect memorability. However superior attention based on these qualities should increase encoding of the videos and hence improve memorability. Thus a key prediction based on these principles is that a U-shaped relationship should emerge in which the most globally coherent video images and the most locally distinctive images should both be more memorable compared to the video frames that fall in-between these extremes – i.e. those that are neither particularly globally coherent nor locally distinctive.
This work in this paper was carried out in the context of the
2018 MediaEval Predicting Media Memorability task and we refer
the reader to the task description for prior art [
In this run, we evaluated the performance of a neural network to
run on the precomputed features provided by the task organisers.
These features include C3D features, HMP, HOG Descriptors and
more. The complete list can be found in [
For our end-to-end system we used 3 keyframe images from the raw
videos as inputs. At each epoch, we selected one frame randomly
from the video as a form of data augmentation. For the architecture,
we tried two standard models: VGG16 [
Visual saliency models generate a probability map highlighting
image regions that most attract human attention. Here, this
information is explored for the task of predicting media memorability.
More precisely, a saliency map for each frame of video is computed
with the SalGAN model [
The maps are used to spatially weight the activations of the last convolutional layer of Inception-v3 pre-trained on Imagenet. For that, video frames are resized to 300 × 300 resolution, and forwarded to Inception-v3 to generate convolutional volumes of 7 × 7 × 2048 (the first two dimensions correspond to the spatial resolution, and the last one the number of channels or depth of the layer).
Saliency maps are downsized to 7 × 7, normalised to contain values between 0-1, and element-wise multiplied to the convolutional activations. Global average pooling is applied on the channel dimension to obtain a final representation of 2048 dimensions. The hypothesis here is that the denser the saliency map the more human attention the images draw, and consequently the more memorable they may be.
This 2048 long vector was then fed into a neural network, similar to how precomputed features were used in Section 2.1. 2.4
In this approach we used human reaction to a second viewing of a video keyframe, to train a classifier for memorability, a true humanin-the-loop experiment. The middle frame was extracted for each video clip in the test set and a participant was shown these images at high speed (4 Hz) on a computer screen while simultaneously recording their EEG (Electroencephalography) signals.
Each of the 2000 test set extracted images were presented twice. Following completion of the first viewing, EEG signals were bandpassed between 0.5 Hz and 10 Hz, re-referenced to a common average reference and the mean voltage between 300ms and 600ms following each image presentation calculated for the Pz channel (baselined to -250ms to 0 ms prior to image presentation). The participant then viewed the images a second time with similar EEG data recording and processing and the values averaged for the two presentations of each image, which formed the submission scores.
These parameters were selected as they are known to correspond
both to a time region and electrode location in which a P300
eventrelated potential in this type of task is typically observed where
attention is elicited [
A final technique we incorporated was to use our own version of an
image aesthetics classifier as described in [
In each of the approaches above we made predictions for the entire training set, as well as the entire testing set after training had completed. One limitation to note is that due to the time consuming nature of EEG labelling in Section 2.4, only a subset of the training dataset (2,000 videos) was used in this ensemble run. We used predictions from each of the above approaches, and trained a linear model on this subset of the training data to identify which were the most important predictors. We then used these weights to combine the values on the test set, which generated this run. 3
The performance results of our submissions are shown in Table 1 and illustrated in Figure 1.
The results show that the run based on direct neural/EEG feedback from the human participant was the worst, as expected, and A.F. Smeaton et al. part of the reason might be because training was done with on only 2,000 images, with only one participant. It is definitely worth scaling up this approach to see performance with more data.
The run based on our saliency was a bit better than the neural run, especially for long-term memorability. The ordering of runs by performance among the provided features, ensemble and endto-end submissions has contradictions across runs, across long vs. short term memorability, and across the metric used but the end-to-end seems to have performed best, which is surprising.
Overall, our results seem poor for the above reason or because of insuficient tuning of parameter settings in our experiments.
This work was partially supported by Science Foundation Ireland under the SFI Research Centres Programme grant number SFI/12/RC/2289.