=Paper=
{{Paper
|id=Vol-2670/MediaEval_19_paper_14
|storemode=property
|title=The
Predicting Media Memorability Task at MediaEval 2019
|pdfUrl=https://ceur-ws.org/Vol-2670/MediaEval_19_paper_14.pdf
|volume=Vol-2670
|authors=Mihai Gabriel
Constantin,Bogdan Ionescu,Claire-Hélène
Demarty,Ngoc Q. K. Duong,Xavier
Alameda-Pineda,Mats Sjöberg
|dblpUrl=https://dblp.org/rec/conf/mediaeval/ConstantinIDDAS19
}}
==The
Predicting Media Memorability Task at MediaEval 2019==
https://ceur-ws.org/Vol-2670/MediaEval_19_paper_14.pdf
The Predicting Media Memorability Task at MediaEval 2019
Mihai Gabriel Constantin1 , Bogdan Ionescu1 , Claire-Hélène Demarty2 ,
Ngoc Q. K. Duong2 , Xavier Alameda-Pineda3 , Mats Sjöberg4
1 University Politehnica of Bucharest, Romania
2 InterDigital, France
3 INRIA, France
4 CSC, Finland
ABSTRACT of this task, including the methods used by all the participants and
In this paper, we present the Predicting Media Memorability task, their results, can be found in the proceedings of the 2017 MediaEval
which is running for the second year at the MediaEval 2019 Bench- workshop.1
marking Initiative for Multimedia Evaluation. Participants are re-
quired to create systems that are able to automatically predict the 2 TASK DESCRIPTION
memorability scores of a collection of videos, which should repre- The 2019 Predicting Media Memorability task is a continuation of
sent the “short-term” and “long-term” memorability of the samples. last year’s task [5]. Participants are required to create systems that
We will describe all the aspects of this task, including its main char- can predict the memorability score for video samples. Just like in
acteristics, a description of the development and test data sets, the the previous settings of this task, ground truth data contains scores
ground truth, the evaluation metrics and the required runs. for both “short-term” and “long-term” memorability, created via
memory performance tests. These two different objectives follow
1 INTRODUCTION psychological and human subject studies, such as [16, 17], that
analyze the effect that time has on visual memory. While short-time
The latest developments in multimedia information processing have
annotations measure the immediate retention of samples, long-time
led to the development of systems and methods that can predict
annotations measure retention after a longer period of time, usually
the way humans perceive and react to images and videos, i.e., in-
ranging from hours to days [16, 18] and may be appropriate for
fering interestingness, aesthetics, emotional content, etc. [7]. Such
different types of applications. Therefore two subtasks are proposed
processing tools are gaining importance on media platforms, social
to participants:
networks and recommender systems considering that the amount
of available data is continually growing so does the need to filter • The prediction of short-term memorability - scores were
media content according to a wide variety of factors. Memorability measured a few minutes after the memorization process.
is one of these factors. Furthermore, the analysis of video memora- • The prediction of long-term memorability - scores were
bility is a domain of media processing with a wide array of possible measured 24-72 hours after the memorization process.
applications such as content retrieval, education, summarization,
advertising, content filtering, and recommendation systems. The 3 DATA DESCRIPTION
study of memorability attracted different research communities, in- The proposed dataset consists of 10, 000 7-second videos without
cluding psychologists, behavior specialists, and computer scientists. sound, split into 8, 000 videos for the development set (devset) and
Early human-based studies on visual memory capabilities indicated 2, 000 for the testing set (testset). Participants must train their sys-
a massive storage capacity for visual data [19, 22], also showing tems on the devset and submit runs containing memorability scores
that, even in a long-term study, subjects are able to retain specific for the testset. Ground truth scores and information regarding the
details of images, not just the general gist [3]. Also noteworthy are number of annotators are provided for each video sample in the
studies showing that memorability is an intrinsic property of im- devset, for both subtasks.
ages [13]. Computer vision scientists used these results and created We provided some pre-computed features that could help teams
methods for the prediction of image memorability [2, 9, 14, 15] and, get their systems started and provide easier access to the task to
more recently, video memorability [4, 6, 11, 21]. Recent studies also a broader community of researchers. First, some frame-based fea-
show that style transfer can be used to increase image memora- tures were extracted, for each video, analyzing the first, middle
bility [23, 24]. However, in many of these examples, the authors and last frames. Among these frame-based features are: Histogram
used different datasets or different splits, thus making it hard to of Oriented Gradients (HoG) [8], calculated on 32 × 32 windows
compare methods and draw a clear set of conclusions with regards for grayscale frames, Local Binary Patterns (LBP) [12], calculated
to the accuracy of individual approaches [6]. The Predicting Media for patches of 8 × 15 pixels, Color histogram in HSV space and
Memorability task addresses this problem, and, starting with last ORB features [20]. Also, we extracted the output of the fc7 layer
year’s competition [5], creates a common benchmarking protocol of InceptionV3 [25]. Another set of handcrafted features are the
and provides a dataset for short-term and long-term video memora- Aesthetic Visual Features (AVF) [10], representing color, texture and
bility using common definitions. Details regarding the first edition object-based descriptors, aggregated by the mean and median val-
ues extracted every 10 frames in a video. Second, we also extracted
Copyright 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution
4.0 International (CC BY 4.0). 1 http://ceur-ws.org/Vol-2283/
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France M.G. Constantin et al.
video-level features representing the final category of visual de- protocol and some factors concerning video memorability can be
scriptors. They have the role of motion or temporal descriptors that found in [4].
analyze the video as a whole and naturally represent the movement Overall, the ground truth files are composed of the short-term
in these samples. We provide the Histogram of Motion Patterns and long-term memorability scores described above and the number
(HMP) [1] and the output of the final classification layer of the of annotators for both subtasks, for each movie individually.
convolutional neural network C3D model [26]. Finally, each video
is accompanied by a short caption-like title or description text, 5 RUN DESCRIPTION
that can be used if necessary as tag-like or textual features by the Teams are required to submit a run to each of the two subtasks, i.e.,
participants. short-term memorability required run, and long-term memorability
required run. In total, 10 runs can be submitted, 5 to each subtask.
4 GROUND TRUTH AND ANNOTATION For the two required runs, all information can be used in the
PROTOCOL development of the system, meaning provided features, ground-
truth data, video sample titles, features extracted from the visual
As we previously mentioned, memorability annotations are cre-
content and even external data. However, the only exception, in
ated via performance tests for both the short-term and long-term
this case, is that the required short-term memorability run must
memorability subtasks and partially inspired by the work of [14].
not use long-term memorability score annotations and the required
The participants to these tests were shown a set of target samples
long-term memorability run must not use short-term memorability
(videos that did repeat after a certain time) and distractor samples
score annotations. For the rest of the runs, a maximum of 4 per
(videos that did not repeat, having the role of fillers).
subtask, everything is permitted, including using cross-annotations
In the short-term phase, participants to these tests viewed 40
between the subtasks.
target videos that reappeared in the testing phase and 140 distractor
videos that are played only once, adding up to a sequence of 180
6 EVALUATION
total videos. In the long-term phase, after 24-72 hours, the same
participants viewed 40 videos repeated from the previous distractor Three classic metrics will be extracted from the submitted runs and
collection and another 120 new distractor videos, adding up to a returned to the participating teams: Spearman’s rank correlation,
sequence of 160 videos. The videos that repeat do so in a variable Pearson correlation and Mean squared error; however, we will use
manner. Each repetition appears after a randomly chosen interval the Spearman’s rank correlation as the official metric. This choice
ranging from 45 to 100 videos. Participants were asked to press the comes from the desire to make comparisons between methods,
space key each time they considered a repetition of a video sample allowing for the normalization of the output of different systems by
occurred. Each sample from the dataset received between 13 and 38 taking into account monotonic relationships between ground truth
annotations from the participants and in general more annotations and system output. Though primarily a prediction task, the use of
were made for the short-term subtask, given that it proved difficult Spearman’s rank as the official metric will allow for the evaluation
to collect data after an extended period from the first viewing. In of the systems based on the ranking of different video samples from
order to assess the permanent attention of the annotators, control the testset.
videos were repeated after a random number of videos between
three and six. 7 CONCLUSIONS
We also applied specific correction protocols for the generation In this paper we presented the 2019 Predicting Media Memorability
of the final memorability scores, inspired by the work of [15]. In the task, running for its second yeat at the MediaEval Benchmarking Ini-
case of short-term annotation, in the first step, we calculated the tiative. We created a framework that allows the comparative study
percentage of memory test participants that correctly recognized of different approaches for predicting short-term and long-term
the repetition of each sample, therefore obtaining an initial score in memorability, based on a common video sample dataset, devset-
the interval [0,1]. However, given that these figures do not take into testset split, annotations, and metric. Details regarding the methods
account the interval between the first viewing of the sample and employed by participants and their results can be found in the
its second appearance, a score normalization protocol, similar to proceedings of the 2019 MediaEval workshop.
the one presented in [15], was applied. The correlation between the
repetition interval and memorability scores was previously studied ACKNOWLEDGMENTS
in [14], where, in a paper on image memorability, the authors con- We would like to thank Ricardo Manhães Savii (Federal University
cluded that scores decrease when the interval grows, but that the of São Paulo) for providing the features that accompany the dataset.
ranks of the samples tend to remain unchanged. We confirmed this This work was partially supported by the Romanian Ministry of
observation on our short-term memory tests too; indeed, a linear Innovation and Research (UEFISCDI, project SPIA-VA, agreement
correlation existed between short-term memorability scores and the 2SOL/2017, grant PN-III-P2-2.1-SOL-2016-02-0002).
interval between the repetitions of the video sample, and therefore
we decided to apply a linear correction to the initial scores. How- REFERENCES
ever, the same observation was not valid in the case of long-term
[1] Jurandy Almeida, Neucimar J Leite, and Ricardo da S Torres. 2011.
memorability, where the second annotation was carried out 24 to Comparison of video sequences with histograms of motion patterns.
72 hours after the short-term stage of the experiment; therefore no In 2011 18th IEEE International Conference on Image Processing. IEEE,
correction was applied. More insights about the dataset, annotation 3673–3676.
Predicting Media Memorability MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
[2] Yoann Baveye, Romain Cohendet, Matthieu Perreira Da Silva, and [19] Ronald A Rensink, J Kevin O’Regan, and James J Clark. 1997. To see
Patrick Le Callet. 2016. Deep learning for image memorability predic- or not to see: The need for attention to perceive changes in scenes.
tion: The emotional bias. In Proceedings of the 24th ACM international Psychological science 8, 5 (1997), 368–373.
conference on Multimedia. ACM, 491–495. [20] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R Bradski.
[3] Timothy F Brady, Talia Konkle, George A Alvarez, and Aude Oliva. 2011. ORB: An efficient alternative to SIFT or SURF.. In ICCV, Vol. 11.
2008. Visual long-term memory has a massive storage capacity for Citeseer, 2.
object details. Proceedings of the National Academy of Sciences 105, 38 [21] Sumit Shekhar, Dhruv Singal, Harvineet Singh, Manav Kedia, and
(2008), 14325–14329. Akhil Shetty. 2017. Show and recall: Learning what makes videos
[4] Romain Cohendet, Claire-Hélène Demarty, Ngoc Q. K. Duong, and memorable. In Proceedings of the IEEE International Conference on
Martin Engilberge. 2019. VideoMem: Constructing, Analyzing, Predict- Computer Vision. 2730–2739.
ing Short-term and Long-term Video Memorability. In International [22] Roger N Shepard. 1967. Recognition memory for words, sentences,
Conference on Computer Vision (ICCV). and pictures. Journal of verbal Learning and verbal Behavior 6, 1 (1967),
[5] Romain Cohendet, Claire-Hélène Demarty, Ngoc Q. K. Duong, Mats 156–163.
Sjöberg, Bogdan Ionescu, and Thanh-Toan Do. 2018. Mediaeval 2018: [23] Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda-
Predicting media memorability task. In Proceedings of the MediaEval Pineda, Elisa Ricci, and Nicu Sebe. 2017. How to Make an Image More
2017 Workshop. Memorable? A Deep Style Transfer Approach. In ACM International
[6] Romain Cohendet, Karthik Yadati, Ngoc Q. K. Duong, and Claire- Conference on Multimedia Retrieval.
Hélène Demarty. 2018. Annotating, understanding, and predicting [24] Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda-
long-term video memorability. In Proceedings of the 2018 ACM on Pineda, Elisa Ricci, and Nicu Sebe. 2019. Increasing Image Memora-
International Conference on Multimedia Retrieval. ACM, 178–186. bility with Neural Style Transfer. ACM Transactions on Multimedia
[7] Mihai Gabriel Constantin, Miriam Redi, Gloria Zen, and Bogdan Computing Communications and Applications (2019).
Ionescu. 2019. Computational understanding of visual interestingness [25] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and
beyond semantics: literature survey and analysis of covariates. ACM Zbigniew Wojna. 2016. Rethinking the inception architecture for
Computing Surveys (CSUR) 52, 2 (2019), 25. computer vision. In Proceedings of the IEEE conference on computer
[8] Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients vision and pattern recognition. 2818–2826.
for human detection. In n Computer Vision and Pattern Recognition, [26] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and
2005. CVPR 2005, Vol. 1. 886–893. Manohar Paluri. 2015. Learning spatiotemporal features with 3d con-
[9] Jiri Fajtl, Vasileios Argyriou, Dorothy Monekosso, and Paolo Re- volutional networks. In Proceedings of the IEEE international conference
magnino. 2018. Amnet: Memorability estimation with attention. In on computer vision. 4489–4497.
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition. 6363–6372.
[10] Andreas F Haas, Marine Guibert, Anja Foerschner, Sandi Calhoun,
Emma George, Mark Hatay, Elizabeth Dinsdale, Stuart A Sandin, Jen-
nifer E Smith, Mark JA Vermeij, and others. 2015. Can we measure
beauty? Computational evaluation of coral reef aesthetics. PeerJ 3
(2015), e1390.
[11] Junwei Han, Changyuan Chen, Ling Shao, Xintao Hu, Jungong Han,
and Tianming Liu. 2014. Learning computational models of video
memorability from fMRI brain imaging. IEEE transactions on cybernet-
ics 45, 8 (2014), 1692–1703.
[12] Dong-Chen He and Li Wang. 1990. Texture unit, texture spectrum, and
texture analysis. IEEE transactions on Geoscience and Remote Sensing
28, 4 (1990), 509–512.
[13] Phillip Isola, Devi Parikh, Antonio Torralba, and Aude Oliva. 2011.
Understanding the intrinsic memorability of images. In Advances in
Neural Information Processing Systems. 2429–2437.
[14] P Isola, Jianxiong Xiao, A Torralba, and A Oliva. 2011. What makes
an image memorable?. In Proceedings of the 2011 IEEE Conference on
Computer Vision and Pattern Recognition. IEEE Computer Society, 145–
152.
[15] Aditya Khosla, Akhil S Raju, Antonio Torralba, and Aude Oliva. 2015.
Understanding and predicting image memorability at a large scale. In
Proceedings of the IEEE International Conference on Computer Vision.
2390–2398.
[16] James L McGaugh. 2000. Memory–a century of consolidation. Science
287, 5451 (2000), 248–251.
[17] Jaap MJ Murre and Joeri Dros. 2015. Replication and analysis of
EbbinghausâĂŹ forgetting curve. PloS one 10, 7 (2015), e0120644.
[18] James S Nairne and Addie Dutta. 1992. Spatial and temporal uncer-
tainty in long-term memory. Journal of Memory and Language 31, 3
(1992), 396–407.