The Predicting Media Memorability Task at MediaEval 2019 Mihai Gabriel Constantin1 , Bogdan Ionescu1 , Claire-Hélène Demarty2 , Ngoc Q. K. Duong2 , Xavier Alameda-Pineda3 , Mats Sjöberg4 1 University Politehnica of Bucharest, Romania 2 InterDigital, France 3 INRIA, France 4 CSC, Finland ABSTRACT of this task, including the methods used by all the participants and In this paper, we present the Predicting Media Memorability task, their results, can be found in the proceedings of the 2017 MediaEval which is running for the second year at the MediaEval 2019 Bench- workshop.1 marking Initiative for Multimedia Evaluation. Participants are re- quired to create systems that are able to automatically predict the 2 TASK DESCRIPTION memorability scores of a collection of videos, which should repre- The 2019 Predicting Media Memorability task is a continuation of sent the “short-term” and “long-term” memorability of the samples. last year’s task [5]. Participants are required to create systems that We will describe all the aspects of this task, including its main char- can predict the memorability score for video samples. Just like in acteristics, a description of the development and test data sets, the the previous settings of this task, ground truth data contains scores ground truth, the evaluation metrics and the required runs. for both “short-term” and “long-term” memorability, created via memory performance tests. These two different objectives follow 1 INTRODUCTION psychological and human subject studies, such as [16, 17], that analyze the effect that time has on visual memory. While short-time The latest developments in multimedia information processing have annotations measure the immediate retention of samples, long-time led to the development of systems and methods that can predict annotations measure retention after a longer period of time, usually the way humans perceive and react to images and videos, i.e., in- ranging from hours to days [16, 18] and may be appropriate for fering interestingness, aesthetics, emotional content, etc. [7]. Such different types of applications. Therefore two subtasks are proposed processing tools are gaining importance on media platforms, social to participants: networks and recommender systems considering that the amount of available data is continually growing so does the need to filter • The prediction of short-term memorability - scores were media content according to a wide variety of factors. Memorability measured a few minutes after the memorization process. is one of these factors. Furthermore, the analysis of video memora- • The prediction of long-term memorability - scores were bility is a domain of media processing with a wide array of possible measured 24-72 hours after the memorization process. applications such as content retrieval, education, summarization, advertising, content filtering, and recommendation systems. The 3 DATA DESCRIPTION study of memorability attracted different research communities, in- The proposed dataset consists of 10, 000 7-second videos without cluding psychologists, behavior specialists, and computer scientists. sound, split into 8, 000 videos for the development set (devset) and Early human-based studies on visual memory capabilities indicated 2, 000 for the testing set (testset). Participants must train their sys- a massive storage capacity for visual data [19, 22], also showing tems on the devset and submit runs containing memorability scores that, even in a long-term study, subjects are able to retain specific for the testset. Ground truth scores and information regarding the details of images, not just the general gist [3]. Also noteworthy are number of annotators are provided for each video sample in the studies showing that memorability is an intrinsic property of im- devset, for both subtasks. ages [13]. Computer vision scientists used these results and created We provided some pre-computed features that could help teams methods for the prediction of image memorability [2, 9, 14, 15] and, get their systems started and provide easier access to the task to more recently, video memorability [4, 6, 11, 21]. Recent studies also a broader community of researchers. First, some frame-based fea- show that style transfer can be used to increase image memora- tures were extracted, for each video, analyzing the first, middle bility [23, 24]. However, in many of these examples, the authors and last frames. Among these frame-based features are: Histogram used different datasets or different splits, thus making it hard to of Oriented Gradients (HoG) [8], calculated on 32 × 32 windows compare methods and draw a clear set of conclusions with regards for grayscale frames, Local Binary Patterns (LBP) [12], calculated to the accuracy of individual approaches [6]. The Predicting Media for patches of 8 × 15 pixels, Color histogram in HSV space and Memorability task addresses this problem, and, starting with last ORB features [20]. Also, we extracted the output of the fc7 layer year’s competition [5], creates a common benchmarking protocol of InceptionV3 [25]. Another set of handcrafted features are the and provides a dataset for short-term and long-term video memora- Aesthetic Visual Features (AVF) [10], representing color, texture and bility using common definitions. Details regarding the first edition object-based descriptors, aggregated by the mean and median val- ues extracted every 10 frames in a video. Second, we also extracted Copyright 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 http://ceur-ws.org/Vol-2283/ MediaEval’19, 27-29 October 2019, Sophia Antipolis, France MediaEval’19, 27-29 October 2019, Sophia Antipolis, France M.G. Constantin et al. video-level features representing the final category of visual de- protocol and some factors concerning video memorability can be scriptors. They have the role of motion or temporal descriptors that found in [4]. analyze the video as a whole and naturally represent the movement Overall, the ground truth files are composed of the short-term in these samples. We provide the Histogram of Motion Patterns and long-term memorability scores described above and the number (HMP) [1] and the output of the final classification layer of the of annotators for both subtasks, for each movie individually. convolutional neural network C3D model [26]. Finally, each video is accompanied by a short caption-like title or description text, 5 RUN DESCRIPTION that can be used if necessary as tag-like or textual features by the Teams are required to submit a run to each of the two subtasks, i.e., participants. short-term memorability required run, and long-term memorability required run. In total, 10 runs can be submitted, 5 to each subtask. 4 GROUND TRUTH AND ANNOTATION For the two required runs, all information can be used in the PROTOCOL development of the system, meaning provided features, ground- truth data, video sample titles, features extracted from the visual As we previously mentioned, memorability annotations are cre- content and even external data. However, the only exception, in ated via performance tests for both the short-term and long-term this case, is that the required short-term memorability run must memorability subtasks and partially inspired by the work of [14]. not use long-term memorability score annotations and the required The participants to these tests were shown a set of target samples long-term memorability run must not use short-term memorability (videos that did repeat after a certain time) and distractor samples score annotations. For the rest of the runs, a maximum of 4 per (videos that did not repeat, having the role of fillers). subtask, everything is permitted, including using cross-annotations In the short-term phase, participants to these tests viewed 40 between the subtasks. target videos that reappeared in the testing phase and 140 distractor videos that are played only once, adding up to a sequence of 180 6 EVALUATION total videos. In the long-term phase, after 24-72 hours, the same participants viewed 40 videos repeated from the previous distractor Three classic metrics will be extracted from the submitted runs and collection and another 120 new distractor videos, adding up to a returned to the participating teams: Spearman’s rank correlation, sequence of 160 videos. The videos that repeat do so in a variable Pearson correlation and Mean squared error; however, we will use manner. Each repetition appears after a randomly chosen interval the Spearman’s rank correlation as the official metric. This choice ranging from 45 to 100 videos. Participants were asked to press the comes from the desire to make comparisons between methods, space key each time they considered a repetition of a video sample allowing for the normalization of the output of different systems by occurred. Each sample from the dataset received between 13 and 38 taking into account monotonic relationships between ground truth annotations from the participants and in general more annotations and system output. Though primarily a prediction task, the use of were made for the short-term subtask, given that it proved difficult Spearman’s rank as the official metric will allow for the evaluation to collect data after an extended period from the first viewing. In of the systems based on the ranking of different video samples from order to assess the permanent attention of the annotators, control the testset. videos were repeated after a random number of videos between three and six. 7 CONCLUSIONS We also applied specific correction protocols for the generation In this paper we presented the 2019 Predicting Media Memorability of the final memorability scores, inspired by the work of [15]. In the task, running for its second yeat at the MediaEval Benchmarking Ini- case of short-term annotation, in the first step, we calculated the tiative. We created a framework that allows the comparative study percentage of memory test participants that correctly recognized of different approaches for predicting short-term and long-term the repetition of each sample, therefore obtaining an initial score in memorability, based on a common video sample dataset, devset- the interval [0,1]. However, given that these figures do not take into testset split, annotations, and metric. Details regarding the methods account the interval between the first viewing of the sample and employed by participants and their results can be found in the its second appearance, a score normalization protocol, similar to proceedings of the 2019 MediaEval workshop. the one presented in [15], was applied. The correlation between the repetition interval and memorability scores was previously studied ACKNOWLEDGMENTS in [14], where, in a paper on image memorability, the authors con- We would like to thank Ricardo Manhães Savii (Federal University cluded that scores decrease when the interval grows, but that the of São Paulo) for providing the features that accompany the dataset. ranks of the samples tend to remain unchanged. We confirmed this This work was partially supported by the Romanian Ministry of observation on our short-term memory tests too; indeed, a linear Innovation and Research (UEFISCDI, project SPIA-VA, agreement correlation existed between short-term memorability scores and the 2SOL/2017, grant PN-III-P2-2.1-SOL-2016-02-0002). interval between the repetitions of the video sample, and therefore we decided to apply a linear correction to the initial scores. How- REFERENCES ever, the same observation was not valid in the case of long-term [1] Jurandy Almeida, Neucimar J Leite, and Ricardo da S Torres. 2011. memorability, where the second annotation was carried out 24 to Comparison of video sequences with histograms of motion patterns. 72 hours after the short-term stage of the experiment; therefore no In 2011 18th IEEE International Conference on Image Processing. IEEE, correction was applied. More insights about the dataset, annotation 3673–3676. Predicting Media Memorability MediaEval’19, 27-29 October 2019, Sophia Antipolis, France [2] Yoann Baveye, Romain Cohendet, Matthieu Perreira Da Silva, and [19] Ronald A Rensink, J Kevin O’Regan, and James J Clark. 1997. To see Patrick Le Callet. 2016. Deep learning for image memorability predic- or not to see: The need for attention to perceive changes in scenes. tion: The emotional bias. In Proceedings of the 24th ACM international Psychological science 8, 5 (1997), 368–373. conference on Multimedia. ACM, 491–495. [20] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R Bradski. [3] Timothy F Brady, Talia Konkle, George A Alvarez, and Aude Oliva. 2011. ORB: An efficient alternative to SIFT or SURF.. In ICCV, Vol. 11. 2008. Visual long-term memory has a massive storage capacity for Citeseer, 2. object details. Proceedings of the National Academy of Sciences 105, 38 [21] Sumit Shekhar, Dhruv Singal, Harvineet Singh, Manav Kedia, and (2008), 14325–14329. Akhil Shetty. 2017. Show and recall: Learning what makes videos [4] Romain Cohendet, Claire-Hélène Demarty, Ngoc Q. K. Duong, and memorable. In Proceedings of the IEEE International Conference on Martin Engilberge. 2019. VideoMem: Constructing, Analyzing, Predict- Computer Vision. 2730–2739. ing Short-term and Long-term Video Memorability. In International [22] Roger N Shepard. 1967. Recognition memory for words, sentences, Conference on Computer Vision (ICCV). and pictures. Journal of verbal Learning and verbal Behavior 6, 1 (1967), [5] Romain Cohendet, Claire-Hélène Demarty, Ngoc Q. K. Duong, Mats 156–163. Sjöberg, Bogdan Ionescu, and Thanh-Toan Do. 2018. Mediaeval 2018: [23] Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda- Predicting media memorability task. In Proceedings of the MediaEval Pineda, Elisa Ricci, and Nicu Sebe. 2017. How to Make an Image More 2017 Workshop. Memorable? A Deep Style Transfer Approach. In ACM International [6] Romain Cohendet, Karthik Yadati, Ngoc Q. K. Duong, and Claire- Conference on Multimedia Retrieval. Hélène Demarty. 2018. Annotating, understanding, and predicting [24] Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda- long-term video memorability. In Proceedings of the 2018 ACM on Pineda, Elisa Ricci, and Nicu Sebe. 2019. Increasing Image Memora- International Conference on Multimedia Retrieval. ACM, 178–186. bility with Neural Style Transfer. ACM Transactions on Multimedia [7] Mihai Gabriel Constantin, Miriam Redi, Gloria Zen, and Bogdan Computing Communications and Applications (2019). Ionescu. 2019. Computational understanding of visual interestingness [25] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and beyond semantics: literature survey and analysis of covariates. ACM Zbigniew Wojna. 2016. Rethinking the inception architecture for Computing Surveys (CSUR) 52, 2 (2019), 25. computer vision. In Proceedings of the IEEE conference on computer [8] Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients vision and pattern recognition. 2818–2826. for human detection. In n Computer Vision and Pattern Recognition, [26] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and 2005. CVPR 2005, Vol. 1. 886–893. Manohar Paluri. 2015. Learning spatiotemporal features with 3d con- [9] Jiri Fajtl, Vasileios Argyriou, Dorothy Monekosso, and Paolo Re- volutional networks. In Proceedings of the IEEE international conference magnino. 2018. Amnet: Memorability estimation with attention. In on computer vision. 4489–4497. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6363–6372. [10] Andreas F Haas, Marine Guibert, Anja Foerschner, Sandi Calhoun, Emma George, Mark Hatay, Elizabeth Dinsdale, Stuart A Sandin, Jen- nifer E Smith, Mark JA Vermeij, and others. 2015. Can we measure beauty? Computational evaluation of coral reef aesthetics. PeerJ 3 (2015), e1390. [11] Junwei Han, Changyuan Chen, Ling Shao, Xintao Hu, Jungong Han, and Tianming Liu. 2014. Learning computational models of video memorability from fMRI brain imaging. IEEE transactions on cybernet- ics 45, 8 (2014), 1692–1703. [12] Dong-Chen He and Li Wang. 1990. Texture unit, texture spectrum, and texture analysis. IEEE transactions on Geoscience and Remote Sensing 28, 4 (1990), 509–512. [13] Phillip Isola, Devi Parikh, Antonio Torralba, and Aude Oliva. 2011. Understanding the intrinsic memorability of images. In Advances in Neural Information Processing Systems. 2429–2437. [14] P Isola, Jianxiong Xiao, A Torralba, and A Oliva. 2011. What makes an image memorable?. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, 145– 152. [15] Aditya Khosla, Akhil S Raju, Antonio Torralba, and Aude Oliva. 2015. Understanding and predicting image memorability at a large scale. In Proceedings of the IEEE International Conference on Computer Vision. 2390–2398. [16] James L McGaugh. 2000. Memory–a century of consolidation. Science 287, 5451 (2000), 248–251. [17] Jaap MJ Murre and Joeri Dros. 2015. Replication and analysis of EbbinghausâĂŹ forgetting curve. PloS one 10, 7 (2015), e0120644. [18] James S Nairne and Addie Dutta. 1992. Spatial and temporal uncer- tainty in long-term memory. Journal of Memory and Language 31, 3 (1992), 396–407.