This paper provides a description of the MediaEval 2017 “Emotional Impact of Movies task". It continues to build on previous years' editions. In this year's task, participants are expected to create systems that automatically predict the emotional impact that video content will have on viewers, in terms of valence, arousal and fear. Here we provide a description of the use case, task challenges, dataset and ground truth, task run requirements and evaluation metrics.
Afective video content analysis aims at the automatic recognition
of emotions elicited by videos. It has a large number of applications,
including mood based personalized content recommendation [
While major progress has been achieved in computer vision for
visual object detection, scene understanding and high-level concept
recognition, a natural further step is the modeling and
recognition of afective concepts. This has recently received increasing
interest from research communities, e.g., computer vision, machine
learning, with an overall goal of endowing computers with
humanlike perception capabilities. Thus, this task is proposed to ofer
researchers a place to compare their approaches for the prediction
of the emotional impact of movies. It continues to build on previous
years’ editions [
The task requires participants to deploy multimedia features and
models to automatically predict the emotional impact of movies.
This emotional impact is considered here to be the prediction of
the expected emotion. The expected emotion is the emotion that
the majority of the audience feels in response to the same
content. In other words, the expected emotion is the expected value
of experienced (i.e. induced) emotion in a population. While the
induced emotion is subjective and context dependent, the expected
emotion can be considered objective, as it reflects the more-or-less
unanimous response of a general audience to a given stimulus [
This year, two new scenarios are proposed as subtasks. In both
cases, long movies are considered and the emotional impact has to
be predicted for consecutive 10-second segments sliding over the
whole movie with a shift of 5 seconds:
(1) Valence/Arousal prediction: participants’ systems are
supposed to predict a score of expected valence and arousal
for each consecutive 10-second segments. Valence is
deifned as a continuous scale from most negative to most
positive emotions, while arousal is defined continuously
from calmest to most active emotions [
The dataset used in this task is the LIRIS-ACCEDE
dataset1. It contains videos from a set of 160 professionally made
and amateur movies, shared under Creative Commons licenses that
allow redistribution [
The continuous part of LIRIS-ACCEDE [
The test set consists of a selection of 14 movies other than the selection of the 160 original movies. They are between 210 and 6,260 seconds long (mean = 2045.2sec ± 2450.1sec SD). The total length of the 14 selected movies is 7 hours, 57 minutes and 13 seconds.
In addition to the video data, participants are also provided with
general purpose audio and visual content features. To compute
audio features, movies have first been processed to extract
consecutive 10-second segments sliding over the whole movie with a shift
of 5 seconds. Then, audio features have been extracted from these
segments using openSmile toolbox2 [
Beyond audio features, for each movie, image frames were extracted every one second. For each of these images, several general purpose visual features have been provided. They have been computed using LIRE library3, except CNN features (VGG16 fc6 layer) that have been extracted using Matlab Neural Networks toolbox4. The visual features are the following: Auto Color Correlogram, Color and Edge Directivity Descriptor, Color Layout, Edge Histogram, Fuzzy Color and Texture Histogram, Gabor, Joint descriptor joining CEDD and FCTH in one histogram, Scalable Color, Tamura, Local Binary Patterns, VGG16 fc6 layer. 4
Annotations are provided to participants for the 30 movies from the development set. Thus, for each movie, a first file contains valence and arousal values for consecutive 10-second segments sliding over the whole movie with a shift of 5 seconds, and a second file contains the indication whether these segments are supposed to induce fear (value 1) or not (value 0). 4.1
In order to collect continuous valence and arousal annotations,
16 French participants had to continuously indicate their level of
arousal while watching the movies using a modified version of the
GTrace annotation tool [
Then, the continuous valence and arousal annotations from the
participants have been down-sampled by averaging the annotations
over windows of 10 seconds with a shift of 1 second overlap (i.e., 1
value per second) in order to remove the noise due to unintended
movements of the joystick. Finally, these post-processed continuous
annotations have been averaged in order to create a continuous
mean signal of the valence and arousal self-assessments. The details
of this processing are given in [
Fear annotations for the second subtask were generated using a tool specifically designed for the classification of audio-visual media allowing to perform annotation while watching the movie (at the same time). The annotations have been realized by two well experienced team members of NICAM5 both of them trained in classification of media. Each movie has been annotated by 1 annotator reporting the start and stop times of each sequence in the movie exptected to induce fear. From this information, the 10-second segments sliding over the whole movie with a shift of 5 seconds have been labeled as fear (value 1) if they intersect one of the fear sequences and as not fear (value 0) otherwise. 5 6
Participants can submit up to 5 runs for each of the two subtasks, so 10 runs in total. Models can rely on the features provided by the organizers or any other external data.
Standard evaluation metrics are used to assess systems performance. The first subtask can be considered as a regression problem (estimation of expected valence and arousal scores) while the second subtask can be seen as a binary classification problem (the video segment is supposed to induce/not induce fear).
For the first subtask, the oficial metric is the Mean Square Error (MSE), which is the common measure generally used to evaluate regression models. However, to allow a deeper understanding of systems’ performance, we also consider Pearson’s Correlation Coeficient. Indeed, MSE is not always suficient to analyze models eficiency and the correlation may be required to obtain a deeper performance analysis. As an example, if a large portion of the data is neutral (i.e., its valence score is close to 0.5) or is distributed around the neutral score, a uniform model that always outputs 0.5 will result in good MSE performance (low MSE). In this case, the lack of accuracy of the model will be brought to the fore by the correlation between the predicted values and the ground truth that will be also very low.
For the second subtask, the oficial metric is the Mean Average Precision (MAP). Moreover, Accuracy, Precision, Recall and F1-score are also considered to provide insights into systems behaviours. 7
The Emotional Impact of Movies Task provides participants with a comparative and collaborative evaluation framework for emotional detection in movies, in terms of valence, arousal and fear. The LIRIS-ACCEDE dataset has been used as development and test sets. Details on the methods and results of each individual team can be found in the papers of the participating teams in the MediaEval 2017 workshop proceedings.
This task is supported by the CHIST-ERA Visen project ANR-12CHRI-0002-04. 5http://www.kijkwijzer.nl/nicam Emotional Impact of Movies Task