This paper provides a description of the MediaEval 2018 “Emotional Impact of Movies task". It continues to build on last year's edition, integrating the feedback of previous participants. The goal is to create systems that automatically predict the emotional impact that video content will have on viewers, in terms of valence, arousal and fear. Here we provide a description of the use case, task challenges, dataset and ground truth, task run requirements and evaluation metrics.
Afective video content analysis aims at the automatic recognition
of emotions elicited by videos. It has a large number of applications,
including mood based personalized content recommendation [
While major progress has been achieved in computer vision for
visual object detection, scene understanding and high-level concept
recognition, a natural further step is the modeling and
recognition of afective concepts. This has recently received increasing
interest from research communities, e.g., computer vision, machine
learning, with an overall goal of endowing computers with
humanlike perception capabilities. Thus, this task is proposed to ofer
researchers a place to compare their approaches for the prediction
of the emotional impact of movies. It continues to build on last
year’s edition [
The task requires participants to deploy multimedia features and
models to automatically predict the emotional impact of movies.
This emotional impact is considered here to be the prediction of
the expected emotion. The expected emotion is the emotion that
the majority of the audience feels in response to the same
content. In other words, the expected emotion is the expected value
of experienced (i.e. induced) emotion in a population. While the
induced emotion is subjective and context dependent, the expected
emotion can be considered objective, as it reflects the more-or-less
unanimous response of a general audience to a given stimulus [
This year, two scenarios are proposed as subtasks. In both cases, long movies are considered.
(1) Valence and Arousal prediction: participants’ systems have
to predict a score of expected valence and arousal
continuously (every second) along movies. Valence is defined on a
continuous scale from most negative to most positive
emotions, while arousal is defined continuously from calmest
to most active emotions [
The dataset used in this task is the LIRIS-ACCEDE dataset1. It
contains videos from a set of 160 professionally made and
amateur movies, shared under Creative Commons licenses that allow
redistribution [
A total of 44 movies (total duration of 15 hours and 20 minutes) selected from the set of 160 movies are provided as development set for both subtasks with the annotations according to fear, valence and arousal. A complementary set of 10 movies (11 hours and 29 minutes) is available for the first subtask with the valence and arousal annotations.
The test set consists of 12 other movies selected from the set of 160 movies, for a total duration of 8 hours and 56 minutes.
In addition to the video data, participants are also provided with
general purpose audio and visual content features. To compute
audio features, movies have first been processed to extract
consecutive 5-second segments sliding over the whole movie with a shift
of 1 second. Then, audio features have been extracted from these
segments using openSmile toolbox2 [
Beyond audio features, for each movie, image frames were extracted every one second. For each of these images, several general purpose visual features have been provided. They have been computed using LIRE library3, except CNN features (VGG16 fc6 layer) that have been extracted using Matlab Neural Networks toolbox4. The visual features are the following: Auto Color Correlogram, Color and Edge Directivity Descriptor, Color Layout, Edge Histogram, Fuzzy Color and Texture Histogram, Gabor, Joint descriptor joining CEDD and FCTH in one histogram, Scalable Color, Tamura, Local Binary Patterns, VGG16 fc6 layer. 4
As mentioned in the previous section, the development set contains a part that is common to both subtasks with valence, arousal and fear annotations (44 movies), and an additional part only concerning the first subtask, with valence and arousal annotations (10 movies).
For each movie from the development set for the first subtask, a file is provided containing valence and arousal values for each second of the movie.
Moreover, for all movies from the development set for the second subtask, a file is provided containing the beginning and ending times of each sequence in the movie inducing fear. 4.1
In order to collect continuous valence and arousal annotations, a
total of 28 French participants had to continuously indicate their
level of valence and arousal while watching the movies using a
modified version of the GTrace annotation tool [
Then, the continuous valence and arousal annotations from the
participants have been down-sampled by averaging the annotations
over windows of 10 seconds with a shift of 1 second overlap (i.e., 1
value per second) in order to remove the noise due to unintended
movements of the joystick. Finally, these post-processed continuous
annotations have been averaged in order to create a continuous
mean signal of the valence and arousal self-assessments, ranging
from -1 (most negative for valence, most passive for arousal) to +1
(most positive for valence, most active for arousal). The details of
this processing are given in [
Fear annotations for the second subtask were generated using a tool specifically designed for the classification of audio-visual media allowing to perform annotation while watching the movie (at the same time). The annotations have been realized by two well experienced team members of NICAM5 both of them trained in classification of media. Each movie has been annotated by 1 annotator reporting the start and stop times of each sequence in the movie expected to induce fear. 5 6
Participants can submit up to 5 runs for each of the two subtasks, so 10 runs in total. Models can rely on the features provided by the organizers or any other external data.
Standard evaluation metrics are used to assess systems performance. The first subtask can be considered as a regression problem (estimation of expected valence and arousal scores) while the second subtask can be seen as a binary classification problem (the video segment is supposed to induce/not induce fear).
For the first subtask, the oficial metric is the Mean Square Error (MSE), which is the common measure generally used to evaluate regression models. However, to allow a deeper understanding of systems’ performance, we also consider Pearson’s Correlation Coeficient. Indeed, MSE is not always suficient to analyze models eficiency and the correlation may be required to obtain a deeper performance analysis. As an example, if a large portion of the data is neutral (i.e., its valence score is close to 0.5) or is distributed around the neutral score, a uniform model that always outputs 0.5 will result in good MSE performance (low MSE). In this case, the lack of accuracy of the model will be brought to the fore by the correlation between the predicted values and the ground truth that will be also very low.
For the second subtask, as the goal is to detect time sequences inducing fear, the oficial metric is the Intersection over Union of time intervals. 7
The Emotional Impact of Movies Task provides participants with a comparative and collaborative evaluation framework for emotional detection in movies, in terms of valence, arousal and fear. The LIRIS-ACCEDE dataset has been used as development and test sets. Details on the methods and results of each individual team can be found in the papers of the participating teams in the MediaEval 2018 workshop proceedings.
This task is supported by the CHIST-ERA Visen project ANR-12CHRI-0002-04. 5http://www.kijkwijzer.nl/nicam Emotional Impact of Movies Task