=Paper=
{{Paper
|id=Vol-1739/MediaEval_2016_paper_2
|storemode=property
|title=The MediaEval 2016 Context of Experience Task: Recommending Videos Suiting a Watching Situation
|pdfUrl=https://ceur-ws.org/Vol-1739/MediaEval_2016_paper_2.pdf
|volume=Vol-1739
|dblpUrl=https://dblp.org/rec/conf/mediaeval/RieglerSLGH16
}}
==The MediaEval 2016 Context of Experience Task: Recommending Videos Suiting a Watching Situation==
https://ceur-ws.org/Vol-1739/MediaEval_2016_paper_2.pdf
The MediaEval 2016 Context of Experience Task:
Recommending Videos Suiting a Watching Situation
Michael Riegler1 , Concetto Spampinato2 , Martha Larson3 , Pål Halvorsen1 , Carsten Griwodz1
1
Simula Research Laboratory and University of Oslo, Norway
2
University of Catania, Italy
3
Delft University of Technology and Radboud Univeristy, Netherlands
{michael, paalh, griff}@simula.no, cspampin@dieei.unict.it, m.a.larson@tudelft.nl
ABSTRACT The next sections of the paper cover related work, and pro-
The Context of Experience Task at MediaEval 2016 is de- vide more details on in-flight-distractors influencing viewer
voted to recommending multimedia content suiting a watch- experience. We close with a brief description of the data
ing situation. Specifically, the task addresses the situation set and the task. The description is brief since this infor-
of viewers watching movies on an airplane. The goal of the mation has been provided in detail elsewhere. Specifically,
task is to use trailer-content and textual metadata in order the first description was published in a short paper in the
to estimate whether movies are fitting to watch in flight, as proceedings of MediaEval 2015 [7], which served to launch
judged by the crowd. The context of an airplane often falls the task. Additional information was published in [6]. Fi-
short of an ideal movie-watching situation (noise, lack of nally, in order to stimulate cross-benchmark collaboration,
space, interruptions, stale air, stress from turbulence) and the task was also offered as part of the Joint Contest on Mul-
the device can also impact user experience (small screens, timedia Challenges Beyond Visual Analysis at ICPR 2016,
glare, poor audio quality). The task explores the notion and a paper published that contains a short description and
that some movies are generally better suited to these condi- some insights on results [4].
tions than others, and that a component of this suitability
is independent of viewers’ personal preferences. 2. RELATED WORK
Although our ultimate aim is to provide viewers with
1. INTRODUCTION multimedia content for a particular context, we differ from
context-aware movie recommendations as addressed by [8,
The Context of Experience Task at the Multimedia Eval-
9]. Context of Experience assumes that the experience of
uation (MediaEval) 2016 Benchmark tackles the challenge
viewing a movie interacts with the context in which a movie
of predicting the multimedia content that users find most
is viewed. Instead, we admit that a movie is actually able
fitting to watch in specific viewing situations. When re-
to change the viewer’s perception of the context. We em-
searchers in the area of recommender systems or multimedia
phasize that addressing the challenge of recommending for
information retrieval consider the situations in which viewers
users’ Contexts of Experience means not ‘just’ matching
consume multimedia content, such as movies, they generally
movies with users’ personal taste, but rather also helping
assume comfortable watching conditions. This assumption
users accomplish goals that they want to achieve by con-
is understandable, since people do frequently enjoy movies
suming movies. These goals may include distracting them-
in the quiet, privacy and comfort of their own living rooms,
selves from discomfort and making time pass more quickly.
together with friends and loved ones, relaxing in arm chairs
We also note that the focus of recommender system research
and on the couch. However, movie watching is certainly not
on personalization often leads to neglect of cases in which
limited to such situations. In fact, people might choose to
context might have a strong impact on preference relatively
watch movies exactly because they are in an uncomfortable,
independently of the personal tastes of specific viewers, an
stressful situation and would benefit from distraction.
idea echoed in [5]. Particularly strong influence of context
Our ultimate goal is to build recommender systems that
can be expected in the stressful situations that are the focus
support people in finding content that helps them through
of our interest.
tough times, i.e., moments at which they are under psycho-
Context of Experience is obviously closely linked to the
logical stress or in physical discomfort. We envisage such
area of Quality of Experience of multimedia content. In [10],
contexts to include dentist offices and hospitals. However,
Physical context, Social Cultural Context and Task are all
here, we focus our effort on on a context that does not in-
identified as context-related factors that contribute to the
volve either physical pain or extreme psychological distress:
user’s perception of quality of experience.
we chose the context of air travel. Specifically, the Con-
Within the MediaEval benchmark1 , the Context of Expe-
text of Experience Task requires participants to use features
rience Task follows upon other tasks that have been devoted
derived from video content and from movie metadata in or-
to predicting the impact of content on viewers or listeners.
der to predict movies that are appropriate to watch on an
These include an Affect Task on predicting viewer experi-
airplane.
enced boredom [12], the Emotion in Music task [1], a current
task on the affective impact of movies [11, 2], and a current
Copyright is held by the author/owner(s). task on Predicting Media Interestingness [3].
MediaEval 2016 Workshop, Oct. 20-21, 2016, Hilversum, Nether- 1
lands. http://www.multimediaeval.org/
3. MOVIES ON A PLANE
On a plane, we assume that movie viewers share the com-
mon goal, which we consider to be a viewing intent, of re-
laxing, passing time and keeping themselves occupied while
being confined in the small and often crowded space of an
airplane cabin.
Figure 1 provides an impression of a screen commonly
used on an airplane and some situations that can occur dur-
ing a flight that can influence the watching experience of the
viewers. Subfigure 1(a) shows the optimal situation without
a distraction and a acceptable video quality. The other sub-
figures illustrate distracters that impact the movie viewing
(a) The ideal situation while watching a movie on a plane. experience. These examples illustrate how a person’s expe-
rience of a movie during the flight can be heavily influenced
by the context.
4. TASK AND DATA
The objective of the task is to classify each movie as either
+goodonairplane or -goodonairplane. Task participants are
asked to form their own hypothesis about what they think
is important for people viewing movies on an airplane, and
then to design an approach using appropriate features and
a classifier or decision function.
The task data set consists of a list of movies, including
links to descriptions and video trailers, pre-extracted fea-
tures and metadata. Movies were collected between Febru-
(b) A flight attendant serving the neighboring passenger. ary and April 2015 from movie lists of a major international
airline, i.e., KLM Royal Dutch Airlines. The set contains
an equal number of non-airline movies, sampled with sim-
ilar distributional properties (e.g., year). We do not pro-
vide video files for the trailers because of copyright restric-
tions. The pre-extracted visual features are Histogram of
Oriented Gradients (HOG) gray, Color Moments, local bi-
nary patterns (LBP) and Gray Level Run Length Matrix.
The audio descriptors are Mel-Frequency Cepstral Coeffi-
cients (MFCCs). Task participants are also allowed to col-
lect their own data such as full length movies, and more
metadata, e.g., user comments. The development set con-
tains 95 and the test set contains 223 movies. The data set is
balanced 50/50 between +goodonairplane/-goodonairplane.
(c) The movie is stopped for an announcement. The ground truth consists of user judgments gathered on
CrowdFlower. In total, 548 different workers participated
and at least five judgments per movie were collected.
For the evaluation, we use the metrics precision, recall
and weighted F1 score. We chose these metrics instead of
error rate because the task is related to recommendation.
For the purposes of recommendation, a ranked list is often
needed. Also, recall is an interesting and important part of
the evaluation. A baseline was created using a simple tree
based classifier (precision 0.629; recall of 0.573; F1 score
0.6). As mentioned above, more information is available in
the other papers that have been published discussing the
data set and the task [7, 6, 4]. We hope that the Context
of Experience Task can help to raise awareness of the topic
and also provide an interesting and meaningful use case to
(d) Glare on the screen makes it almost impossible to see what
is going on. inspire more work in this area.
Figure 1: The four images show the ideal situation 5. ACKNOWLEDGMENT
compared to three distracting situations that can This work is partly funded by the FRINATEK project
occur during a flight. ”EONS” (#231687) and the BIA project PCIe (#235530)
funded by the Norwegian Research Council and by the EC
FP7 project CrowdRec (#610594).
6. REFERENCES
[1] A. Aljanaki, Y.-H. Yang, and M. Soleymani. Emotion
in Music Task: Lessons learned. In this proceedings.
[2] E. Dellandréa, L. Chen, Y. Baveye, M. Sjöberg, and
C. Chamaret. The MediaEval 2016 Emotional Impact
of Movies Task. In this proceedings.
[3] C.-H. Demarty, M. Sjöberg, B. Ionescu, T.-T. Do,
H. Wang, N. Q. K. Duong, and F. Lefèbvre.
MediaEval 2016 Predicting Media Interestingness
Task. In this proceedings.
[4] H. J. Escalante, V. Ponce-López, J. Wan, M. A.
Riegler, B. Chen, A. Clapés, S. Escalera, I. Guyon,
X. Baró, P. Halvorsen, H. Müller, and M. Larson.
ChaLearn joint contest on multimedia challenges
beyond visual analysis: An overview. In Proceedings of
the International Conference on Pattern Recognition,
2016, to appear.
[5] R. Pagano, P. Cremonesi, M. Larson, B. Hidasi,
D. Tikk, A. Karatzoglou, and M. Quadrana. The
contextual turn: From context-aware to
context-driven recommender systems. In Proceedings
of the 10th ACM Conference on Recommender
Systems, RecSys ’16, pages 249–252, 2016.
[6] M. Riegler, M. Larson, C. Spampinato, P. Halvorsen,
M. Lux, J. Markussen, K. Pogorelov, C. Griwodz, and
H. Stensland. Right inflight?: A dataset for exploring
the automatic prediction of movies suitable for a
watching situation. In Proc. of Multimedia Systems,
MMSys ’16, pages 45:1–45:6, 2016.
[7] M. Riegler, M. Larson, C. Spampinato, J. Markussen,
P. Halvorsen, and C. Griwodz. Introduction to a task
on Context of Experience: Recommending videos
suiting a watching situation. In Proceedings of the
MediaEval 2015 Workshop, Wurzen, Germany,
September 14-15, 2015.
ceur-ws.org/Vol-1436/Paper5.pdf.
[8] A. Said, S. Berkovsky, and E. W. De Luca. Putting
things in context: Challenge on context-aware movie
recommendation. In Proceedings of the Workshop on
Context-Aware Movie Recommendation, CAMRa ’10,
pages 2–6, 2010.
[9] A. Said, S. Berkovsky, and E. W. De Luca. Group
recommendation in context. In Proceedings of the 2nd
Challenge on Context-Aware Movie Recommendation,
CAMRa ’11, pages 2–4, 2011.
[10] R. Schatz, T. Hoßfeld, L. Janowski, and S. Egger.
From Packets to People: Quality of Experience as a
New Measurement Challenge, pages 219–263. Springer
Berlin Heidelberg, Berlin, Heidelberg, 2013.
[11] M. Sjöberg, Y. Baveye, H. Wang, V. L. Quang,
B. Ionescu, E. Dellandréa, M. Schedl, C.-H. Demarty,
and L. Chen. The MediaEval 2015 Affective Impact of
Movies Task. In Proceedings of the MediaEval 2015
Workshop Wurzen, Germany, September 14-15, 2015.
http://ceur-ws.org/Vol-1436/Paper1.pdf.
[12] M. Soleymani, M. Larson, T. Pun, and A. Hanjalic.
Corpus development for affective video indexing. IEEE
Transactions on Multimedia, 16(4):1075–1089, 2014.