The Context of Experience Task at MediaEval 2016 is devoted to recommending multimedia content suiting a watching situation. Speci cally, the task addresses the situation of viewers watching movies on an airplane. The goal of the task is to use trailer-content and textual metadata in order to estimate whether movies are tting to watch in ight, as judged by the crowd. The context of an airplane often falls short of an ideal movie-watching situation (noise, lack of space, interruptions, stale air, stress from turbulence) and the device can also impact user experience (small screens, glare, poor audio quality). The task explores the notion that some movies are generally better suited to these conditions than others, and that a component of this suitability is independent of viewers' personal preferences.
The Context of Experience Task at the Multimedia Evaluation (MediaEval) 2016 Benchmark tackles the challenge of predicting the multimedia content that users nd most tting to watch in speci c viewing situations. When researchers in the area of recommender systems or multimedia information retrieval consider the situations in which viewers consume multimedia content, such as movies, they generally assume comfortable watching conditions. This assumption is understandable, since people do frequently enjoy movies in the quiet, privacy and comfort of their own living rooms, together with friends and loved ones, relaxing in arm chairs and on the couch. However, movie watching is certainly not limited to such situations. In fact, people might choose to watch movies exactly because they are in an uncomfortable, stressful situation and would bene t from distraction.
Our ultimate goal is to build recommender systems that support people in nding content that helps them through tough times, i.e., moments at which they are under psychological stress or in physical discomfort. We envisage such contexts to include dentist o ces and hospitals. However, here, we focus our e ort on on a context that does not involve either physical pain or extreme psychological distress: we chose the context of air travel. Speci cally, the Context of Experience Task requires participants to use features derived from video content and from movie metadata in order to predict movies that are appropriate to watch on an airplane.
The next sections of the paper cover related work, and
provide more details on in- ight-distractors in uencing viewer
experience. We close with a brief description of the data
set and the task. The description is brief since this
information has been provided in detail elsewhere. Speci cally,
the rst description was published in a short paper in the
proceedings of MediaEval 2015 [
Although our ultimate aim is to provide viewers with
multimedia content for a particular context, we di er from
context-aware movie recommendations as addressed by [
Context of Experience is obviously closely linked to the
area of Quality of Experience of multimedia content. In [
Within the MediaEval benchmark1, the Context of
Experience Task follows upon other tasks that have been devoted
to predicting the impact of content on viewers or listeners.
These include an A ect Task on predicting viewer
experienced boredom [
(c) The movie is stopped for an announcement. (d) Glare on the screen makes it almost impossible to see what is going on.
On a plane, we assume that movie viewers share the common goal, which we consider to be a viewing intent, of relaxing, passing time and keeping themselves occupied while being con ned in the small and often crowded space of an airplane cabin.
Figure 1 provides an impression of a screen commonly used on an airplane and some situations that can occur during a ight that can in uence the watching experience of the viewers. Sub gure 1(a) shows the optimal situation without a distraction and a acceptable video quality. The other subgures illustrate distracters that impact the movie viewing experience. These examples illustrate how a person's experience of a movie during the ight can be heavily in uenced by the context. 4.
The objective of the task is to classify each movie as either +goodonairplane or -goodonairplane. Task participants are asked to form their own hypothesis about what they think is important for people viewing movies on an airplane, and then to design an approach using appropriate features and a classi er or decision function.
The task data set consists of a list of movies, including links to descriptions and video trailers, pre-extracted features and metadata. Movies were collected between February and April 2015 from movie lists of a major international airline, i.e., KLM Royal Dutch Airlines. The set contains an equal number of non-airline movies, sampled with similar distributional properties (e.g., year). We do not provide video les for the trailers because of copyright restrictions. The pre-extracted visual features are Histogram of Oriented Gradients (HOG) gray, Color Moments, local binary patterns (LBP) and Gray Level Run Length Matrix. The audio descriptors are Mel-Frequency Cepstral Coe cients (MFCCs). Task participants are also allowed to collect their own data such as full length movies, and more metadata, e.g., user comments. The development set contains 95 and the test set contains 223 movies. The data set is balanced 50/50 between +goodonairplane/-goodonairplane. The ground truth consists of user judgments gathered on CrowdFlower. In total, 548 di erent workers participated and at least ve judgments per movie were collected.
For the evaluation, we use the metrics precision, recall
and weighted F1 score. We chose these metrics instead of
error rate because the task is related to recommendation.
For the purposes of recommendation, a ranked list is often
needed. Also, recall is an interesting and important part of
the evaluation. A baseline was created using a simple tree
based classi er (precision 0:629; recall of 0:573; F1 score
0:6). As mentioned above, more information is available in
the other papers that have been published discussing the
data set and the task [
This work is partly funded by the FRINATEK project "EONS" (#231687) and the BIA project PCIe (#235530) funded by the Norwegian Research Council and by the EC FP7 project CrowdRec (#610594).