We propose a Context of Experience task, whose aim it is to explore the suitability of video content for watching in certain situations. Speci cally, we look at the situation of watching movies on an airplane. As a viewing context, airplanes are characterized by small screens and distracting viewing conditions. We assume that movies have properties that make them more or less suitable to this context. We are interested in developing systems that are able to reproduce a general judgment of viewers about whether a given movie is a good movie to watch during a ight. We provide a data set including a list of movies and human judgments concerning their suitability for airplanes. The goal of the task is to use movie metadata and audio-visual features extracted from movie trailers in order to automatically reproduce these judgments. A basic classi cation system demonstrates the feasibility and viability of the task.
The challenge of the Context of Experience task is to automatically predict viewers' judgments on whether video content is suitable for a particular watching situation. Ultimately, the aim is to build a recommender system that would provide viewers with recommendations of content for a given context. Currently, the majority of work on video content recommendation focuses on personal preferences, and overlooks cases in which context might have a strong impact on preference relatively independently of the personal tastes of speci c viewers. Particularly strong in uence of context can be expected in psychologically stressful or physically uncomfortable situations.
For our task, we choose one such situation, with which
a large number of people have quite frequent experience:
watching movies on an airplane. In this situation, a large
majority of viewers share a common goal, which we
consider to be a viewing intent. The goal is to pass time as
pleasantly and meaningfully as possible, while con ned in
the small space of an airplane cabin, which is characterized
by a number of distractors. We take the large number of
websites discussing movies to watch on airplanes (e.g., [
For the task we provide the participants a list of movies,
including links to descriptions and video trailers. The
assignment of the task is to classify each movie into
+goodonairplane / -goodonairplane classes. Therefore, the ground
truth of the task is derived from two sources: A list of movies
actually used by a major airline1, as well as user judgments
on movies that are collected via a crowdsourcing tool2. Task
participants should form their own hypothesis about what
is important for users viewing movies on an airplane, and
design an approach using appropriate features and a
classi er, or decision function. Figure 1 gives an impression of
a screen commonly used on an airplane and the very
speci c attributes regarding size and quality of the video. The
value of the task lies in understanding the ability of
contentbased and metadata-based features to discriminate the kind
of movies that people would like to watch on small screens
under stressful or somehow not normal situations. Since the
multimedia content that users watch on ights can in uence
their well being and overall experience this task is related
to the quality of multimedia experience work like for
exam1http://www.klm.com/travel/no_en/prepare_for_
travel/on_board/entertainment/onboard_movies.htm
2https://crowdflower.com/
ple [
To tackle the task it can be addressed by leveraging techniques from multiple multimedia-related disciplines, including such as social computing (intent), machine learning (classi cation), multimedia content analysis, multimodal fusion, and crowdsourcing. Further we hope that it will be useful for content provider, since the exploitation of intent in combination with users' satisfaction could lead to more sophisticated ways to develop methods of providing a better service to the users. 3.
The data set we provide is released, including titles and links, that allow participants to gather online metadata and trailers for movies. We do not, as already mentioned, provide the video les because of copyright restrictions. Movies are collected based on movie lists from a major international airline, in our case, KLM Royal Dutch Airlines. The nal list of movies is a merged set of movies collected between February and April 2015. The video data set contains both positive and negative samples, whereas the negative examples are carefully sampled from IMDB in order to create a fair and representative negative class. The data set is split into a training set and a test set. In order to collect user judgments, we use an existing system that has been built for the purpose of collecting user feedback of this sort. We evaluate systems both with respect to the airline's choice of movies, and the crowd's choice of airline-suitable movies. Votes about the labels collected by crowdsourcing are considered as the authoritative labels. For this reason, crowdworkers are asked to rank a small set of movies with respect to how strongly they would like to watch this video on an airplane. This ranking is then combined to create the class for each movie in the training and test data.
Technical details. Overall, the data set contains 318 movies. Links to trailers are collected from IMDB and YouTube. Participants are also allowed to collect their own data such as full length movies, more metadata and user comments, etc. The goal of systems that are developed to address this task should be to automatically identify appropriate content, i.e., whether a movie should be recommended for being watched on an airplane or not. To achieve this goal, the methods should not require manual or crowdsourced input. The data set contains extracted visual, audio and text features. Furthermore, we provide metadata collected from IMDB including user comments. The visual features that are provided are: Histogram of Oriented Gradients (HOG), Color Moments, Local Binary Patterns (LBP) and Gray Level Run Length Matrix. The audio descriptors are MelFrequency Cepstral Coe cients (MFCC). The development set contains 95 labelled movies. The test data contains 223 movies without labels.
Features used Metadata + user ratings Only user ratings Only visual information Only metadata
Evaluation. For the evaluation we use the standard metrics Precision, Recall and weighted F1 score. Negative and positive classes in both data sets are balanced. Participants are asked to submit a predicted class for each movie in the test data set. The metrics then are calculated and provided to the participants. For a transparent and fair procedure, the labels used for the evaluation will be released together with the results.
Initial results. To con rm the viability of the task, and
show the possibilities opened by this data set, we carried
out some basic classi cation experiments. For the classi
cation we used the Weka library. As classi er we choose
the rule based PART classi er. This classi er uses separate
and conquer to generate a decision list. From this, it builds
a decision tree where the best leaves are used as rules for
the classi er [
Finally, the last run con rms that metadata without user ratings is able to yield performance above the naive baseline. An information gain based analysis of all features ranked genre, publication year, country, language and runtime as the top ve features.
The task is challenging due to the complex relationship between the multimedia content, and viewers' perceptions and reception. We hope that the novel use case will inspire researchers to investigation of user intent and context of experience. Understanding user intent and what users need in order to have the best experience is an important emerging topic in the area of multimedia research.
This work is funded by the Norwegian FRINATEK project "EONS" (#231687) & the EC project CrowdRec (#610594).