This paper provides a three-step framework to predict user assessment of the suitability of movies for an inflight viewing context. For this, we employed classifier stacking strategies. First of all, using the different modalities of training data, twenty-one classifiers were trained together with a feature selection algorithm. Final predictions were then obtained by applying three classifier stacking strategies. Our results reveal that different stacking strategies lead to different evaluation results. A considerable improvement can be found for the F1-score when using the label stacking strategy.
A substantial amount of research has been conducted in
recommender systems that focus on user preference prediction. Here,
taking contextual information into account can have significant
positive impact on the performance of recommender systems [
The MediaEval Context of Experience task focuses on a specific type of context: the viewing context of the user. The challenge considers predicting the multimedia content that users find most fitting to watch in a specific viewing condition, more specifically, while being on a plane.
The dataset for the Context of Experience (CoE) task[5] contains metadata and pre-extracted features for 318 movies [6]. Features are multimodal and include textual features, visual features and audio features. The training set contains 95 labeled movies, which are labeled as 0 (bad for airplane) or 1 (good for airplane).
A set of initial experiments has been conducted in order to evaluate the usefulness of the various modalities in the CoE dataset [6]. A rule-based PART classifier was employed to evaluate the feature performance in terms of Precision, Recall and F1 Score, the result can be found in Table 1.
Ensemble learning uses a combination of different classifiers,
usually getting a much better generalization ability. This
particularly is the case for weak learners, which can be defined as learning
algorithms that perform just slightly better than random guessing
by themselves, but can be jointly grouped into an algorithm with
arbitrarily high accuracy [
Metadata + user rating Metadata + visual
0.371 0.447 0.524 0.581 0.584
0.609 0.476 0.516 0.6 0.6
F1 0.461 0.458 0.519 0.583 0.586
Therefore, we were interested in taking a multimodal classifier stacking approach to the given problem, and use a combination of multiple weak learners to ‘boost’ them into a strong learner.
The process can be separated into three stages: classifier selection, feature selection and classifier stacking. 3.1
First of all , we want to select base classifiers that will be useful candidates in a stacking approach. For this, we use the following classifier selection procedure: 1. Initialize a list of candidate classifiers. For each modality, we consider the following classifiers: k-nearest neighbor, nearest mean, decision tree, logistic regression, SVM, bagging, random forest, AdaBoost, gradient boosting, and naive Bayes. We do not apply parameter tuning, but take the default parameter values as offered by scikit-learn1. 2. Perform 10-fold cross-validation on the classifiers. As input data, we use the training data set and its ground truth labels, per single modality. For the audio MFCC features, we set NaN values to 0, and calculate the average of each MFCC coefficient over all frames. 3. If Precision and Recall and F1-Score > 0.5, keep the candidate classifier on the given modality as base classifier for our stacking approach.
The selected base classifiers and their relevant modalities can be found in Table 2. It should be noted that the performance of Bagging and Random forest is not stable. This is because Bagging tries to use different subset of instances in each run and RandomForest tries to use different subsets of instances and features in each run. 3.2
For each classifier and corresponding modality, a better-performing subspace of features may optimize results further. Since we have
k-Nearest neighbor Nearest mean classifier
Decision tree
Logistic regression SVM (Gaussian Kernel)
Bagging Random Forest
AdaBoost Gradient Boosting Tree
Naive Bayes k-Nearest neighbor SVM (Gaussian Kernel) k-Nearest neighbor
Decision tree
Logistic regression SVM (Gaussian Kernel)
Random Forest
AdaBoost Gradient Boosting Tree
Logistic Regression Gradient Boosting Tree
Modality metadata metadata metadata metadata metadata metadata metadata metadata metadata textual textual textual visual visual visual visual visual visual visual audio audio
In previous research, classifier stacking (or metalearning) has
been proved beneficial for predictive performance by combining
different learning systems which each have different inductive bias
(e.g. representation, search heuristics, search space) [
We considered all prediction results by the 21 selected base classifiers, and then applied the three different classifier stacking strategies to the test data using 10-fold cross-validation. As results for label stacking vs. label attribute stacking were comparable on the training data, we only consider voting vs. label stacking on the test data.
All obtained results, on the training (development) and test dataset, are given in Table 3. On the training data, we notice significant improvement can be found in terms of Precision, Recall as well as F1 score in comparison to results obtained on individual modalities. The voting strategy results in the best precision score, but has bad performance in terms of recall. On the contrary, label stacking has higher recall and the highest F1 score.
Considering results obtained on the test dataset, we can conclude that label stacking is more robust than the voting strategy. For voting strategy, a significant decrease can be found in terms of precision on test set. This is because majority vote (and Bayesian averaging) tendency to over-fit derives from the likelihood’s exponential sensitivity to random fluctuations in the sample, and increases with the number of models considered. Meanwhile, label stacking strategy performs reasonable well on test data.
Voting (cv)
Label Stacking (cv) Label Attribute Stacking (cv)
Voting (test) Label Stacking (test)
0.94 0.72 0.71 0.62 0.62
0.57 0.86 0.79 0.80 0.90
In our entry for the MediaEval CoE task, we aimed to improve classifier performance by a combination of classifier selection, feature selection and classifier stacking. Results reveal that employing a ensemble approach can considerably increase the classification performance, and is suitable for treating the multimodal Right Inflight dataset.
The larger diversity of base classifiers is able to produce a more robust ensemble classifier. On the other hand, a blending of multiple classifiers may also have some drawbacks, e.g computational costs, and difficulty in traceable interpretation.
We expect better results for our method can still be obtained through parameter tuning, and by applying more robust classifier stacking methods, such as feature weighted linear stacking [7]. Advances in distributed and parallel knowledge discovery, pages 81–114. MIT/AAAI Press, 2000. [5] M. Riegler, , C. Spampinato, M. Larson, P. Halvorsen, and C. Griwodz. The mediaeval 2016 context of experience task: Recommending videos suiting a watching situation. In Proceedings of the MediaEval 2016 Workshop, 2016. [6] M. Riegler, M. Larson, C. Spampinato, P. Halvorsen, M. Lux, J. Markussen, K. Pogorelov, C. Griwodz, and H. Stensland. Right inflight? A dataset for exploring the automatic prediction of movies suitable for a watching situation. In Proceedings of the 7th International Conference on
Multimedia Systems, pages 45:1–45:6. ACM, 2016. [7] J. Sill, G. Takacs, L. Mackey, and D. Lin. Feature-weighted linear stacking. arXiv:0911.0460, 2009.