This paper will present our results for the MediaEval 2016 Predicting Media Interestingness task. We proposed an approach based on video descriptors and studied several machine learning models, in order to detect the optimal con guration and combination for the descriptors and algorithms that compose our system.
Interestingness is the ability to attract and hold human
attention, this concept is gaining importance in the eld
of computer vision, especially since the growing importance
and market value of social media and advertising. Even
though the concept of interest might seem the result of a
subjective viewer judgment, important progress has been
made towards both an objective and context-based model for
interest. Generally, in the eld of computer vision two
directions arose regarding this topic: pure visual interestingness
(based on multimedia features and ideas [
In this context, the MediaEval 2016 Predicting Media
Interestingness Task [
As previously stated, to determine the interestingness of
images and video, we have experimented with a classic
machine learning approach. First, the raw data is converted
to content descriptors which should capture as best as
possible the visual interestingness features of the data. Then,
a supervised classi er is learned on these features using the
labeled examples. Finally, the actual evaluation is carried
out by feeding the classi er the unlabeled data. Regarding
the content descriptors, we used the ones provided by the
task organizers [
Several visual features were used as descriptors, many of
them being used in the literature for some computer vision
tasks. The provided computed features were: color
histogram of the Hue-Saturation-Value (denoted histo),
Histogram of Oriented Gradients (HoG) descriptors computed
over densely sampled patches, dense Scale Invariant Feature
Transform (SIFT) with a codebook of 300 codewords and a
three layered spatial pyramid (denoted dsift), Local Binary
Patterns (LBP), GIST computed with the output of
Gaborlike features (denoted gist) and the fc7 and prob layers of
AlexNet (denoted cnnfc7 and cnnprob). All these features
are presented and detailed in [
For the image subtask, each image is represented with a content descriptor. For the video subtask, each video contains a certain number of images. To determine the nal descriptor we use the simple averaging of the frames descriptors, leading in the end to a global descriptor per video. 2.2
The learning is achieved using a Support Vector Machine (SVM) binary classi er. For all trained SVM models we used polynomial, RBF and linear kernels. For the polynomial kernels we used all the combinations of the following degrees : 1, 2, 3*k where k 2 [1; :::; 10] and the gamma coe cients were set as 2k where k 2 [0; :::; 6]. For the RBF kernel combinations we had values for the cost parameter of 2k where k 2 [ 4; :::; 8] and gamma coe cients with values in 2k where k 2 [ 4; :::; 8]. We also tried di erent weights, considering the fact that the devset data, both for images and for videos, was unbalanced, the ratio of uninteresting to interesting samples being almost 10 to 1. Run run1 run2 run3 run4 run5
Subtask image image video video video
The task data consists of a development data intended to train the approaches and a test data for the actual benchmarking. The devset was extracted from 52 trailers, manually segmented, thus obtaining 5054 segments. For the image subtask one key-frame was used from each segment, while for the video subtask the whole segment was used. By annotating all the data a total of 473 interesting images and 420 interesting videos were obtained, with a provided interestingness score for calculating the mean average precision. The testset consisted of 26 trailers divided into 2342 segments. We performed a number of experiments on devset and selected the best combinations to be run on testset. 3.1
Using a 10-fold cross-validation, we chose the best results for the descriptor-classi er combinations based on precision, with a recall better than 0.03. For those best combinations we calculated the mean average precision. We have experimented with many di erent combinations of descriptors and SVM kernels. The best performing combination was generally the polynomial SVM. A high number of training runs, especially with the RBF or linear kernels, tended to classify all or almost all (low recall) the samples as non-interesting. In the case of weight-based training for the RBF kernel the recall tended to grow, but the precision was below that of the polynomial SVMs.
Table 1 lists the best ve results for each of the two
subtasks, giving details regarding the best coe cient
combination used. As shown, the estimated MAP on the devset
was better for the image subtask than for the video subtask.
The MAP scores were calculated by using LibSVM's
decision values/prob estimates output result for indicating the
interestingness score of each sample [
The teams were allowed to submit 5 runs, so we chose the best 2 descriptor-classi er combinations for the image subtask and the best 3 combinations for the video subtask. This time the training of the SVM learning systems was done on the entire devset, using the optimal degree and gamma parameters obtained in our previous experiments. The submitted runs were the following : run1 - image subtask with HSV Histogram + GIST, SVM with degree = 18 and gamma = 2, run2 - image subtask with DSIFT + GIST, SVM with degree = 3 and gamma = 32, run3 - video subtask with GIST + CNNProb, SVM with degree = 9 and gamma = 4, run4 - video with CNNFc7 + HSV Histogram, SVM with degree = 3 and gamma = 4 and run5 - video with DSIFT + CNNProb, SVM with degree = 24 and gamma = 64.
The nal results, as returned by the task organizers are presented in Table 2. The best results were a 0.1714 MAP on run1 for the image subtask and a 0.1629 MAP on run5 for the video subtask. With the single exception being run5, the MAP results on testset were below the estimated MAP on devset. 4.
In this paper we presented several models for predicting and scoring multimedia interestingness. Our best MAP results on the testset were 0.1714 for the image subtask and 0.1629 for the video subtask. These results seem to indicate that the task in very challenging, one possible reason for this being the subjective nature of this eld of study.