This paper describes the system developed by team IITB for MediaEval 2017 Predicting Media Interestingness Task. We propose a new method of training based on pairwise comparisons between frames of a trailer. The algorithm gave very promising results on the development set but did not impress on test set. Our highest achieved MAP@10 on test set is 0.0911 (Image subtask) and 0.0525 (Video subtask), based on a systems submitted last year ([4, 6]).
The MediaEval 2017 Predicting Media Interestingness Task
[
Previous systems on this task discuss in detail several
relevant inherent problems. Further, they also point towards
the usefulness of CNN features: in particular, they report
features from AlexNet's fc7 layer performing reasonably well
with simple classi ers [
Given the training data feature matrix X 2 RN F consisting of N examples, each described by a F -dimensional vector, we rst standardize it and apply principal component analysis (PCA) to reduce its dimensionality. The transformed feature matrix Z = (zi)i 2 RN M is used to experiment with various classi ers. Here M depends on the number of top eigenvalues we wish to consider.
For our system we use AlexNets's fc7 [
We adopted the following two methods for training:
1. Feed every frame/video's feature vector to the classi er
where it learns to predict the interestingness label of
the frame as in [
For the second training method, pairwise comparisons are made . First, from each trailer, we generate all possible pairs of frames. This ensures that only frames/videos of the same trailer are being compared. Considering T trailers having ni number of frames/videos in them, we get N1 = Pii==1T n2i pairs. Representation of each pair is done by concatenating the feature vectors of each frame/video. The feature vector of each being of size M , after concatenating we get nal feature vector of size 2M . This procedure yields a feature matrix Znew 2 RN1 2M . Output labels for an ordered pair of frames/videos (I1; I2) is assigned as follows: y = (1; 0;
(1) 2.3
For the rst two runs which are based on [
Ranking of the frames/videos according to their interestingness in a particular trailer is determined from the predicted results of all the pairwise comparisons by generating penalty scores si for each of them and ordering them from lowest to highest with lowest corresponding to most interesting frame/video. The scores are determined using the following algorithm (referred as P1): 1. Initialize the penalty scores si = 0 for each i 2. Iterate over results of all pairwise comparisons: for each pair indexed by fk; lg, let r(k; l) denote the prediction of classi er. The following update is performed: su = su + j Prfr(k; l) = 1g
Prfr(k; l) = 0gj where u denotes the index of less interesting frame/video predicted, Prf:g denotes the probability and j:j the absolute value
This essentially increases the penalty score for the less interesting according to the con dence the classi er has in its prediction. The con dence value of the classi er for a given pair is treated as absolute di erence between Prfr(k; l) = 1g and Prfr(k; l) = 0g. We also try a variant of the above algorithm in one of our runs wherein the update equation is: su = su + 1 (referred as P2)
Interestingness classi cation: We opt for a simple method for binary classi cation of each image as interesting or not: We classify the top 12% ranked images as interesting. We chose top 12% images as it's slightly higher than the average number of interesting images, which is about 9%. It's important to note that since we generate ranking of frames, choosing only top 12% images has no particular signi cance as the o cial metric remains una ected by it. 3.
The training dataset consisted of 7396 frames extracted
from 78 movie trailers with about 392,000 pairs of frames,
while the test data consisted of 2435 frames extracted from
30 movie trailers. [
Our results on the development set for various approaches are given in Table 1. The run submission results are given in Table 2. The tables gives the mean average precision (MAP) - the o cial metric MAP@10, of di erent runs corresponding to the method of training and the classi er used.
We experimented with the CNN features provided and used
PCA to bring down the number of dimensions to 200.
Additionally, we used a non pairwise (NP) and a pairwise strategy
(P1, P2) for training and prediction as described in previous
section. These methods were used to train SVM (rbf kernel)
[
Due to large number of pairs involved in training, we could not experiment with classi ers such as SVM in our presented approach (P1, P2) because of large training time. We experimented with the following classi ers. (1) Logistic regression with l2 penalty, (2) Random Forest, (3) Logistic regression with l1 penalty. (3) gave slightly better results than the other two and was the fastest in training, hence we went with logistic regression-l1 penalty as our classi er.
However the results on the test set, were unexpected. Logistic regression using pairwise comparisons gave the best results on the development set for both the tasks. On the test set it isn't impressive where the best result is for nonpairwise logistic regression (Image subtask) and non-pairwise SVM-rbf kernel (Video subtask).
There could be various possible reasons for the discrepancy in the results on development and test set. (i) Viewing the classi er as a neural network, it may require more ne tuning of the weights of fc7 layer of AlexNet, or a more complex network instead of a single neuron so that it generalizes better. (ii) Though improbable, it's possible there are some discrepancies in the sources of development and test set which result in poor generalization. 4.
In summary, we proposed a new system for interestingness
prediction in images and videos. It essentially di ers in the
method of training based on pairwise comparisons of images.
This helps in capturing interestingness of an image in a
local context. Although our system gave impressive results on
development set, it failed to perform well on the test set.
Some improvements on current system can be improving its
complexity or ne tuning the last layer of AlexNet for better
input representation. The e ciency of the training can also
be improved by selecting pairs more intelligently ([