In MediaEval 2016, we focus on the image interestingness subtask which involves predicting interesting key frames of a video in the form of a movie trailer. We speci cally propose three di erent deep models for this subtask. The rst two models are based on ne-tuning two pretrained models, namely AlexNet and MemNet, where we cast the interestingness prediction as a regression problem. Our third deep model, on the other hand, depends on a triplet network which is comprised of three instances of the same feedforward network with shared weights, and trained according to a triplet ranking loss. Our experiments demonstrate that all these models provide relatively similar and promising results on the image interestingness subtask.
Understanding and predicting interestingness of images
or video shots have been proposed as a recent problem in
computer vision literature [
Deep convolutional neural networks (CNNs) have
revolutionized the computer vision eld in recent years, obtaining
state-of-the-art results in many di erent problem domains.
In our submission, we tested three di erent CNN models,
which are all based on the popular AlexNet architecture [
For our rst model, we ne-tune AlexNet [
Our second model is based on the recently proposed
MemNet model [
Our third model also follows the AlexNet architecture,
but di ers from our previous models in that we employ a
di erent training procedure. Speci cally, we consider a deep
triplet network which is composed of three instances of the
AlexNet model where the weights are shared across the
instances. We employ a ranking loss similar to that of [
Considering a triplet network allow us to learn a 1Dembedding space for images, where the triplet ranking loss function enforces an interesting frame to be close by to other interesting frames and far away from the uninteresting ones: L(x; x+; x ) = max(0; D(x; x+) D(x; x ) + M ) (1) where x, x+, x denote the anchor, positive and negative samples given as inputs, D( ) represents the distance between the interestingness scores and M represents the margin. In terms of optimization, one critical point is the triplet selection procedure since we observe that using all possible triplets in the training is costly and might lead to a local minima. For this reason, we use a hard negative mining strategy, which is commonly used in similar works. During training, we only ne-tune the fully connected layers as in our previous models, while all the weights are initialized with AlexNet weights. We interrupted training at the 10000th epoch. 2.4
Thus far, we have described our CNN models which can be used to compute real valued interestingness scores for each key frame of a given video sequence, where these interestingness scores correspond to con dence values. However, the task also requires classifying a frame as interesting or not, in addition to predicting their interestingness scores.
A simple and straightforward way to convert real valued outputs to class labels is to introduce a thresholding procedure. However, choosing a single appropriate threshold value is not easy; in fact, we observe that it is very video sequence dependent. Figure 1 shows the ground truth distributions of the con dence values for the interesting (blue) and uninteresting (orange) frames over all video sequences in the training set. As can be seen, these distributions have a large overlap, demonstrating that a single threshold value won't work.
Next, we analyzed the ratio between interesting and uninteresting key frames per each training video. As shown in Table 1, the ratio is, on average, about 1:9. Hence, given a test video sequence, we sort all its key frames according to their predicted interesting scores and classify the top 10% frames as interesting.
We submit three di erent runs for the image subtask. While the rst run uses our ne-tuned AlexNet, the second one uses predictions from our ne-tuned MemNet model. Lastly, the third run includes the results of our proposed triplet network. All these models are trained by using the provided training data. However, we split it into two as training and validation splits using a ratio of 80% and 20% to deal with over tting. In our experiments, as the size of the training data is relatively small, we decided to update the weights of only the fully connected layers of the pretrained models.
The performances of our models are evaluated by considering the accuracy and the mean average precision (mAP) scores. Table 2 summarizes our results on the test set. As can be seen, the mAP scores of all the proposed models are not very high, demonstrating that interestingness prediction is not a trivial task. We note that the accuracy values being high are somewhat misleading since the training data is highly unbalanced (see Table 1). Hence, we additionally show the confusion matrices for all of our runs in Table 3.
To sum up, the growth of the visual media on the Internet has led to an increased need for understanding and predicting interestingness of images and video shots, and in this work, within the proposed deep models, we treat each key frame of a given video as an independent sample. One possible future direction could be to process each key frame in the context of a local temporal neighborhood or the whole video, by extending our models to process multiple key frames simultaneously. Another extension could be to consider a multi-task learning scheme, which involves jointly classifying key frames as interesting or not and estimating an interestingness score based on a regression-based loss function, which eliminates the need for post-processing the regression scores.
Acknowledgement.
This work is partially supported by the Scienti c and Technological Research Council of Turkey (Award #113E497). 4.