TUD-MMC at MediaEval 2016: Predicting Media Interestingness Task Cynthia C. S. Liem Multimedia Computing Group, Delft University of Technology Delft, The Netherlands c.c.s.liem@tudelft.nl ABSTRACT Data MAP video ground truth on image set 0.1747 This working notes paper describes the TUD-MMC entry to the image ground truth on video set 0.1457 MediaEval 2016 Predicting Media Interestingness Task. Noting that the nature of movie trailer shots is different from that of pre- Table 1: MAP values obtained on development set by swapping ceding tasks on image and video interestingness, we propose two ground truth annotations of image and video. baseline heuristic approaches based on the clear occurrence of peo- ple. MAP scores obtained on the development set and test set sug- gest that our approaches cover a limited but non-marginal subset of the interestingness spectrum. Most strikingly, our obtained scores 2. CONSIDERATIONS on the Image and Video Subtasks are comparable or better than In designing our current method, several considerations coming those obtained when evaluating the ground truth annotations of the forth from the task setup and provided data were taken into account. Image Subtask against the Video Subtask and vice versa. First of all, interestingness assessments only considered pairs of items originating from the same trailer. Therefore, given our cur- 1. INTRODUCTION rent data, scored preference between items can only meaningfully The MediaEval 2016 Predicting Media Interestingness Task [3] be assessed within the context of a certain trailer. As a conse- considers interestingness of shots and frames in Hollywood-like quence, we choose to only focus on ranking mechanisms restricted trailer videos. The intended use case for this task would be to auto- to a given input trailer, rather than ranking mechanisms that mean- matically select interesting frames and/or video excerpts for movie ingfully can rank input from multiple trailers. previewing on Video on Demand web sites. Secondly, the use case behind the currently offered task consid- Movie trailers are intended to raise a viewer’s interest in a movie. ered helping professionals to illustrate a Video on Demand (VOD) As a consequence, they will not be a topical summary of the video, web site by selecting interesting frames and/or video excerpts of and they are likely to be constituted by ‘teaser material’ that should movies. The frames and excerpts should be suitable in terms of make a viewer curious to watch more. helping a user to make a decision on whether to watch a movie or In our approach to this problem, we originally were interested in not. As a consequence, we assume that selected frames or excerpts assessing whether ‘interestingness’ could relate to salient narrative should not only be interesting, but also representative with respect elements in a trailer. In particular, we wondered whether criteria to the movie’s content. for connecting production music fragments to storylines [5] would Thirdly, the trailer is expected to contain groups of shots (which also be relevant factors in rater assessment of interestingness. may or may not be sequentially presented) originating from the However, the rating acquisition procedure for the task did not same scenes. involve full trailer watching by the raters, but rather the rating of Finally, binary relevance labels were no integral part of the rating isolated pairs of clips or frames. As such, while ideas in [5] largely procedure, but added afterwards. As a consequence, finding an considered the dynamic unfolding of a story, a sense of overall sto- appropriate ranking order will be more important in relation to the ryline and longer temporal dynamics could not be assumed in the input data than providing a correct binary relevance prediction. current task. When manually inspecting the ground truth annotations, we We ultimately decided on pursuing a simpler strategy: the cur- were struck by the inconsistency between ground truth rankings rently presented approaches investigate to what extent the clear on the Image Subtask vs. that obtained for the Video Subtask. To presence of people, as approximated by automated face detection quantify this inconsistency, given that annotations were always pro- results, indicate visual environments which are more interesting to vided considering video shots as individual units (so there were as a human rater. The underlying assumption is that close-ups should many items considered per trailer in the Image Subtask as in the attract a viewer’s attention, and as such may cause larger empathy Video Subtask), we mimicked the evaluation procedure for the case with the shown subject or its environment. It will be interesting ground truth would be swapped. In other words, we computed the to consider to what extent this currently proposed heuristic method MAP value for the Image Subtask in case the ground truth of the will compare against more agnostic direct machine learning tech- Video Subtask (including confidence values and binary relevance niques on the provided labels. indications) would have been a system outcome, and vice versa. Results are shown in Table 1: it can be noted the MAP values are indeed not high. As we will discuss at the end of the paper, this Copyright is held by the author/owner(s). phenomenon will be interesting to investigate further in future con- MediaEval 2016 Workshop, Oct. 20-21, 2016, Hilversum, Netherlands. tinuations of the task. 3. METHOD Run name MAP As mentioned, we assess interestingness on the basis of (clearly) image_hist 0.1867 visible people. We do this for both Subtasks, and simplify the no- image_histface 0.1831 tion of ‘visible people’ by employing face detection techniques. video_hist 0.1370 While these techniques are not perfect (and false negatives, or video_histface 0.1332 missed faces, are prevalent), it can safely be assumed that when a face is detected, the face will be clearly recognizable to a human Table 2: MAP values obtained on development set. rater. Both for the Image and Video Subtask, we follow a similar strat- Run name MAP egy, which can be described as follows: image_hist 0.2202 image_histface 0.2336 1. Employ face detectors to identify those image frames that video_hist 0.1557 feature people. For each of these, store bounding boxes for video_histface 0.1558 all positive face detections. Table 3: Official task evaluation results: MAP values obtained 2. In practice, the amount of frames with detected faces is rel- on test set. atively low. Assuming that frames in which detected faces occur are part of scene(s) in the trailer which are important (and therefore may contain representative content of inter- We sort the obtained confidence values, and apply an (empirical) est), we consider the set of all frames with detected faces, threshold to set binary relevance. For the hist run, all items with and calculate the mean HSV histogram Hf over it. a confidence value higher than 0.75 are deemed interesting; for the histface run, the threshold is set at 0.6. 3. For each shot s in the trailer, we consider its HSV histogram Hs and calculate the histogram intersection between Hs and 3.2 Video Subtask Hf as similarity value: For the Video Subtask, in parallel to our approach for the Image |Hf |−1 Subtask, we consider HSV color histograms and face detections. X sim(Hs , Hf ) = min(Hs (i), Hf (i)). For this, we can make use of released precomputed features. How- i=0 ever, in contrast to the Image Subtask, these features now are based on multiple frames per shot. 4. Normalize the similarity scoring range to the [0, 1] interval In case of the HSV color histograms [4], we take the average to obtain confidence scores. The ranking of shots according histogram per shot as representation. For face detection, we use the to these scores will be denoted as hist. face tracking results based on [1] and [2], and consider the sum of all detected face bounding box areas per shot. 5. Next to considering histogram intersection scores, for each The binary relevance threshold is set at 0.75 for the hist run, shot, we consider the bounding box area of detected faces. and at 0.55 for the histface run. If multiple faces are detected within a shot, we simply sum areas. 4. RESULTS AND DISCUSSION 6. The range of calculated face areas also is scaled to the [0, 1] Results of our runs as obtained on the development and test set interval. are presented in Tables 2 and 3, respectively. The results on the test 7. For each shot, we take the average of the normalized set constitute the offical evaluation results of the task. histogram-based confidence score and the normalized face Generally, it can be noted that MAP scores are considerably area score. These averages are again scaled to the [0, 1] in- lower for the Video Subtask than for the Image Subtask. Also look- terval, establishing an alternative confidence score which is ing back to the results in Table 1, it may be hypothesized that the boosted by larger detected face areas. The ranking of shots Video Subtask generally is more difficult than the Image Subtask. according to these scores will be denoted as histface. We would expect for temporal dynamics and non-visual modali- ties to play a larger role in the Video Subtask; aspects we are not Both for the Image and Video Subtask, we submitted a hist considering yet in our current approach. and histface run. Below, we give further details on what feature When comparing the obtained MAP against the scores seen in detectors and implementation details were used per subtask. Table 1, we notice that our scores are comparable, or even better. Furthermore, comparing results for the test set vs. the development 3.1 Image Subtask set, we see that scores slightly improve for the test set, suggest- For the Image Subtask, each shot is represented by a single ing that our modeling criteria were indeed of certain relevance to frame. The HSV color histograms for each frame are taken out ratings in the test set. of the precomputed features for the image dataset [4]. For future work, it will be worthwhile to further investigate how No face detector data was available as part of the provided universal the concept of ‘interestingness’ is, both across trailers, dataset. Therefore, we computed detector outcomes ourselves, us- and when comparing the Image Subtask to the Video Subtask. The ing the head detector as proposed by [7], and employing a detection surprisingly low MAP scores when exchanging ground truth be- model as refined in [6]. The features were computed employing the tween Subtasks may indicate that human rater stability is not opti- code released by the authors1 . This head detector does not require mal, and/or that the two Subtasks are fundamentally different from frontal faces, but also is designed to detect profile faces and the one another. Furthermore, as part of the quest for a more specific back of heads, making it both flexible and robust. definition of ‘interestingness’, a continued discussion on how inter- 1 estingness can be leveraged for a previewing-oriented use case will http://www.robots.ox.ac.uk/~vgg/software/ headmview/ also be useful. 5. REFERENCES [1] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2005. [2] M. Danelljan, G. Häger, F. Shahbaz Khan, and M. Felsberg. Accurate scale estimation for robust visual tracking. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014. [3] C.-H. Demarty, M. Sjöberg, B. Ionescu, T.-T. Do, H. Wang, N. Q. Duong, and F. Lefebvre. MediaEval 2016 Predicting Media Interestingness Task. In Proc. of the MediaEval 2016 Workshop, Hilversum, The Netherlands, October 2016. [4] Y.-G. Jiang, Q. Dai, T. Mei, Y. Rui, and S.-F. Chang. Super Fast Event Recognition in Internet Videos. IEEE Transactions on Multimedia, 177:1–13, 2015. [5] C. C. S. Liem, M. A. Larson, and A. Hanjalic. When Music Makes a Scene — Characterizing Music in Multimedia Contexts via User Scene Descriptions. International Journal of Multimedia Information Retrieval, 2:15–30, 2013. [6] M. Marin-Jimenez, A. Zisserman, M. Eichner, and V. Ferrari. Detecting People Looking at Each Other in Videos. International Journal of Computer Vision, 106(3):282–296, February 2014. [7] M. Marin-Jimenez, A. Zisserman, and V. Ferrari. Here’s looking at you, kid." Detecting people looking at each other in videos. In British Machine Vision Conference, 2011.