The Violent Scenes Detection task aims at evaluating algorithms that automatically localize violent segments in both Hollywood movies and short web videos. The de nition of violence is subjective: \the segments that one would not let an 8 years old child see in a movie because they contain physical violence". This is a highly challenging problem because of the strong content variations among the positive instances. In this year's evaluation, we adopted our recently proposed classi cation method to fuse multiple features using Deep Neural Networks (DNN). The method was named regularized DNN. We extracted a set of visual and audio features, which have been observed useful. We then applied the regularized DNN for feature fusion and classi cation. Results indicate that using multiple features is still very helpful, and more importantly, our proposed regularized DNN offers signi cantly better results than the popular SVM. We achieved a mean average precision of 0.63 for the main task and 0.60 for the generalization task.
Three kinds of audio-visual features were extracted, which have been observed useful in 2013.
We extracted trajectory-based motion features according
to our previous work [
F O H V F H B M V F
e p a h S j a r T V F
G O H F M j a r T F O H F M j a r T H B M F M j a r T 5
3
4
2
ture. In total, there are seven trajectory-based features,
including four baseline FV and three dimension-reduced
TrajMF features. See [
The other two kinds of features include Space-Time
Interest Points (STIP) [
We adopted both SVM and deep neural networks (DNN) for classi cation.
SVM: 2 kernel was adopted for the bag-of-words features (STIP and MFCC), and linear kernel was used for the others. For feature fusion, kernel-level average fusion was used for the trajectory-based features, while score-level average late fusion was adopted to combine trajectory features with STIP and MFCC.
WL−1 n,3 l = L l = F l = E ......
......
...... x n,1 x n,2
DNN: We also adopted a new DNN-based classi er
proposed in our recent work [
Temporal score smoothing has been proved to be effective as incorrect predictions on a short clip may be eliminated by considering predictions on nearby clips. All the videos were rst partitioned uniformly into 3-second long clips. A smoothed prediction score of a clip is simply the average value of the scores in a three-clip window.
As we need to output segment level predictions (not on the xed-length clip-level), we need to merge continuous clips if they are all determined to contain violence or no violence. This was done if their violence scores were all above or below a threshold, and the new score of the merged segment was set to be the average value of clips.
We submitted 5 runs for official evaluation. As shown in Figure 1, Run 1 and Run 2 used SVM and DNN respectively. Run 2 did not use FV encoding of the HOG, HOF and MBH features, as the dimensionality of these three features are too high, which would jeopardize the performance of DNN when there is insufficient training data. Run 3 is the score fusion of Run 1 and Run 2. Run 4 is the score-smoothed version Feature Fusion
WE
1 Feature Extraction 0% 0.5% of Run 3 (smoothing was performed before merging), while Run 5 is the direct fusion of SVM and DNN without using any smoothing and merging functions.
The official results are summarized in Figure 3. We see that, although some features were not used in DNN, the performance of DNN (Run 2) is still signi cantly better than SVM. This clearly con rms the effectiveness of deep networks. Directly fusing DNN and SVM incurs a small performance drop (Run 3). This may be due to the sub-optimal parameters used in the fusion process. Another fusion setting (Run 5) without using score merging improves the main task performance but still hurts the result of the generalization task, showing that DNN has better generalization capability than the SVM, and thus fusing SVM with DNN will always degrade the performance of the generalization task. Finally, the results of Run 4 indicate that both smoothing and merging are useful for the main task. It is not surprising that smoothing does not work for the generalization task, because, compared with the long movies used in the main task, the test clips are short and are relatively temporally more consistent.