Figure 1 gives an overview of our system. In this short paper, we briefly describe each of the key components. For the task definition, data and evaluation metric, interested readers may refer to [1].
Three kinds of audio-visual features were extracted, which have been observed useful in 2013.
We extracted trajectory-based motion features according to our previous work [2]. A main difference is that the new improved dense trajectories (IDT) [4] were used as the basis to replace the original dense trajectories. Four baseline features, histograms of oriented gradients (HOG), histograms of optical flow (HOF), motion boundary histograms (MBH) and trajectory shape (TrajShape) descriptors were computed. These features were encoded using the Fisher vectors (FV) with a codebook of 256 codewords. We further computed our proposed TrajMF [2] based on the HOG, HOF and MBH, by considering the motion relationships of the trajectories. As the dimension of the original TrajMF is very high, we employed the expectation-maximization principal component analysis (EM-PCA) [3] for dimension reduction, generating a 1500-dimensional representation for each fea- ture. In total, there are seven trajectory-based features, including four baseline FV and three dimension-reduced Tra-jMF features. See [2] for more details.
The other two kinds of features include Space-Time Interest Points (STIP) [5] and Mel-Frequency Cepstral Coefficients (MFCC). The STIP describes the texture and motion features around local interest points, which were encoded using the bag-of-words framework with 4000 codewords. Here we randomly sampled 300k features and used k-means to generate the codebook. The MFCC is a very popular audio feature. It was extracted from every 32ms time-window with 50% overlap. The bag-of-words was also adopted to quantize the MFCC descriptors, using 4000 codewords.
We adopted both SVM and deep neural networks (DNN) for classification.
SVM: χ 2 kernel was adopted for the bag-of-words features (STIP and MFCC), and linear kernel was used for the others. For feature fusion, kernel-level average fusion was used for the trajectory-based features, while score-level average late fusion was adopted to combine trajectory features with STIP and MFCC.
Feature Extraction
We also adopted a new DNN-based classifier proposed in our recent work [6,7]. The aforementioned fusion methods used for the SVM classifiers neglect the hidden patterns shared among the different features. To capture the relationships of distinct features, we constructed a regularized DNN for video classification. Specifically, as shown in Figure 2, in the regularized DNN, a layer of neurons were first used to perform feature abstraction separately for each input feature. After that, another layer was used for feature fusion with carefully designed structural-norm regularization on network weights, which can identify feature relationships. Finally, the fused representation was used to build a classification model in the last layer. With this special network, we are able to fuse features by considering both feature correlation and feature diversity, as well as perform classification simultaneously. See [6,7] for more details.
Temporal score smoothing has been proved to be effective as incorrect predictions on a short clip may be eliminated by considering predictions on nearby clips. All the videos were first partitioned uniformly into 3-second long clips. A smoothed prediction score of a clip is simply the average value of the scores in a three-clip window.
As we need to output segment level predictions (not on the fixed-length clip-level), we need to merge continuous clips if they are all determined to contain violence or no violence. This was done if their violence scores were all above or below a threshold, and the new score of the merged segment was set to be the average value of clips.
We submitted 5 runs for official evaluation. As shown in Figure 1, Run 1 and Run 2 used SVM and DNN respectively. Run 2 did not use FV encoding of the HOG, HOF and MBH features, as the dimensionality of these three features are too high, which would jeopardize the performance of DNN when there is insufficient training data. Run of Run 3 (smoothing was performed before merging), while Run 5 is the direct fusion of SVM and DNN without using any smoothing and merging functions.
The official results are summarized in Figure 3. We see that, although some features were not used in DNN, the performance of DNN (Run 2) is still significantly better than SVM. This clearly confirms the effectiveness of deep networks. Directly fusing DNN and SVM incurs a small performance drop (Run 3). This may be due to the sub-optimal parameters used in the fusion process. Another fusion setting (Run 5) without using score merging improves the main task performance but still hurts the result of the generalization task, showing that DNN has better generalization capability than the SVM, and thus fusing SVM with DNN will always degrade the performance of the generalization task. Finally, the results of Run 4 indicate that both smoothing and merging are useful for the main task. It is not surprising that smoothing does not work for the generalization task, because, compared with the long movies used in the main task, the test clips are short and are relatively temporally more consistent.
This work was supported in part by a National 863 Program (#2014AA015101), the National Natural Science Foundation of China (#61201387), and the Science and Technology Commission of Shanghai Municipality (#13PJ1400400, #13511504503, #12511501602).