We present a comprehensive evaluation of performance of visual feature representations for MediaEval 2011 - Violent Scenes Detection Task. As for global features, color moments, color histogram, edge orientation histogram, and local binary patterns are used. As for local features, keypoint detectors such as Harris Laplace, Hessian Laplace, Harris A ne, Dense Sampling are used to extract keypoints and SIFT and COLOR SIFT are used as descriptor of the region around these keypoints. The results obtained by our runs are presented. The demo is available at: http://satoh-lab.ex.nii.ac.jp/users/ledduy/Demo-MediaEval
We have developed NII-KAORI-SECODE, a general framework for semantic concept detection, and used it to participate several benchmarks such as IMAGECLEF, MEDIAEVAL, PASCAL-VOC, IMAGE-NET and TRECVID. The purpose is to evaluate performance of various visual feature representations for concept detection-like task. In this framework, rst features are extracted from keyframes, then concept detectors using these features are learned by using SVM with 2RBF kernel. The probability output scores of the learned concept detectors are used for ranking. We consider the Violent Scenes Detection Task [1] as a kind of concept detection task and use NII-KAORI-SECODE framework for evaluation of performance of global and local features. The results show that using global features are more e ective than using local features and combination of global features and local features can help to improve the overall performance. 2.
We evaluate both global features and local features. The global features include color moments, color histogram, edge orientation histogram, and local binary patterns. The local feature is based on the BOW model in which the SIFT descriptor is extracted at interest points detected by Harris Hessian Laplace and multi-scale dense sampling detector. 3. 3.1
Since global features do not capture spatial information, to overcome this problem, a grid n m is usually used to divide the input image into non overlapping sub-regions. The features extracted from these regions are concatenated to form the feature vector for the image. 3.2
Local binary patterns and edge orientation histogram are extracted from gray scale image. For color moments and color histogram, color spaces including HSV, RGB, Luv, and YCrCb are used. 3.3
For color histogram, we only use 8-bin histogram for each channel. For edge orientation histogram, we quantize orientations into histograms of 12+1 bins, 18+1 bins, 36+1 bins, and 72+1 bins. For local binary patterns, we quantize binary patterns into histograms of 10, 30, and 59 bins.
Each combination of feature type, granularity, quantization, and color space forms one feature con guration. The feature con gurations evaluated in this study are described in Table 1.
Local binary patterns (LBP) Edge orientation histogram (EOH)
Local features (harhes, harlap, heslap, hara , hesa , dense, phow csift, sift, oppsift, rgbsift, rgsift) 20 20 15 20 44
3x3 8x3 500 visual words 4.
LibSVM is used to train SVM classi ers. The extracted features are scaled to [0; 1] using the svm-scale tool of LibSVM. The 2RBF kernel is used as similarity measure. The optimal (C; g) parameters for learning SVM classi ers are found by conducting a grid search with 5-fold cross validation on a subset of 3,000 samples strati ed selected from the original dataset.
For each shot, one keyframe is used for training and testing. The set of keyframes are provided by the organizer. To generate training data, shots falling into positive segments are considered as positive shots. The other shots are considered as negative shots. We apply the trained classi er to the keyframes of the test set. The output scores of keyframes are considered as scores of shots and used for ranking. We use the threshold = 0:02 for binary decision.
We submitted 6 runs and the details of performances are shown in Table 2. AED cost is the cost de ned by the task's organizer and MAP is mean average precision. The results show that using global features are more e ective than using local features and combination of global features and local features can help to improve the overall performance.
Figure 1 shows ROC curves of the submitted runs. The details of other runs are available at http://satoh-lab.ex.nii.ac.jp/users/ledduy/Demo-MediaEval.
Using one keyframe per shot for training and testing needs to be reconsidered since in many cases, keyframes are not related to violent scenes. We tried using 10 keyframes per shot for training and testing. However, the performance is even worse than that of using only one keyframe. The reason could be multiple keyframes per shot in training data make large variations. Annotation and ground truth are based on segments while shots are used for experiments. Many shots are very short, e.g. less than 1 second and might be easily classi ed as non-violent shots based on the de nition. Future work is to study how to use multiple keyframes per shot to represent violent scenes. Simple sampling does not work.