We present a comprehensive evaluation of shot-based visual and audio features for MediaEval 2013 - Violent Scenes Detection A ect Task. To obtain visual features, we use global features, local SIFT features and motion features. For audio features, the popular MFCC is employed. Besides that, we also evaluate the performance of mid-level features which is constructed using visual concepts. We combined these features using late fusion. The results obtained by our runs are presented.
We have developed NII-KAORI-SECODE, a general framework for semantic concept detection, and used it to participate in several benchmarks such as IMAGECLEF, MEDIAEVAL, PASCAL-VOC, IMAGE-NET and TRECVID. In this year, we evaluate performance for concept detectionlike task using shot-based feature representations only. Our previous works show that using the shot-based features not only reduce the computational cost but also improve the performance.
We consider the Violent Scenes Detection (VSD) Task [
We use the same framework for evaluating both objective and subjective tasks (just di erent annotations). Our results show that the combined runs using all visual, audio and midlevel features achieved the best performance. 2.
We use feature from di erent modalities to test if they are complementary for violent scenes detection. Currently, we have developed our VSD system to incorporate still image feature, motion feature, and audio feature. 2.1
We use both global and local features for VSD because they capture di erent characteristics of images. For global feature, we use Color Histogram (CH), Color Moment (CM), Edge Oriented Histogram (EOH), and Local Binary Pattern (LBP). For local feature, we use popular SIFT with both Hessian Laplace interest points and dense sampling at multiple scales. For dense sampling, besides the standard SIFT descriptor, we also use Opponent-SIFT and C-SIFT. For interest point detector, we only use normal SIFT descriptor. We also employed the bag-of-words model with a codebook size of 1000 and the soft-assignment technique to generate a xed-dimension feature representation for each keyframe. Beside encoding the whole image, we also divided it into grids of 3x1 and 2x2 to encode spatial information. Finally, in order to generate a single representation for each shot, we employed two pooling strategies: average pooling and max pooling.
Trajectories are obtained by tracking the densely sampled points in the optical ow elds. We use Motion Boundary Histogram (MBH) to describe each trajectory. This feature descriptor is know to perform well for handling camera motion. For motion feature we use Fisher vector encoding after reducing feature dimension using PCA. The codebook size is 256, trained using a Gaussian Mixture Model (GMM). The nal feature dimension is 65,536. 2.3
We use the popular MFCC for extracting audio feature. We choose a length of 25ms for audio segments and a step size of 10ms. The 13d MFCCs along with each rst and second derivatives are used for representing each audio segment. Raw MFCC features are also encoded using Fisher vector. We use GMM to build the codebook with 256 clusters. We also apply PCA to reduce feature dimension, resulting feature descriptors of 12,288 dimensions.
Beside low-level features, we also investigate how to use related violent information as mid-level feature to detect violent scenes. We use only seven violent concepts to create attributes: re, rearms, cold arms, car chase, gore, blood, and ght. We use low-level image feature to train the attribute classi ers on the VSD development set of 2011. For each image, we apply these attribute classi ers to get score values corresponding to each attribute. After that, we concatenate all these values to form the mid-level representation of each image. We then train our mid-level classi er on the VSD development and test set of 2012. Finally, this classier is used for testing on this year's set. The detailed work ow is shown in Figure 1.
LibSVM is used for training and testing at shot level (based on shot boundaries provided by the organizers). To generate training data, shots which fall into positive segments more than 80% will be considered as positive shots. The remaining shots are considered as negative. Extracted features are scaled to [0; 1] using the svm-scale tool of LibSVM. For still image features, we use a chi-square kernel to calculate the distance matrix. For audio and motion feature, which are encoded using sher vector, a linear kernel is used. The optimal gamma and cost parameters for learning SVM classi ers are found by conducting a grid search with 5-fold cross validation on the training dataset.
We employ a simple late fusion strategy on the aforementioned low-level and mid-level features, giving equal weights to the di erent factors. We submitted ve runs in total: (R5) Fusion of all 4 global features and 5 local features; (R4) Fusion of motion feature (dense trajectories + MBH) and audio feature (MFCC); (R3) The run using mid-level feature; (R2) Fusion of R4 and R5; and (R1) Fusion of R3, R4 and R5.
The detailed performance for each submitted run is shown in Figure 2. We report the performance of both objective and subjective tasks. For each task, we report two evaluation metrics: overall MAP and MAP100, which is the MAP at top 100 return shots. Our best run is the fusion run of all global, local, motion and audio feature (R1). This observation con rm the bene t of combining multiple features for violent scenes detection. Among all submitted runs, the run using mid-level (R3) performs the worst. However, it can be complementary for combining with other low-level features (R1). The combined run using motion feature and audio (R4) feature did not achieve good results as expected. In fact, its performance is lower than the combined run of still image features (R5). This can be due to minor motion in each shots and/or noise in audio signals.
Our future study includes investigating the contribution of motion features and audio features. The result of mid-level features is also promising. Currently, we only use 7 visual concepts for constructing mid-level features. In the future, we will incorporate audio concepts using audio feature.
This research is partially funded by Vietnam National University Ho Chi Minh City (VNU-HCM) under grant number B2013-26-01.