This paper describes the participation of the TUB-IRML group to the MediaEval 2014 Violent Scenes Detection (VSD) a ect task. We employ low- and mid-level audio-visual features fused at the decision level. We perform feature space partitioning of training samples through k -means clustering and train a di erent model for each cluster. These models are then used to predict the violence level of videos by employing two-class support vector machines (SVMs) and a classi er selection approach. The experimental results obtained on Hollywood movies and short Web videos show the superiority of mid-level audio features over visual features in terms of discriminative power, and a further enhanced performance resulting from the fusion of audio-visual cues at the decision-level. Finally, the results also demonstrate a performance gain obtained by partitioning the feature space and training multiple models, compared to a unique violence detection model.
The MediaEval 2014 VSD task aims at detecting violent
segments in movies and short Web videos. Detailed
description of the task, the dataset, the ground truth and evaluation
criteria are given in the paper by Sjoberg et al. [
We represent the audio content using mid-level representations, whereas the visual content is represented at two di erent levels: low-level and mid-level.
Mid-level audio representations are based on
MelFrequency Cepstral Coe cient (MFCC) features extracted
from the audio signals of video segments of 0.6 second length.
We experimentally veri ed that the 0.6 second time window
was short enough to be computationally e cient and long
enough to retain su cient relevant information. In order
to generate the mid-level representations, we apply an
abstraction process which uses an MFCC-based Bag-of-Audio
Words approach with a sparse coding scheme. We employ
the dictionary learning technique presented in [
We use motion-related descriptors for the visual
representation of video segments. One of the motion
descriptors is Violent Flow (ViF) which is proposed for real-time
detection of violent crowd behaviors. We compute a ViF
descriptor for each video segment to represent statistics of
ow-vector magnitude changes over time. For a detailed
explanation of the computation of this descriptor, the reader
is referred to [
We also use static content representations. More
speci cally, we employ a ect-related static visual
descriptors. We compute mean and standard deviation of
saturation, brightness and hue in the HSL color space. We also
compute the colorfulness of the keyframe of video segments
using the method in [
Mid-level visual representations are based on
histogram of oriented gradient (HoG) and histogram of oriented
optical ow (HoF) features extracted from the visual content
of video segments of 0.6 second length. HoG and HoF
descriptors are densely sampled and computed for subvolumes
of video segments (HoG descriptors are subsampled every 6
frames and HoF descriptors are subsampled every 2 frames
as recommended in [
Violence Detection Model \Violence" is a concept which can audio-visually be expressed in diverse manners. Therefore, learning multiple models for the \violence" concept instead of a unique model constitutes a more judicious choice. This justi es that we rst perform feature space partitioning by clustering video segments in our training dataset and learn a di erent model for each violence sub-concept (i.e., cluster). We use twoclass SVMs in order to learn violence models (i.e., one SVM for each sub-concept).
In the learning step, the main issue is the problem of imbalanced data. This is caused by the fact that, in the training dataset, the number of non-violent video shots is much higher than the number of violent ones. We choose, in the current framework, to perform random undersampling to balance the number of violent and non-violent samples (with a balance ratio of 1:2).
In the test phase, the main challenge is to combine the classi cation results of the violence models. We perform a classi er selection to solve this. More precisely, we rst determine the nearest cluster to a video segment of the test set using Euclidean distance measures. Once the classi er for the video sample is determined, the output of the chosen model is used as the nal prediction for that video sample.
Table 1 reports the mean Average Precision (MAP2014) and MAP@100 metrics on the Hollywood movie dataset and the Web video dataset. We observe that the mid-level audio representation based on MFCC and sparse coding (Run1 ) provides promising performance and outperforms all other representations (Run2 and Run3 ) that we use in this work. We also note that the performance is further improved by fusing these mid-level audio cues with low- and mid-level visual cues at the decision level by linear fusion (Run4 ). However, the fusion of a ect-related color features with the other audio-visual features does not help improving the performance (Run5 ).
The results on the Web video dataset in terms of MAP metrics even demonstrate superior results compared to the ones obtained on the Hollywood movie dataset. Therefore, we can conclude that our violence detection method generalizes particularly well to other contents, including other types of video content not used for training the models. Another interesting observation is that a ect-related color features seem to provide better results in terms of MAP metrics on the Web video dataset in comparison to the Hollywood movie dataset (Run3 ).
Table 1 also provides a comparison of our methods in terms of MAP2014 and MAP@100 metrics with an SVMbased unique violence detection model (i.e., a model where no feature space partitioning is performed). We can conclude that our method outperforms the SVM-based detection method where the feature space is not partitioned and all violent and non-violent samples are used to build a unique model.
In this paper, we presented an approach based on feature space partitioning for the detection of violent content in movies and short Web videos at the video segment level. We employed low- and mid-level audio-visual features to represent videos. We showed that the mid-level audio representation based on MFCC and sparse coding provides promising performance in terms of MAP2014 and MAP@100 metrics and also outperforms our visual representations. We also fused these mid-level audio cues with low- and midlevel visual cues at the decision level using linear fusion for further improvement and achieved better results than unimodal video representations in terms of the MAP metrics. We observed from the overall evaluation results that our method performs better when violent content is better expressed in terms of audio features (a typical example would be a gun shot scene). Hence, as a future work, we need to extend/improve our visual representation set with more discriminative representations. Another possibility for future work is to further investigate the feature space partitioning concept and optimize the distribution or number of subconcepts in order to enhance the classi cation performance of our method.
The research leading to these results has received funding from the European Community FP7 under grant agreement number 261743 (NoE VideoSense).