The MediaEval 2014 Violent Scenes Detection task challenged participants to automatically nd violent scenes in a set of videos. We propose to rst predict a set of midlevel concepts from low-level visual and auditory features, then fuse the concept predictions and features to detect violent content. With the objective of obtaining a higly generic approach, we deliberately restrict ourselves to use simple general-purpose descriptors with limited temporal context and a common neural network classi er. The system used this year is largely based on the one successfully employed by our group in 2012 and 2013, with some improvements and updated features. Our best-performing run with regard to the o cial metric received a MAP2014 of 45.06% in the main task and 66.38% in the generalization task.
The MediaEval 2014 Violent Scenes Detection task [
Our system this year is largely based on the one
successfully employed by us in 2012 [
Our system builds on a set of visual and auditory features, employing the same type of neural network classier at di erent stages to obtain a violence score for each frame of an input video. First, we perform feature extraction at the frame level. The resulting data is then fed into a multi-classi er framework that operates in two steps. The rst step consists of training the system using ground truth data. Training is performed at two levels. At mid-level, a bank of classi ers is trained using ground truth related to concepts that are usually present in the violent scenes, e.g., presence of \ re", presence of \gunshots", or \gory" scenes. Then, high-level violence detection is ensured by a nal classi er that is fed either with the previous concept predictions and/or the low-level content descriptors. The violence classi er is also trained on the provided ground truth for the violent segments. The nal step consists of classifying the new unlabeled data (e.g. the test set) which is achieved by employing the previously trained multi-classi er framework. These steps are detailed in the following. 2.1
Visual (225 dimensions): For each video frame, we
extract several standard color and texture-based descriptors,
such as: Color Naming Histogram, Color Moments, Local
Binary Patterns, Color Structure Descriptor, and Gray Level
Run Length Matrix. Also, we compute the Histogram of
Oriented Gradients, that exploits the local object appearance
and shape within a frame by using the distribution of edge
orientations. For a more detailed description of the visual
features, see [
Auditory (29 dimensions): In addition, we extract a set of low-level auditory features: amplitude envelop, rootmean-square energy, zero-crossing rate, band energy ration, spectral centroid, spectral ux, bandwidth, and Mel-frequency cepstral coe cients. We compute the features on frames of 40 ms without overlap to make alignment with the 25-fps video frames trivial. 2.2
For classi cation, we use multi-layer perceptrons with a single hidden layer of 512 units and one or multiple output units. All units use the logistic sigmoid transfer function. The input data is normalized by subtracting the mean and dividing by the standard deviation of each input dimension. Training is performed by backpropagating cross-entropy error, using random dropouts to improve generalization. We follow the dropout scheme of [2, Sec. A.1] with some minor modi cations to the parameters.
For the concept training set of 18 movies, each video frame
was annotated with the 10 di erent concepts as detailed in
[
The nal violence predictor is trained using both low-level features and all mid-level concept predictions as inputs. For comparison, we also train classi ers to predict violence just from the features or just from the concepts.
Training the violence detector requires inputs that are similar to those that will be used in the testing phase, thus using the concept ground-truth for training will not work. Instead we use the concept prediction cross-validation outputs on the training set (see previous section) as a more realistic input source { in this way the system can learn which concept predictors to rely on.
The nal violence prediction score is generated by applying a sliding median lter as temporal smoothing. We used a lter length of 5 seconds (125 frames), this was selected from experimenting in the training set. The nal detection as violent or non-violent is generated by thresholding the prediction score. The thresholds were determined by maximizing the MAP2014 performance measure in the training set using cross-validation.
We submitted ve runs for both the main task and the generalization task. Table 1 details the results for all our runs. The rst ve lines show our runs submitted to the main task, the next ve lines are those for the generalization task. The second column indicates which input features were used, 'a' for auditory, 'v' for visual, and 'c' for concept predictions. Multiple feature modalities indicate that they were integrated using early fusion. For the main task, the auditory features achieved the highest MAP2014 result. Concept detectors and visual features performed poorly in the main task, and fusing them with the auditory features did not improve the results above the audio-only result. In contrast, in the generalization task all combinations perform similarly, except for the concepts which have a clearly better result.
Another observation is that all results have a strong imbalance between precision and recall. Our analysis indicates that this is not due to a poor selection of the violence judgment threshold (in fact our thresholds are relatively close to optimal), but instead due to the new MAP2014 measure favoring high recall.
Table 2 shows the movie speci c results for each of our main task runs. Interestingly auditory features perform particularly well on the anime movie \Ghost in the Shell", while the visual features perform strongly on \8 Mile", a drama movie with more realistic violence such as st ghts etc. \Jumanji" and \Braveheart" are the two movies with the poorest results. This can perhaps be explained by the fact that they di er from the training set more than the other movies. In particular \Braveheart" depicts brutal medieval ghts, which are not represented in the training set. 4.
Our results show that violence detection can be done well using general-purpose features and generic neural network classi ers, without engineering domain-speci c features. The selection of feature modalities is highly dependent on the type of material, for Hollywood-style movies auditory features performed best, while concepts are useful for the more mixed style found in YouTube videos. Based on the results, we can also conclude that our violence detection framework generalises well: even though it was trained on only feature length movies it performs accurate violence detection on YouTube videos as well.
We received support by the Academy of Finland, grants no. 255745 and 251170, ESF POSDRU/159/1.5/S/132395 InnoRESEARCH programme, the EU-FP7 project no. 601166 and the Austrian Science Fund (FWF): P22856 and P25655.