This paper presents the work done in Technicolor and INRIA regarding the A ect Task at MediaEval 2011. This task aims at detecting violent shots in movies. We studied a bayesian network framework, and several ways of introducing temporality and multimodality in the framework.
Audio modality Five audio features were extracted from the audio stream from 40 msec frames with 20 msec overlap. These features were then averaged over video shots, as the task is to be performed at the shot level. The audio features are the energy, the centroid, the asymmetry, the zero crossing rate (ZCR) and the atness.
This work was partly achieved as part of the Quaero Program, funded by OSEO, French State agency for innovation.
K2 [
We used the Bayes Net Toolbox1.
Considering the temporal structure of movies we decided to try several types of temporal integrations: we used contextual features2 over n 2 [ 5; +5], and two types of temporal ltering over 5 samples, that are used to smooth decisions: decision maximum vote This intervenes once the decision has been taken and consists in taking the maximum decision over a few samples. 1http://code.google.com/p/bnt/ 2Considering Xt = [x1; ; xK ] the feature vector for sample at time t, the contextual features vector becomes Xtc = [Xt n; ; Xt; ; Xt+n].
Description A: Me V: Ma A: Me V: Me
Ma Me Me Me Ma Ma Ma Ma
MC 0.761 0.774 0.784 0.805 0.840 0.843 0.892 0.943 0.950 0.967 0.998 1.009
F 0.397 0.391 0.305 0.295 0.297 0.354 0.284 0.268 0.255 0.251 0.266 0.276 probability averaging This intervenes before taking the decision, by directly averaging the samples probabilities of being violent.
As for multimodal fusion, two cases were considered: late fusion and early fusion. For early fusion, we simply fused the features from both modalities before learning, while for late fusion, we fused the probability of both modalities for the ith shot si using: 8 max(Pvsai ; Pvsvi ) if both are violent < min(Pvsai ; Pvsvi ) if both are non violent : Pvsai Pvsvi otherwise Pfsuised(Pvsai ; Pvsvi ) = (1) where Pvsai (respectively Pvsvi ) is the probability of being violent for the audio (respectively video) modality for the ith shot.
This rule gives high scores when both audio and video nd a violent segment, a low score if they both do not and an intermediate score if only one answers yes.
This section describes the runs submitted to the MediaEval 2011 A ect Task. For the audio and video experiments, we chose to submit the two best runs according to the MediaEval cost and to the false alarm vs missed curve using cross validation on the learning set, while for the multimodal runs (namely, early and late fusion), we chose the best ones according to both metrics. The selected runs and their results are presented in table 1.
Most of the obtained scores have values lower than < 1 which is better than the simple case where each sample is classi ed as violent, i.e. false alarm rate is 100% and missed detection rate is 0%.
The analysis of the produced graphs yields nice and encouraging observations on the quality of the structure learning algorithms. Firstly, the links between features may be easily interpreted. The ZCR and centroid are linked as they represent the same information, the activity is linked to the shot length as the shot detector used tends to oversegment when the activity is high, and nally blood is not connected to violence, which seems logical considering that the presence of blood in the violent scenes highly depends on the movie and the chosen de nition for violence. As for EF, while we thought it would improve the results and nd correlations between audio and video, it seems that for non contextual data, only the video features are linked to the violence node, and that for contextual data the links are messy. This and the better results obtained using LF tend to indicate that the features used in both modalities are from di erent levels and cannot be compared as such. Secondly, it seems that the algorithms produce a strict temporal structure, i.e. the features from time t = n are linked together and not to features from di erent times unless they are in chains. There are four chains in the graphs: atness, energy, activity and blood. It is easy to see that these features have a temporal structure. On the other hand, the ash feature is connected only to the violence node and forms no chain, which is again logical as the ash feature only detects high luminance variations, and has therefore no well de nite temporal structure. The use of contextual seems to provide good and promising results, which tends to con rm the importance of the temporal structure of movies. The depth used for this evaluation has been chosen arbitrarily, however it should be interesting to also consider other depths. On the downside, it seems that these results depend on the algorithm used for learning the BN structures: FAN and non contextual data seem to work better, while K2 and contextual data seem to give the best results.
This concludes the preliminary analysis that can be inferred from the evaluation.
This paper presents a simple framework based on temporal integration, multimodality and Bayesian network. First, it is experimentally shown that the structure learning algorithm output logical graph with respect to the provided data: they are able to capture the links between features and provide a coherent temporal structure. It is also shown that early fusion with features that have di erent nature yields to poor results, while late fusion seems to be more promising. Second, the use of contextual data seems to improve the result. This work provides a promising baseline for future work on the subject. We have several improvement ideas. We want to add features from the text modality, as we think it also contains important information on the violent nature of the video shots. We also want to investigate more the contextual data and test other structure learning algorithms.