Audio signal processing is moving towards detecting and/or defining rare/anomalous sounds. The application of such an anomaly detection problem can be easily extended to audio surveillance systems. Thus, a rare sound event detection method for road trafic monitoring is proposed in this paper, including detection of hazardous events, i.e., road accidents. The method is based on combining anomaly detection techniques, such as variational autoencoders (VAE) and Interval-valued fuzzy sets. The VAE is used to calculate the reconstruction error of the input audio segment. Based on this reconstruction error, a fuzzy membership function, composed of an optimistic/upper component and a pessimistic/lower component, is calculated. Finally, a probabilistic method for interval comparison is used to calculate the membership score, hence to evaluate the interval-valued fuzzy sets. Finally, classification into anomalous/normal events is obtained by defuzzification. Results show that with a careful parameter setting, the proposed method outperforms the state-of-the-art one-class SVM for anomaly detection.
Anomaly/outlierness/novelty can be defined in diferent ways [
This problem can be formalized either as a classification task for all perceived events, or as detection of only anomalous/outlier/novel events. In either case, two major issues make this task dificult: first, background noise that fully or partly masks all events, making the resulting signals highly variable; secondly, the rareness of the “interesting” events, such as car accidents, which makes them more dificult to model accurately for scarcity of data.
This implies that not only classes are fuzzy, but the membership itself to any class is afected
by a degree of uncertainty. In this case, interval-valued fuzzy sets [
We state the problem as a classification task based on generative models where the final
decision is taken by comparing the inferred interval-valued memberships to the diferent classes,
using a classical metric of interval comparison, named degree of preference [
SED is a relatively young discipline, that has emerged since nearly a decade. Sound recognition methods in general proceed by segmenting signals into fixed-length, possibly overlapping frames of relatively short duration (fractions of a second). For anomalous SED, anomaly detection and supervised/unsupervised recognition methods are then applied on the obtained, fixed-size feature vectors.
Several methods have been built around generative models, such as hidden Markov models
using Gaussian mixture models. Examples of this approach are Ntalampiras et al. [
Unsupervised learning has often been preferred to cope with the issues described.
Selfsupervised neural networks, such as autoencoders, are well suited to this task. We can mention
Wei et al. [
As mentioned, the method uses multiple generative models that learn individual classes, and
compares interval-valued memberships by using the degree of preference. It proceeds as follows:
• In the training phase, a dedicated VAE model is learnt on each subset containing only
one type of events, i.e. Normal or Anomalous.
• In the test phase, the RMSE error is calculated between the input, i.e. the feature vector
representing the signal, and the reconstructed output of each VAE model.
• For each input signal , the output error , of each VAE (1 ≤ ≤ and 1 ≤ ≤ , for
samples and classes) is used to compute a fuzzy membership function, that provides
a measure of closeness of the signal to the event class on which the VAE model had been
trained. In our case, for each input sample = 2 interval membership functions are
computed, corresponding to the Normal category and the Anomalous one.
• The membership function associated to each event category, i.e. Normal/Anomalous,
is composed of a low/pessimistic component and an upper/optimistic component,
respectively. The values of both components form the interval-valued fuzzy membership
function interval (cf. Figure 2).
• Finally, interval comparison is applied using a probabilistic method [
The variational autoencoder (VAE) is a reconstruction network learning a compressed representation of the input to reconstruct the output. The encoding layer stores the parameters of a probability distribution, e.g., mean and variance, representing the input in a latent space. Then, the decoder uses the probability distribution to generate an approximated reconstruction of the input data. Hence the encoder approximates the probability distribution of the identity function. Given a feature vector , the VAE aims to find the probability of with respect to its representation ,
∫︁ () =
(|) () .
The network has parameters of () (average and variance) as its hidden parameters. Using variational inference on a maximum likelihood ojective, the encoder output is trained so that its probability approximates (|). The reconstruction RMSE can then be obtained as follows: = √︂ ∑︀=1( − ′)2 , (1) (2) where and ′ ( = 1, . . . , ) are the input and the output feature vectors for each autoencoder. To compensate for class imbalance, a priori class probabilities are used to compute thresholds.
In the present work, the VAE employs convolutional layers. The input features are extracted
from the spectrogram, i.e. Mel-frequency cepstral coeficients (MFCC) and log-Energy, with
their first and second derivatives ( Δ and Δ-Δ). The choice of these features is motivated by
their proved performance in the state-of-the-art methods of SED [
The membership of each input signal to each event is computed from on the corresponding VAE’s output error , , and its value is the interval between two membership components: a) Pessimistic/Lower membership , , minimum when the sample is an outlier w.r.t. class , i.e. , > , and b) Optimistic/Upper membership , , maximum when the sample is classified in class , i.e. , < (cf. (3)). , ( , ) = {︃ 1
, if , ≤ − 0 if , >
⎧ , ( , ) = ⎨ 2 ⎩ 1 if , ≤
, if < , ≤ 2 − 0 if , > 2 (3)
For each class model , the reconstruction error , is used to generate the interval membership
, = [ , ( , ), , ( , )]. To make the final decision, intervals must be compared for each
∈ {1, 2}. Interval comparison is a particular case of fuzzy number comparison, broadly
investigated since several years [
Interval comparison aims to rank real intervals. The heuristic approach developed in [
The degree of preference Π( > ) of = [1, 2] over = [1, 2] is defined in [
(2 − 1) + (2 − 1) ︂{ if ≡ then if 2 < 1 then Π( > ) = Π( > ) = 0.5, Π( > ) = 1.
We employ this comparison to rank class memberships , , ∈ {1, }. The defuzzification for the final decision simply consists in choosing the “least preferred” (minimum-error) one:
Event() = arg =m1,i..n., {Π(, > ,̸= )} ,
Diferent audio trafic datasets are suggested in the literature, such as AXA database [
The main parameter adjustment concerns the setting of the thresholds . Diferent values were experimentally optimized. Thresholds were pondered using the complementary of the proportion of each class as a weighting coeficient. Thus, the threshold for each class = 1, . . . , of each VAE’s error was set as the baseline VAE’s threshold 0 pondered by the weight = 1 − , where is the proportion of samples of Class . Table 1 summarizes the values. (4) (5) (6)
The experimental work aims to detect audio events on roads. To do so, features were extracted
from the selected audio database, MIVIA DB [
Regarding the first step, i.e. feature extraction, data augmentation was realized to cope with the issue of rareness of Anomalous samples, so that more data is obtained through the segmentation of the audio signals into short frames, with a duration of 250 ms, with a high overlap rate, i.e. 75%. Nevertheless, it is worth noting that all training segments, whether belonging to Normal or Anomalous, contain background street noise.
Regarding neural networks training, the VAE network was constructed using convolutional layers, using an input feature vector made of log-energy and MFCC features, along with their ifrst and second derivatives ( Δ and Δ-Δ). 80% of the extracted data were utilized for training and validation, whereas test was realized on the remaining 20%.
The evaluation results are listed in Table 2. These results correspond correspond to a state-ofthe-art method, i.e. OC-SVM (used for benchmarking), and to the proposed method (event-based VAE with fuzzy membership). For the latter, the values of the event weights were { }=1,..., were varied to find the tradeof between data distribution and the global performance. For evaluation purposes, standard metrics were calculated, i.e. overall accuracy (), precision ( ), recall () and 1 scores, defined as in (7): = , = , 1 = 2 , + (7) where , and ( ∈ {1, 2}) are the number of ground-truth, estimated and correctly detected events for Normal and Anomalous class, respectively.
The results mentioned in Table 2 show the eficiency of using an interval-valued fuzzy membership function to improve anomaly detection. The main advantages of using such a method can be summarized as follows: • The proposed methods outperforms the state-of-the-art OC-SVM, in terms of overall accuracy and balance between class-based metrics. • Overall accuracy rates are enhanced, reaching 95% for the proposed method, vs. 84% for OC-SVM. Also, the precision, recall and F1 score obtained are more balanced between Normal and Anomalous classes, notwithstanding their disproportional distribution. • The efect of using unbalanced weights is more evidenced, with higher accuracy when is higher for the Anomalous class.
This paper presented a novel method of anomaly detection, based on interval-valued fuzzy sets. A direct application in road trafic surveillance allows detecting hazardous events such as car accidents using audio signals. The proposed method is based on combining two anomaly detection tools, i.e. auto-regressive VAE’s and interval-valued fuzzy sets. Finally, a probabilistic interval comparison method, denoted as degree of preference, is utilized for defuzzification, i.e. detecting the corresponding class.
The main results can be summarized as follows: a) Spectrogram-extracted features are the most suitable to approach such a problem; b) unbalanced weights, where the least abundant class receives the highest weight, contribute to enhance the results; and c) interval-valued fuzzy sets seem more eficient than crisp one-class SVM to detect anomaly. As an outlook, the proposed method could be further improved in two directions: either by making it semi-supervised, as only normal data can be collected and trained, or fully unsupervised, by not using labels any more.
This work was carried out in the framework of the project Xpert funded by the University of Genova.