This paper presents a method that uses convolutional neural networks for audio recognition in an open-set scenario. The audio sounds in an open-set scenario are usually out of the training data distribution, which necessitates a model that can recognize the known classes while rejecting the unknown ones. We propose a convolutional approach for recognizing audio events, that can effectively address open-set recognition by adding inclusion probabilities of extreme value machines. Extensive experiments conducted showed that our proposed method outrivals representative existing methods under the open-set regime.
CNNs with the SoftMax activation function have been used in closed-set recognition tasks and
demonstrated an outstanding performance in many applications in literature, such as speech sound
recognition [
The proposed CNN architecture includes several convolutional layers [
Each audio signal is composed of different frequencies and different energy amplitudes, with quick
variations within a short time. There is a need to define and represent audio signals such that a robust
recognition system can be built. We have chosen the log-Mel spectrogram since it has proven to be
suitable to model the human auditory system and is used in many speech and audio recognition tasks
[
The architecture block of the CNN is composed of convolution layers and pooling layers. A
convolution layer applies filters to the input then takes the inner product and adds the bias. Each filter
has its own bias and weights. A pooling layer reduces the dimensionality of the subsequent layers. It is
applied to each convolution feature map independently. Please refer to [
When we apply more convolution and pooling techniques to feature maps at higher levels, their resolution decreases. Before feeding the features to the output layer, they must be integrated across all frequency bands. On top of the last CNN layer, fully connected hidden layers are formed. The SoftMax is the output layer used as an activation function to predict probability over the class labels. Each number in the softmax function's output is inferred as the probability of belonging to each class. However, for open-set recognition, SoftMax cannot work well, so we propose to replace it with the EVM to determine the probability of the output for each class.
For closed-set recognition, let us assume the known classes {C1, C2 ,..., CN } , where Nk is the number of known classes. The final layer has the same size as the number of known classes. We denote the representation of this final network layer as y = f (x) , where f denotes the network as a function. When an audio data point x arrives, the SoftMax function to label this sound is defined as follows: p(Ci | x, x N ) = SoftMaxi ( y) =
exp(xi ) Nj exp(x j ) ()
The SoftMax function assigns a certain probability to each training class by computing the maximum
SoftMax probability, which is suitable for the optimization of the deep network in the closed-set
recognition. In open-set settings, we need to consider x N , where the class CN +1 corresponds to a novel
class. The crucial step is to find a suitable value for thresholding between known and unknown classes.
Some previous works such as [
We assess the effectiveness of our proposed algorithm by computing similarities after aligning the recognition outputs with a reference ground truth. The evaluation utilizes cross-validation, which allows evaluation of the accuracy of data that may not be part of the training dataset. Two fundamental assumptions have been used in the DCASE/ AASP challenges [20] to evaluate how individual audio sounds are classified: • Segment-based evaluation: the system output and ground truth are compared for each segment length. • Event-based evaluation: the system output is considered the same within all ranges (duration) of the event. This means that event labels in the recognition output will be compared to the ground truth events.
Let us consider a binary classification, where the labels consist only of positives or negatives. Based on true labels and predicted labels, we divide the metrics into four intermediate statistics: true-positive (TP), false-positive (FP), true-negative (TN), and false-negative (FN). A count is made for each category. Applying this to a multi-class problem, every single classifier that produces a “positive” or “negative” prediction can be “true” or “false” depending on the corresponding ground-truth label.
The recognition accuracy (RA) can be described as the ratio of the correctly labeled predictions to the whole pool:
RA =
TP + TN
TP + FP + FN + TN
Precision (Pr) is the ratio of predicted positive samples that are calculated correctly (true) divided by all predicted positive samples, while recall (Re) is the fraction of predicted positive samples correctly detected from all ground truth positive samples (labels). For multi-class classification, there are two ways of computation: macro-averaging and micro-averaging [21].
Prmacro = 1 N TPi N i=1 TPi + FPi
N TPi i=1 Prmicro = N (TPi + FPi ) i=1
Remacro = 1 N TPi N i=1 TPi + FNi
N TPi i=1 Remicro = N (TPi + FNi ) i=1
The F1-measure includes both precision and recall merged in a single score, which is computed as the harmonic mean between precision and recall. The F1-measure is computed as:
F1- measure = 2PrRe
The confusion matrix CM (i, j) summarizes the performance of multi-class recognition. It depicts the various ways in which the classification model gets confused when making predictions. Each column of the matrix represents a real class, whereas each row represents a predicted class. The diagonal of this matrix (i = j) reveals the correct prediction.
Extensive experiments were conducted using Python. A Keras [22] implementation of CNNs was used, with TensorFlow [23] as the backend. First, we carried out closed-set recognition experiments where the audio dataset is separated into training and testing datasets.
To model the classifiers, we applied the 5-folding cross-validation technique where a total of 80% is used as the training dataset, while the remaining 20% of the data is used for testing. The experiment is conducted on the DCASE2016 dataset. This dataset consists of audio recorded in everyday life, which includes 11 sound classes that were recorded in an office environment: clearing throat, coughing, speech, drawer, keyboard, keys drop, knock, laughter, page-turning, phone ringing, and a door slam.
The classification output is evaluated to be correct or not according to the ground truth. We did not perform comparisons with other algorithms in this part, because the experiments in a closed-set regime aim to evaluate the ability of the algorithm to differentiate among recognized classes. The comparison will be conducted later in the open-set recognition part. As can be noticed from Fig. 1, the event-based confusion matrix discloses that most of the classes have been recognized very well except door-slam and phone-ring classes, whose accuracies were 60% and 70 %, respectively. Fig. 2 shows the confusion matrix after applying frame-based recognition. The right column reveals the percentage accuracy of each class. Most of the classes have been recognized correctly, and some misclassification can be also observed that is because the similarities among these classes are high. The sound class has a great impact on the results, as expected. For example, the door-slam class was the hardest class to recognize, probably because of the short length of such sounds.
Yk
Yt
The experiments in this section were performed to recognize audio sounds where the testing set also includes classes that may not be part of the training dataset. These experiments measure the capability to discriminate known classes from novel classes and to discriminate known classes from one another. The level of openness for a classification task can be defined as:
Openness = 1− 2 Yt Yk + Yu ()
Where the subscripts k , t and u are for the known, target, and unknown label sets, as defined in Fig. 3. The testing dataset is defined as Ytest = Yk Yt Yu , while the training dataset is the combination Ytrain = Yt Yk , and the unknown classes are its complement Yu = y | y Ytrain and y Ytest . If we set Yk = Yt these yields Ytrain = Yk .
We used varying degrees of openness and followed k-fold cross-validation to obtain robust evaluation
metrics. The experiments were performed by generating different amounts of openness. Our experiments
were conducted for several evaluations in which we examined how well our proposed algorithm
performs in comparison to other representative algorithms: W-SVM [
The parameters of all previous algorithms were set according to the corresponding paper. To ensure a fair comparison, all of the algorithms were run on the same dataset and the same distribution of classes.
The experimental results are depicted in Fig.4 for event-based recognition and in Fig.5 for framebased recognition. It is clear that there are considerable differences in the performance among the different methods. Looking at these figures, all the methods, in general, suffer from a performance decrease if the openness increases. However, our proposed algorithm performs relatively well compared to the other methods, in terms of determining the novel classes and discriminating among the known classes. The OSmIL algorithm had the worst performance. As expected, the performance of event-based measures outperforms the performance of frame-based measurements.
In this work, we presented a CNN network architecture that is efficient for robust audio open-set recognition. Extensive testing was done to distinguish between known and unknown audio classes. Our proposed method overall outperformed representative previous work across a wide range of openness levels. For further work, more research should be done to see how well the proposed CNN performs on large real-world audio datasets. Experiments and algorithmic modifications for incremental learning should also be performed.
[19] A. Bendale and T. E. Boult, “Towards open set deep networks,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2016, pp. 1563–1572. doi: 10.1109/CVPR.2016.173. [20] A. Diment, T. Heittola, and T. Virtanen, “Sound event detection for office live and office synthetic AASP challenge,” Proc IEEE AASP Chall. Detect. Classif Acoust Scenes Events WASPAA, 2013, Accessed: Nov. 11, 2016. [Online]. Available: http://c4dm.eecs.qmul.ac.uk/sceneseventschallenge/abstracts/OL/DHV.pdf [21] K. Zhang, H. Su, and Y. Dou, “Beyond AP: a new evaluation index for multiclass classification task accuracy,” Appl. Intell., vol. 51, no. 10, pp. 7166–7176, Oct. 2021, doi: 10.1007/s10489-02102223-7. [22] J. Moolayil, Learn Keras for Deep Neural Networks: A Fast-Track Approach to Modern Deep
Learning with Python. Berkeley, CA: Apress, 2019. doi: 10.1007/978-1-4842-4240-7. [23]M. Abadi et al., “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,”
ArXiv Prepr. ArXiv160304467, 2016. [24] P. R. Mendes Júnior et al., “Nearest neighbors distance ratio open-set classifier,” Mach. Learn., vol. 106, no. 3, pp. 359–386, Mar. 2017, doi: 10.1007/s10994-016-5610-8. [25] S. Dang, Z. Cao, Z. Cui, Y. Pi, and N. Liu, “Open set incremental learning for automatic target recognition,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 7, pp. 4445–4456, Jul. 2019, doi: 10.1109/TGRS.2019.2891266.