The issue of overconfidence in out-of-distribution data in current deepfake audio detection models has resulted in poor robustness, making it dificult for models to achieve good results in the test set with large distribution diferences, such as Audio Deepfake Detection Challenge (ADD 2023) Track 1.2 audio fake game detection task. In this paper, the Energy-based Open-World Softmax (EOW-Softmax) is introduced to calibrate model confidence and achieves good results in the challenge. Additionally, the paper presents a range of data augmentation methods, including vocoder-generated training data, to efectively improve the performance of the deepfake audio detection models. By fusing the scores of a variety of single systems based on Squeeze-and-Excitation residual neural network (SENet), light convolutional neural network (LCNN) and Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks (AASIST), the proposed system achieves the second place in the challenge.
previous challenges as it aims to go beyond binary
classification. Track 1 of ADD 2023, which is comprised of
In recent decades, the progress of deep learning and ar- two tasks: the audio generation task and the fake audio
tificial intelligence generated content (AIGC) has led to detection task. For the Track 1.1 generation task (FG-G),
significant advancements in speech synthesis and voice participants aim to create fake audio that can deceive
conversion technologies. Deep learning models have the fake detection model of Track 1.2. The Track 1.2
the ability to generate incredibly realistic audio. How- detection task (FG-D) requires participants to identify
ever, the malicious use of deepfake has also raised con- fake utterances, with a particular focus on identifying
cerns. As a result, the detection of deepfake audio has fake samples generated in Track 1.1. Both tasks have two
become an important topic of interest. Four ASVspoof rounds of evaluations. Track 2 focuses on localizing
maChallenges [
In this section, data augmentation methods and confidence calibration is presented. In both rounds, large performance gains are achieved by creating training data with vocoders. In the second round, the confidence calibration further improve the performence efectively.
Discriminators based on deep neural networks often
suffer from overfitting problems, producing overconfident
predictions [24]. In the speech anti-spoofing task faced
with a large amount of out-of-distribution (OOD) data,
overconfident false predictions seriously afect the
performance [
The purpose of the energy function is to map a Ddimensional data point to a scalar. With it is possible to assign low energy to the observed variable configuration and high energy to the unobserved variable. The probability density () for ∈ Rin an energy based model (EBM) can be written as
can be efective in improving model robustness in the absence of suficient training data. Three data augmentation methods are used to enhance the performance. () = exp(− ()) (1)
There is obvious noise and reverberation in the speech ( )
of the evaluation sets. Therefore, the most commonly
used methods for speech data angmentation, noise and where ( ) = ∫︀ exp(− ()) represents the
normalreverberation addition from the MUSAN [
RawBoost [
In order to enhance the robustness of CMs to
unknown deepfake algorithms, data augmentation can
be performed by doing copy-synthesis on genuine
speech through various vocoders [
To allow ℎ ()[ +1] to encode uncertainty, the score of ℎ ()[ + 1] needs to be correlated with the marginal data distribution. When the input comes from within the training data distribution (), the model should be confident in its decisions. Therefore, ℎ ()[ + 1] should be low, while ∑︀
=1 ℎ ()[] should be high. If the input features deviate from the distribution of the training data, the model should remain uncertain about its decision. As a result, ℎ ()[ + 1] should be high to indicate greater uncertainty, which naturally yields to low ∑︀ =1 ℎ ()[]. By designing learning objectives
distribution ().
() = log ℎ ()[ + 1] (3) Then, the objective function is defined as
minE() [− log ℎ ()[]]
+ E ¯() [− log ℎ ()[ + 1]] where >
0 is a hyper-parameter. The first term is the maximum log-likelihood objective for the -way classification task using the true label ; the second term can be viewed as the maximum log-likelihood objective for identifying data sampled from ¯(). ¯() denotes the model distribution of the frozen parameters in this iteration. Optimizing Eq.4 can make the sum of the softmax scores of the original class proportional to the marginal density (), which in turn makes the + 1-th softmax score negatively correlated with ().
The ADD 2023 Challenge datasets are utilized in all experiments. There are 27,084 pieces of audio in the training set, 3,012 of which are genuine. And the development set has 28,324 pieces of audio, of which 2,307 are genuine. while the development set is used for performance validation during training. All data augmentation methods described in Section 2.1 are utilized. 21,276 pieces of fake audio are generated through four vocoders.
method capable of distinguishing between genuine and serves as the evaluation metric for this task. Specifically, WEER is defined as follows 2:
= 1 + 2 where
= 0.4 and = 0.6 represents the weights for the two equal error rates (EERs) obtained in round 1 and round 2 of Track 1.2, respectively. through EBM, ℎ ()[ + 1] can capture the marginal
features are extracted with Blackman window of 400 frame length and 512 Fast Fourier Transform (FFT) points. Only low-frequency part (0 − are fed into neural networks [26].
(4) (ResNet) with SE block [
monly used classifiers for spoof speech detection. The SENet implemented here is SENet34, and the number of channels is (16, 32, 64, 128). SENet can assign weights to the features of diferent channels through attention mechanisms, thereby enhancing features that are more important for fake audio detection. The A-softmax is used as the loss function with = 4. The features are extracted with torchaudio [27].
The end-to-end anti-spoofing CM AASIST 3 is the same
as [
pooling layer. The loss function is cross-entropy loss with weights of (0.1, 0.9) for spoof and bonafide class.
The features fed in the STFT-LCNN system are
extracted with librosa [28], and the loss function is
crossentropy loss and center loss. The factor of center loss
combined with the cross-entropy loss is 0.05. The
proposed system based on LCNN is similar to the LCNN
baseline system [29] for the ASVspoof 2021 Challenge
and the system presented in the first place of ASVspoof
2021 [
The wide ResNet used for EOW Softmax 4 is modified based on the [25], with depth of 22 and wide factor of 2.
Adam optimizer is adopted with 1 = 0.9, 2 = 0.98, = 10−9
and weigth decay 10−4 . The learning rate is initialized as 0.0003. For ResNet based systems, the learning rate increases linearly for the first 1000 warm-up Uncertainty-Modeling-for-Confidence-Calibration Solely the training set is employed to train all systems, the selection of critical features, reduces the number of deepfake audio. The weighted equal error rate (WEER) fully connected layers. The 128-dimensional embeddings steps and then decreases proportionally to the inverse fake audio using vocoders can be an extremely efective square root of the step number. StepLR is used as sched- data augmentation technique. uler for LCNN with step size of 10 epochs and coeficient 0.5. Adam optimizer with learning rate of 10−4 and co- 3.3.2. Results of EOW Softmax sine annealing learning rate decay are utilized in AASIST.
All models are trained with 100 epochs, and the model with the lowest loss on the development set is selected as the final model for evaluation.
In the first round, two systems, STFT-SENet and AASIST, are implemented and the scores are fused with weights of 0.75 and 0.25 . In the second round, four systems of STFT-SENet, AASIST, STFT-LCNN and STFTResNet-EOW are implemented, and the score fusion weights are 0.06375, 0.02125, 0.732, 0.183. The fusion weights are manually selected based on the performance.
duce the confidence of the model and alleviate the overiftting problem. Compared to vanilla trained LCNN and SENet, EOW-Softmax demonstrates great performance on the evaluation set of round 2 with a significant amount of OOD data. But in the second round EER is slightly higher than LCNN. The efectiveness of this approach is also validated on the evaluation set from the first round after the challenge. The results are presented in Tabel 2.
3.3.1. Results of data augmentation
The efects of two data augmentation methods vary significantly. Rawboost exhibits minimal impact on the evaluation set in both systems. In contrast, the addition of noise and reverb online is more efective for LCNN, but less efective than RawBoost for SENet. A plausible explanation for this discrepancy can be attributed to the pooling layers and MFM operations in LCNN, which may result in the model paying less attention to input feature details than SENet. Additionally, the presence of noise in the test set, which difers from the noise used for data augmentation, may cause performance degradation of SENet, while LCNN may benefit from it.
After identifying the most efective data augmentation approach for each system, the fake audio generated by vocoders is incorporated into the training process. The use of vocoders to generate training data proved to be highly efective for both systems, resulting in a significant relative reduction in the EER of LCNN and SENet by 29% and 53%, respectively. This suggests that when faced with limited training data, transforming genuine audio into
A comparison of score distribution histograms, as depicted in Figure 1, reveals that EOW-Softmax successfully calibrates the confidence of model predictions. The scores obtained by the SENet are more extreme than those obtained using EOW-Softmax. However, the scores obtained by EOW-Softmax seem to be dificult to clearly distinguish between genuine and fake. And the efect of EOW-Softamx on other models is still worth exploring. 3.3.3. Results of submitted systems
Track 1.2. Several single systems that efectively leverage various data augmentation methods to achieve strong performance are presented. Of particular note is the introduction of EOW-Softmax, which addresses the challenge of deepfake audio detection systems exhibiting overconifdence in OOD data. The EOW-Softmax based system successfully calibrates confidence, mitigates overfitting, and improves overall robustness. The final system submitted achieves the second place in the challenge.