Audio deepfake detection (ADD) is a prominent problem in artificial intelligence. With diverse spoofing attacks emerging continually, generalization of ADD algorithms in the face of unknown domains and robustness in complex environments become key points for this field. However, when only limited and low-quality learning data is available, as in the case of ADD 2023 Challenge Track 1.2, it is an open issue to achieve good generalization and robustness. In this paper, we propose a Shufle Mix Aggregation and Separation Domain Generalization (SM-ASDG) method which enables single-domain generalization. Specifically, we first design a pre-processing module to improve the robustness of the method against low-quality data. Next, we split the single domain into multiple data domains via the proposed data shufle module. Finally, a well-generalized feature space is constructed through the designed feature extractor and MixStyle domain classifier. The proposed SM-ASDG obtain the weighted equal error rate (WEER) of 23.17% on ADD Challenge Track 1.2, which achieves the Top-5 rank in the challenge.
tion and further strategies for improving generalizability are required.
Audio deepfake detection (ADD) is an important yet chal- To this end, several methods propose the domain
inlenging task, which has raised several concerns due to its variant representation learning (DIRL) strategy [
In recent years, several works [
To address generalization and robustness issues, some to construct ADD models with good generalizability and
methods [
… ADD 2023 Dataset
Pre-processing
Low-pass Filter Amplitude Adjustment
Noise Augmentation
Shuffle Mix Aggregation and Separation Domain Generalization (SM-ASDG)
Data Shuffle … … …
Feature Extractor
MixStyle Domain Classifier
Feature Space
Triplet Loss BCE Loss Adversarial
Loss
Audio
Score
to eliminate the efects of noise and other factors on We further adjust the amplitude of signals due to the
the model and to improve the robustness of the algo- observation that the amplitude of genuine speech is
difrithm. Second, the data shufle module is introduced to ferent from that of spoofed speech. In the training set,
approximate a multi-source domain situation by splitting we observe that the genuine speech has higher
amplithe single domain. Then, we construct a feature extrac- tude than spoofed speech. This may cause the model
tor based on W2V2-XLS-R [
To address the efect of codec variabilities, we first adopt
a low-pass filter [
To improve the generalization ability of the model, we divide the training data into three diferent domains randomly. Randomly shufling the domains enriches the style information of each domain, allowing the domain adversarial loss to aggregate all real speech from various styles. In the experimental section, we further verify that randomly shufling the domains is more efective than direct grouping the validation set into one domain and the training set into two domains.
We first extract features via a W2V2 based front-end, which is trained using a contrastive method with a masked feature encoder. The front-end feature extractor has a feature extractor with seven CNN layers to process speech signals of diferent lengths, followed by a Transformer network with 24 layers, 16 attention heads, and an embedding size of 1024 to obtain context representations. Consequently, the last hidden states from the
ConvFilter
After get the W2V2 feature from feature extractor, we propose a MixStyle Domain Classifier to generate the feature space by optimizing three diferent loss function. The detailed architecture is described in Table 1, which is modified on the traditional LCNN [ 16]. In the architecture, MFM means the Max-Feature-Map layer to select the critical channels for ADD task of the feature and BN means Batch Normalization. After MixStyle domain classifier, we get the feature space of the shape (16,512).
Through the mixing of training instance styles, we can implicitly synthesize novel domains, which results in increased domain diversity of the source domains and ultimately improves the generalizability of the trained model. Given an input batch , we first random choose a reference batch ˜ from . Then, Mixstyle computes the mixed feature statistics as follow: mix = () + (1 − ) (˜), = () + (1 − )( ˜), where is the weight sample from the Beta distribution Beta(, )
. We set to 0.1 in our paper. Then, the style normalized feature is computed by the mixed feature statistics,
BCE loss. First, our main task is binary classification, which is to determine whether the features obtained are genuine or spoofed. We use several FC layer to down sample the feature from 512 to 1 and compute Binary Cross Entropy (BCE) to classify. It is worth mention that the feature normalization and weight normalization is used for this process, which will balance the numerical values of features and weights from speech signals across diferent domains, facilitating the convergence of the model.
Triplet loss. Our proposed ASDG strategy is that the real speech from diferent domain should be aggregated and the spoof one will be separate. The triplet mining method is suitable for the goal, which is defined as follow: (3) (4) = ∑︁ ‖ ( ) − ()‖22 −
⃦⃦ () − )︁ ⃦⃦⃦ 22
︁( ⃦
+ , where ,
, represent the anchor sample, real sample, and fake sample. By minimizing , the euclidean distance between the anchor and the real sample may get closer while the anchor may get further away from the fake sample. We set to 0.1 which is a margin value.
Adversarial loss. In the feature space, the distribution of real speech should be aggregated regardless of domain.
Thus, we design a single-side domain discriminator with Gradient Reverse Layer (GRL) [17]. Let () denotes the distributions of real feature and denotes the domain of . The adversarial loss function of the domain discriminator is defined as follows:
minmax (, ) = − ∼ (),∼ 3 =1 ∑︁ ( = ) ( (()) , where denotes the domain label. The feature generator is trained to learn a robustness feature to spoof the domain discriminator in order to maximize . In the meantime, the discriminator is trained to identify the feature domain by minimizing. To achieve this goal, we (1) use the Gradient Reversal Layer (GRL), which reverses the gradient during back propagation by multiplying negative dynamic coeficients. This makes the discriminator unable to identify the domain of the real feature, which leads to the aggregation of genuine speech in the feature space without being divided by domains.
Total loss. The total loss for our system is defined as follow: = + 1 + 2, (5) MixStyle() = + .
(2) − ()
() where 1 and 2 set to 0.1 to balance the value of three Table 2 diferent losses. By utilizing the loss, we can con- Performance comparison with the state-of-the-art ADD modstruct an optimal classification feature space for ADD els on the ADD-FG dataset. task, where genuine speech signals from diverse domains Method Feature 1 2 are clustered together while fake speech signals are separated from them.
SM-ASDG
Mel W2V2 W2V2
All experiments are conducted on the ADD 2023 Audio FG-D datasets. There are 27,084 audio clips in the training set and 28,324 audio clips in the development set. We divide the dataset as 90%/10% for training and validation, respectively. The audio amplitude in the training set is inconsistent and contains noise, and there are repeated tail segments without valid information. The audio situation in the test set is much more complex, including noise, reverberation, background music, and a large number of silent clips. Therefore, how to improve the generalization and robustness of methods is the core challenge.
Weighted equal error rate (WEER) is used as the evaluation metric, which is defined as better performance than manual feature. This is due to that the W2V2 is trained on a large amount of real utterances from diferent source domains which can enhance the diferential capability in complex scenarios. Moreover, results show that our SM-ASDG model outperforms all backbone models. = 1 + 2, (6) Impact of MixStyle. We further investigate the impact of the MixStyle units. As shown in Table 3, “SM-ASDG where = 0.4 and = 0.6 represent the weights for w/o MIX” denotes removing the MixStyle layer from equal error rate (EER) obtained in round 1 (1) and our full model. It can be observed that the performance round 2 (2) of ADD Challenge Track 1.2, respec- of the model decreases by 2.91% in round1 and 4.50% tively. in round2 with the removal of MixStyle. This demonstrates the efectiveness of our MixStyle domain classifier 3.2. Implementation details module. This is due to the fact that the bottom layer of CNN corresponds to style information and the top layer All training audio files are trimmed or padded to 4s. For corresponds to label information. MixStyle enables the baseline AASIST, the input is the raw waveform of about diversification of the bottom style information of LCNN. 4s (64000 samples). For baseline Resnet18, we use 80- That is, our model can generate diverse new styles of real dimensional LFCCs with a shape of (80,404) as front-end. speech and fake speech to enhance the ability of domain During training, the parameters of W2V2 front-end are generalization. frozen. After front-end, we can get the last hidden states Visualization for feature To analyze the efect of the vector with shape of (201, 1024) as input of back-end. For MixStyle and our proposed ASDG backbone, we visutraining strategy, the Adam optimizer is adopted with alize the distribution of diferent hidden features using 1 = 0.9, 2 = 0.999, = 10−9 and weight decay is T-SNE [20]. As shown in Figure 2, we randomly select 10−4 . The learning rate is initialized as 10−5 and halved 360 samples for three source data domains. In each doevery 5 epochs. main, we select 60 samples for real utterances and 60 for fake utterances. Figure 2 (a) and (b) demonstrate 3.3. Ablation studies on architecture that the hidden feature distributions become more distinct after applying MixStyle, indicating that MixStyle facilitates the diversification of the bottom style information in LCNN. The feature space depicted in Figure 2 (c) aligns with our conception of an ideal feature space by ASDG, where genuine speech signals are clustered together, while synthetic ones are segregated.
Impact of backbone models and features. We first investigate the impact of the backbone models and features.
As shown in Table 2, we compare with three baseline
backbone models: AASIST [
It can be observed that the W2V2 based feature shows
Real
Fake (a) (b)
(c)
To improve the robustness and generalization of the
model, we explore a series of pre-processing strategies,
including data shufle, noise augmentation, low-pass
filtering, amplitude adjustment, and region of interest (ROI)
detection. Pre-processing strategy 1 2
Does data shufle help? We first explore the eficacy SM-ASDG w/o MIX 26.97 27.09
of data shufle strategy. As shown in Table 4, we design SM-ASDG w/o [MIX+AMP] 27.53 27.80
two domain segmentation schemes, namely data shuf- SM-ASDG w/o [MIX+AMP+LP] 28.05 29.24
lfe and direct division. The direct division refers to the SM-ASDG w/o [MIX+AMP+LP+NA] 37.07 38.67
directly using the test set and validation set as two sep- SM-ASDG with ROI 26.25 26.45
arate source domains. The two domain segmentation SM-ASDG 24.06 22.59
schemes are used in four variant, namely ASDG model
and the ASDG model with diferent data augmentation
strategies. “Rawboost” denotes the raw data boosting of the model. So we ultimately choose RM strategy as
and augmentation strategy [
information for distinguishing real and fake audio [22]. Therefore, simply eliminating silent segments does not improve the generalization and robustness of the model. In this paper, we propose SM-ASDG, a novel shufle mix aggregation and separation domain generalization method for single domain ADD. The proposed method achieves a WEER of 23.17% in ADD 2023 track 1.2 final ranking, which is one of the top-5 performing methods. The outstanding robustness and generalization of the proposed SM-ASDG model is due to our carefully designed preprocessing module, data shufle and MixStyle domain classification module. In future works, we plan to embed more high-level semantic features of audio, such as sentiment features, into the model to further improve generalization.