In this paper, we describe our submitted systems to the ADD2023 Challenge Track 3-Deepfake algorithm recognition (AR). This task requires not only identifying known deepfake algorithms in closed-set but also distinguishing unknown algorithms. By closed-set classification experiments, we select the output of the pre-trained wav2vec2.0-base model as acoustic features. Then, we apply the ECAPA-TDNN model to recognize diferent deepfake algorithms and determine whether the samples belong to the unknown algorithms by threshold. Besides, we adopt data augmentation to improve the generalization and robustness of our model. We evaluate our system on the ADD2023 Challenge Track 3 and achieve a 75.41% F1-score. Our submission ranked third in the deepfake algorithm recognition track of the ADD2023 Challenge.
robustness of the model is one of the key objectives for deepfake detection systems.
Speech synthesis and voice conversion technologies [1] However, there are many algorithms for generating
are evolving rapidly, thanks to the development of deep deepfake audio. While detecting spoof audio, we would
neural networks. While these advanced technologies pro- like to know by which algorithms this fake audio is
vide convenience for application, they also threaten the generated. Recently, the second Audio Deepfake
Detecidentity of speaker [2]. For the safety of speakers, audio tion C
In past work, several handcrafted features are used recognition [24], which requires not only correctly classifor deepfake audio detection, including MFCC, iMFCC, fying the types of algorithms that appear in the training LFCC, DCT-DFTspec, log-linear filterbank, and CQT fea- set but also distinguishing the unknown algorithms. tures [8, 9, 10, 11, 12, 13, 14]. Previous works indicate In the field of open-set recognition, advanced methods that LFCC or CQT is more suitable for deepfake audio have been proposed, e.g. OpenMax [25], OpenGAN [26] detection. In addition, some works [15, 16, 17] adopt raw and OpenHybrid [27]. These methods specifically design waveforms as the input to construct end-to-end deepfake a novel module for identifying the open-set samples. Bedetection systems. Moreover, various data augmentation sides, distinguishing the open-set samples based on the technologies are also applied to develop detection sys- comparison of threshold and posterior probabilities is tems. Codec augmentation are used in [8, 18, 19, 20, 21]. one of the most general methods. It is also interesting to Noise augmentation, reverberation, and cutmix are also note that S. Vaze et al. [28] proves a positive correlation essential technologies in anti-spoof systems [20, 21, 22]. between closed-set accuracy and open-set recognition It demonstrates that improving the generalization and performance.
In this paper, we describe our submission for ADD2023 Challenge Track 3 in detail. First, motivated by [28], we conduct classification experiments on closed training set for selecting the optimal input acoustic features.
Finally, we choose the output of wav2vec2.0 [29] for
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License the experiment of open-set recognition. The pre-trained CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) wav2vec2.0 model extracts powerful acoustic features, Noisy Original Mixed
Add Noise & Reverb Utterance Mixup
and then these features are fed into ECAPA-TDNN [30] is extracted as a 512-dimensional vector representation, to classify deepfake algorithms. In addition, we exploit and the stride is 20ms. Then, the vector representations several data augmentation, including noise [31], rever- are processed by a series of transformer blocks. With beration [32], and utterance mixup [33], to generate aug- the help of self-attention module and feed-forward netmented samples. In our experiments, all augmented and work, the context and content information is shown in original samples form a larger training batch for improv- representations, which is conducive to learning more ing the generalization and robustness of our model. Ulti- informative representations. mately, we achieve a 75.41% F1-score, which ranks 3rd The wav2vec2.0 is pre-trained in a self-supervised manin ADD2023 Challenge Track 3. ner using contrastive loss. The objective is to predict the true quantized latent speech representation for a masked time step within a set of distractors. The output of pre2. Methods trained wav2vec2.0 can be applied to downstream supervised learning tasks associated with speech.
In our deepfake algorithm recognition system, we adopt wav2vec2.0-base as the pre-trained feature extractor, aiming at extracting more helpful information from the waveform.
Our deepfake algorithms recognition system is illustrated in Fig. 1. We apply several data augmentation (add noise & reverb [31, 32] and utterance mixup [33]) to obtain augmented samples. These samples are used along with the original samples for training. They are fed into the ifxed wav2vec2.0-base model to extract acoustic features and then are classified by ECAPA-TDNN. In this section, 2.1.2. Classification Model we introduce the model’s architecture, data augmentation ECAPA-TDNN [30] is a classical model in the field of methods, and open-set recognition approaches we used. speaker recognition and verification. As front-end systems of speaker recognition, deepfake detection systems 2.1. The Architecture of Model are also required to process speaker-related information. Therefore, we think that ECAPA-TDNN is also suitable 2.1.1. Feature Extractor for deepfake algorithm recognition systems. Wav2vec2.0 can learn powerful representations from After extracting the representations from wav2vec2.0, speech audio [29]. After pre-training with a large dataset, we adopt ECAPA-TDNN to aggregate the features further. it has been demonstrated the feasibility of speech recog- The SE-Res2Block achieves channel attention that uses nition with limited amounts of labeled data. a global context incorporated in the frame layers. Then,
The raw speech signal is first fed into a feature en- attentive statistic pooling is used to aggregate utterancecoder which consists of several convolution layers (CNN), level embeddings about the deepfake algorithm. After where the kernel widths and strides behave as the win- the embedding of input speech is obtained, it is fed into dow length and hop of the Short-Time Fourier Trans- an algorithm classifier. Simple, this classifier is achieved form (STFT). Following wav2vec2.0, every 25ms of audio by a fully-connected layer.
In this work, we employ several data augmentation methods in order to improve the generalization and robustness of the learned representations. 2.2.1. Add Noise and Reverb We use the MUSAN dataset [31] and the RIR dataset [32] to randomly add noise or reverb to the input audio. We get a random number that follows a uniform distribution between 0 to 1 and choose diferent data augmentation methods depending on . Specifically, the relationship between and diferent data augmentations is shown in Table 1. In our work, the augmented samples and the original samples form a larger batch that is utilized together to optimize the network. 2.2.2. Utterance Mixup
In the training procedure, we follow the closed-set setting, and the model is optimized to perform well under the multi-classes classification task. The objective is to accurately distinguish the deepfake algorithms in the training set so that a compact category distribution is obtained.
During the testing phase, we have to solve an open-set recognition task. We first calculate the prototypes of known algorithm categories in the training set, = 1
∑︁ () || ∈
(4) where is a set of all training samples labeled with deepfake algorithm and || is its cardinality. is the model we use with parameter .
After obtaining prototypes, we calculate the cosinesimilarity between test embedding and prototypes .
The category corresponding to the maximum similarity simimax is the classification result we determine. For unknown algorithm recognition, if the maximum similarity is lower than a pre-defined threshold , this sample is regarded as an open-set category, i.e. unknown deepfake algorithm. This process can be expressed as, result = {︃known class if simimax ≥ unknown class if simimax ≤ (5) Then, the loss function is also modified to, = 1 + (1 − ) 2 ℒ = ℒ 1 + (1 − )ℒ 2 where ℒ represents the loss calculated with the label corresponding to sample .
Mixup [33] is one of the common methods of data augmentation in the field of computer vision. We introduce This is one of the most widely applied methods of openutterance mixup to speech deepfake algorithm recog- set recognition. The method is simple and intuitive, so nition. The utterance mixup aims to simulate multiple we adopt it to distinguish unknown deepfake algorithms. deepfake algorithm audio and improve model generaliza- Meanwhile, this threshold determination method has to tion. guarantee the accuracy of the closed-set categories.
To generate the mixed audio from multiple deepfake algorithms, we randomly select two utterances 1, 2 3. Experimental Setup from each training batch and mix them according to the following equation,
= 1 + (1 − ) 2 where mixing coeficient ∼ Beta(, ). Obviously, the mixed sample contains two diferent deepfake algorithms, which means the mixed samples belong to two diferent algorithmic categories. Therefore, the label of mixed sample is modified as follows, (1) (2) (3)
T
In the test set, there are 79490 test utterances in total. deepfake algorithms of the training set using ResNet18
It contains eight categories, seven of which are the same with Time-Frequency Attention Pooling [34]. The results
as the training and development datasets. Besides, one of closed-set classification are presented in Table 3. It
additional deepfake algorithm category is added as an is not hard to obverse that when LFCC and wav2vec2.0
open-set class. T
Our network consists of wav2vec2.0-base and ECAPATDNN. In our experiments, we employ an open-source, 4.2. Open-set Recognition Results self-supervised pre-trained wav2vec2.0-base model. The output of wav2vec2.0-base is 768-dimensional features. We compare the performance of various input features Thus the input dimension of ECAPA-TDNN needs to and classification models for open-set recognition. We be modified to match. In the training phase, we fix the adopt grid search for unknown algorithm thresholds parameters of the pre-trained wav2vec2.0-base model and present the optimal results in Table 4. The reand optimize only the ECAPA-TDNN model. sults indicate that the output of pre-trained wav2vec2.0
For each training utterance, we randomly crop 4 sec- captures higher-level semantic information and more onds of audio to construct batches with a batch size of 64. detailed information from waveform than LFCC. It is An SGD algorithm is utilized with a momentum of 0.9 more beneficial for deepfake algorithm recognition. The and weight decay of 1e-4 to optimize our model over 50 ResNet18 exploits 2D convolution to process features epochs. The initial learning rate of 0.1 is used for training during the first 25 epochs, declining to 0.01 between the 25th and 40th epochs and 0.001 for the remaining 10 epochs. Moreover, the hyper-parameter in utterance mixup is set to 1.0 in our experiments, and we select the optimal by grid searching.
This paper has presented our proposed systems for t
To improve the generalization and robustness of the
model, we applied various data augmentation in the
training process. In Table 5, we study the influences of the
different data augmentations on performance. After adding
noise and reverbing, the performance is significantly
improved. The larger value of means that more
augmented samples caused by noise and reverberation are
generated. We find that the model makes further progress
as the number of augmented samples increases. Utilizing
utterance mixup on the basis of noise addition and
reverberation, we further improve the performance. With
the help of several data augmentation, the optimal
F1score of 75.41% is achieved. This result ranks 3rd in t