This paper describes our NPU-ASLP system for the Deepfake Algorithm Recognition (AR) task in the Audio Deepfake Detection 2023 Challenge. This task is an open-set classification problem focusing on identifying the specific algorithms used to create the deepfake speech utterances. In this task, we introduce a deepfake AR system with contributions in data augmentation, model architecture, fine-tuning strategy, and model ensemble. We first generate training data by applying various data augmentation techniques to the deepfake speech. We then utilize ResNet101 and a long-term temporal-frequency transformer module to better capture audio context dependencies. Moreover, we employ pre-trained WavLM for better feature extraction. Additionally, our content-invariant fine-tuning strategy improves performance. Finally, model ensemble with diferent representation combinations further enhances performance. Experiments show that our system achieves an F1-score of 0.7355 on the evaluation set, and ranks fourth in the challenge.
tifying the specific algorithms used to create deepfake
speech utterances. With the rapid advancements in deep
In recent years, the progress of deep learning has resulted learning as speech synthesis and voice conversion,
deepin significant advancements in speech synthesis [
The objective of the second edition of the Audio Deep- ods, leveraging the recent advancements in deep learning
fake Detection Challenge (ADD 2023) [
The Deepfake Algorithm Recognition (AR) task in We adopt the noise and room impulse response clips
ADD 2023 is an essential challenge that focuses on iden- and simulate training data by adding distortions to the
deepfake speech utterances. We further apply random
sampling, time stretching, time and frequency masking to
the mixed audio to better reveal the artifacts in the
deepfake speech utterances. As for the model architecture,
on the augmented data, which better captures long-term
temporal-frequency dependencies and further aggregates
global hierarchical contextual information. Moreover,
we leverage the pre-trained audio-LLM WavLM [
Finally, we utilize model ensemble to imporove the robustness and generalization ability of our system.
Our system comprises three main modules designed to
improve the performance of deepfake audio detection.
We select ResNet101 [
When inputting an audio to be detected, log mel spectra and latent representations are extracted first. Two alternative combination techniques, namely feature concatenation and multi-channel fusion, are employed to derive the input features. The combined features are then fed into the ResNet101-TFT network to generate predicted probabilities. To determine the classification, we establish a hyperparameter threshold denoted as T and compare the maximum probability of prediction against this threshold. If the maximum probability is below T, the prediction is labeled as an unknown counterfeit, while exceeding the threshold identifies it as belonging to a specific category. The details are illustrated in Figure 1.
ResNet [
A N S B HM
A N S B HM
Temporal Modeling
N B N B i-BURG leuR irneLa i-BRUG leuR irneaL
adaptive frequency modeling branch, use two adaptive weights and . The module incorporates an improved transformer, which includes a multi-head self-attention (MHSA) module and a GRU-based feed-forward network.
Residual connections and batch normalization are also
employed. To better model long-term dependencies, the
feed-forward network uses a bi-directional GRU (Bi-GRU)
instead of the first fully connected layer in the traditional
transformer [
The training and development datasets of ADD 2023
Challenge Task 3 consist of 22,397 and 8,400 audio clips,
respectively. We rearrange the training set and
development set in a ratio of 9 to 1. All audios are in
single2.4. Utilization of Pre-trained WavLM channel 16KHz 16-bit format, ranging from 1 to 105
seconds. In the training and development datasets, there are
Recently, following the success of the language model a total of 7 categories, with 1 category representing real
BERT on textual data representation, various models audio and the remaining 6 categories representing
counwhich learn audio data representation have been devel- terfeit audio. Each of these categories is labeled from 0
oped. The objective is to train a system for a single task to 6. The test dataset, on the other hand, comprises 8
requiring an extensive representation of the underlying categories, including the 7 categories found in the
trainaudio data. Once the network is trained, the last few ing and development datasets and an additional category
layers are then removed to get a system building an ex- representing an unknown counterfeit. To better leverage
tensive vectorial representation of audio data, which can the training data, we clip long-duration deepfake speech
then be used as input features. WavLM [
To further reveal the artifacts in the deepfake speech utterances and make the deep learning based model robust, our data augmentation pipeline comprises two strategies. One is that we generate mixed audio by mixing speeches, noises, and room impulse responses. One original deepfake speech audio is added by a piece of noise recording with a signal-to-noise ratio (SNR) drawn from a Gaussian distribution with N(−5, 20)dB with an 80% possibility. Then, this audio is convolved with a piece of room impulse response with a possibility of 50%. The other is that random sampling, time stretching, time, and frequency masking are applied to the mix audios generated by the first strategy with a certain probability. Specifically, one mix audio is upsampled or downsampled ranging in [8kHz, 16kHz, 24kHz], then processed with speed shifting and temporal and frequency masking.
We extract input features from the augmented audios in diferent hierarchies. To be exact, we facilitate 128dimensional log Mel spectrograms extracted with 1024 samples of Hann window and 512 samples shift as low × ×1 , we further exploit the level common features F pre-trained WavLM to extract high level general latent × ×1 . We explore two diferent fearepresentations Rℎℎ ture combination techniques: one is concatenating lowlevel features and high-level features along the feature dim as feature concatenation resulting in the input features I × ×1 , the other is combining low-level features and high-level features as multi-channel representation fusion resulting in the input features I × ×2 , in which T denotes time-frames and F denotes the feature dim.
The augmented training data are clipped into 4-sec long segments. AdamW optimizer is adopted to train the model and ReduceLROnPlateau learning rate scheduler is applied to control the learning rate. Batch size is set to 48 and the upper limit of training epochs is set to 100. The optimization criteria are Cross Entropy loss and the Kullback-Leibler divergence loss [23] between the prediction probabilities and target labels as we set the output size of the last fully connected layer in our deepfake AR system to be 7, the highest logit indicates the predicted class. The targets are set to be the same size, where the probability of the true class is 1 and the others are 0. The hyperparameters and in the TFT module are initialized to 1. Early stop is utilized when the accuracy rate on the dev set stops improving for 10 epochs. gradient clipping is employed to avoid overfitting and encourage the models to learn more robust and generalizable representations.
We intuitively perceive the Deepfake Algorithm Recognition as a content-invariant classification task, due to the fact that humans tend to assess the authenticity of an audio through aspects such as pitch, tone, naturalness, emotions, and subjective auditory perception, rather than the actual content of the audio. To this end, we draw on the experience of the mixup strategy [24, 25, 26] from computer vision and introduce a content-invariant finetuning strategy. Specifically, we fully exploit the oficial dataset provided by ADD Challenge in two ways. One is that we simulate multi-speaker audios by mixing different deepfake utterances with the same label with a signal-to-noise ratio (SNR) drawn from a Gaussian distribution with N(−5, 20)dB, and the other is that we randomly clip deepfake utterances with the same label into 1-2 secs and concatenate together to obtain new audio sequences. We then employ the generated data to finetune our model and achieve better results on the evaluation set, the details are discussed in Section 4.
Model ensemble [27, 28] is a machine learning technique that combines the predictions of multiple models to achieve better performance than any individual model, which reduces the impact of individual model biases and errors and improves the overall accuracy and robustness of the predictions.
As the test dataset includes an additional category for an unknown counterfeit, which is not present in the training and development datasets, we set the output size of our deepfake AR system to 7. To detect the unknown counterfeit, we compare the posterior probability output P with a hyperparameter threshold T. If the posterior probability P is less than the threshold T, we consider the audio to be the unknown counterfeit, otherwise, the audio is classified into the 7 known categories. The hyperparameter T is tuned by the performances on the evaluation set.
= ∑︁ ∑︀ =1 −1 + + (1)
In model ensemble, we use weighted averaging [29] to combine the predictions of multiple models and obtain the ensembled posterior probability, demonstrated in Equation (1). The ensemble prediction probability is calculated by taking a weighted average of the individual model predictions. The weights are assigned based on the accuracy of each model on the development set, which ensures that the more accurate models are given