This paper presents our learned lessons from the ADD2023 track3, Deepfake Algorithm Recognition (AR). In recent years, speech synthesis has made remarkable progress, where it has become increasingly dificult for human listeners to diferentiate between synthesized speech and genuine human speech. Previous research has demonstrated that improving the recognition of deepfake algorithms can significantly enhance spoofing detection. Therefore, the primary focus of this paper is to investigate deepfake algorithm recognition, with experiments conducted based on the ADD2023 challenge. Inspired by speaker verification, we approach deepfake algorithm recognition as an open-set task. We propose a center-based similarity maximum method for determining the category of deepfake algorithms. Finally, by combining the scores from multiple models at the score level, we achieve an impressive F1 score of 0.8312 on the evaluation set.
algorithm that a fake utterance is generated from. The
evaluation set consists of both known and unknown
deepText-to-speech (TTS) and voice conversion (VC) systems fake algorithms, making the task an open-set recognition
have made significant progress in recent years, thanks problem. To address this challenge, we propose a
centerto the progress made in deep learning techniques and based maximum similarity evaluation method inspired by
the availability of large-scale corpora [
In recent years, significant eforts have been devoted 2. Related Work
to the audio anti-spoofing task, with notable challenges
such as ASVSpoof [
In this paper, we evaluate the performance of deep- ifer module determines whether the test audio is bonafide
fake algorithm recognition, focusing on the ADD2023 or spoofed based on the extracted features. Therefore, the
track3 [
to increase the inter-class distance while reducing the
intra-class distance. The OC-softmax is also adopted
to customizes metric parameters for the diferent class
distributions of bonafide and spoof audios [
The second Audio Deepfake Detection challenge (ADD
2023)[
Automatic Speaker Verification (ASV) is a voice identity technology that determines whether two speech segments belong to the same person. To evaluate the performance of an ASV system, a trial list is provided, where each trial consists of an enrollment segment and a test segment. In the deep learning stage, we usually extract the speaker embedding, a fixed -dimensional vector, from variablelength waveform. Then, we calculate the similarity between the enrollment embedding and the test embedding, which represents the probability that they belong to the same person. ASV is an open-set task, meaning that all test speakers are unseen during the training stage. On the other hand, deepfake algorithm recognition is a semi-open task, where both seen and unseen classes exist during the evaluation stage.In this case, we can adopt the method of speaker verification for deepfake algorithm recognition.
In this paper, we regard the deepfake algorithms recognition as a speaker verification task. Accordingly, we propose a center-based maximum similarity method to determine the category of test audio.
Figure 1. We employ a supervised learning approach to train a closed-set classification model on the training dataset. Subsequently, we feed all variable-length audio samples from the training set into the encoder of trained model, obtaining fixed-length -dimensional embeddings z ∈ R. These embeddings correspond to the final linear outputs before the classifier. The classifier output can be seen as the result of inner product between the test embedding and the linear weight w ∈ R, where w can be consider the centroid . When z and w are normalized, the classifier output is the cosine similarity between z and w. However, the training process is not always stable, which may make center shifting. To overcome this, we calculate the average embedding for each class, obtaining stable intra-class centroids: c =
1 ∑︁
z, =0 where c represents the embedding centroid of class , and z, corresponds to the embedding of the -th sample from class . Next, we evaluate the test embedding z by scoring it against all intra-class centroids. The scores are denoted as ∈ and ∈ {0,1,2,3,4,5,6}.
= cos(z,c) (2)
where cos(·) indicates the cosine similarity between
test embedding and various intra-class centroids. During
the training process, we employ a loss function with
angular margin, specifically ArcFace [
Finally, if any score surpasses the predefined centroid threshold, the test audio is assigned to the class with the highest score. Conversely, if all scores fall below the threshold, the test audio is considered as belonging to an unknown class 7.
{︃argmax∈ (),
max() > 0.75, 7, max() ≤ 0.75.
The value 0.75 is a hyperparameter that has been obtained through tuning on the development set. (1) (3)
a two-stage training approach: pre-training and large margin fine-tuning (LMFT). During the first stage, we set the scalar and margin of the angle softmax to 32 and 0.2, respectively. In the LMFT stage, we increase the margin to 0.4. To capture the distinctive features of deepfake audio, we leverage various network architectures. Below are the features from the input audio. These features are then specific details of each network utilized in our system: fed into the ECAPA-TDNN model for further processing ResNet34SimAM-ASP: We adopt the ResNet34SimAM- and embedding extraction. The resulting embedding has ASP [19] as our baseline system. This network utilizes a dimension of 256. the residual modules of ResNet and is equipped with the SimAM (Simple attention module). We use a thin ResNet 4.2. Data processing and augmentation structure with a backbone width of {32, 64, 128, 256} to prevent overfitting on small-scale training data. The For our input features, we adopt the log Mel-filterbank, output feature maps are then processed using attentive log-spectrum, and waveform. The log Mel-filterbank statistics pooling (ASP) [20]. The output embedding has energies have a dimension of 80, and the log-spectrum a dimension of 256. has a dimension of 257. The frame length is set to 25ms, ResNet34-GSP: ResNet34-GSP is based on the standard and the hop size is set to 10ms.
ResNet backbone, which is a well-known framework To diversify the training samples, we apply on-the-fly for 2D Convolutional Neural Networks (2D-CNNs) in data augmentation techniques [26]. We employ two types image recognition tasks. We use the same backbone of augmentation methods: 1) Adding noise: We use the width as ResNet34SimAM-ASP. The encoding layer of MUSAN dataset [27], specifically the non-speech parts, to ResNet34-GSP is based on global statistic pooling (GSP), add noise to the audio samples during training; 2) Adding and the embedding has a dimension of 256. convolutional reverberation: We utilize the RIR Noise ResNet34SE-ASP: ResNet34SE-ASP is similar to datasets [28] to simulate convolutional reverberation ResNet34SimAM-ASP in terms of architecture but efects and apply them to the training data. replaces the SimAM attention module with a squeezeand-excitation (SE) module. The SE module aggregates 4.3. Dataset analysis global channel information as attention weights for all feature maps. The embedding dimension is also 256. 4.3.1. Statistic Analysis ECAPATDNN-ASP: ECAPA-TDNN [21] is a 1D-CNN model that has achieved great success in speaker verification. The SE-Res2Block is used to capture speaker features, and Multi-layer Feature Aggregation (MFA) is employed to process concatenated information before feeding it to the ASP. The resulting embedding has a dimension of 256.
LCNN: LCNN is a network commonly used for audio
anti-spoofing, particularly in the ASVSpoof 2021
challenge [
AASIST-SAP: AASIST [
AASIST operates on raw waveforms to extract meaningful
high-dimensional spectro-temporal feature maps. It then
extracts graph nodes from the feature maps in both the
temporal and frequency domains [
By utilizing Wav2vec 2.0 and WavLM, we extract acoustic
in the train set is 6.58 seconds, and in the development set, it is 6.84 seconds. Therefore, we randomly divide the samples into 6-second segments during training to ensure consistency in the input size. The duration distribution of the training and development sets can be observed in Figure 2.
Additionally, we observe that while the sampling rate of the training set is uniformly set to 16kHz, the development set contains audio samples with both 24kHz and 16kHz sampling rates. The sample rate of the 3rd and 4th classes in the development set is 24kHz. Some audio samples in the development set were originally at an 8kHz sampling rate but were upsampled to 16kHz. We notice that these upsampled samples may lack high-frequency information. To maintain consistency, we downsample or upsample all audio samples to a 16kHz sampling rate during training and testing.
However, the evaluation set exhibits some statistical diferences compared to the training set. The evaluation set includes audio samples with sampling rates of 16kHz, 24kHz, and 44.1kHz. On the other hand, the duration distribution of the evaluation set can be seen in Figure 2. Although there are certain statistical inferences present in the evaluation set, we choose not to exploit these statistical inference in decision to ensure the generalizability of our method. 4.3.2. Visualization Analysis In order to assess the similarity between the training and test data distributions, we utilized a trained embedding extractor to extract embeddings from the train, development, and evaluation sets. Subsequently, we employed a Gaussian Mixture Model (GMM) with 512 components to perform one-class modeling on the train set. By calculating the log-likelihood for all embeddings on the one-class GMM, we obtained score distributions of three sets as shown in Figure 3.
Upon analyzing the results, we observe that the score distributions of the evaluation set and development set are relatively similar. However, there is a significant disparity between the score distribution of the evaluation set and the training sets. This observation highlights the considerable challenge presented by the evaluation set in terms of testing the generalization performance of our proposed method.
In addition, we also present the t-SNE visualization of the training set and the evaluation set in Figure 4. The embeddings t-SNE used are extracted by ResNet34SimAMASP model. The label of evaluation set is marked by center-based maximum similarity method. We have observed that, except for the samples at the margins, most of the known classes can be recognized efectively. However, the unknown class does not exhibit a clear intra-class center and is not easily detected. Therefore, we speculate that the unknown class may not consist of a single category but rather a collection of multiple unknown classes. (a) t-SNE of embeddings in the (b) t-SNE of embeddings in the training set evaluation set
Table 1 presents the final performance of various systems using center-based maximum similarity method. Firstly, by comparing diferent models with the addition of the development set (e.g., ID 1&2, ID 4&5, and ID 8&9), no clear improvement or decrease in performance was observed. This could potentially be attributed to the limited amount of data, which makes the models prone to overfitting and results in unstable performance. Based on the score distribution depicted in Figure 3, the score trends of evaluation set and development set are similar.
Therefore, in order to select the best model of each system, many models are not trained using the development set.
Secondly, we analyze the impact of acoustic features on the recognition performance. Since diferent models exhibited similar performance trends with diferent features, the focus is mainly on comparing model ID 4 and model ID 6 in terms of diferent features. It was found that the spectrum feature tended to overfit on the development set, indicating that the performance on the development set alone might not fully reflect the generalization performance of the models. In addition, performance of the SSL-based models are also not satisfactorily.
Considering the utilization of the center-based max- set. Finally, by model fusion under the score level, we imum similarity method, the inter-class similarity matrix achieved the 0.8312 F1-score in the evaluation set of is computed for diferent models, as shown in Figure 5. ADD2023 Track3.
By combining Table 1 and Figure 5, it is observed that better results are achieved when the inter-class similarity distances are larger. Therefore, the best single model of 7. Acknowledgments each system is selected by considering both the inter-class similarity matrix and the results on the development set. This research is funded in part by the National Natural
Model fusion is performed at the score level to achieve Science Foundation of China (62171207) and MaShang complementary performance between models. Several Consumer Finance Co.Ltd. Many thanks for the compuoptimal models with diferent modeling approaches are tational resource provided by the Advanced Computing selected. Finally, an F1 score of 0.8312 is achieved on the East China Sub-Center. evaluation set.
tion as speaker verification and propose the center-based maximum similarity method to determine the test audio category. We select the best single model according to the intra-class similarity matrix and result of development