This paper describes a deepfake algorithm recognition system submitted to the Audio Deep Synthesis Detection (ADD) Challenge Track 3, which aiming to recognize the algorithms of the deepfake utterances. Given the complex noise present in the testing data and the existence of unknown deepfake algorithms, we propose a manifold-based multi-model fusion approach for open-set recognition. This approach constructs a manifold space to fuse the deep embedding features extracted by diferent models and computes the geodesic distance between the manifold spaces of diferent deepfake algorithms to distinguish unknown deepfake methods. Experimental results demonstrate the efectiveness of the proposed strategy in multi-model fusion. The proposed system obtained the F1-score of 0.7934 in ADD Track 3 testing.
The rest of this paper is organized as follows: Section 2 describes the task. Section 3 presents the related work and illustrates our proposed method. Results and discussions are reported in Section 4. Finally, the paper is concluded in Section 5.
To handle the complexity of the testing data, we explored three categories of features: raw waveform, hand-crafted features, and pre-trained features. Our expectation was that a combination of these features would be able to capture the divergences among diferent deepfake algo
Based on the findings in literature [
There are 22,400 training data, 8,400 developing data, cepstral coeficients (CQCC), linear frequency cepstral
and 79,490 testing data. In addition to containing un- coeficients (LFCC), and log power magnitude
spectroknown categories, the noise of the testing data is much gram (Spec) [
Metrics for this track is the macro-average precision, Due to the complexity of the testing data and the
recall, and F1-score. scarcity of available training data, we utilized a
pretrained model to extract essential speech features.
Recently, some pre-trained speech models, including
3. System description Wav2vec 2.0 [
First, by observing the training data, we found that the audio were sampled at 16 or 24 kHz, and the volume of the audio varies relatively widely. Thus, the whole audio were uniformly resampled to 16 kHz and normalized.
Then, by examining the testing data, compared to the
relatively clean training and developing data, the noise
interference in the test data was more complicated, then
data augmentation was performed on training and
developing data with MUSAN [
Finally, some completely silent segments with zerovolume were found in these datasets. Although this may be a characteristic of some deepfake methods, the silent segments that appear at the beginning and the end of the audio were cropped out considering the generalized application of the model.
In our work, we utilized three diferent deep networks: rawnet2, SE-Res2Net50, and HuBERT.
rawnet2 [
SE-Res2Net50 [
HuBERT is a self-supervised learning pre-trained
model and is available in several versions. We utilized
the chinese-hubert-large [
To ensure the best performance, we selected the final model for testing from the above mentioned models with the highest F1-score in the developing dataset.
To classify the categories of deepfake audio and
identify unknown deepfake means, we adopted the manifold
space and manifold distance. Firstly, the manifold space
of each deepfake category was constructed using the
ONPE method [
(1, 2) = ‖Θ‖ 2, ‖Θ‖ 2 = [ 1, 2, . . . , ], (1) where the geodesic distance (1, 2) was calculated based on the principal angles [ 1, 2, . . . , ] between spaces (1, 2), which were obtained from the orthonormal basis matrix (obtained by ONPE) and singular value decomposition.
((,) − ) (,) = ∑︀6 =0((,) − ) , = ((,0), (,1), . . . , (,6)), (,) = −( , ), (2) (3) (4) where (,) represents the similarity score between the testing data and the deepfake category , while (,)( = 0, 1, . . . , 6) represents the negative of the geodesic distance between the testing data manifold space and the deepfake method manifold space .
To efectively improve the final recognition results, we conducted model fusion at three levels.
First is the label layer fusion. In the output scores of rawnet2, SE-Res2Net50 and HuBERT models, the index corresponding to the maximum score was set to be the output label. A threshold was set for open-set recognition based on model training and validation. The output labels were secondary adjusted and those with scores less than the threshold were considered as unknown label 7. Finally, three sets of recognition label values were thus obtained for the testing data. The mode of the three sets of labels was used as the fused label. When all three sets of labels were diferent, the result from HuBERT model was chosen as the fused result because it had the best performance.
Next is the score-level fusion. A common score fusion
method is conducted by calculating the mean of multiple
sets of scores. As discussed in literature [
Finally, feature-level fusion is performed, as shown in Figure 2. For diferent models, the 256-dimensional output of the penultimate layer was connected as the embedding features, and then used to construct the manifold space for each class, and the spatial distance was calculated as the similarity score.
Specifically, training data for each deepfake method were input to the trained models to obtain × 256× feature matrices. The feature matrices were then processed by ONPE to obtain the manifold space of the deepfake method. Next, the testing data were segmented into segments of length 3 with a shift of 1, and audio segments were obtained. The segments were input to the three trained models to obtain × 256 × feature matrices. The feature matrices were processed by ONPE to obtain the manifold space of the testing data. The geodesic distance and softmax score between manifold space of training data and manifold space of testing data were calculated as the final fusion score. If the maximum score was higher than the threshold, the index corresponding to the maximum score was set to be the output label, otherwise the label was set to 7 as a new label, and the threshold was fine-tuned by testing data.
results, in fact it was less efective than the approach of
Se-Res2Net50 with LFCC, probably because the model
was relatively simple and overfitting was more serious.
Maybe we should match the model with appropriate
subsequent classification networks so as to train a model
with excellent discriminative ability. To visualize the
efectiveness of the proposed method, we also used
tSNE [
Secondly, in terms of fusion strategies, it can be seen that manifold-based feature-level fusion got the best performance, while the score-level fusion by inference augmentation performed better than common score fusion method (shown as F21 vs F22 and F41 vs F42). As our trained rawnet2 model got poor performance and it pulled down the overall performance in label-level fusion with the other two models (shown as F1), it was not considered in the subsequent score-level fusion and feature-level fusion.
According to the results of score-level fusion and feature-level fusion, it indicated that there was complementary information among the diferent models, and by constructing the manifold space and measuring the geodesic distance, further discriminative information was extracted, thus enhancing the overall recognition performance.
Thirdly, in data augmentation, shown as B11 to B22, due to the variability of background noise between the training and testing data, by adding noise to the training data was efective in improving the model performance. However, unexpectedly, the performance of the HuBERT model trained on the augmented data was not as good as that of the HuBERT model trained on the original training data (shown as B31 vs B32). One possible reason was that the training data of the pre-trained models already contain rich noisy data, which itself can shield the efect of noise on speech. In addition, due to the time constraint of the competition, all models were obtained by training a set of parameters and no parameter tuning was performed, which may also be a reason.
Finally, it should be noted that, F3 in Table 1 with a F1-score of 0.7352, was the best result we submitted to ADD Track 3 during the competition and is ranked 5th. After the competition, when we conduct supplementary experiments on data augmentation, a better result was found as B31. Then we conduct relevant fusion experiments and obtained results shown as F4 and F5, with the best result up to 0.7934, which so far can rank 3rd in the competition. Despite this, the conclusion that featurelevel fusion was better than fractional-level fusion was consistent.
The existing fake audio recognition systems often rely on three types of architectures: handcrafted features with classifiers, end-to-end classification models, and pre-trained feature extractors with classifiers. In ADD Track 3, we explored three models and three multi-model fusion strategies. Experiments demonstrated the efectiveness of the proposed manifold-based feature-level fusion strategy. And the proposed score-level fusion by inference augmentation provided an attempt to solve the fusion of models with an overfitting tendency. In addition, we experimented the efect of data augmentation on model performance enhancement. Finally, the proposed model fusion method obtained the F1-score of 0.7934 in ADD Track3 testing.