The rapid advancement of spoofing algorithms necessitates the development of robust detection methods capable of accurately identifying emerging fake audio. Traditional approaches, such as finetuning on new datasets containing these novel spoofing algorithms, are computationally intensive and pose a risk of impairing the acquired knowledge of known fake audio types. To address these challenges, this paper proposes an innovative approach that mitigates the limitations associated with finetuning. We introduce the concept of training low-rank adaptation matrices tailored specifically to the newly emerging fake audio types. During the inference stage, these adaptation matrices are combined with the existing model to generate the final prediction output. Extensive experimentation is conducted to evaluate the eficacy of the proposed method. The results demonstrate that our approach efectively preserves the prediction accuracy of the existing model for known fake audio types. Furthermore, our approach ofers several advantages, including reduced storage memory requirements and lower equal error rates compared to conventional finetuning methods, particularly on specific spoofing algorithms.
seen types of fake audio. However, finetuning the model
on the new dataset can disrupt the knowledge model
In recent years, there has been a significant concern sur- learned from the old dataset, leading to a decrease in the
rounding the issue of audio forgery attacks. Detection recognition accuracy of the model for fake audio
genermodels for detecting fake audio, based on handcrafted ated by known spoofing algorithms, which is known as
features [
The most intensive way to improve the detection accu- than finetuning the whole model for improving the recogracy of the model against new spoofing algorithms is to nized accuracy of the unseen fake audio. During training ifnetune the model on a new dataset including those un- on the new dataset that includes those unseen fake audio, we load the source model (SoM), which is a saved model training on the old dataset, and freeze all its parameters. This allows us to solely focus on training two adaptive matrices, namely A and B, as introduced by the LoRA algorithm. When performing inferences on the new dataset, we load the SoM together with the two adaptive matrices. Conversely, when dealing with the old
Load
Save SoM Load AB (a) Save BB
Ac Save Bc
SoM Load
PredictionA Load AB Load Ac BB Load Bc Load
PredictionB (b)
PredictionC
SoM, efectively evading the risk of damage to the knowl- downstream task. During model inference, LoRA
simuledge obtained from known instances of fake audio in the taneously loads the large-scale pre-trained model and
old dataset. Additionally, our approach boasts an advan- the two matrices A and B trained specifically for that
tage in terms of storage memory consumption, as it only downstream task. Compared to existing methods that
necessitates the storage of the two low-rank adaptive introduce adapter layers [
Contribution: We propose a method based on Low- times.
Rank Adaption to address the issue of low recognized accuracy when models encounter new and unknown types of fake audio in fake audio detection. Compared 3. Methodology to the commonly used fine-tuning approach, our method requires lower storage space and avoids forgetting the knowledge learned from existing known types of fake audio. Additionally, experimental results demonstrate that our method achieves higher recognized accuracy for certain unknown types of spoofing algorithms compared to finetuning.
Low-Rank Adaption (LoRA)[
In each downstream task, LoRA first loads the large-scale pre-trained model but freezes all its parameters. Then, it initializes matrices A and B as zero matrices and trains only these two matrices using the training dataset of the When facing unknown types of fake audio generated by unknown algorithms, the accuracy of deep neural networks would significantly decrease compared to the known types included in the training set. The finetuningbased methods may damage the learned knowledge of the model, leading to a reduction in the detection accuracy of known types of fake audio. To address this problem, we propose a new method based on LoRA to improve the recognized ability of the model to detect unknown types of fake audio. The training and inference processes of our method are illustrated in Fig 1a and 1b, respectively.
We consider a model designed for fake speech detection,
where there exists an initial dataset A containing some
fake audio generated by known spoofing algorithms, and
two additional datasets B and C, both of which contain
new types of fake audio not present in dataset A. We first
train a source model (SoM) on dataset A. As SoM has
not seen the new types of fake audio in datasets B and C,
it can achieve good recognition performance on dataset
A, but its performance would significantly decrease on
datasets B and C. To improve the detection accuracy of
new types of fake audio in dataset B, we load the saved
SoM trained on dataset A, freeze all its parameters, and we enable our model to achieve self-incremental
learnintroduce two low-rank adaptive matrices A and B ing within limited storage space, thus defending against
specifically designed for dataset B. Both A and B are emerging attacks from new spoofing algorithms.
initialized as all-zero matrices with rank , which is
much lower than the rank of SoM. During the training
process on dataset B, we simultaneously feed data into 4. Experiment
both SoM and the adaptive matrices A and B . The
outputs from both components are summed to generate 4.1. Datasets
the output of the model ℎ, as shown in Equation 1. Three fake audio datasets are selected for our
experiments, including the ASVspoof2019LA [
ASVspoof2019LA is a dataset widely used in the field of fake audio detection. It was created as part of an international challenge that aimed to evaluate the performance of automatic speaker verification systems in detecting spoofing attacks. The dataset consists of a large collection of both genuine and spoofed speech recordings, where spoofed speech refers to artificially generated or manipulated audio designed to deceive speaker verification systems.
ASVspoof2015 is another important dataset used in fake audio detection research. The dataset contains both genuine and spoofed speech recordings, with various types of spoofing attacks, such as speech synthesis, voice conversion, and replay attacks. ASVspoof2015 ofers a diverse range of spoofing techniques, making it a valuable resource for studying and developing robust countermeasures against fake audio.
In-the-Wild is a commonly used collection of realworld audio recordings that encompass a broad range of environments and scenarios. Unlike the aforementioned datasets that focus on specific spoofing attacks, In-the-Wild captures audio data from various sources and situations encountered in everyday life. This dataset aims to simulate the challenges faced by fake audio detection systems when dealing with uncontrolled and unpredictable acoustic conditions. We divide the genuine and fake audios of the In-the-Wild dataset into two subsets.
One-third is used to build the training set, and the rest is used as the evaluation set. where the represents the input batch of data and the ℎ is the output state of the model. While the parameters of SoM remain unchanged during the training on dataset B, we optimize the parameters of the two low-rank adaptive matrices A and B to learn new features for the new types of fake audio and optimize the detection performance of the model on these types. Once the training on dataset B is completed, we only need to save the two low-rank adaptive matrices, A and B , instead of the entire model. The same process is repeated on dataset C, allowing us to train two additional low-rank adaptive matrices, A and B , specifically tailored to the new types of fake audio in dataset C.
Our method follows a similar inference process across diferent datasets, as shown in Fig 1b. When the model predicts a fake audio type belonging to Dataset A, we only load the SoM. Since the parameters of the SoM are frozen and not involved in the training process on Datasets B and C, the detection performance on Dataset A is not disrupted by the learned features from the new fake audio in Datasets B and C. When the model predicts a fake audio type from Dataset B, both the source model SoM and the low-rank adaptive matrices A and B are loaded into the model, and the output follows Equation 1. Although the parameters of the SoM are not updated during training, the model can learn the features of the new fake audio type by training the parameters of the two adaptive matrices. As a result, compared to using only the source model SoM, the detection accuracy of the model is significantly improved. Similarly, when the model faces fake audio types from Dataset C, we can load the SoM and the adaptive matrices A and B trained specifically for Dataset C.
Overall, our algorithm provides a low-cost incremental learning method for the model. As audio spoofing algorithms continue to evolve, we can view Dataset A as consisting of the known spoofing algorithms and set a time period , after which we collect new fake audio generated by emerging spoofing algorithms to build Dataset B.
We apply our method to incrementally learn the model on Dataset B. After another time period of 2, we repeat the process to construct Dataset C and perform incremental learning on Dataset C. Through this approach, 4.2. Experimental Setup In our experiments, the Low-Level Cepstral Coeficients (LFCC) [19] feature has been selected as the input feature extracted from each audio. The classifier is the Squeeze-and-Excitation Network (SENet) [20] with three sub-layers. All of them include three basic blocks introduced by the SENet. There is one conv2d layer before each sub-layer. The input dim and output dim of the first conv2d are 1 and 128, respectively. The second conv2d and the third have input and output dim 128 and 256, 256 and 512, respectively. The kernel sizes of them are 9, 7,
Dataset EER(%)
ASVspoof2019LA 6.51
ASVspoof2015 In-the-Wild
51.77 51.71 4.3. Only trained on ASVspoof2019LA We first test the recognized performance of the deep neural network against fake audio generated by known and unknown algorithms, respectively. We consider the datasets ASVspoof2019LA, ASVspoof2015, and In-theWild as datasets A, B, and C, respectively, as shown in Fig 1. We train the model only on the training set of ASVspoof2019LA and evaluate it on the evaluation sets of the three datasets. The experimental results are shown in Table 1. The results indicate that the model has high accuracy when faced with known types of fake audio that have appeared in the training set, but its recognized performance will degrade considerably when faced with fake audio generated by new and unknown spoofing algorithms.
EER(%)
SoM Finetuning Our method
ASVspoof2019LA 6.51 35.31 6.51
ASVspoof2015 In-the-Wild 51.77 51.71 45.13 1.39 2.38 1.25 4.4. The comparison on EER between finetuning and our method on learning between two datasets In this section, we compare the recognized performance between our method and finetuning on two-dataset learning condition. We set two experimental situations: the ifrst is the model first trained on the ASVspoof2019LA and then trained on the ASVspoof2015; the second is the model first trained on the ASVspoof2019LA and then trained on the In-the-Wild. The results of these two experiments are shown in Table 2a and 2b, respectively.
From the second column of the two tables, we can easily observe that training on the new dataset is really beneficial for the detection of new fake audio generated from new spoofing algorithms. However, from the comparison in the first column, we can observe that finetuning on the new dataset will definitely disrupt the learned knowledge from the known types of fake audio (6.51 → 49.03, 6.51 → 33.05). Compared to finetuning, our method freeze the parameters of the SoM and only trained two adaptive matrices to learn new knowledge from the new dataset. In this case, the recognized performance of the known fake audio types will remain unchanged even after training on the new dataset 1b, which is evaluated in our results shown in Table 2. From the comparison on the learning performance between our method and finetuning in the second column, we can also see that our method achieves a higher recognized accuracy in ASVspoof2019LA → ASVspoof2015, which shows that our method has a positive efect on the learning on specific unknown spoofing algorithms. 4.5. The comparison on EER between finetuning and our method on learning among three datasets To evaluate the efectiveness of our method in multidataset learning, we also compare the recognized performance between our method and finetuning on three datasets learning condition. In our experiment, we first trained our model in the ASVspoof2019LA and saved the completed source model SoM. After that, we trained
the SoM first on the ASVspoof2015 and then on the Inthe-Wild, and saved the adaptive matrices AB, BB and AC, BC, respectively. The inference process is shown This work is supported by the National Natural in Fig 1b and the comparison result is illustrated in Table Science Foundation of China (NSFC) (No.61831022, 4. From the comparison of the first two rows in the re- No.U21B2010, No.62101553, No.61971419, No.62006223, sult, we can observe that finetuning on new datasets will No.62276259, No.62201572, No. 62206278), Beijing Mureduce the recognized accuracy on old datasets, which nicipal Science and Technology Commission, Adminshows that the detection accuracy of the known types of istrative Commission of Zhongguancun Science Park fake audio will considerably decrease after finetuning on No.Z211100004821013, Open Research Projects of Zhethe unknown types of fake audio. However, we can see jiang Lab (NO. 2021KH0AB06). that our method still remains unchanged in old datasets ASVspoof2019LA and ASVspoof2015 and achieves lower References EER than finetuning on the final dataset. 4.6. The comparison on storage memory
between finetuning and our method In order to improve the recognition performance of the model, we train it according to the process illustrated in Fig 1. After training on the new datasets ASVspoof2015 and In-the-Wild, we compare the storage memory between the whole model and adaptive matrices saved by ifnetuning and our method, respectively, which has been shown in Table 3. The experimental result shows that our method achieves a marked success in squeezing storage memory. Under the setting illustrated in Sec 4.2, our method greatly reduces the number of trainable parameters and the storage memory requirement by about 30 times, which makes the model can be easily applied in many strict memory constraint situations.
In this paper, we propose a method to address the problem of low detection accuracy of models facing newly emerging fake audio generated by new spoofing algorithms. In the training process, we train two low-rank adaptation matrices A and B specifically for these new types of fake audio. During inference, we simultaneously load the existing model and these adaptation matrices, and combine their prediction outputs as our final prediction output. The experimental results demonstrate that our method does not degrade the prediction accuracy of the existing model for known types of fake audio because the existing model parameters are not modified during training on the new dataset. Moreover, our method has a lower storage memory requirement and lower equal error rates on some specific spoofing algorithms compared to finetuning. These findings encourage further investigation into countering the ever-evolving landscape of audio spoofing while maintaining the learned knowledge of known types of fake audio.