Self-supervised speech models are a rapidly developing research topic in fake audio detection. Many pre-trained models can serve as feature extractors, learning richer and higher-level speech features. However,when fine-tuning pre-trained models, there is often a challenge of excessively long training times and high memory consumption, and complete fine-tuning is also very expensive. To alleviate this problem, we apply low-rank adaptation(LoRA) to the wav2vec2 model, freezing the pre-trained model weights and injecting a trainable rank-decomposition matrix into each layer of the transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared with fine-tuning with Adam on the wav2vec2 model containing 317M training parameters, LoRA achieved similar performance by reducing the number of trainable parameters by 198 times.
information for identifying fake audio. In addition to
wav2vec2, other studies have investigated various
selfSelf-supervised pre-training has become a popular re- supervised models as potential feature extractors for the
search topic in recent years and has already been applied spoof detection task [
Q + LoRA K + LoRA K V +
LoRA
× ∈ ℝ
+ d
B
r
A
tuning methods are not systematically studied with self- were inspired by [
Inspired by Hu et al. [
The main contributions of this study can be summa- output after low-rank decomposition as follows: rized as follows: • To our best knowledge, we are the first to apply the LoRA method to pre-trained models for fake audio detection. • The study found that our proposed method can significantly reduce the number of model parameters while maintaining high performance and improving training eficiency.
Fine-tuning a pre-trained model is a common method used to further improve its performance on a specific task. However, the requirement that the newly fine-tuned model has the same number of parameters as the original model results in a significant increase in computing resources and higher costs. To address this issue, we ℎ = ℎ + ∆ = ℎ + (1)
In this study, we applied the LoRA technique at three
diferent locations of the multi-head attention layer in
the wav2vec2 transformer, specifically at the query, key,
and value vectors in the self-attention module, as shown
in Figure 1. Unlike the fully fine-tuned model, LoRA
introduces no inference latency and roughly converges to
training the original model’s performance within
approximately the same number of training iterations [
3.1. Dataset
ASVspoof 2019 LA [
The LA data set contains three subsets: the training
set, the development set, and the evaluation set.
Table 1 details the number of real and fake audio of the
ASVspoof2019 LA dataset. The training set and
development set mainly include four TTS and two VC algorithms,
namely A01-A06. To better evaluate the performance of
the system, unseen spoofing attacks were added to the
evaluation set, including two known spoofing attacks
(A16 and A19) and 11 unseen spoofing attacks (A07-A15,
A17, and A18).
3.2. Evaluation Metrics
In this work, in order to evaluate the results of
diferent fake audio detection systems, the equal error rate
(EER) is used as the evaluation metrics. Previously, EER
is used in the ASVspoof challenges and ADD 2022
challenge [
( ) = ( ) = #{ ℎ > } #{ }
(2) #{ ℎ < } #{ } 3.3. Experimental Setup We first focus on the efect of rank r on model performance. We found that the number of model parameters is positively correlated with the value of R, indicating that the model’s parameter count increases with increasing r. Surprisingly, the model achieved the best EER of 1.30 when R was set to 4, outperforming the results of the other three ranks. It is worth noting that even with a very low value of r, r=2, the model’s EER was still better than when the model’s parameters were fixed, demonstrating the efectiveness of combining LoRA with Wav2vec2.
However, when the rank of LoRA was set to 8 and 16, the EER results were slightly worse than the rank of 4. This may be due to injecting a rank that was too large, leading to an increase in the number of model parameters and an increase in overfitting.
Table 3 shows the efect of diferent weight types on the
results. We found that, for all weight types and low-rank
matrix ranks, the LoRA model outperforms the baseline
model in terms of EER, indicating that LoRA’s low-rank
matrix adaptation technique can efectively improve the
performance of ASVspoof detection. Furthermore, for
each weight type, as the low-rank matrix rank r increases,
the performance of the LoRA model slightly improves,
but the improvement gradually decreases. This suggests
the paper[
For the backend classifier, we chose light convolution indicating that applying LoRA to the query and value
neural network (LCNN), which is the baseline system for matrices of the Transformer can lead to better
perforASVspoof2019 and 2021. Other settings refer to the paper mance.
[
To train the model, we use the Adam optimizer with a model performance. As expected, the ASVspoof deteclearning rate of 1 × 10−5 . Due to the limitation of GPU tion system exhibits a continuous decrease in EER performemory, we set the batch size to 16. The model is trained mance with an increase in audio length. Longer speech for 50 epochs. The training set is used to train the model, signals provide more information, facilitating the diferthe development set is used to select the model with the entiation of genuine from spoofed speech. The poorest best performance, and finally, the evaluation set is used EER performance is observed at the shortest input length for evaluation. of 1 second, with a score of 10.27%. This highlights the challenge of the ASVspoof detection task in short audio scenes. In contrast, the EER performance of the ASVspoof 4. Results and Discussion detection system drops to 1.18% when the input length increases to 8 seconds, approaching the optimal performance. Combining the findings in Table 6, we conclude that increasing the audio length can improve the performance of the ASVspoof detection system, but it also leads to higher memory requirements during training.
Therefore, it is crucial to strike a balance between model performance and training eficiency.
Based on the aforementioned experimental results, we
set the rank r to 2, the length of the input audio to 4
seconds, and apply the LoRA method to the and
weight matrices. Adapter [
Table 6 presents the results of our proposed method Commission,Administrative Commission of Zhongguanon improving training eficiency, showing that LoRA out- cun Science Park No.Z211100004821013, Open Research performs global fine-tuning in terms of training time Projects of Zhejiang Lab (NO. 2021KH0AB06). and memory eficiency. Specifically, for all audio lengths, LoRA’s training time is significantly shorter than global References ifne-tuning, and this improvement becomes more pronounced with increasing batch sizes. For instance, when the audio length is 1 second, LoRA’s training time only takes 46 seconds with a batch size of 16 samples, while global fine-tuning requires 1 minute and 37 seconds. Additionally, compared to global fine-tuning, LoRA can handle larger batch sizes without running out of memory.
For example, when the audio length is 4 seconds, LoRA can handle a maximum batch size of 8, while global finetuning runs out of memory with a batch size of 4. Overall, these results suggest that LoRA is a promising approach that can improve the training eficiency of the wav2vec2 model and lower the training threshold of hardware.
In conclusion, the paper proposes a novel low-rank adaptation method (LoRA) to improve the eficiency and performance of the wav2vec2 model for the fake audio detection task. The experimental results show that both global and local fine-tuning can improve the performance compared to the fixed pre-trained model parameters. However, the proposed LoRA method significantly reduces the number of trainable parameters while achieving performance comparable to global fine-tuning, indicating that LoRA can maintain high performance while reducing the parameter count. Additionally, the paper finds that the performance of LoRA is limited by the low-rank matrix rank, and therefore using a higher rank does not significantly improve performance. Among each weight type, applying LoRA to the query and value matrices (2019).