Thanks to recent advancements in end-to-end speech modeling technology, it has become increasingly feasible to imitate and clone a user's voice. This leads to a significant challenge in diferentiating between authentic and fabricated audio segments. To address the issue of user voice abuse and misuse, the second Audio Deepfake Detection Challenge (ADD 2023) aims to detect and analyze deepfake speech utterances. Specifically, Track 2, named the Manipulation Region Location (RL), aims to pinpoint the location of manipulated regions in audio, which can be present in both real and generated audio segments. We propose our novel TranssionADD system as a solution to the challenging problem of model robustness and audio segment outliers in the trace competition. Our system provides three unique contributions: 1) we adapt sequence tagging task for audio deepfake detection; 2) we improve model generalization by various data augmentation techniques; 3) we incorporate multi-frame detection (MFD) module to overcome limited representation provided by a single frame and use isolated-frame penalty (IFP) loss to handle outliers in segments. Our best submission achieved 2nd place in Track 2, demonstrating the efectiveness and robustness of our proposed system.
hashing. Neither of these two works is suitable for sit- convolution with filter size and kernel size set to be 256
uations where the forged region is extremely similar to and 3 respectively. After each convolutional layer, batch
its neighbor, such as a segment forged by noise addition normalization and Relu activation are applied.
only. Wu et al. [
To solve these problems, we propose TranssionADD 2.2. Data Augmentation
system, which specifically addresses poor generalization
and outlier problems of the location model. Firstly, we In this study, there are two major dificulties: 1) the data
innovatively convert the task into a sequence tagging is very limited and imbalanced; 2) the training and test
problem, predicting the label of each frame and merg- sets have huge disparities in terms of their domain. The
ing frames with the same label to locate audio segments. model, therefore, sufers from underperformance. To
Secondly, we use RCNN-BLSTM to extract spatial and address this issue, we propose a two-stage data
augmentemporal features frame by frame. To avoid the network’s tation method.
poor learning of contextual information, we use multi- Before-training augmentation. Both the training
frame detection (MFD) module to compress multi-frame set and dev set are insuficient and lack diversity.
Thereinformation. Then we address data insuficiency by apply- fore, we propose three data augmentation methods:
ing data augmentation including voice conversion, pitch
shift, etc. Finally, to solve outliers from isolated frames,
we introduce a penalty strategy called isolated-frame
penalty (IFP) to constrain this situation. Our submission
won second place in the ADD 2023. More explicitly, our
system mainly contributes to the following parts:
• Voice conversion. Use VC [
pus to entire audio segments. • Insertion. Insert a real audio clip with diferent lengths randomly selected from other audio or the audio itself. • We design a novel location model that draws inspiration from the sequence tagging task, specifically the Name Entity Recognition (NER) task. • We utilize data augmentation method both before
training and during training. • We apply MFD module to improve robustness and
add IFP loss to deal with outliers. • Experiments show our proposed method per
forms better than the baseline.
In this section, we introduce details and rationales of our proposed method, which mainly consists of four parts: 1) RCNN-BLSTM backbone; 2) data augmentation; 3) multiframe detection module; and 4) isolated-frame penalty loss.
As shown in Figure 1, we use mel-spectrograms extracted
from audio as input representations, and then we build a
basic model based on residual convolutional neural
network (RCNN) and bi-directional long short-term
memory (BLSTM). Some of our structures refer to the
design of RawNet2 [
RCNN primarily focuses on single-frame information but
lacks a broader context. To further extract more
efective contextual information, we draw inspiration from
wav2vec2.0 [
Linear + repeat
Linear BLSTM
MFD
X N x0 x1 x2 x3 x4 ...
x8 x9
... xn RCNN encoder
+ BatchNorm
Relu Conv1D
BatchNorm
Relu Conv1D (stride=2) BatchNorm
Relu Conv1D (stride=5) (a) Our model framework (b) RCNN encoder (c) MFD a classification network using the estimated label as the simultaneously calculating the diference between curtarget. rent frame and its surrounding frames (1 to 3 before and
Precisely, as shown in Figure 1 , we use two convo- after). Particularly, for frame , its predicted probabillutional layers to build the downsampling network and ity is ˆ, and we consider the previous and following one dense layer to form the classification network. The frames, ∈ {1, 2, 3} . Correspondingly, we can obtain iflter size of both convolutional layers is set to 128. The the regularization term . stride values are set to be 5 and 2; the kernel sizes are sMetFDto ibsea7ddaendd t3o, rtheseptercatiinvienlyg. lTohsse. cFrionsasl-leyn,twroepryepl oliscsaotef () = ||ˆ − ∑︀=1(ˆ2− + ˆ+) || (1) and add the outputs of the MFD module to the outputs Finally, we calculate the regularity constraint of total of BLSTM for each frame, to enhance the classification N frames and divide the sum by 3 as the final regularity prediction accuracy for each frame. constraint term . The formula is as follows:
In traditional NER tasks, one entity corresponds to one =1 or several word(s); while in ADD tasks, one fake/real seg- Then, our total loss involves losses of multiple modules, ment corresponds to tens or hundreds of frames. Conse- and its expression is as follows: quently, when applying regular NER to audio frame-level predictions, a challenge arises wherein the model pre- = + + * (3) dicts labels that appear on a significantly higher number of discontinuous frames, which we refer to as outliers. Where and represent the cross-entropy Therefore, in order to constrain the case where the pre- loss for the single frame and each MFD module, respecdicted value is diferent from the surrounding values, we tively, while serves as the constraint applied to isolated design an additional loss function called isolated-frame small fragments. It is important to note that the weight penalty (IFP) loss. assigned to the IFP loss is a hyperparameter.
Specifically, we devise a penalty calculation method: = ∑︁((1) + (2) + (3))/3 (2)
Our dataset is a publicly available Mandarin corpus provided by organizers of ADD 2023. Some samples are all real or all fake, and some are partially fake or partially real.
We compare mel-spectrograms and MFCCs, and finally we choose mel-spectrograms as low-level acoustic representation due to the better performance. We use a 16 kHz sampling rate for all experiments. And window size of fast Fourier transform (FFT), hop size, and number of output bins are set to 800, 160, and 80 respectively.
Before training, we augment the training set and dev set using techniques such as VC, noise addition, and insertion. During training, we use a 0.2 random rate to add Gaussian noise with less than 15 dB signal-to-noise ratios (SNR) and change the audio pitch by pitch shift.
We first train the model on one NVIDIA 3090 GPU, with 64 batch sizes and 100 epochs. The models are optimized by Adam with a learning rate of 0.0001 and weight decay of 10−5 . The hyper-parameter weight , in the isolatedframe penalty (IFP) loss, is set to be 0.1 as default, based on simple tuning experiments. 3.2.4. Metrics As follow, is used to measure the model’s ability to correctly distinguish between genuine and fake audio, and 1_ is used to measure the model’s ability to correctly identify fake areas from fake audios. The final score is the weighted sum of and 1_.
= 0.3 * + 0.7 * 1_ (4)
Moreover, we consider audio should consist of long enough consecutive frames with the same label and isolated frames are possible to be outliers. Therefore, relying solely on and 1_ may not adequately capture the occurrence of fragmented and isolated segments in audio frame predictions. To evaluate the eficacy of dealing with outliers, we introduce a simple auxiliary evaluation metric called "iso-rate," as defined in Eq. 5. If a segment is shorter than 6 frames (60ms), we classify it as an isolated segment (), and then compute the proportion of these isolated segments appearing across the entire dataset. A lower iso-rate indicates that the evaluated system has fewer isolated segments and addresses outliers better.
iso-rate = (5)
From Table 1 above, our system achieves significantly higher scores on the development set than on the test set which may contains more variable and complicated samples. To address this problem, we introduce the augmented development set (aug-dev) as an essential reference to evaluate the model’s performance.
We also conduct ablation studies to demonstrate the effectiveness of each part in our method. Specifically, we conduct individual experiments by removing diferent components, including before-training augmentation (BefAug), in-training augmentation (InAug), multi-frame detection module (MFD), and isolated-frame penalty (IFP).
From the multi-metric comparison in Table 2, we observe that the IFP significantly reduces iso-rate while improving the correct detection score, as evident from the second and third rows. Additionally, the removal of MFD module, as indicated in the fourth row, results in a significant decrease in correct detection score, indicating its notable contribution, despite limited impact on iso-rate. Moreover, the fifth and sixth rows demonstrate that using data augmentation both before-training and in-training (BefAug & InAug) leads to a significant improvement in performance, especially on more various augmented development set and test set.
In summary, we propose a basic RCNN-BLSTM backbone to classify each frame of audio. Then we enhance the stability and generalization ability of our base model through diverse data augmentation techniques. We also innovatively design a multi-frame detection (MFD) module and isolated-frame penalty (IFP) loss to improve the representation of local features and constrain outliers, resulting in improved model performance. These three aspects are crucial for our success in manipulated region detection challenge. In the future, to further improve the detection capabilities of our system, we will focus on enhancing the model’s robustness against noise.