This paper is a detailed description of a multi-stage audio synthesis system that participated in Track1.1 (spoofing) in ADD Challenge 2023 in which we play the role of an attacker. These stages include a fully end-to-end text-to-speech model, a fully end-to-end any-to-many voice conversion model, and an adversarial attacking model. We believe that such a design can reduce the possibility of artifact exposure at multiple levels and dimensions so that it is closer to real speech and is becoming hard to extinguish. Besides, we adopt post-processing methods to further improve our spoofing capability against detection methods for non-speech parts. Our system won 3rd place in the total score of Track1.1 in this challenge, especially, the performance in attacking black-box systems ranked 1st.
is judged to be more authentic.
The system designed in our paper adopts a multi-stage Audio deep synthesis techniques have been able to gener- speech generation method as shown in Figure 1. The first ate high-quality speech whose authenticity is dificult for stage is a state-of-art text-to-speech (TTS) module based humans to recognize. Meanwhile, as the quality of deeply on hierarchical feature modeling, the second stage is an synthesized speech approaches human’s natural voice, end-to-end any-to-many voice conversion (VC) module anti-spoofing systems have emerged and been continu- leveraging intermediate features from automatic speech ously upgraded for security purposes. So audio deepfake recognition (ASR), and the third stage is a gradient-based and anti-spoofing are in a game of attacking and defend- adversarial attacking (AA) module. In addition, we also ing. So far, there have been several authoritative chal- come up with some countermeasures against non-speech lenges for speech deepfake and anti-spoofing tasks such part detection, which we call the post-processing (PP) as ASVSpoof-2017/2019/2021[1, 2, 3] and ADD-2022[4]. module.
T
Track 1.1 of this challenge focuses on speech spoofing with so many stages? The motivation along with reattacks against anti-spoofing systems. Each participating lated works will be discussed in Section 2. The system team is required to design a speech generation system, in- framework and detailed methods adopted in diferent put the given text and speaker information, and generate components will be described in Section 3. Training setthe corresponding fake speech. Synthesized audio will tings will be described in Section 4. Experiments and our be sent to a group of black-box anti-spoofing systems rankings in the ADD Challenge will be demonstrated in (from submissions of other tracks) and a white-box anti- Section 5. spoofing system (the oficial baseline) for detection, and the deception success rate (DSR) will be used to measure the performance of a synthesizing system. A higher DSR 2. Related works and motivation means stronger deception ability, in other words, it can be an indication that the audio generated by the system TTS and VC are techniques that convert certain types of inputs to human speech waveforms – TTS accepts raw text or phoneme sequences, while VC takes in the voice of a source speaker. With the development of deep learning, research on TTS and VC based on deep neural networks has been carried out in large numbers.
TTS technology has made remarkable progress in its goal of generating natural speech which is close to human speaking[6, 7]. Representative contributions can be
Sentence
Word Phone
Hierarchical latent representation Text
Hier-TTS Text-to-speech (TTS)
Voice conversion (VC)
E2EVC
Target speaker PPG intermediate features
Attacking attempt
White-box Anti-spoofing
System
Return
Gradient characteristics Project Gradient
Descent Adversarial attacking (AA)
Post-processing (PP) Silence Replace
Global Additive Real silence sample
Parabolic crossfading
Final Waveform based series[12, 13], fully end-to-end VITS[14], difusion- through the hierarchical VAE structure, so that the genbased models[15, 16], etc. erated speech has good long-term features at multiple
Considering VC, models based on generative adversar- levels such as sentence level, phrase level, word level, ial networks (GAN) along with their variants, such as and frame level.
StarGAN-VC[17] and CycleGAN-VC[18] use a generator Additionally, machine learning models may misclassify or a conditional generator that transforms the source examples that are only slightly diferent from correctly speaker’s voice features to the target speaker’s directly. classified examples drawn from the data distribution[ 33]. Auto-encoder-based VC systems such as AutoVC[19] and In many cases, a wide variety of models with diferent VQVC[20] learn to reproduce their input as the output, architectures trained on diferent subsets of the training exhibiting efectiveness in disentangling speaker iden- data misclassify the same adversarial example. This sugtity information from linguistic content. Benefiting from gests that adversarial examples expose fundamental blind automatic speech recognition (ASR) systems pre-trained spots, so we can take advantage of this characteristic to with large corpora, models based on phonetic posteri- create specific samples to attack the classification system, orgrams (PPGs) are considered to have an advantage in which is called an adversarial attack (AA). The repreextracting speaker-independent acoustic features from sentative methods include Fast Gradient Sign Method source speech[21, 22]. (FGSM)[33], and the Projected Gradient Descent (PGD)
From the perspective of spoofing, we understand that algorithm[34] involved in this paper is exactly a variant
most anti-spoofing systems do not target specific arti- of FGSM. Such an AA stage may further compensate
facts, but use more general features[23, 24, 25, 26]. There- for the shortcomings of the aforementioned generative
fore, when synthesizing speech, we will try our best to models, hiding possible exposed artifacts under
adversarmove in a direction that is more authentic to human per- ial perturbations especially confronting white-box
antiception. But in the last year or two, the interpretability of spoofing systems.
antispoofing has become stronger and stronger, and some This is not enough. Some anti-spoofing systems even
representative works have focused on the detection of pay attention to non-speech parts like breath sounds
artifacts generated by vocoders[27, 28]. Therefore, in this and silent segments because generative models usually
article, we consider making the VC stage a complete end- lack attention to these non-speech parts[35, 36]. Our
to-end form[29], and adopt certain strategies to alleviate VC system can reduce the false creation of unreal
harartifacts such as chessboard efects. In addition, some monic structures in breath sounds because its front end
counterfeiting studies have begun to notice the inade- is an ASR-based phoneme classifier. Nevertheless,
requacy of the acoustic model, for example, detecting some garding silence, it is no
The system designed is a multi-stage waveform generation pipeline. The first stage is a state-of-art text-tospeech TTS module based on hierarchical feature modeling. This module analyzes the input text to predict prosody and style features at the word level, phrase level, and sentence level, and finally generates speech waveforms with high naturalness in long-term features. The second stage is an end-to-end any-to-many VC module leveraging PPG features extracted from a pre-trained ASR. We introduce the desired voiceprint of the target speaker at this stage and transfer the aforementioned waveforms to the target speaker’s timbre. The third AA stage is optional: It is dificult for us to impose this attack on the unknown black-box antispoofing systems, so we skip the stage; but confronting the white-box baseline system whose model is known, we use the adversarial attack method to further improve our spoofing ability. Finally, the generated speech waveform will also be processed in the PP stage to enhance the resistance to the detection of non-speech parts.
We adopt Hier-TTS [32] as the very first stage of the whole model that analyzes and processes text inputs and returns raw speech waveform. Because of its powerful ability to model the unified space of text and audio, HierTTS can reconstruct factors such as prosody and style better than common TTS models, therefore it may achieve better performance against some recent anti-spoofing systems which take inner-speaker long-time consistency of prosody or style into consideration.
Figure 2 is the overall structure which contains a Hierarchical Audio Encoder (HAE), a Hierarchical Context Encoder (HCE), and a Hierarchical Audio Decoder (HAD). Hier-TTS introduce five latent variables at diferent temporal resolution, including sentence, word, subword, phoneme, and frame level. First, HAE extracts the hierarchical latent variables from the linear spectrogram in
The VC model features the structure proposed in [29], which can be divided into five components as shown in Figure 3. In short, it incorporates a conformer encoder from an auto-speech-recognition (ASR) model and a series of transformer blocks on one stream. On another stream, a posterior encoder upon linear spectrograms is constructed. Outputs from those two modules are constrained to be subject to the same distribution, hence reducing the gap between the generated latent variables and the distribution of the real features. Then a re-parameterization module followed by a GAN-based decoder is leveraged to convert hidden features to the target waveform. This waveform is then post-processed to become the final generated speech. For the ASR, we utilize an encoder a hybrid CTC/Attention model[37] to extract the features of phonetic posteriorgrams(PPGs) from given speech.
We use the non-causal WaveNet[38] residual blocks as the posterior encoder. Such residual block consists of several dilated convolutions with skip connection and gated activation. For multi-speaker VC, we additionally feed speaker embedding into residual blocks through the method of global conditioning.
The conversion decoder is almost essentially the HiFiGAN[39] generator. The generator is a stack of convolution blocks, which include transpose convolution layers and a multi-receptive field fusion module (MRF). The MRF is composed of a series of residual blocks that have receptive fields of diferent sizes. To avoid possible checkerboard artifacts [40] caused by the transpose convolution process, we rebuild the upsampling layer using temporal nearest interpolation followed by a 1D convolution. As with GAN-based vocoders, we also add a discriminator D that attempts to distinguish audio generated by the generator G from the ground truth. Similar to the design in Hifi-GAN[ 39] and MelGAN[41], the discriminators include a multi-period discriminator(MPD) and a multi-scale discriminator (MSD). MPD is a mixture of window-based sub-discriminators, where diferent pe
target
word sub− − − − sample sample sample
predict
Feature combination Calculating KL Divergence For training For inference
+1 = ∏︁ ( + (∇(, , )))
(1) + Where represents the original sample, denotes sample label/class, is the number of iterations, is the maximum perturbation range, is the perturbation size riodic patterns of waveform are operated. MSD directly for each iteration, and L represents the loss function of functions on diferent scales, which consecutively evalu- the target model. The PGD attack generates the adverates audio samples at diferent levels that help to capture sarial perturbation that maximizes the sample loss value consecutive patterns and long-term dependencies. through multiple iterations. Adversarial training can be understood as solving an optimization problem with an 3.3. Adversarial attacking external minimum and an internal maximum. It solves decision boundaries that are robust to maximum adverProjected Gradient Descent (PGD) [34] is considered to sarial perturbations. For speech, adversarial attacks can be the strongest attack method based on gradient infor- target both the original audio and the acoustic features mation. When the model is robust to PGD attacks, it is used by the model, which need to be considered at the robust to most gradient-based attack methods. The PGD same time during adversarial training. The optimization attack uses the gradient of the model to the input to find function for adversarial training is shown below: the disturbance that maximizes the loss value. A typical PGD attack on network is mathematically expressed as the following recursive formula: , ℎ ( ) = (,)[ ∈ (, + , )]
(2) Among them, ( ) is the external minimum optimization function, which optimizes the network parameters for the most adversarial example of a clean sample, so that the network can be classified correctly. Source 1 12 (2 − )
> 0 ≤ ≤ 2() =
︂{ for from 0 to : for from 0 to ℎ: () ←
1(, )() ℎ() ←
2()ℎ() Connect(,,ℎ)= ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ () + ℎ( + −
) () 0 ≤ < − , −
≤ < , ℎ( + −
) ⎩ ≤
≤ + ℎ − where and ℎ denote the raw silent segments to be concatenated.
and ℎ represent the lengths of these two segments. and in the formulas are the subscripts of the time dimension iteration. stands for the overlapping duration. The final result is calculated by Connect(·).
Further more, we assume that a decline in speech quality may cause the performance of some anti-spoofing systems to degrade correspondingly and thus increase the probability of a successful attack. We reduce the
Mel Spectrogram
Conformer Encoder ×N
Transformer
Blocks ×N
Latent Representation Re-parameterize
Coding Vector
Decoder (Generator) recordings of the target speaker. We crop those real seg- for from 0 to +ℎ- :
Similar to what is proposed in our previous work[29], we introduce some postprocessing countermeasures against silence detection and check-out employed by antispoofing. We experiment with two methods of appending silence: In the first approach, we leverage a non-neural voice-activity-detecting(VAD) system that quickly finds silent segments in fake speech and gets the timestamps.
Then we randomly select real silent segments from the ments to the same length as calculated from the timestamps given by VAD and then simply replace them. The other is global superposition without relying on VAD.
We also randomly select multiple real silent segments and normalize their amplitudes to an average level. After that, we connect them end to end through the algorithm of parabolic cross-fading till the total length surpasses the length of generated audio. Finally, we directly paste this stitched silent sample onto our synthetic speech like additive noise. The parabolic cross-fading algorithm is formulated as follows: 1(, ) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎩ ⎪ 12 ( −
1 < − , − ≤ ≤ 2 − )( − ) (3) speech quality of speech by adding white Gaussian noise anchored in: and using the signal-to-noise ratio (SNR) to characterize 1. A complete end-to-end training method has been our speech quality. We notice that when the SNR is too adopted–the boundary between the acoustic model and low, the intelligibility of speech drops so rapidly that it the vocoder is ambiguous so the part that acts as a becomes nearly unintelligible and cannot pass ASR, so vocoder in the synthesis pipeline is somewhat diferent there is probably a trade-of and we manage to maintain from a traditional autoregressive/GAN/DPM model. This the SNR at a relatively high level after perturbation. may have led to anti-spoofing models submitted by many teams that have not seen the patterns of our generated speech. 4. Experiments and results 2. We have considered the existence of some long-term artifacts based on human hearing perception, and the We tested the spoofing ability of our synthesized audio on HierTTS and E2EVC models adopted have been designed several diferent mainstream anti-spoofing models along to suppress these artifacts to a certain extent. with various training features. For each anti-spoofing 3. AA and PP methods have increased the complexity model, we sent 1000 pieces of the generated waveform of generating speech, masking some artifacts and causing to perform our test, and the equal error rate (EER) was possible bias in the feature extraction of the anti-spoofing considered to be the performance indicator. Results are model. Methods that use non-speech parts as the main shown below in Table 1. Judging from the results, AA basis for detection are also targeted. and PP methods have achieved some efects in the face For the given AsvSpoof baseline released by the chalof a bunch of anti-spoofing systems. lenge organizers, the adversarial attack method we adopt
In the original design that was abandoned, we also has achieved some efect. ∼ 1e-5 and 0.99 represent the tried to incorporate a vocoder of Difusion Probabilis- average detection score before and after adding the adtic Models (DPM) after the VC module, because in dif- versarial attack respectively. We are informed that the fusion models, a segment of generated signal is iter- judging threshold is 0.1, so the performance against baseated from white noise. Unlike most vocoders that use a line is supposed to be DSR 100%. However, according to deconvolution-like upsampling method, difusion models the Challenge results, our spoofing performance against will be less likely to produce artifacts in the frequency the white-box baseline remains only DSR 23%. The redomain. However, the actual efect is not obvious or may sults are inconsistent. Our follow-up tests suggest that reduce performance, according to Table 2, which demon- the PGD noise used in the AA method may cause some strates the results of this part of the ablation experiment. negative efects after being enhanced by PP approaches,
Our team’s system achieved the 3rd place in the overall but we are not confident enough. score in this spoofing track. It is worth noting that our We understand that in any case, there is still room synthesized speech had the highest DSR% against all for improvement in the attack capabilities of white-box black-box anti-spoofing systems in two rounds of testing. and black-box systems, such as further dismantling the Table 3 gives the specific performance data of all teams. GAN-like vocoder structure of the complete end-to-end Our team is A03. VC part, or solving the problem that PGD may play a
We believe that the reasons why our multi-stage sys- negative role. tem is more powerful against black boxes are mainly
This paper is a detailed description of a multi-stage audio
synthesis system that participated in Track1.1 (spoofing)
in ADD C