This paper presents our eforts to build a robust ASR model for the shared task and prepared speech & Speech Emotion Recognition in Portuguese (SE&R 2022). The goal of the challenge is to advance the ASR research for the Portuguese language, considering prepared and spontaneous speech in diferent dialects. Our method consist on fine-tuning an ASR model in a domain-specific approach, applying gain normalization and selective noise insertion. The proposed method improved over the strong baseline provided on the test set in 3 of the 4 tracks available.
The performance of Automatic Speech Recognition
systems (ASRs) has increased significantly with the
development of modern neural network topologies and the
use of massive amount of data to train the models [1].
Although the accuracy of recent models improved for
high-resource languages, such as English, the
development of ASR models in other languages is still a dificult
task using the same technologies [
Wav2vec 2.0 (Figure 1) is inspired by previous works in unsupervised pre-training for speech recognition, that is, Wav2vec [7] and Vq-Wav2vec [4]. During pre-training the model learns speech representations solving a contrastive task which requires identifying the correct quantized latent speech representations of a masked time step among a set of distractors. After the self-supervised pretraining, the model can be fine-tuned on labeled data in a supervised task, like ASR, adding a randomly initialized linear projection on top of the context network with classes and a loss function specific to the task at hand, like CTC, for instance.
Proceedings of the First Workshop on Automatic Speech Recognition for Spontaneous and Prepared Speech & Speech Emotion Recognition in Portuguese (SE&R 2022), co-located with PROPOR 2022. March 21st, nEvelop-O 2022 (Online). [9] demonstrated that the fine-tuning of the Wav2vec 2.0 model achieves state-of-the-art (SOTA) results only using publicly available datasets.
ASR model is the quality and the domain of the data [
SOTA in ASR in Portuguese, for both prepared and spontaneous speech, the shared-task Automatic Speech Recognition for spontaneous and prepared speech & Speech Emotion Recognition in Portuguese (SE&R 2022) introduces a new baseline for ASR and a new dataset in Portuguese [9]. The Corpus of Annotated Audios (CORAA ASR), a large corpus of spontaneous and prepared speech, is composed by various subsets in Portuguese with diferent characteristics. The baseline achieves a Word Error Rate (WER) of 24.18% on CORAA ASR test set, a dificult dataset containing samples with low quality, noise, and a variety of domains and dialects.
In this work, we investigate the fine-tuning of the baseline model [9] proposed by the shared task, a fine-tuned model based on the Wav2vec 2.0 XLSR-53 [15], using only public available Portuguese datasets, including the CORAA ASR dataset. We conducted several experiments in diferent domains for the challenge and explored the use of selective noise insertion and audio normalization during training. This work is organized as follows: Section 2 discuss the proposed methods and Section 3 shows and discusses the obtained results. Finally, Section 4 presents the conclusions of this work.
We used several publicly available datasets in Portuguese. Besides CORAA ASR, most of them is composed by prepared speech. In general, we opted to use all the data in the gathered datasets for training, except the dev part of the CORAA ASR, as presented in Table 1. The datasets used in this work are: • CETUC [13]: contains approximately 145 hours of Brazilian Portuguese speech distributed among 50 male and 50 female speakers, each pronouncing approximately 1,000 phonetically balanced sentences selected from the CETEN-Folha1 corpus; • Common Voice (CV) 7.0 [16]: is a project proposed by Mozilla Foundation with the goal to create a wide open dataset in diferent languages. In this project, volunteers donate and validate speech using the oficial site 2; • Multilingual LibriSpeech (MLS) [14]: a massive dataset available in many languages. The MLS is based on audiobook recordings in public domain like LibriVox3. The dataset contains a total of 6k hours of transcribed data in many languages. The set in Portuguese used in this work4 (mostly Brazilian variant) has approximately 284 hours of speech, obtained from 55 audiobooks read by 62 speakers; • Multilingual TEDx [17]: a collection of audio recordings from TEDx talks in 8 source languages. The Portuguese set (mostly Brazilian Portuguese variant) contains 164 hours of transcribed speech. 1https://www.linguateca.pt/cetenfolha/ 2https://commonvoice.mozilla.org/pt 3https://librivox.org/ 4http://www.openslr.org/94/ • Corpus of Annotated Audios (CORAA ASR) v1 [9]: is a public available dataset that contains 290.77 hours of validated pairs (audio-transcription) in Portuguese (mostly Brazilian Portuguese variant) and is comprised by five other corpora: ALIP [18], C-ORAL Brasil I [19], NURC-Recife [20], SP2010 [21] and TEDx Portuguese talks.
line model of [9]. For each experiment, we trained the model for 5 epochs, using a batch size of 192, and using Adam [22] where the learning rate is 3−05 that is warmed up for the first 400 updates, then linearly decayed for the remained. For the experiments, we used a NVIDIA TESLA V100 32GB, a NVIDIA TESLA Tesla P100 16GB and a NVIDIA A100 80GB, depending on the type of audio pre-processing used. The code to replicate the results is available at https://github.com/alefiury/SE-R_ 2022_Challenge_Wav2vec2.
In total, we conducted five main experiments to test our methods: • Experiment 1: Wav2vec 2.0 XLSR-53 - Base: For this experiment, the model was fine-tuned considering the whole train set, but did not receive neither normalization nor noise addition; • Experiment 2: Wav2vec 2.0 XLSR-53 - Norm: The model was fine-tuned with the whole train set with normalization. For the normalization, the mean gain of all the audios in the train set was considered; • Experiment 3: Wav2vec 2.0 XLSR-53 - Norm and SNA: The model was fine-tuned with gain normalization and selective noise addition. The audios were normalized considering the mean gain of all the audios in the train set, and those audios pertaining to datasets that were considered to have a low presence of noise, namely MLS and CETUC, received randomly one of the following 5 possible types of noises: additive noise, being music or nonspeech noises from the MUSAN Corpus [23], Room impulse responses [24], Addition or reduction of gain, Pitch shift and Gaussian noise; • Experiment 4: Wav2vec 2.0 XLSR-53 - Norm + Prepared Speech: Model fine-tuned based on the ifnal trained model of Experiment 2, but considering just the prepared speech data from the CORAA ASR dataset, trained for more 5 epochs; • Experiment 5: Wav2vec 2.0 XLSR-53 - Norm + Spontaneous Speech: Model fine-tuned based on the final trained model of Experiment 2, but considering just the spontaneous speech data
Table 4 compares the baseline with our selected models in the test set. The model Wav2vec 2.0 XLSR-53 - Norm 3. Results and Discussion + Prepared Speech surpassed the strong baseline in the Prepared Speech PT_BR, Prepared Speech PT_PT and the The shared-task consists of 4 tracks. Each track has a Mixed tracks. As seen in the results based on the dev set, domain specific scenario, that includes prepared speech the fact that most of the data of the datasets that were and spontaneous speech. In this regard, we conducted added to the train set are comprised of prepared speech, a prior analysis (Section 3.1) using the dev set to select might have contributed to the increase in performance the best approaches based on 3 of the 4 tracks available: in this domain in both Portuguese variants. Lastly, even Mixed, Prepared Speech PT_BR and Spontaneous Speech. though we were not able to surpass the baseline model in The best models were selected and then submitted for the Spontaneous Speech track, we achieved competitive evaluation. Our final results are presented in Section 3.2. results with both submitted models.
Overall, our models did not show a huge improvement in After selecting and submitting the best results, we perperformance when compared to the baseline model. Even formed some additional experiments to further explore though we fine-tuned a model that is considered the state- our proposed methods using the dev set. We tried to use of-the-art in Brazilian Portuguese, we suspect that the text correction in the outputs of the ASR models, and number of training epochs might have been insuficient train the normalized models for a longer period of time to obtain an increase in performance, or that the baseline using early stopping considering the prepared speech and model might have already reached a local optima. spontaneous speech data from the CORAA ASR dataset.
Furthermore, as presented in Table 2, the model that The text correction was done with an additional postwas fine-tuned using prepared speech clearly improved processing step, using a KenLM [25] model. For the difthe results in the Prepared Speech subset (and conse- ferent tasks, we used 2 diferent KenLM models: one for quently the Mixed subset). The same phenomenon could spontaneous speech, which was built using subsets of not be seen in the Spontaneous Speech subset. A possible the CORAA ASR dataset containing spontaneous speech explanation to this fact is that most of the data of the phrases. And the other one was built considering wikipedia datasets that were added to the train set are comprised in portuguese texts, as proposed by [3]. Both were 4of prepared speech, which might have contributed to the grams. We found that this post-processing of the predicincrease in performance in this particular domain. An- tions from the ASR models did not improve the results other possible explanation is the low number of training from the dev set on the Prepared Speech PT_BR and epochs used to train the models. Spontaneous Speech tracks, as can be seen in Table 5, in
Additionally, the noise insertion did not gave further fact they were worse. One possible explanation is that improvement in performance. Nevertheless, the results some of the decoder hyper-parameters did not work well of the SDA model in some more noisy subsets of the with our ASR models. Another possibility is that the CORAA ASR dataset, like ALIP and NURC, for instance, 4-gram trained with spontaneous text was built with a showed some interesting and promising results when small amount of text, which might have decreased the compared to the baseline. These results are shown in performance of the model. However, the results on the Table 3. Prepared Speech PT_PT track were much better compared to the previous experiments. This result suggests that the LM might improve results when there are few domain data used to train the Wav2vec model, since most of our training data was composed by Brazilian Portuguese audios.
Furthermore, as we had suspected earlier, the model with gain normalization that were trained considering the spontaneous speech data from the CORAA ASR corpus and for a longer period of time performed better in its respective subtrack, strengthening our hypothesis that our results did not improve in the main experiments due to a low number of training epochs.
with additional public available data in multiple domains and using normalization, even for a few epochs, can improve performance. With our results we were able to improve on the test set in 3 of the 4 tracks available over the strong baseline provided.
As future works, we plan to train a ASR model using a dynamic noise insertion approach that do not depend on choosing specific datasets previously.
This research was funded by CEIA with support by the Goiás State Foundation (FAPEG grant #201910267000527)5. We also would like to thank Cyberlabs Group6 for the support for this work.