We describe our system participating in the SwissText/KONVENS shared task on low-resource speech-to-text (Plüss et al., 2020). We train an end-to-end neural model based on Mozilla DeepSpeech. We examine various methods to improve over the baseline results: transfer learning from standard German and English, data augmentation, and post-processing. Our best system achieves a somewhat disappointing WER of 58.9% on the held-out test set, indicating that it is currently challenging to obtain good results with this approach in a low-resource setting.
Recently, end-to-end models like DeepSpeech1
have been introduced as an alternative to
traditional HMM-DNN based models like Kaldi
In this paper, we examine how well DeepSpeech performs in a truly low-resource setting like Swiss Language
Dataset Swiss German
SwissText Shared Task
ArchiMob German English
Voxforge TUDA-De M-AILABS MCV_v4 LibriSpeech MCV
Size [h]
German, where less than 100 hours of annotated
data are available. Previous speech recognition
systems for Swiss German
We used DeepSpeech version 0.6.0 for all experiments.2 2.1
To train the Swiss German DeepSpeech model, we utilized the following publicly available datasets as showed in Table 1.
For Swiss German we used the official data provided by the shared task (Plüss et al., 2020). The corpus contains 70 hours of spoken Swiss German (predominantly in the Bernese dialect) and some Standard German speech from the parliament of the canton of Bern3. We additionally use the ArchiMob (Samardžic´ et al., 2016) corpus, which represents German linguistic varieties spoken within the territory of Switzerland and contains long samples of transcribed text in Swiss German. The corpus contains 57 hours and is 2https://github.com/mozilla/DeepSpeech/releases/tag/v0.6.0 3https://swisstext-and-konvens-2020.org/low-resource-speech-to-text/ available under creative commons licence 4.0.4
As the amount of data is probably not sufficient
to train a good model, we will experiment with
transfer learning from standard German. Publicly
available datasets include Voxforge5, TUDA-De
As there has been previous work on
transfer learning models starting with a different
language
We trained and tested our models on a compute server having 56 Intel(R) Xeon(R) Gold 5120 CPUs @ 2.20GHz, 3 Nvidia Quadro RTX 6000 with 24GB of RAM each. Typical training time with augmentation for the SwissText dataset is 1.5 hours, for German 12 hours, and for English 30 hours. Without augmentation, the training time was approximately 10% less than with augmentation. 2.3
We cleaned the data by using only the allowed set of characters listed by the shared task. We converted all transcriptions to lower case and further ensured that all audio clips are in wav format. The resulting samples were split into training (70%), validation (15%), and test data (15%). The preprocessing scripts can be referenced at GitHub 9 4https://www.spur.uzh.ch/en/departments/research/textgroup/ArchiMob. html
5http://www.voxforge.org/home/forums/other-languages/german/ open-speech-data-corpus-for-german 6https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/ 7https://voice.mozilla.org/en 8http://www.openslr.org/12/ 9https://github.com/AASHISHAG/deepspeech-swiss-german Hyperparameter Batch Size Dropout Learning Rate English a b German a b
Value
For the acoustic model, we use the best
hyperparameters as reported by
We use a probabilistic 3-gram language model
based on KenLM
As the baseline model, we train DeepSpeech with the setup described above and using only the Swiss German data provided by the shared task. The model achieved a WER of 71.5%. As expected, DeepSpeech is not able to simply train a suitable model based on this amount of training data.
We try to improve over those results using data augmentation and transfer learning as discussed in the remainder of this section. 3.1
Augmentation is a useful technique for better generalization of machine learning models. Inspired by Park et al. (2019), Mozilla DeepSpeech has implemented several augmentation techniques like frequency masking, time masking, speed scaling, and pitch scaling. We used all the augmentation approaches with default hyperparameters, which can be referenced here.12 Augmentation actually 10https://www.statmt.org/europarl/ 11https://github.com/mozilla/DeepSpeech/releases/tag/v0.6.0 12https://deepspeech.readthedocs.io/en/v0.7.0/TRAINING.html# training-with-augmentation 71.5 increases model error from 71.5% to 74.3%. However, we further test the impact of augmentation in our transfer learning results discussed below. As we have discussed above, end-to-end training of automated speech recognition systems requires massive data. As we only have 70 hours of training data available from the shared task, we experiment with transferring the model from different starting points. Table 3 gives an overview of the results. Transferring from about 2,500 hours of English data gives about the same results are starting from about 1,000 hours of German data even if standard German is closer to Swiss German than English. However, the best results are achieved when starting with English, transferring to German and then transferring to Swiss. Data augmentation in this case improves results a bit for a final WER of 61.5%. 4
When analyzing the errors made by DeepSpeech, one issue stands out: truncated output. Quite a lot of output texts are much shorter than the source transcript. Table 4 shows some examples. The performance of the model will be seriously impacted by not producing long enough output sentences. It might be informative to only look at output text that is about the same length as the original transcript. Figure 1 displays the distribution of samples with a certain ratio of sample length to source sample length in characters. The figure shows that almost all DeepSpeech outputs are shorter than the original. If we only look at the samples that are about the same size as expected (with a ratio higher than 0.75, which is still about half of all samples), we find that WER improves from 61.5% to 47.7%. This means that when the model outputs a string that is approximately of the correct length, it is actually much better than the 1.00 0.80 1.00 0.67 results in Table 3 indicate.
The length of the output is partly controlled by the model’s hyperparameters. We want to find a sequence c that maximizes the combined objective function:
Q(c)=log(P(cjx))+a log(Plm(c))+b wordcount(c)
where a and b controls the trade-off between the
acoustic model, the language model constraint,
and the length of the sentence. The term Plm
indicates the probability of the sequence c according
to the language model. The weight a constrains
the relative contributions of the CTC network and
the language model and the weight b determines
the count of words in the recognized transcription
By changing the relative weight of acoustic model and language model by optimizing a and b , we can improve the model a bit as shown in the optimized model examples in Table 4. However, we were not able to eliminate the problem altogether. 70.2 Consequently, WER only improves from 61.5. to 57.1 (with augmentation). As the model with the optimized hyperparameters and without augmentation is still a bit better, we submitted that one in the shared task. It achieved a WER of 58.9% on the held-out test set. 5
The baseline system trained only on the SwissGerman data yields a quite high word error rate of 71.5. Data augmentation strategies implemented in DeepSpeech did not result in consistent improvements. Transfer learning has a much higher impact reducing the word error rate by over 10 percent points when transferring an English model to German and finally transferring to Swiss German. The best model yields a WER of 56.6% on our test set (58.9% in the public ranking based on the hidden test set of the shared task). When analyzing the results, the model seems to suffer from truncated output which we can somewhat improve by hyperparameter tuning. Overall, the results show that training an end-to-end neural speech recognition system with DeepSpeech in a low-resource setting remains challenging.