GermEval 2020 Task 4: Low-Resource Speech-to-Text

Abstract

We present the results and findings of GermEval 2020 Task 4 on Low-Resource Speech-to-Text. Participants were asked to build a system translating Swiss German speech to Standard German text and minimize its word error rate. The task was based on a new dataset for Swiss German to Standard German speech translation, which contains 74 hours of sentence-level speech-text-pairs. 3 teams participated, with the winning contribution reaching a word error rate of 40.29 %.

1 Introduction

Speech-to-text methods for well-resourced languages like English or Standard German work very well. Lu¨scher et al. (2019) set the current state-of-the-art on the popular LibriSpeech testother benchmark (Panayotov et al., 2015) with a word error rate (WER) of as low as 5 %. This is in stark contrast to the situation for Swiss German, the most frequently used spoken language in Switzerland, for which almost no publicly available training data for speech-to-text is available. Apart from the comparatively low number of speakers of around 5 million, the main reason for this is the lack of a standardized writing system. Thus, the official written language in the German part of Switzerland is Standard German, not Swiss German. Despite this, a lot of Swiss German speakers write in Swiss German, especially in informal conversations using messaging apps, but they resort to phonetical writing in their local dialect. The multitude of different written variants for each word makes direct speech-totext almost impossible. Therefore, for most use cases, a speech-to-text method for Swiss German has to simultaneously translate to Standard German. This combination of speech recognition and translation is also referred to as speech translation.

For this shared task, we built a new dataset for Swiss German to Standard German speech translation. We provide data for training containing audio and text as well as data for testing containing only the audio. The testing data has to be transcribed and submitted on the task website for evaluation.

The remainder of this paper is structured as follows: The task and the evaluation of submissions are described in section 2. Our speech translation dataset is introduced in section 3. An overview of the submissions and results of this task can be found in section 4. Finally, section 5 wraps up the paper and gives directions for future work. 2

Task Description and Evaluation

The goal of the task is to build a sentence level

Swiss German to Standard German speech translation system. The submission with the lowest WER wins. We chose WER as opposed to the BLEU score (Papineni et al., 2002) , which is often used for machine translation and speech translation, for 2 main reasons. Firstly, manual inspection of a sample of our dataset showed that in most cases, the Standard German transcription does not deviate much from the Swiss German speech. Secondly, we only have a single reference transcription per utterance but BLEU would require multiple reference transcriptions to work well. Additionally, there is not a lot of margin for alternative transcriptions.

Apart from working with the provided dataset for training, participants were encouraged to explore data augmentation methods and transfer learning approaches to build a better model in this low-resource setting.

At evaluation time, the following preprocessing steps are taken before comparing a submission against the reference transcriptions:

The structure of the data is the same as in the

Mozilla Common Voice project4. Table 2 gives a short description of the metadata provided in the TSV file. 4

Results • Transform to lower case

3 teams participated in the shared task and submit

• Remove punctuation , ; : . ? ! ted a solution. Table 3 shows an overview of the results on the public part of the test set. • Remove leading and trailing whitespace The team in first place, Bu¨chi et al. (2020), achieved a WER of 40.29 %. Their approach is Numbers are not normalized in any way. based on a CNN acoustic model called Jasper (Li et al., 2019) . It is first trained on additional Stan3 Data dard German data and then fine-tuned on the task For this task, we created the, to the best of our data. The model was trained using the CTC loss knowledge, first publicly available dataset1 for function (Graves et al., 2006) . To further imSwiss German to Standard German speech trans- prove the results, a language model and data auglation. It consists of 70 (train) + 4 (test) hours mentation methods were applied. Table 4 shows of mostly Swiss German and some Standard Ger- some examples of true and predicted sentences man speech from the parliament of the canton of and the corresponding WER. Sentence 1 is a good Bern and corresponding Standard German tran- prediction, especially considering that the words scriptions. The Swiss German speech is predom- ”gefa¨hrdet ist” are missing in the recording due inantly in the Bernese dialect. Some parliament to an alignment error. In sentence 2, the alignmembers speak in Standard German, hence the ment is perfect, but the model chooses the word small part of Standard German speech. Speech- ”Fu¨hrungskontrolle” rather than “Feuerungskontext-pairs consist of a single sentence and were at- trolleur”. This is a seldomly used word in Swiss tained from the raw data using a fully automated German and therefore hard to get right. Sentence alignment procedure described in (Plu¨ss et al., 3 is actually a good prediction of what can be 2020) . The raw data consisted of audio recordings heard in the recording, but the sentence was reof full meetings, usually between 2 and 3 hours, formulated in the transcription. In this case, the and the transcript in a PDF file2. While the align- BLEU score with multiple reference transcriptions ment quality is fairly good, it is certainly not per- would better fit the task. Finally, the predicted senfect. The most common errors are missing or ad- tence number 4 does not make too much sense and ditional words at the beginning or end of a speech shows that the model still has considerable potenutterance compared to the transcript. Table 1 lists tial for improvement. a few examples of this. This obviously makes the The team in second place, Kew et al. (2020), task harder and leads to non-avoidable mistakes. achieved a WER of 45.45 %. They follow a DNNThe transcripts were pre-processed as follows: HMM approach for the acoustic model using a time delay neural network. No additional speech• Transform to lower case to-text data is used. They create a pronunciation lexicon specifically adapted to this task. Like • Replace or remove all characters except a-z, Bu¨chi et al., they use a language model and apply a¨, o¨, u¨, 0-9, space, punctuation , ; : . ? ! data augmentation methods.

The team in third place, Agarwal et al. (2020), • Remove leading and trailing whitespace achieved a WER of 58.93 %. Their approach is based on DeepSpeech (Hannun et al., 2014) , an Details of the replacement and removal operations end-to-end deep learning system. They use cascan be found in our code on GitHub3. caded transfer learning, first training the model 1https://drive.switch.ch/index.php/s/PpUArRmN5Ba5C8J with English data, then transferring to Standard 2https://www.gr.be.ch/gr/de/index/sessionen/sessionen.html German, then finally to Swiss German. The Archi3https://github.com/festivalhopper/germeval-2020-task4/blob/master/transcript preprocessing.py 4https://voice.mozilla.org/en/datasets Sentence in Recording Sentence in Transcript ...der Fall. Wir Motiona¨re wurden zusammen Wir Motiona¨re wurden zusammen mit anderen mit anderen Interessengruppen sehr schnell Interessengruppen sehr schnell eingeladen. eingela...

Das Pricing des Stroms ist relativ klar, es ist Das Pricing des Stroms ist relativ klar, es ist geregelt und die Gewinnspanne garantiert. Ich geregelt und die Gewinnspanne garantiert. will...

Mob (Samardzˇic´ et al., 2016) dataset is used as additional Swiss German training data. Like the other participants, they use a language model and apply data augmentation methods.

5

Conclusion

We have described GermEval 2020 Task 4 on

Low-Resource Speech-to-Text. The task used a newly created dataset for Swiss German to Standard German speech translation described in section 3. 3 teams participated in the task, with the winning team reaching a WER of 40.29 %. This is a good result given that few research has been done on this topic and considering the alignment errors apparent in the dataset due to the fully automated alignment procedure. An open question is how well this model would generalize to other Swiss German to Standard German speech translation datasets or to a Standard German speech-totext task.

The evaluation of the results of all teams indicates that data augmentation methods and language models work well in this low-resource setting. More details about the individual systems can be found in their respective system description papers, which are published in the SwissText & KONVENS 2020 proceedings.

We have made the dataset publicly available5 to the research community beyond the GermEval competition, hoping to facilitate future research on this important topic.

In future work, we plan to minimize errors made by the automatic alignment procedure and substantially increase the dataset size by aligning additional raw data.

Acknowledgments

First of all, we would like to thank the parliamen

tary services of the canton of Bern for their work on the transcription of the debates and for publishing recordings and transcripts on their website. Without them, this task would not have been possible.

We would also like to thank the GermEval 2020 organizers for hosting the Low-Resource Speechto-Text task and for replying promptly to all our inquiries.

We especially thank the GermEval 2020 Task 4 participants for their interest in the shared task, for their participation, and for their timely feedback, which have helped us make the shared task

5https://drive.switch.ch/index.php/s/PpUArRmN5Ba5C8J

True Sentence insbesondere kann der kanton mit finanziellen zuschu¨ssen steuernd eingreifen, wenn die versorgungssicherheit gefa¨hrdet ist. der feuerungskontrolleur, der von den gemeinden gewa¨hlt und eingesetzt wird, ist neutral. dabei ist zu beachten, dass der sinn dieser bru¨ckenangebote auch von der mehrheit nicht infrage gestellt wird. wissenschaft la¨uft nicht so, dass ein ma¨zen, wie ein hansjo¨rg wyss und vor allem nicht er, mit seiner fachkompetenz , so in wissenschaftliche forschung reinreden wu¨rde.

Predicted Sentence WER in % insbesondere kann der kanton mit fi- 28.57 nanziellen zuschu¨sse steuernd eingreifen die versorgungssicherheit die fu¨hrungskontrolle die von den gemeinden gewa¨hlt und eingesetzt wird ist neutral das ist vielleicht in die debatte auch wichtig dass der sinn dieser bru¨ckenangebote ist auch von der mehrheit nicht infrage gestellt es la¨uft nicht so dass denen ans ju¨rg weise und von er mit seiner fachkompetenz einsprechen in wissenschaftliche forschung wissen

References

Aashish Agarwal and Torsten Zesch . 2020 . Ltl-ude at low-resource speech-to-text shared task: Investigating mozilla deepspeech in a low-resource setting . In preparation.

Matthias Bu ¨chi, Malgorzata Anna Ulasik, Manuela Hu¨rlimann, Fernando Benites, Pius von Da¨niken, and Mark Cieliebak . 2020 . Zhaw-init at germeval 2020 task 4: Low-resource speech-to-text . In preparation.

Alex Graves , Santiago Ferna´ndez, Faustino Gomez, and Ju¨rgen Schmidhuber . 2006 . Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks . In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 369-376 , New York, NY, USA. Association for Computing Machinery.

Awni Hannun , Carl Case, Jared Casper , Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng . 2014 . Deep speech: Scaling up endto-end speech recognition .

Tannon Kew , Iuliia Nigmatulina, Lorenz Nagele, and Tanja Samardzˇic´. 2020 . Uzh tilt: A kaldi recipe for swiss german speech to standard german text . In preparation.

Jason Li , Vitaly Lavrukhin , Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M. Cohen , Huyen Nguyen, and Ravi Teja Gadde. 2019 . Jasper: An end-to-end convolutional neural acoustic model .

Christoph Lu ¨scher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer , Ralf Schlu¨ter, and Hermann Ney. 2019 . Rwth asr systems for librispeech: Hybrid vs attention - w/o data augmentation . In INTERSPEECH.

V. Panayotov , G. Chen , D. Povey , and S. Khudanpur . 2015 . Librispeech: An asr corpus based on public domain audio books . In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5206 - 5210 .

Kishore Papineni , Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002 . Bleu: a method for automatic evaluation of machine translation .

Michel Plu ¨ss, Lukas Neukom , and Manfred Vogel . 2020 . Forced alignment of swiss german speech to standard german text . In preparation.

Tanja Samardzˇic´ , Yves Scherrer, and Elvira Glaser . 2016 . Archimob - a corpus of spoken swiss german . In Language Resources and Evaluation (LREC 2016 ), Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016 ), pages 4061 - 4066 . s.n.