We present the results and findings of the 2nd Swiss German speech to Standard German text shared task at SwissText 2022. Participants were asked to build a sentence-level Swiss German speech to Standard German text system specialized on the Grisons dialect. The objective was to maximize the BLEU score on a test set of Grisons speech. 3 teams participated, with the best-performing system achieving a BLEU score of 70.1. German dialects spoken in Switzerland, see Plüss et al. [1]. the Swiss German, Standard German, French, and Italian ∗Corresponding author.
of Swiss German Speech to Standard German text and
can be viewed as a speech translation task with similar
source and target languages, see Plüss et al. [
This task has two predecessors. The 2020 task [
11 hours are in Grisons dialect. From version 9.0 of the
Swiss German dialect landscape. The winning contribu- Italian, all of which are oficial languages of Switzerland.
tion by Arabskyy et al. [
200 [
The goal of this task is to build a system able to trans- scribed Grisons speech by 11 speakers, of which 8 are
late Swiss German speech to Standard German text and
optimize it for the Grisons dialect. To enable this, we
provide the Swiss German labeled datasets SDS-200 [
2.2. Evaluation
evaluation script, which uses the NLTK [9] BLEU implementation, is open-source1. The private part of the test
Final ranking of the shared task. The BLEU column shows the achieves a BLEU score of 65.3 on the 2021 task test set
BLEU score on the private 50 % of the test set. [
The test set contains the characters a-z, ä, ö, ü, 0-9, as the use of models pre-trained on large amounts of and spaces, and the participants’ models should support unlabeled speech as demonstrated by the teams Stucki exactly these. Punctuation and casing are ignored for et al. and Nafisi et al., who employed XLS-R models [ 10]. the evaluation. Numbers are not used consistently in The addition of an LM seems to be especially important the test set, so sometimes they are written as digits and for XLS-R models. The main diference between Nafisi et sometimes they are spelled out. We create a second ref- al. and Stucki et al. is that the latter add an LM, leading erence by automatically spelling out all numbers and use to a relative improvement of 23 % BLEU. both the original and this adjusted reference in the BLEU On the other hand, none of the 3 participating teams score calculation. Participants were advised to have their made a significant efort to optimize their system for models always spell out numbers. All other characters the Grisons dialect. The best approach to create an ASR are removed from the submission (see evaluation script system optimized for a specific dialect remains to be for details). Participants were therefore advised to re- found in future work. Incorporating the provided French place each additional character in their training set with and Italian data for training is another possible direction a sensible replacement. for future research.
baseline. Table 1 shows the final ranking.
Our baseline achieves a BLEU score of 70.1. We use the
model Transformer Baseline described in Plüss et al. [
Stucki et al. achieve a BLEU score of 68.1. They use an XLS-R 1B model [10], pre-trained on 436K hours of unlabeled speech in 128 languages, not including Swiss German. They fine-tune the model on SDS-200 and SwissDial. A KenLM 5-gram LM [11] trained on the German Wikipedia is employed.
Nafisi et al. achieve a BLEU score of 55.3. They use an XLS-R 1B model [10], pre-trained on 436K hours of unlabeled speech in 128 languages, not including Swiss German. They fine-tune the model on SDS-200. No LM is employed. N. Kapotis, J. Hartmann, M. A. Ulasik, C. Scheller, Y. Schraner, A. Jain, J. Deriu, M. Cieliebak, M. Vogel, Sds-200: A swiss german speech to standard german text corpus, in: Proceedings of the Language Resources and Evaluation Conference, 2022. [7] P. Dogan-Schönberger, J. Mäder, T. Hofmann, Swissdial: Parallel multidialectal corpus of spoken swiss german, CoRR abs/2103.11401 (2021). URL: https: //arxiv.org/abs/2103.11401. a r X i v : 2 1 0 3 . 1 1 4 0 1 . [8] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, G. Weber, Common Voice: A MassivelyMultilingual Speech Corpus, in: Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020. [9] S. Bird, E. Klein, E. Loper, Natural language processing with Python: analyzing text with the natural language toolkit, ” O’Reilly Media, Inc.”, 2009. [10] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, M. Auli, Xls-r: Selfsupervised cross-lingual speech representation learning at scale, arXiv abs/2111.09296 (2021). [11] K. Heafield, Kenlm: Faster and smaller language model queries, in: Proceedings of the sixth workshop on statistical machine translation, 2011.