This paper describes the winning approach in the Shared Task 3 at SwissText 2021 on Swiss German Speech to Standard German Text, a public competition on dialect recognition and translation. Swiss German refers to the multitude of Alemannic dialects spoken in the German-speaking parts of Switzerland. Swiss German differs significantly from standard German in pronunciation, word inventory and grammar. It is mostly incomprehensible to native German speakers. Moreover, it lacks a standardized written script. To solve the challenging task, we propose a hybrid automatic speech recognition system with a lexicon that incorporates translations, a 1st pass language model that deals with Swiss German particularities, a transfer-learned acoustic model and a strong neural language model for 2nd pass rescoring. Our submission reaches 46.04% BLEU on a blind conversational test set and outperforms the second best competitor by a 12% relative margin.
While general speech recognition has matured to a
point where it surpasses human performance on
specific datasets
In previous studies, Garner et al. (2014) tackled
this challenge by training hybrid models
(HMMGMM, HMM-DNN, and KL-HMM) to transcribe
Walliserdeutsch, a Swiss-German dialect spoken
in the south-western alpine canton of
Switzerland and further used a phrase-based machine
translation model to translate it to standard
German. Following this, other researchers explored
techniques to add the translation step in the
lexicon by directly mapping Swiss-German
pronunciation to standard German. Stadtschnitzer and
Schmidt (2018) estimated Swiss German
pronunciations from a standard German speech
recognition model using a data-driven technique, and
trained stronger TDNN-LSTM based acoustic
models. Kew et al. (2020) and Nigmatulina et al. (2020)
trained transformer-based G2P models from
standard German to Swiss pronunciations and trained a
Kaldi-based TDNN+ivector system using the WSJ
recipe1. Yet a third approach is to directly
apply end-to-end deep learning models. Büchi et al.
(2020) and Agarwal and Zesch (2020) at
SwissText 2020 used the Jasper architecture
In this paper, we describe our proposal to solve the challenging task of transcribing Swiss German speech to standard German text. It won the competition at SwissText 2021 with a large margin to other competing systems. 2
In this section, we propose our changes to a
conventional hybrid
To train our proposed model, we utilized a selection of publicly available and internal datasets. Our starting point was the Swiss Parliament Corpus V2 dataset (Plüss et al., 2020) shared as part of the SwissText 2021 competition. It covers 293 hours and contains recordings from the local parliament of the Kanton Bern. Its transcripts are in standard German while the audio covers Swiss German (predominantly in the Bernese dialect). The dataset has been preprocessed by the publishers with the purpose of cleaning its annotations and ensuring a good match between audio content and transcription. It is provided with a choice of different preprocessing flavors. We used the train_all split. In addition, we used a 493-hour internal dataset representing a media domain encompassing conversational speech from interviews, discussions, podcasts and others. A subset (around 50 hours) of the data is annotated with both Swiss transliterations as well as standard German. The remaining data has only been annotated with standard German. Additionally, we used an internal high German dataset encompassing around 10k hours to pre-train our model.
In terms of test data, the SwissText 2021 competition was accompanied by a 13 hours conversational test set covering Swiss German speakers from all German-speaking parts of Switzerland. The encountered dialectal distribution is claimed to closely match the real distribution in Switzerland. The set was not disclosed to the participants. Hence, for the analysis in this paper, we report our numbers on a publicly available test set part of the dataset from the Bernese parliament (Plüss et al., 2020). It comprises 6 hours of dialectal speech. 2.2
We propose to incorporate the translation from Swiss German to standard German as part of the lexicon. However, this leads to a complex and often ambiguous mapping from phoneme to grapheme sequences, which is very different from languages with a direct relation between writing scheme and pronunciation (e.g. English or standard German). Subsequently, statistical models that map graphemes to phonemes (G2P) trained on Swiss German data incorporating such translations yield much noisier output with significantly higher phone error rates as compared to G2Ps for standard languages. To mitigate this problem, we construct the lexicon in several stages.
In a first step, we make use of parallel corpora
encompassing Swiss and standard German
annotations to extract word mappings between Swiss and
standard German. Sophisticated filtering methods
help to ensure a high quality of these mappings. We
opt for frequency filtering and filtering based on
vicinity in a word embedding space
In a second step, a standard German G2P model
is applied to convert Swiss German transliterations
into corresponding phone sequences. This results
in a dictionary that maps standard German words to
Swiss pronunciations. Jointly with existing Swiss
German lexicon resources
We evaluate the quality of the resulting G2P model on a manually labeled test set. Those cover mappings from standard German words to Swiss German phone sequences and encompass a variety of relevant categories such as diminuation, shortening or translation. Refer to Table 1 for samples of the assessed categories.
The Swiss G2P model allows to find suitable
pronunciations for the relevant word inventory present
in the acoustic and language model training
corpora. However, to further increase the quality of
the given pronunciations, data-driven lexicon
learning techniques
fragt f hr a_ g ax t
riecht sh m oe k c ax t
fragst f hr a_ k sh
riechst sh m oe k sh
Diminuation erdmännchen e_r t m eh n l i_ gläschen g l e_ s l i_
Shortening gymnasium
g ih m i_ schwimmbad b a_ d ih
Translation
Variability
kopf
g hr ih n t
kneipe
b ai ts
kannst
k a sh
zweites
ts v ai t
Incorporating the translation from Swiss German to
standard German as part of the lexicon introduces
significant ambiguity in the decoding process. To
counteract, we suggest using a strong standard
German language model (LM) which helps to produce
accurate hypotheses. We employ a first pass
countbased LM to output up to 100 sentence hypotheses
and a second pass neural LSTM (long short-term
memory) LM
Furthermore, we make some adjustments to better deal with Swiss German particularities, as described in the following paragraphs.
Compounds: German is a compounding language and tends to compose words (particularly nouns) of several smaller subwords. The resulting chains of word stems can lead to an infinitely large vocabulary size with words that occur very infrequently throughout the corpus. This spreading of probability mass weakens the LM. We hence decompound all compounded words in the training corpus and split them into subwords.
Clitics: Swiss German tends to merge words
beyond compounding, not preserving word
stems
The acoustic model is trained with 80 dimensional
log-mel filterbank features, with a processing
window of 25ms and 10ms frame shift. The feature
vector from the previous frame is concatenated with
the current frame to obtain a 160 dimensional
vector. We used a LC-BLSTM (latency controlled
bidirectional long short-term memory) based acoustic
model, that is popularly applied in speech
recognition for controlling decoding latency to a few
frames
The results are reported in terms of BLEU
An ablation study of the proposed approaches is presented in Table 2. All of the performance gains in this section will be reported as relative percentage improvements, while aforementioned table contains absolute numbers.
We first evaluate the effect of transfer learning on the results with the Swiss Parliament training set. It can be observed that it significantly helps to improve both WER and BLEU. In particular, the transfer-learned model (row 2, Table 2) improves over the model trained from scratch (row 1, Table 2) by 8:4% WER and 11:6% BLEU. The result shows 1 2 3 4
Swiss Parliament + Transfer Learning + Internal Data + 2nd Pass Rescoring
Further adding additional internal training data shows additional gains in performance. As such, we observe that the WER improves by 2:3% and BLEU by 2:5%.
Finally, 2nd Pass rescoring is applied as described in Section 2.3 to reorder the top 100 hypotheses. It can be observed from row 4, Table 2 that rescoring helps to improve the performance by 5:8% WER and 8:9% BLEU.
Our submission to SwissText 2021 achieves 46:04% BLEU on the official SwissText blind test set. This leads to a 12% relative margin in BLEU with respect to the second best competitor which was 40:99%.
The acoustic models have been trained using 8 GPUs for 25 epochs. This results in a total training time of around 400 GPU-hours when training on Swiss Parliament only and about 1200 GPU-hours when adding the internal data. 4
In this paper, we described a speech recognition system that achieves strong results on the task of recognizing Swiss German dialect and translating it into standard German text. We proposed a hybrid ASR system with a lexicon that incorporates translations, a 1st pass language model that deals with Swiss German word compounding and clitics, an acoustic model that is transfer-learned from standard German resources and a strong neural language model for 2nd pass rescoring to smoothen translation artifacts. Furthermore, we provided an ablation study that allows to infer the effect of adding training data, performing transfer learning and 2nd pass rescoring. Our submission reached 46.04% BLEU on a challenging conversational test set and outperformed all competing approaches by a large margin.
In terms of future work, we would like to investigate word re-orderings as part of the translation, which our current model does not actively support. For instance, Swiss German frequently moves verbs in relative clauses to different positions with respect to the standard German word order. Furthermore, sequence discriminative training is a promising route for exploration as well as using unsupervised data for acoustic model training.