This paper presents the contribution of ZHAW-InIT to Task 4 ”Low-Resource STT” at GermEval 2020. The goal of the task is to develop a system for translating Swiss German dialect speech into Standard German text in the domain of parliamentary debates. Our approach is based on Jasper, a CNN Acoustic Model, which we fine-tune on the task data. We enhance the base system with an extended Language Model containing in-domain data and speed perturbation and run further experiments with post-processing. Our submission achieved first place with a final Word Error Rate of 40.29%.
Automatic Speech Recognition (ASR) is defined
as mapping audio signals to text. A particular
challenge for ASR arises if a language does not
have a standardized writing system, as is the case
for Swiss German. In German-speaking
Switzerland, Swiss German is the default spoken language
on most occasions, from formal to informal;
however, the language of reading and writing is
Standard German (“medial diglossia”, Siebenhaar and
Wyler (1997)). Swiss German is increasingly used
for writing in informal contexts, especially on
social media, but users usually write phonetically in
their local dialect
The Shared Task ”Low-Resource STT” at GermEval 2020 aimed exactly at a specific Swiss case of CL-STT: translating Swiss German dialect spoken in an official context to written Standard German.
In our approach, we applied a general
characterbased ASR system
A data set of 36’572 utterances with a total duration of 69.8 hours was made available for training the systems and a 4 hour test set was used for evaluating solutions. The training data consists of a set of recordings of debates held in the parliament of the canton of Bern, with utterances produced by 191 speakers. None of these 191 speakers occur in the test set. The audio recordings contain mostly Swiss German dialect speech with the majority of the utterances being spoken in Bernese dialect; however, there are also some recordings of Standard German speech as well as a few English utterances. Each utterance contains one sentence and has an average duration of 6.9 seconds.
All recordings have been manually transcribed
into Standard German, while the alignment
between audio and transcripts was performed
automatically by the task organizers
The transcript accuracy is measured with the
Word Error Rate (WER), which is the standard
ASR evaluation metric. It is computed as the sum
of the number of insertions, deletions and
substitutions between predicted and reference sentences
divided by the number of words in the reference
The most recent developments in both ASR and
machine translation involve generalized methods
that can be relatively easily ported across the two
tasks, such as the encoder-decoder architecture.
One of the most prominent, ”Listen, Attend and
Spell” (LAS)
Li et al. (2019) propose a convolutional network
with residual connections, with state-of-the-art
results on the LibriSpeech and Wall Street Journal
ASR data sets. The network predicts a character at
each step (of 20 ms) and a Connectionist
Temporal Classification (CTC) loss
While usually tens of thousands of hours of
audio are required for achieving state-of-the-art
ASR performance, some approaches target
languages where only a few hours of data are
available
Although there are approaches which directly
target the speech translation setup
This section describes the initial system used to establish a base for our experiments. Important concepts as well as parameters crucial for the experiments are explained. 4.1
We normalized all texts before training the Acoustic Models and Language Models. This step was necessary to have a standardized set of possible characters, which in this case were the letters az, a¨, o¨ and u¨. Normalization was performed in multiple steps, starting by lower-casing the whole text and splitting it into sentences. All punctuation symbols were removed, except for points and commas which might be used as decimal point or for ordinal numbers. Numbers were transliterated to words. Common abbreviations and symbols were replaced by their spoken form (e.g. ”%” by ”Prozent” or ”kg” by ”Kilogramm”). Letters with diacritics other than a¨, o¨, and u¨ were replaced by their counterpart without diacritics. Finally, any remaining unknown symbols were removed without replacement. 4.2
An Acoustic Model was used to predict
linguistic units based on an audio signal. For this
purpose, Jasper
The model consists of convolutional layers structured in blocks and sub-blocks. A model B R is defined by the number of blocks B and number of sub-blocks R. Every sub-block consists of a 1D-convolution, batch-normalization, ReLU, and dropout. The input of each block is connected to the last sub-block by a residual connection. We applied the Dense Residual configuration, which is shown in Figure 1, where the output of each block is additionally added to the inputs of all following blocks. For pre- and post-processing one and three additional blocks were used, respectively.
During training, the CTC loss
In order to get transcriptions from the Acoustic
Model output, beam search was applied. Beam
search tries to find the most probable text sequence
given probabilities of characters over time.
Additionally, a Language Model was used to re-rank
the beam search hypotheses. A Language Model
penalizes words that are not known and assigns a
probability to each word given the words
preceding it. The weight of the Language Model is
controlled with parameter . A parameter is used as
the word insertion bonus to prevent the preference
of long words. The Language Model we used was
a 6-gram model trained with KenLM
The Acoustic Model requires a large amount of data for training. Therefore, Standard German speech data as listed in Table 1 was used to create a pre-trained model1. Based on the given data sets, a combined version was created. Training, development and test splits were kept if given in the original data sets. Otherwise, custom splits were created with a size of 15% for test and validation, but with a maximum of 15000 seconds.
For the size of the model the configuration 10 5 was used. The model was trained with an initial learning rate of 0.015 on batches of size 64 for a total of 100 epochs. 4.5
The pre-trained model was used as a base for finetuning using the task specific data. The first few blocks serve as acoustic feature extraction. Since acoustic features of Standard German and Swiss German are very close, only weights of the postprocessing blocks as well as the last three or five intermediate blocks were updated, depending on 1Accessible through german-asr/megs. https://github.com/ the experiment as described in Section 5.2. Apart from the frozen blocks, the same hyperparameters were used as for the pre-training. The model was trained for another 100 epochs for fine-tuning (see Figure 2 for Word Error Rate progression over the 100 epochs). The acoustic models were trained on a NVIDIA DGX-1 system. Pre-training with about 540 hours of Standard German took approximately 197 hours using two NVIDA Tesla V100 GPUs, while fine-tuning of the acoustic model (AM-A5x5-SP) with about 70 hours of Swiss German speech required approximately 21 hours with one V100 GPU. The time for inference was much lower and took only about two minutes per 4 hours of speech on a NVIDA Titan X GPU. Applying the language model (LM extended) required some additional computation time. However, this took only a few minutes on a recent system for training as well as for decoding in combination with the beam search algorithm. 5
We describe the experiments we conducted in order to improve the baseline system in Section 5.2, present the results we obtained in Section 5.3 and reflect on them in 5.4. 5.1
The data set provided as part of the Shared Task was split into internal train, development and test sets. The train set consisted of 32’978 utterances, the development set contained 1’778 utterances, while the test set comprised 1’816 utterances. This split approximates 90% training, 5% development, 5% testing. A single speaker could not occur in different sets and the utterance lengths were taken into account for splitting.
The experiments consisted in fine-tuning the baseline system with the use of additional text data and, in one case, in applying transcript postprocessing.
Acoustic Models The baseline Acoustic Model (called ”AM base” below) was fine-tuned on the internal train set, first on three blocks (model ”AM-E 3x5”) and in the second version on five blocks (model ”AM-E 5x5”). In the last step of Acoustic Model fine-tuning, the baseline model was re-trained on the complete official train set (internal train, development and test sets combined), which resulted in the model called ”AMA 5x5”. Additionally, we trained a model with the internal training set without applying any pretraining (model ”AM-NOPRE”).
Language Models The language modelling setup is described in Section 4.3. We used two different Language Models (LMs). The basic Language Model (”LM base”) consists of corpora 1-3 in Table 2. Since these corpora are from different domains than the task data, we injected additional data to fine-tune the LM: corpus 4 is a collection of 11’576 press releases by the Federal Chancellery (Bundeskanzlei). These were scraped from https: //www.bk.admin.ch/bk/de/home/ dokumentation/medienmitteilungen. msg-id-<ID>.html using a custom script, where consecutive <ID>s up to the most recent press release were queried and the content was subsequently extracted using XPath. Corpus 5 consists of the internal training set transcripts. The LM trained on all available corpora (1-5) is referred to as ”LM extended”.
Article Post-processing During development
we noticed that there was a considerable amount
of errors due to incorrectly predicted articles (e.g
”der”, ”die”, ”das”) (see Section 5.4 for more
details). We identified individual definite and
indefinite articles in a predicted utterance, removed
them, and queried the top 5 predictions of a BERT
model
In total, nine experiments were conducted with
the goal to investigate system performance of the
various models. The details of the experiments
are presented in Table 3. The very first
experiment (”base”) was performed without any
finetuning or post-processing on the base model, while
the second one (”AMext3x5”) aimed at
evaluating the predictions from the ”AM extended 3x5”
model without applying any Language Model.
In the third experiment we evaluated the model
trained only on the internal Swiss German train
set without any pre-training on Standard
German (”AMno pretrain”). The next two
experiments consisted in introducing and extending
the Language Model (”AMch3x5 LMbase” and
”AMch3x5 LMext”). Following that, we
investigated data augmentation possibilities. In
addition to SpecAugment which is used in all
experiments, we applied speed perturbation
The results of all experiments were evaluated on the internal test set, except for the last one, ”AMall5x5 sp LMext”, where the internal test set was used for training the models. The five bestperforming versions were submitted for evaluation on the public test set of the Shared Task. Table 3 provides an overview of all results.
Eventually, we achieved 40.29% WER on the official test set. Our best performing system is a combination of the baseline Acoustic Model retrained on 5 blocks with Swiss German data, speed perturbation, and a Language Model fine-tuned on in-domain data from Switzerland. 5.4
The two largest performance improvements were achieved by fine-tuning the Acoustic Model on the task-specific data (”AMext3x5” vs ”base”: WER reduced by 38% absolute) and by using a general-purpose Language Model during decoding (”AMext3x5 LMbase” vs ”AMext3x5”: WER reduced by 7.64% absolute). Both of these are standard practices in ASR and hence these improvements are neither surprising nor particularly
We identified articles as one distinct source of errors: around one sixth of substitution errors were articles; hence, we decided to address these during post-processing (model ”AMch3x5 sp LMext artc”). Our method using BERT (see Section 5.2) did not improve performance. There are several reasons for this. First, while some articles were indeed improved with this method, often there was insufficient context to accurately determine the correct article. Domainspecific abbreviations (e.g. party names such as SVP, EVP) also proved difficult. Second, we observed a number of article errors that are due to the non-exact nature of the transcription. These are linguistic or stylistic changes and improvements of the spoken text and can therefore not be addressed by our method. For example: changing a spoken definite article to an indefinite one, using plural instead of singular, transcribing a spoken ”es” with ”das”, or inserting an extra article into a coordinated noun phrase.
Finally, there is also a challenge that relates to the specific language variety in this task: articles in Swiss German are rather difficult to detect as they usually consist of single phonemes which are assimilated to the following noun. This means that articles may be missed at an earlier stage of processing and will not be present in the output passed to the post-processing.
Our extended Language Model brought a nearly 1% absolute WER improvement (”AMch3x5 LMext” vs ”AMch3x5 LMbase”), which is less than we expected. However, this can be explained by the rather small amount of additional data - corpora 4 and 5 (see Table 2) Language Model LM base LM extended LM extended LM extended LM extended LM extended LM extended only account for 2% of all sentences passed to the LM. Using more in-domain data in the LM could lead to a larger effect.
Further small improvements were obtained by using speed perturbation (”AMch3x5 sp LMext” vs ”AMch3x5 LMext”: -0.7% absolute on our internal test set and -0.85% on the task test set) and retraining five Jasper blocks instead of three (”AMch5x5 sp LMext” vs ”AMch3x5 sp LMext”: -0.4% absolute on our internal test set and -0.6% absolute on the task test set).
We also note that our performance on the task test set is consistently better than the one on our internal test set. 6
Before we conclude, we would like to reflect on the properties of the task data and their repercussions for WER results.
Our analysis of the errors and the data showed that properties of the data often lead to an increase in WER, where the ASR model provides an adequate transcription but is ”punished” by data artefacts. We identified the following main issues: We noticed that transcriptions in the training set are inconsistent with respect to numerals, which are written as either numbers or words, so that transcribing the numeral four as ”vier” when the reference transcript has ”4” will lead to a substitution error. Since there is no consistency in the writing of numerals (e.g. always using words, always using numbers, using words when smaller than ten, etc), this leads to errors that we could not prevent.
Transcripts are polished (e.g. speech disfluencies such as repetitions, hesitations, and false starts are removed) and reformulated so they become more readable, which means they do not exactly represent the spoken text. For example, in training set item 19940.flac, the speaker starts by saying ”mer hie enne” (DE: ”wir hier drin”, EN: ”we in here”), but this was transcribed as ”wir in diesem Saal” (EN: ”we in this chamber”), leading to three errors (two substitutions and one deletion) when transcribed faithfully to the spoken utterance by the model.
We also note issues with the segmentation of audio files, which, according to the task organizers, was performed automatically. This leads to insertion errors (when extra audio is included beyond what is transcribed) or deletion errors (when portions of the audio are missing) of the model that cannot be mitigated.
Given the observed discrepancies between the
speech and transcript, additional evaluation
measures might be considered. In CL-STT, BLEU
scores are used for evaluation. Even though this
metric has been criticized, it might fit the setup
of this task better, since the paraphrasing might
not be unique. Further, measures considering
semantics and synonyms
In this paper, we presented our contribution to the
Shared Task on Low-Resource STT at GermEval
2020. Our solution consists of a CNN acoustic
model based on Jasper
Neil Zeghidour, Qiantong Xu, Vitaliy Liptchinsky, Nicolas Usunier, Gabriel Synnaeve, and Ronan Collobert. 2018. Fully Convolutional Speech Recognition. arXiv preprint arXiv:1812.06864.