Voice assistants understanding dialects would help especially elderly people. Automatic Speech Recognition (ASR) performs poorly on dialects due to the lack of sizeable datasets. We propose three adaptation strategies which allow to improve an ASR model trained for German language to understand Swiss German spoken by a target speaker using as little as 1.5 hours of speaker data. Our best result was a word error rate (WER) of 0.27 for one individual. SwissText 2022: Swiss Text Analytics Conference, June 08-10, 2022, [11]. Our evaluation yields results similar to Plüss et al
Automatic Speech Recognition (ASR) refers to the task of converting an audio signal into its written transcrip- a pre-trained model adapted on a single speaker. tion and finds application, among others, in voice assistants. ASR performs well on so-called well-resourced1 languages, while results on dialects, specifically Swiss German, are poorer. This is particularly inconvenient for the acceptance of applications involving smart assistants for elderly people, for whom it might be a big nuisance to switch to Standard German. ASR for Swiss German is challenging for several reasons: 1. Swiss German has no standardized written form and Standard German is the output of choice, meaning that the system must provide speech translation (ST) rather than mere recognition. For example, the German expression «wollen wir» ferent variants in Swiss German, e.g. «wömmer», «wemmer», «wemmr» or «wämmer». 2. Swiss German dialects are diverse and not geographically well confined. Thus, creating regional models would be challenging.
In recent studies, the application of end-to-end ASR models (from raw audio to the words) based on deep neural networks has shown a considerable performance boost. To achieve good results, a considerable amount of training data is needed [1]. In the case of Swiss German, there is a lack of enough data, variability and the appropriate ground truth. An exception is the recently published Swiss Parliaments Corpus (SPC), which we use in our work [2].
tracted some attention [5, 6, 7, 8, 9]. When looking at Swiss German, we find the work of Plüss et al. [ 2], who claim a WER of 0.289 using a Conformer model on a the SPC dataset. Other researchers combined the SPC with a proprietary dataset to train various ST systems and achieved a WER of 0.5 when using only the SPC dataset [10]. A further approach achieved a WER of 0.39 on the SPC by training a model on a German dataset, transfer learning to the SPC enhanced with a proprietary internal dataset and refining the classification with a re-scoring [2], even if a direct comparison is not possible, and gives interesting insights about diferent fine-tuning strategies.
could be pronounced and written in several dif- Conformer [3] and wav2vec2 [4]. The latter achieved the htp:/ceur-ws.org ISN1613-073
CEUR
Workshop Proceedings (CEUR-WS.org) 1Well-resourced refers to the availability of abundant labeled data corpora to train machine learning algorithms.
We here describe the baseline model, datasets and adaptation approaches. 3.1. Model We base our work on a pre-trained wav2vec2 model2 available from the HuggingFace3 AI community. We denote this model as baseline. The model topology consists of convolutional layers which map the raw audio to latent quantized speech representations and a Transformer structure which maps to context representations. The first pre-training stage involves self-supervised learning and therefore does not need labeled data [4]. For the baseline model, pre-training is done with multilingual data (53 languages) to learn language-independent speech units, followed by supervised training with German data, since Swiss German has strong similarities with German.
In our experiments we used the Swiss Parliaments Corpus (SPC), a Swiss German dataset that contains recordings and transcriptions of the cantonal parliament of Bern (Grosser Rat Kanton Bern) [2]. It contains 293 hours of audio by 198 speakers and represents the biggest Swiss German speech recognition dataset to date. We use a subset of the SPC containing only samples with a high alignment between text and audio4. The audio files contain mostly Swiss German speech, whereas the labels (transcriptions) are in Standard German. We chose SPC mainly because of its size. In comparison to existing Swiss German datasets, such as ArchiMob [12], SPC has more audio data, which allows us to experiment better with various sizes of the single-speaker datasets. In addition, we recorded a new small dataset from a speaker unrelated to the SPC dataset, which allows us to test our approaches on another context. It is based on utterances of the Voxforge speech corpora5. We call this the external speaker (shortened: «ext»). 3.2.1. Dataset Partitioning From the SPC dataset we created convenient partitions for the experiments. In the original corpus we identified the five speakers that have the biggest amount of data. These are the speakers with IDs 82, 145, 177, 186 and 207. Together with our own small external dataset, this yields six datasets. We refer to them as Single Speaker Corpora (SSC). For the approaches which involve a training step with multiple speakers, we extracted a subset of the SPC which excludes the speakers identified above (SPC-without-top5). Among the single speakers, the one with the least amount of data has around 1.5 hours 2The exact model used is wav2vec2-large-xlsr-53-german. 3https://huggingface.co/ 4Intersection over Union (IoU) > 0.9, train_0.9 5http://www.voxforge.org/de/downloads of audio; therefore, for a fair comparison, we limited all six SSCs to 1.5 hours.
Our goal is to adapt the baseline model to perform ASR satisfactorily with a single target speaker. We propose three approaches, which we evaluated on six SSC:
the SPC excluding all target SSCs. 2. Fine-tune the baseline model with data from the target SSC. 3. Combine the two previous approaches by training the baseline model on SPC excluding all target SSCs, then fine-tune with the target SSC.
We conducted several experiments in line with the approaches described in Section 3.3 and evaluated the models against held-out test data, reporting the word error rate (WER). We show the results in Table 1. The best approach for adapting to a single speaker is the last one, i.e., to first train the model on Swiss German data from several speakers and then on the corresponding target speaker dataset. Interestingly, fine-tuning the baseline model only with the target single speaker datasets gives worse results compared to training a model on multiple speakers (SPC-without-top5). However, it is important to note that the SSCs contain only 1.5 hours of data, whereas SPC-without-top5 contains around 176 hours.
The individual improvement of the adaptation depends on the speaker and varies between 1% and 4% on the five SPC speakers and reaches a notable 14% on the external speaker. Speaker 82 has the highest WER when evaluated with the base model but the lowest WER when ifne-tuning with multiple Swiss German speakers and/or the single speaker dataset of speaker 82. In contrast, speaker 207 has the lowest WER when evaluated with the base model, but the highest WER using the other three approaches. The reasons for this behaviour could not be fully determined and further investigations are future work.
Increasing the data for the single speaker training has not led to a significant reduction of the WER. When training the model resulting from the second approach with six hours of audio from speaker 82 instead of only 1.5 hours the WER decreases only by 2%. An identical improvement is observed when training with four hours of speaker 207 instead of 1.5 hours. Decreasing the time used for single speaker training does increase the WER: When training with a third of the external single speaker dataset, the WER increases by 4% and when training with a sixth, the WER increases by 5%.
A remarkable result is the impact of Swiss German finetuning before the single speaker adaptation. Training the baseline model on the full SSC of speaker 82 (six hours), it achieves a WER of only 0.44. In comparison, training the model first with SPC-without-top5 and then finetuning with 1.5 hours of speaker 82, achieves a WER of 0.27 on the same test set, giving an improvement of 17%. Training the baseline model on SPC-without-top5 and then on 0.25 hours of data of the external speaker still performs better than using the model trained only on SPC-without-top5 and the model trained only with 1.5 hours of data of the external speaker.
(Bernese) in the SPC. Furthermore, the SPC was recorded in a parliament and has therefore a certain bias in terms of content. The results can also be influenced by the combination of the chosen metric and the ground truth.
For instance, if the audio contains the phrase session vom september and the label is septembersession, the WER increases if the model predicts the former phrase, even if the two options are semantically identical. In addition, Signal and Information Processing Association AnSwiss German does not have a past simple tense. Con- nual Summit and Conference (APSIPA ASC), 2019, sequently, if the label is written in past simple, there is pp. 628–632. doi:10.1109/APSIPAASC47483.2019. a significant diference in the structure of the spoken 9023130. sentence and the ground truth. [6] R. Imaizumi, R. Masumura, S. Shiota, H. Kiya, Dialect-aware modeling for end-to-end japanese dialect speech recognition, in: 2020 Asia-Pacific 5. Conclusion Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2020, We presented three possible strategies to adapt a pre- pp. 297–301. trained ASR model based on wav2vec2 to enhance the [7] Y. Zhao, J. Yue, X. Xu, L. Wu, X. Li, End-torecognition on a single Swiss-German-speaking individ- end-based tibetan multitask speech recognition, ual. The best strategy appears to be training a baseline IEEE Access 7 (2019) 162519–162529. doi:10.1109/ model with multiple Swiss German speakers and in a second phase fine-tuning with a small amount of data [8] AYC.ZCEhSaSn.g2,M01.9A.b2l9im52it4,0A6..Hamdulla, Error correction from the target speaker. With this strategy, the WER for based on transformer lm in uyghur speech recognisix speakers ranges between 0.27 and 0.41. The improve- tion, in: 2021 IEEE 2nd International Conference on ments of each approach on an external speaker and five Pattern Recognition and Machine Learning (PRML), SPC speakers are similar. 2021, pp. 204–207. doi:10.1109/PRML52754.2021. 9520740. 6. Outlook [9] L. Pan, S. Li, L. Wang, J. Dang, Efective training end-to-end asr systems for low-resource lhasa Our adaptation approaches were tested only on one kind dialect of tibetan language, in: 2019 Asiaof model. It would be interesting to extend the evalua- Pacific Signal and Information Processing Astion to diferent models and examine if the behaviour and sociation Annual Summit and Conference (APresults are similar. Furthermore, the evaluation sample SIPA ASC), 2019, pp. 1152–1156. doi:10.1109/ size n=6 is not quite representative considering the di- APSIPAASC47483.2019.9023100. versity of Swiss German. An evaluation containing more [10] A. Khosravani, P. N. Garner, A. Lazaridis, Learning speakers will allow more solid claims. to translate low-resourced swiss german dialectal speech into standard german text, in: 2021 IEEE Automatic Speech Recognition and Understanding References Workshop (ASRU), 2021, pp. 817–823. doi:10.1109/ [1] D. Jurafsky, J. H. Martin, Speech and Lan- [11] AYS.RAUr5a1b5sk0y3.y,2A02. 1A.g9a6r8w8a2l4, 9S.. Dey, O. Koller, Dialecguage Processing (3rd Edition Draft), USA, 2021. tal speech recognition and translation of swiss gerURL: https://web.stanford.edu/~jurafsky/slp3/ man speech to standard german text: Microsoft’s ed3book_jan122022.pdf. submission to swisstext 2021, 2021. URL: https:// [2] M. Plüss, L. Neukom, C. Scheller, M. Vogel,
Swiss parliaments corpus, an automatically aligned arxiv.org/abs/2106.08126. doi:10.48550/ARXIV. swiss german speech to standard german text [12] 2T1.0S6a.m0a8r1d2ž6ić., Y. Scherrer, E. Glaser, ArchiMob corpus (2021). URL: http://arxiv.org/abs/2010. a corpus of spoken swiss german, in: Proceed02810. arXiv:2010.02810. ings of the Tenth International Conference on Lan[3] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, guage Resources and Evaluation (LREC’16), EuJ. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, R. Pang, ropean Language Resources Association (ELRA), Conformer: Convolution-augmented transformer 2016, pp. 4061–4066. URL: https://aclanthology. for speech recognition, 2020. URL: https:// org/L16-1641. arxiv.org/abs/2005.08100. doi:10.48550/ARXIV.
2005.08100. [4] A. Baevski, H. Zhou, A. Mohamed, M. Auli, wav2vec 2.0: A framework for self-supervised learning of speech representations (2020). URL: http: //arxiv.org/abs/2006.11477. arXiv:2006.11477. [5] J. Sun, G. Zhou, H. Yang, M. Wang, End-to-end tibetan ando dialect speech recognition based on hybrid ctc/attention architecture, in: 2019 Asia-Pacific