Due to the prevalence of bilingualism among Kazakh speakers, code-switching to Russian is common in Kazakh speech. That presents a challenge for monolingual Kazakh-language ASR systems that struggle to transcribe the embedded Russian words. This paper attempts to determine the bene t of bilingual training on matrix language (Kazakh) and embedded language (Russian) monolingual data, as opposed to training on code-switched data only. Speci cally, we evaluate the model's performance on matrix language words and embedded words separately. We make use of two datasets: Kazakh speech with code-switching and Russian speech with no code-switching. We train a monolingual model on each dataset, and a bilingual model on a mixture of the two. The main objective of the experiments is to compare the performance of a model trained on code-switched speech with that of a model trained on full utterances in both languages. Experimental results suggest that bilingual training improves the model's performance on matrix words, and greatly improves its performance on embedded words. We observe an absolute WER improvement of 14.69% in the code-switched words.
Previous attempt at building a bilingual Kazakh{Russian speech recognition
system by Khomitsevich et al. [
Code-switching (also referred to as code-mixing [
Although most state-of-the-art ASR systems are monolingual, the impact of code-switching on ASR performance has recently sparked research interest [5{ 9]. The success so far has, however, been limited, largely due to the challenges outlined above.
Due to the majority of Kazakh speakers being bilingual [
In this paper we attempt to determine the impact of training on both Kazakh and Russian language data on the quality of speech recognition of the embedded Russian segments in Kazakh speech.
The rest of the paper is organized as follows: Section 2 describes the dataset used in this work. Section 3 describes the model architecture and reports the experimental results. Finally, Section 4 concludes the paper and discusses the results. 2
We make use of a proprietary Russian{Kazakh dataset consisting of Kazakh
call centre operator recordings. No data augmentation techniques were used in
the course of this work. Data statistics by language and subset are presented in
Table 1.
The domain of the data is very narrow, containing a signi cant amount of stock
phrases and domain-speci c words. Approximately 10% of words in the Kazakh
language data are code-switched speech. The observed cases of code-switching
include intra-sentential code-switching (insertion of Russian phrases into otherwise
Kazakh sentences), as well as intra-word switching (Russian words conjugated
as if they were Kazakh) [
For training we use 40-dimensional log Mel-scale lter bank energy features with CMN with rst- and second-order derivatives.
All the ASR systems are built using the Kaldi speech recognition toolkit [
(a) Single BLSTM layer (b) Acoustic model
For language modeling, all transcripts available for each set of data are merged and used to train a 3-gram language model. We use graphemic pronunciation dictionaries when compiling the language model into a WFST-decoder.
Acoustic models based on deep Bidirectional Long Short-Term Memory (BLSTM)
recurrent neural networks have been demonstrated to be highly e ective in
various ASR tasks [14{16]. We use identical BLSTM architecture for the acoustic
model for each set of data. Each has three hidden BLSTM layers with
projections [
Each acoustic model is then trained using Natural Gradient for Stochastic
Gradient Descent [
As seen in Table 2, the bilingual model displays higher performance on the Kazakh evaluation set at the expense of signi cant performance loss on the Russian evaluation set.
As the Russian language evaluation set contains no code-switched sentences, it and the Russian monolingual model are not considered further. Instead, we focus on the Kazakh evaluation set for closer examination.
To determine the impact of bilingual training on code-switching, we've collected per-word statistics used in WER calculation (Table 3). Each error| substitution (S), insertion (I), or deletion (D)|is classi ed based on the language the word belongs to. Note that substitutions are thus split into two classes: substitution with a Kazakh word or a Russian word.
Monolingual (kaz) Bilingual
We then calculate WER for matrix and embedded language words separately (Table 4). For the purposes of this calculation, all substitutions are considered to belong to the language of the token of the reference transcription.
The results shown in Table 4 show clear improvement in recognition of the embedded language words. In this paper we presented a bilingual Kazakh{Russian speech recognition system. We observe signi cant WER improvement in the matrix (Kazakh language) data, and 14.69% absolute WER improvement on the embedded (Russian language) data. It is worth noting that this is not the case of more data trivially yielding better results. The bilingual model performs signi cantly worse on the Russian language data alone.
The results indicate that multilingual speech recognition systems are inherently better capable of recognizing code-switched speech than monolingual systems trained on code-switched speech itself. Future directions include investigating end-to-end multilingual systems from the point of view of code-switched segments, developing more sophisticated language modeling for code-switching, and introducing more than two languages. 5
This work was partially nancially supported by the Government of the Russian Federation (Grant 08-08), and by the grant of Ministry of Education and Science of the Russian Federation Goszadanie No. 2.13462.2019/13.2.