This paper explores the potential of using Large Language Models in multilingualism research to accelerate the management and processing of spoken data. The speech-to-text processing of utterances by multilingual speakers are in the focus. Qualitative discussion of the main issues relating to the non-standard language use of bilingual individuals is provided, using Polish-German recordings from the LangGener corpus as an example.
The main challenge in multilingualism research is collecting suficient data from a homogeneous sample of multilingual speakers to achieve robust results. Perfectly balanced bilingualism in speech and writing is extremely rare. Most speakers have significant discrepancies in their writing skills across diferent languages, often because they were not educated in one of them. Consequently, multilingualism research, which is necessarily usage-based, relies mostly on spoken data. Managing spoken data is considerably more challenging than managing written data, which is more consistent, even in less formal varieties, and therefore easier to parse and annotate. To perform any kind of quantitative analysis, spoken data must first be transcribed into digital text for further processing and annotation.
Recent developments in large language models for Automatic Speech Recognition (ASR), in particular the emergence of Whisper [1], have clearly accelerated the transcription process. The business sector is the biggest beneficiary, as any kind of recording can be used for quick documentation of a meeting, compilation of notes, protocols or presentations.
From the perspective of multilingualism research, however, the situation looks diferent due to the diferent expectations of the quality of automatic transcription from business and research. This paper provides an overview of the issues that need to be addressed to enable linguists to use LLM-enhanced ASR eficiently. We describe the targeted standard of transcript in Section 2. We use Polish-German bilingualism as an example and data from the LangGener project [2] (see Section 3.2). The preliminary research presented here is based on three versions of Whisper[1]: small, medium, large, because it can be run locally (see Section 6 for ethical reasons) and is evaluated as best performing among the available transcription tools (see [3] for comparison). Section 4 describes the generally known problems related to the quality of transcripts, while Section 5 presents the issues specific to multilingualism.
In terms of transcripts, the main diference between business and multilingual studies is that businessoriented transcripts should ideally be clean and monolingual to ensure fluent reading. In other words, the focus is on the content’s informativeness. Therefore, an ASR should automatically translate foreign elements into the company’s language — English in many cases — even if the meeting was multilingual. Elements of speech that do not carry essential information, such as repetitions, disfluencies and hesitation markers, should be removed. In other words, a business transcript will resemble standard written language more closely than the actual oral communication.
Transcripts used for research purposes are diferent in nature. In linguistic investigations, the recording is the primary object of research and transcripts approximate all the sounds that speakers make, including hesitation markers and self-corrections, to show how and when oral communication lfows.
The standards for transcripts vary in academic research, depending on the language being documented and the purpose of the research. Underresourced languages without a written tradition or standard orthographic rules are usually transcribed using phonetic transcription systems (for overview, see [4]). Most European languages can be transcribed using simpler systems that have been developed for analysing spoken discourse (for overview, see [5]). Such systems are mostly based on the orthography of the standard language. They vary in terms of the notation used for spoken language phenomena and issues related to transcript quality, as well as the number of annotation layers. The latter approach is preferable for further automatic language processing, such as lemmatisation and morphosyntactic tagging, as well as for automatic queries, which can be conducted according to standard forms. Further annotation layers, such as phonetic transcriptions, can be added afterwards to diferentiate between diferent non-standard pronunciations, which are typical of dialects, heritage, and non-native varieties. In summary, a transcript for research purposes should be a precise representation of speech that is simple enough to be easily searched and processed further.
The following section focuses on the technical side of this article. First, a brief introduction into the Whisper models used is provided together with the configuration parameters. Afterwards, a brief description of the dataset is given.
For this case study, we have decided to use the standard Whisper models[1]. The latest versions as described on the Whisper GitHub page of both the library and the versions of the models were employed1. For this first case study, our goal was to have a base case estimate of how the basic Whisper models will perform. Thus, we have decided to use the auto language detection modes for all the experiments. We have used three diferent model sizes to find the tradeofs between the model size and qualitative results of the transcription. The three models used were • Whisper base with 74 million parameters, • Whisper medium with 769 million parameters and • Whisper large with 1550 million parameters.
More information can be found on the model card on GitHub2 All Whisper models were used in the multilingual versions.
The LangGener Corpus [6] contains recordings of language biography interviews with Polish–German bilinguals. The sample is stratified across two generations: an older generation who lived in Poland in areas that were part of the German Reich before 1945 (called Generation Poland), and late bilinguals who were born in these areas and immigrated to Germany (nowadays known as ’Aussiedler’ or in the project called Generation Germany).
This stratification principle makes the sample interesting for ASR since their speech contains various features of non-standard language, including dialectal features and non-native pronunciation.
Structurally, the corpus contains many phenomena related to multilingualism [7] such as codeswitching, lexical matter and pattern replication. We describe them briefly in Section 5. For a full overview of these features in LangGener, see [8].
One serious flaw that applies to many LLM-enhanced tools is hallucinations. In the case of ASR, this
means transcriptions of text which cannot be aligned with the audio file, observed by [ 9] for Whisper.
Although open.ai claimed improving this issue,3 we still identify hallucinations in the studied data, as
shown in (
The Whisper transcription of the LangGener corpus contains such mistakes too. In particular, it omits disfluencies, broken words, repetitions and other features of conversational data. We find issues relating to incorrect endings, spelling mistakes and homophone structures that are particularly relevant for multilingual data. We therefore return to them in the next section.
Problems strictly related to multilingual transcriptions have received little investigation to date. We are aware of [10], who fine-tuned Whisper for English-Chinese (i.e. biscriptual) transcription. However, this work does not provide an overview of the problems associated with such transcriptions; rather, it contains a technical description of the fine-tuned model.
When studying multilingualism, it is important to avoid translations in transcripts known from business solutions, since the focus of research is precisely on phenomena related to the mixing of two language codes. Typically, one code dominates (the matrix code), while the other is used only occasionally and is embedded within the matrix code.
Whisper does adjust the text to one language relatively rarely, but we find some problematic passages,
for example (
b. Tak, tak. ...na szpruch. To jest... każde dziecko ma prawo na szpruch [Whisper Large] The word Rechtsanspruch ‘legal right’ is not recognized as the embedded German code, and its second part is transcribed with Polish orthography as Polish non-existing word. /S/ is represented as <sz> instead of German spelling requiring <s> at the beginning of word before <k>, <p> or <t>. Further, the phrase Recht da drauf ‘right to it’ is not transcribed but interpreted, and rendered in Polish consistently with the previous mistake, thus, as prawo na szpruch ‘right for szpruch’.
Preferring to unify with the matrix code in the output can also lead to words being replaced with
similar-sounding words from the matrix code, or entirely new linguistic units being formed according
to word-building rules of the matrix code. Thus, in example (
[Whisper Large]
In all three versions of Whisper, the matrix code was usually recognised in our data, and the embedded code was a considerable source of transcription errors. However, we obtained transcripts where the output was recognised as an entirely diferent language, such as Yiddish, as shown in Figure 1 from the transcript of speaker BQ RAC with Whisper Large.
We assume that this may be due to the frequent code switching and unclear pronunciation in the first few seconds of the audio file. This shows that the way in which the recordings are cut before processing afects the accuracy of the automatic transcription. Whisper Base exhibits a similar issue in the same section, but incorporates an even greater variety of languages, ultimately producing a predominantly Germanic transcription, as shown in Figure 2.
In Section 3.2 we mentioned three phenomena that occur particularly frequently in the situations of language contact: Code Switching (CS), Matter Replication (MAT) and Pattern Replication (PAT). Having them shortly explained, we point out the problems they cause for ASR.
Following [7], we classify phenomena related to the direct transfer of embedding language as CS
and MAT. When a single word or phrase takes over the inflection of the matrix language and becomes
fully integrated, we call it MAT. CS is similar to MAT, but it shows the lower grade of grammatical
integration as shown in (
‘because normally, I had only half of the language course.’
CS and MAT are particularly interesting from the perspective of ASR, as they imply mixing of two language codes with distinct phonological systems, and orthographies. This leads to issues regarding orthographic choice and the identification of words as belonging to the lexicon of the matrix or embedded language. Consequently, we observe the frequent mistakes mentioned in the previous section, such as incorrect spellings, interlingual homophones and problems with ending assignment.
MAT in (
b. ta grupa z tymi ganzkleinen Kindern ist dobrze belegt. [Whisper Large]
Frequent CS seems to increase the error in the transcription, as shown in (
my to są trzy # Personalschlüssel ist # ich glaube drei # zwei Vollzeitkräfte i chyba jeszcze trzydzieści # godzin jedna Kraft10 b. To są trzy personalsche SEL, jakieś global drive, zwei Vollzeitkräfte i chyba jeszcze 30 o godzin, bo jednak kraft. [Whisper Large]
The long German compound noun Personalschlüssel ‘stafing ratio, personnel counting’ is most likely recognised as German since the first part of the word is transcribed following German spelling, the German clause ich glaube drei ‘I guess three’ is transcribed as a mix of Polish and English jakieś global drive. Although the phrase zwei Vollzeitkräfte ‘two full positions’ is entirely correctly transcribed, the final part of the utterance trzydzieści godzin jedna Kraft, which is a mix of Polish and German is transcribed (partly incorrectly) with Polish spelling, since kraft is not capitalised.
We also observe here that if a word is spelled incorrectly, this error is consistent throughout the transcript. The word Personalschlüssel, which appears four times in one recording, is consistently misspelled with a made-up expression personalsche SEL each time.
The third mentioned category, PAT refers to the situation where only structures are borrowed into
matrix language. In example (
Nevertheless, the syntax-based prediction may play a role ASR. For example, the Polish modal verb
może ‘can’ is often followed by an infinitive complement. In (
a. jak dziecko nie dostanie, może rodzić swe klage w tej szczypie [Whisper Large] Therefore, we believe that the word-order related PAT are worth inspections in the future. For example, we formulate a working hypothese that the strict German word-order rules regarding the position of the verbal predicate – in the second position in the main sentence or in the last position in the subordinate sentence, frequently copied by bilingual speakers from Polish to German – could pose a potential source of error for ASR too.
Due to protection policies and laws, such as the General Data Protection Regulation in the European Union, personal and sensitive data must be protected and inaccessible to external entities during and after the preparation and publication of data. Nonetheless, data obtained is expected to be publicly available to other researchers in accordance with the current research standards. Typically, the personal and sensitive data are accessible only to authorised project members. To enable LLM-based processing of data that could contain sensitive information, such information would need to be deleted or hidden before 10https://langgener.ijppan.pl/OUT/NF_PAD_II_GD_PL01-49260-57407.wav 11https://langgener.ijppan.pl/OUT/XL_PIL_GP_DE27-77300-82110.wav 12https://langgener.ijppan.pl/OUT/NF_PAD_II_GD_PL01-158897-162417.wav processing on external servers. Alternatively, it can be conducted exclusively locally while maintaining the data protection standards. Of these two options, only local processing is realistic. First, manually processing transcripts to pseudonymise or erase information would be economically unreasonable. Second, an ASR with integrated LLMs benefits from the syntactic and semantic information encoded by name entities, which are usually the subject to pseudonymisation.
Access to data sets appropriate for the fine-tuning ASR is also more problematic than for other LLMenhanced applications. Such data is most frequently subject to data protection regulations, meaning the data sets cannot be shared without major changes being undertaken.
It appears that the quality of state-of-the-art ASR systems is still far from enabling instant multilingual transcriptions without considerable expenditure on post-editing. It is important to point out that grammatical and orthographic errors are easily detectable. Problems with translation and the removal of embedded language from transcriptions are more serious, as they are harder to identify. Therefore, these two issues are given higher priority in additional training. Based on the results of the survey, our future work will focus on fine-tuning Whisper using the bilingual LangGener corpus. Additionally, we will explore ways to incorporate ASR into the Research Data Repository of the University of Hamburg, using the Polish-German example as the prototype. This would enable users to search for recordings based on the words used in them without requiring the uploader to provide a transcript upfront.
This paper provides a very preliminary overview of problems related to ASR in multilingualism research. It is evident that trials on fine-tuning could ofer solutions to these issues. Although multilingual transcripts are the focus of this paper, some of the problems (and therefore potential solutions) may also be relevant to monolingual transcriptions with dialectal features or other vulnerable groups. This work complies with the ACL Ethics Policy. Prior to the current study, we had not taken any actions to pre-train the systems for the needs of the current task.
During the preparation of this work, the authors used DeepL in order to: Grammar and spelling check. After using these tool(s)/service(s), the authors reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.
We are thankful to the principal investigators of the project Language across Generations: Contact Induced Change in Morphosyntax in German-Polish Bilingual Speech, Anna Zielińska and Björn Hansen, for providing us with the material for the survey.
This contribution was partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany´s Excellence Strategy – EXC 2176 ‘Understanding Written Artefacts: Material, Interaction and Transmission in Manuscript Cultures’, project no. 390893796. The research was mainly conducted within the scope of the Centre for the Study of Manuscript Cultures (CSMC) at University of Hamburg.