Conducting “manual” transcriptions and analyses is unsustainable for most historical oral archives because they require a remarkable amount of funds and time. The FONTI 4.0 project aims at exploring the suitability of automatic transcription and information extraction technologies for making historical oral sources available. In this work, we conducted an experiment to test the performance of two commercial speech-to-text services (Google Cloud Speech-to-text and Amazon Transcribe) on digitized oral sources. We created an eight-hour corpus made of manually transcribed and annotated historical speech recordings in TEI format. The results clearly show how audio quality and disturbing elements (e.g., overlaps, foreign words, etc.) impact on the automatic transcription, showing what needs to be improved for implementing an unsupervised transcription chain.
FONTI 4.01 is a project aiming at exploring the suitability of automatic transcription and analysis tools for the preservation of historical oral sources recorded on analog carriers, in particular magnetic tapes. The digitization of an audio archive is a long and expensive task that can require several years. Furthermore, the content of audio recordings needs to be listened and cataloged for making audio recordings retrievable. Archives composed by hundreds or thousands of hours of audio require a huge amount of time, people and funds for making the content accessible and preventing their exploitation. Therefore, automatizing the transcription and the analysis task could drastically reduce the time for making digitized audio recordings accessible.
The project consists in a transcription-chain
(Tchain), firstly defined in (van Hessen et al., 2020),
that differs in two main aspects: (a) in FONTI
4.0, the transcription obtained with
speech-totext (STT) algorithms should not be corrected by
human; (b) an additional restoration step could
be required for digitized audio recordings.
Furthermore, differently from STT evaluation
experiments conducted by
During the digitization process, speed and equalization errors can occur, especially when different speed and equalization configurations are used in different part of the same tape (Pretto et al., 2020). This leads to distortions of the recorded signal that becomes unlistenable. By using the correction workflow and digital filters described in (Pretto et al., 2021a; Pretto et al., 2021b) these errors can be corrected and at least parts of the signal can be saved. This task is essential for making the speech signal suitable for STT algorithms. This paper aims at evaluating the transcription performance of two commercial software on a real use case and identifying potential problems or limitations concerning peculiarities of analog audio recordings. Section 2 describes the corpus, used as ground-truth for the experiment. Section 3 outlines the methodology adopted for this experiment, whereas results are reported in Section 4. Finally, Section 5 presents the authors’ conclusions. 2
The Cinema & Civilta` (C&C) corpus was
conceived within the FONTI 4.0 as ground-truth for
evaluating the performance of STT services on a
real case study. To build the corpus, we
transcribed speech recorded on four magnetic tapes
made available by the Giorgio Cini Foundation
in Venice and digitized at the Centro di
Sonologia Computazionale - CSC
Defining the methodology for the transcription is an important step for the preservation, analysis and access of oral sources. The main difficulty consists in making decisions on how to represent and convey both verbal and non-verbal elements in written form. Because of the absence of a universal standard of transcription (Schorrsidt, 2011), the methodology usually depends on the research aim.
In this research, we decided to complete a verbatim transcription, by reporting every word spoken in the recording including errors, false starts, truncations, and overlaps in Italian, French and Spanish. Using the software ELAN (Lausberg and Sloetjes, 2016), we first segmented audio files extracted from the digitized tapes, making the start and end of each segment coincide with the 2DOI: 10.5281/zenodo.5645827 speaker’s turn of talk. Then, we transcribed each segment while listening to the corresponding part of audio in slow motion. Eventually, we opted for employing automatic transcriptions from Google Cloud Speech-to-text (GCS) and Amazon Transcribe (AT)3, later used in the STT experiment, and correcting the text playing the audio at normal speed. This allowed us to save half the time for each transcription, which previously required a full day of work. Moreover, we were able to retrace and match the identity of the speakers to the voices in the recordings, through the consultation of historical documentation on the conference, and also by comparing voices across the recordings. 2.2
The annotation was employed for the addition of
important metadata to the C&C corpus
regarding different levels of audio quality and the
presence of disturbing elements in the recordings. Our
methodology is in compliance with the Text
Encoding Initiative (TEI) standard guidelines4 for
transcribed spoken material
4tei-c.org/release/doc/tei-p5-doc/en/ html/TS.html (last accessed September 3rd, 2021 5oxygenxml.com (last accessed September 3rd, 2021) <foreign> identifies a word or phrase as belonging to some language other than that of the surrounding text, e.g.: <foreign xml:lang=”fr-FR”> Mesdames, messieurs </foreign>; <shift> marks the point at which some paralinguistic feature of a series of utterances by any one speaker changes, e.g.: Io credo che questo argomento sia <shift feature=”tempo” new=”a”/> particolarmente importante <shift feature=”tempo” new=”normal”/> per vedere; <del> contains a letter, word, or passage indicated as superfluous by the annotator, in this case it was used for false starts, repetitions and truncations, e.g.: in questo <del type=”falseStart”> moden </del> momento (false start) momento di <del type=”repetition”> di </del> crisi (repetition) suggestione di <del type=”truncation”> spettaco </del> di spettacolo (truncation); <anchor> was used to mark overlaps by attaching an identifier to a point within a text, e.g.: a contatto di un <anchor synch=”ovrl6” xml:id=”S06”/> pensiero <anchor synch=”ovrl6e” xml:id=”S06e”/> lo inducono a (interrupted speaker) <anchor xml:id=”ovrl6”/> Io non lo vedo. Chi e` questo? Chi e` questo? <anchor xml:id=”ovrl6e”/> (interrupting speaker); <note> contains notes or citations, and, for the purpose of this research, it was used to annotate the audio quality at the beginning of each segment, e.g.: <note>good </note>; Audio quality annotations (<note>) were assigned to each segment using the the following scale (Samar and Metz, 1988): excellent: speech is completely intelligible; good: speech is intelligible with the exception of a few words or phrases; fair: with difficulty, the listener can understand about half the content of the message; poor: speech is very difficult to understand, only isolated words or phrases are intelligible; bad: speech is completely unintelligible.
The distribution of words (without punctuation and events) for each audio quality annotation is reported in Table 1.
TOT it-IT 9,075 30,571 2,919 1,417 43,984 fr-FR 5,930 2,514 83 23 8,550 es-ES 4,097 800 0 0 4,897 <distinct> identifies any word or phrase which is regarded as linguistically distinct, as in the case of prosodically unified units, e.g.: staccarsi da <distinct type=”pcu”> questa estetica </distinct> e dai pregiudizi;
The STT experiment consisted in testing the
ability of GCS and AT to correctly transcribe
historical recordings. Furthermore, we decided to
investigate the performance of STT transcriptions
obtained from GCS and AT at different levels of
au<vocal> marks any vocalized but not necessarily dio quality and in presence of disturbing elements
lexical phenomenon, e.g.: del nostro mondo in the recordings such as background noise,
over<vocal> <desc>cough</desc> </vocal> laps, code switching etc. (see Section 2.2).
che direi postmoderno.; To analyze the performance of the two STT
systems, we developed a Jupyter notebook able to
<incident> marks any phenomenon or iflter the text by language, audio quality,
disturboccurrence, not necessarily com- ing elements, etc., and select several options, such
municative, for example incidental as tokenization rules. In this experiment, we
denoises or other events affecting com- cided to use only lower case characters, split
aposmunication, e.g.: e` attivita` creatrice, trophes and remove punctuation from both
man<incident><desc>noise</desc></incident> ual and automatic transcriptions. The
groundma non propriamente l’artista; truth and the resulting transcription of the STT
services were canonicalized. The alignment
algorithm works on single utterances and minimizes
the Levenshtein distance
It is important to note that we did not employ this
metric to tell how good a system is, but only that
one is better than the other
In order to avoid the introduction of errors not due to the transcription task, we decided not to use the automatic language recognition feature because it could drastically impact on the performance. Therefore, we cut and divided the audio ifles in different languages and automatically transcribed them separately. In this preliminary work we illustrate and compare mainly WER trends between the two STT systems, calculated on the entire corpus as well as each sub-corpora in relation to audio quality levels and the presence of disturbing elements.
Figure 1 illustrates that the performance of AT are better than GCS in all corpora. The difference between the two systems is small in the Italian sub-corpus, but much wider in the French.
A possible explanation could be the presence of L2 speakers of French whose pronunciation could have negatively affected the recognition performance. Nevertheless, it should be also considered that the Italian sub-corpus is more than vfie times bigger than the French and the Spanish.
STT software performance can be further observed in Table 2: for the transcription of the whole corpus, AT scores a lower WER and finds more correct matches than GCS. On the other hand, deletions in GCS are more than double than in AT, whereas substitutions and insertions are higher in AT than in GCS. In any case, the number of deletions and insertions between AT and GCS are different probably because the two services make use of different language model weights.
Figure 2 shows that transcription performance are very similar in Italian and Spanish with “Excellent” quality, but not in French. For this reason, we cannot impute the bad GCS performance to audio quality. In the Italian sub-corpus, performance are also similar with “Good” quality, but not in the Spanish, where both services performed badly.
The negative impact of audio quality is also evident in the French sub-corpus, despite WER values are much higher than Italian.
Results in Figure 3 display the annotated disturbing events found in the C&C corpus that were assumed to negatively affect the performance of STT software in terms of WER. The element that provides the minor disturbance is shift, although the scored WER value for this tag is higher than the one calculated on the overall evaluation. About the other disturbing elements, they show a major impact on the transcription of both STT services. Overall, AT performance is better with most disturbing elements. The only exception is represented by code-switching events in foreign languages for which GCS had a better performance. 5
In this article we conducted a preliminary research experiment testing the ability of STT software to correctly transcribe digitized historical oral sources on magnetic tape. It should be noted, that since this preliminary work has been conducted on a small sample of data, our results are only indicative of which elements represent the biggest obstacle for STT software performance.
In spite of disturbing elements and the variation of audio quality in the recordings, we demonstrate that with our dataset and in terms of WER, AT performed more accurate transcriptions compared to GCS . On the other hand, GCS was better at recognizing foreign words. Table 2 shows that AT introduces less incorrect words but more insertions and substitutions. This should be taken into consideration when working with automatic information extraction tools (e.g., Named Entity Recognition algorithms) applied to automatic transcriptions. Further analysis should investigate the cause of this trend, to verify if this behavior is also due to alignment or tokenization errors.
With respect to software performance evaluations in relation to variables characterizing analog recordings of speech, we found evidence that audio quality drastically impacts on the number of mismatches. Observations about the incidence of disturbing elements, on the other hand, cannot be generalized since sub-corpora are in three different languages and have three different sizes. Throughout the analyses we noted that the most negative impact on transcription, in terms of the increase of WER, is caused by the presence of some specific recurring elements, i.e.: code-switching (foreign), overlaps and probably even the production of L2 speakers (Figure 3). Nonetheless, given the necessity of preserving historical documents in a more time and cost effective way, we came to the conclusion that researchers working on the preservation of historical recordings will benefit from the use of the T-chain. This is because the reduction by half of the time required for manual transcriptions in slow motion does compensate the lack of accuracy. This means that researchers working on the collection and preservation of oral archives will be able to focus on filling the gap between human and machine output.
Further contributions will be necessary for conducting experiments on L1 and L2 data separately, cross-language testings reducing the Italian subset to the size of the French and Spanish sub-corpora and evaluating the impact of incorrect transcriptions on WER. Language identification through code-switching is another important problem for automatic transcription. Both services recently provided this functionality, but while we are writing this paper, the Google Cloud is still a preview version. As soon as the feature will be available the performance of automatic language recognition algorithms should also be investigated, especially because this feature is essential for automatizing the transcription of entire archives.
This paper is produced under the FONTI 4.0 project, financed by resources from the Regional Operational Program co-financed with the European Social Fund 2014-2020 of the Veneto Region. An important acknowledge is due to Fondazione Giorgio Cini, Venice for making available its precious audio material as well as for its help in the recording analysis, and Matteo Petten o` for his contribution in the Jupyter notebook development.
Hedda Lausberg and Han Sloetjes. 2016. The revised neuroges–elan system: An objective and reliable interdisciplinary analysis tool for nonverbal behavior and gesture. Behavior research methods, 48(3):973–993.
Meredith Moore, Michael Saxon, Hemanth Venkateswara, Visar Berisha, and Sethuraman Panchanathan. 2019. Say what? A dataset for exploring the error patterns that two ASR engines make. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2019-Septe:2528–2532.
Andrew Cameron Morris, Viktoria Maier, and Phil Green. 2004. From wer and ril to mer and wil: improved evaluation measures for connected speech recognition. In Eighth International Conference on Spoken Language Processing.
Niccolo` Pretto, Alessandro Russo, Federica Bressan, Valentina Burini, Antonio Roda`, and Sergio Canazza. 2020. Active preservation of analogue audio documents: A summary of the last seven years of digitization at csc. In Proceedings of the 17th Sound and Music Computing Conference, SMC20, pages 394–398, Torino, Italy.
Niccolo` Pretto, Nadir Dalla Pozza, Alberto Padoan, Anthony Chmiel, Kurt James Werner, Alessandra Micalizzi, Emery Schubert, Antonio Roda`, Simone Milani, and Sergio Canazza. 2021a. A worklfow and novel digital filters for compensating speed and equalization errors on digitized audio open-reel tapes. In Proceedings of Audio Mostly 2021, AM21, Trento, Italy.
Niccolo` Pretto, Edoardo Micheloni, Anthony Chmiel, Nadir Dalla Pozza, Dario Marinello, Emery Schubert, and Sergio Canazza. 2021b. Multimedia Archives: New Digital Filters to Correct Equalization Errors on Digitized Audio Tapes. Advances in Multimedia, 2021:5410218.
Vincent J Samar and Dale Evan Metz. 1988. Criterion validity of speech intelligibility rating-scale procedures for the hearing-impaired population. Journal of Speech, Language, and Hearing Research, 31(3):307–316.
Thomas Schmidt and Kai Wo¨rner. 2014. Exmaralda. In Jacques Durand, Ulrike Gut, and Gjert Kristoffersen, editors, The Oxford handbook of corpus phonology. Oxford University Press.
Thomas Schorrsidt. 2011. A tei-based approach to standardising spoken language transcription. Journal of the Text Encoding Initiative, 1.
Arjan van Hessen, Silvia Calamai, Henk van den Heuvel, Stefania Scagliola, Norah Karrouche, Jeannine Beeken, Louise Corti, and Christoph Draxler. 2020. Speech, voice, text, and meaning: A multidisciplinary approach to interview data through the use of digital tools. In Companion Publication of the 2020 International Conference on Multimodal Interaction, ICMI ’20 Companion, page 454–455, New York, NY, USA. Association for Computing Machinery.