The evolution of artificial intelligence models has made them tools of everyday use in many fields. However, the enormous capabilities demonstrated by these models have, on the one hand, some apparent costs in terms of money, computational resources, or data. On the other hand, there are some hidden costs for end users who rely on models trained by third parties, sacrifice awareness and control of a tool, and try to evaluate its performance in their specific contexts. This is the case of supervised End-to-End (E2E) ASR systems and self-supervised E2E-ASR, also referred to as Large Acoustic Models (LAM). On the one hand, they provide an important starting point for building information systems oriented to speech interaction and, on the other hand, are complex to evaluate, use and adapt in specific contexts.
1. Introduction cesses of manual labelling. On the other hand, it has allowed the creation of a large number of freely availModern Automatic Speech Recognition (ASR) systems, able and open-source general-purpose ASRs, bringing among the other Natural Language Processing (NLP) these systems within the reach of a greater number of systems, achieve remarkable performances thanks to institutions and companies. However, their use remains the computing potential enabled by Deep Neural Net- limited due to the lack of benchmarks oriented towards works (DNN). Indeed, over the last decade, the auto- specific contexts and communication styles. In this work, matic speech recognition community has made great we will analyze the evolution of ASR systems, how the strides [1, 2, 3], moving from traditional hybrid mod- nature of the data used for their training has changed, elling (Acoustic Model+Language Moel) to end-to-end and the limitations of modern ASR systems. Finally, we (E2E) modelling that directly translates an input speech will propose an initiative aimed at collecting high-quality sequence into a sequence of output tokens using a sin- data in Italian aimed at both performance verification gle network, leading to self-supervised E2E models, also and training based on specific communicative styles. referred to as Large Acoustic Models (LAMs), that can model speech without the aid of labelled data. These revolutionary innovations have completely subverted the 2. The evolution of ASR systems traditional architectures of ASR systems used in previous decades. In addition, there has also been a strong impact on the cost-efectiveness and democratization of ASR systems. On the one hand, the change in architecture has made it more economical to collect and create the data sets necessary for training, which previously required the use of a large number of experts in the field of speech analysis involved in long and expensive pro
ASR systems have been the subject of several revolutions, which have impacted their internal architecture and the nature of the data employed for their training. Traditional ASR systems rely on two separate components [9]: The Acoustic Model (AM), which is aimed at converting the voice signal into a sequence of phones, and the Language Model (LM), aimed at transforming the sequence of phones received from the AM, in the most likely and reliable transcription. These two models were initially realised with techniques such as Hidden Markov Models (HMM) or Gaussian Mixture Models (GMM). Then, with the advent of Deep Neural Networks (DNN), both have been realized as supervised DNNs. Still, the output of both components was the same: the AM produces the most likely sequence of phones given the input voice sig
Wav2vec2.0 [6] with Quantization
2020 2021 2021
RNN-T RNN-T CTC CTC
1.8/3.3 1.8/2.9 1.4/2.5 given the input sequence of phones. This means that the bels (character, word piece, etc.) to one label unless a two components had separate objectives and relied on special label separates these. The result is a sequence of ladiferent kinds of high-quality and costly datasets. On the bels shorter or equal to the input vector sequence length. one hand, the AM needs well-aligned sound-to-phone The CTC is one of the most difused decoding techniques. transcriptions. On the other hand, the LM needs a sta- As non-auto-regressive, it is also considered computatistically representative set of phone-to-word samples tionally efective as it requires less time and resources in order to provide meaningful transcription. Providing for training and inference phases. Conversely, the RNNadequate quality data requires highly specialised profes- T (also named Transducer) is an auto-regressive speech sionals to hand label in both cases. This type of ASR transcription technique which overcomes CTC’s limitasystem requires tens to hundreds of hours of speech to tions, i.e., non-auto-regressive and limited label sequence train the AM and a few million words to train an LM length. An RNN-T is a speech transcription technique (depending on the context). The aim is to transcribe which can produce label-transcription sequences longer fairly long sentences with an accuracy linked to specific than the input vector sequence and models long-term application contexts. transcription elements’ inter-dependency. A Transducer
The turning point that led to the recent End-to-End typically comprises two sub-decoding modules: one that ASR (E2E-ASR) [2] was the introduction of the Trans- forecasts the next transcription label based on the previformer [10] network architecture, on which most ac- ous transcriptions (prediction network); the other that tual AI models rely. Compared to traditional systems, combines the encoder and prediction-network outputs in E2E-ASRs, the voice signal is directly converted into to produce a new transcription label (joiner network). its corresponding transcription without any intermedi- These features improve transcription speed and perforate, human-readable format. This evolution results in mance with respect to CTC at the expense of more trainsystems with a single objective needing only one cheaper ing and computational resources required [12]. dataset to be trained since the intermediate phones tran- Finally, the most recent advancement consists of the scription and the alignment parts have been removed. employment of self-supervised training techniques, givThe Transformer architecture [10] opens up the possi- ing rise to what could be defined as the first truly Endbility of building a combination of AM and LM, now to-End ASR, namely Wav2Vec2 [6] and after a while to referred to as the Encoder and Decoder, which directly HuBERT [7], both are also referred to as Large Acousmaps an unaligned sequence of sounds to its transcrip- tic Model (LAM) [13] because of their training process tion. With a few hundred hours of non-aligned tran- which usually involves two main phases. The first one scribed speech through a supervised learning process, is the pre-training phase, during which vast amounts of E2E-ASR systems outperform the previous generation on untranscribed speech data are employed in order to recogaverage by providing an error of up to 5% in the case of nize and discretize hidden acoustic units’ representations pure Transformer-Encoder systems, or up to 4% in the by employing diferent processes such as quantization case of Conformer-Encoder systems [5] (see table 1 for (Wav2Vec2[6]) directly from the raw audio sample, or performance). Clearly, the Decoder module implemen- clustering (HuBERT [7]) on MFCC features. Then, during tation choice strongly impacts E2E ASR performances, the last phase, a transcription module could be trained on such module is usually implemented as a Connectionist smaller datasets (few hours) in order to obtain an error Temporal Classification (CTC) model [ 11] or as a Recur- rate of about 2% (see table 1). rent Neural Network Transducer (RNN-T) [12]. CTC is a non-auto-regressive speech transcription technique which collapses consecutive, all-equal, transcription la3. Self-supervised E2E Solutions (?) to data shortages which is usually unavailable. Another aspect to consider is how and to what extent the democratisation of ASR systems has been impacted. In fact, if, on the one hand, it is possible to obtain much more data for the same cost, on the other hand, the same quantity of resources is no longer suficient, especially for training purposes.
Undeniably, by committing the model to learn all parts automatically, E2E-ASRs overcome the dificulties and cost-inefectiveness of the data preparation and modelling phases of conventional systems, while requiring far more training data [14]. This shift significantly im- 4. High-quality data for pacted ASR systems; on the one hand, it significantly reduced training data costs while increasing their vol- context-specific assessment ume, as shown by the availability of plenty of generalpurpose training datasets [1, 3]. On the other hand, in Clearly, the availability of good-quality and wellspite of the cheapness of training data, ASR systems are categorized data is paramount in the current application now accessible to a wider public. Clearly, these innova- landscape. On the one hand, such data is essential to tions present some expenses, which in this case consist evaluate pre-trained systems in specific contexts with of higher computational costs, longer training times, and speaking styles related to diferent communication situaloss of modularity [3] compared to traditional ASR sys- tions. On the other hand, such data is crucial for training tems. Indeed, adapting such a general-purpose E2E-ASR and fine-tuning modern supervised and self-supervised to specific contexts means, in some cases, updating the E2E ASR. To this end, the Phoné consortium was born Decoder (LM) to a special-purpose field or updating the as a voluntary initiative to collect, verify and distribute Encoder (AM) to handle a special type of speech, which transcribed and non-transcribed Italian speech datasets requires fine-tuning and, in the worst cases, training the in various application contexts. Table 2 shows the actual model from scratch. amount of data collected and verified by the consortium
Then, the advent of Self-supervised systems impacted to provide Italian institutions and companies with adthe adaptability aspects of general-purpose E2E ASR, giv- equate instruments to evaluate these promising tools, ing rise to Large Acoustic Models (LAMs), which basically which are, however, assessed in contexts and communiare Encoders trained on vast amounts of non-transcribed cation styles that do not reflect the target ones. cheaper datasets, compared to data needed by simple Currently, data is divided into two macro-categories, E2E-ASR, which are then combined with an Encoder namely, Transcribed and Untranscribed, to enable the fupart trained on small quantities of language-specific tran- ture training of self-supervised E2E-ASR. Then, datasets scribed data. The result is a large, general-purpose model are further divided into specific communication styles that can be easily deployed in most contexts. Although [20, 21]: they are publicly available and, therefore, freely adaptable, the necessary computational resources are so prohibitive that they are within the reach of a few companies and institutions, even for simple fine-tuning.
A further point to be considered is that the advantages of both simple E2E ASR and Self-supervised ones come at the expense of lower interpretability of systems’ internals, making it dificult to diagnose errors and limiting their usage in critical contexts [3]. However, some studies in the field of eXplainable AI (XAI)[ 15] try to provide explanations and methodologies for analysing behaviours and phenomena modelled by various E2E ASR systems, aiming to make them more interpretable [16, 17, 18, 19], still based on special purpose data.
To summarize, although the innovations introduced by E2E and self-supervised E2E systems have allowed their fast difusion, still their industrial and institutional deployment remains subject to limitations [3] which, in some cases, are strongly related to special-purpose data availability. Indeed, employing a general-purpose E2E ASR system in a specific domain requires evaluation and potential fine-tuning /training on domain-specific data, • Monologic speech involves only one person speaking without interacting with an interlocutor.
This type of speech is characterized by consistency and structuring, as it typically consists of lectures, speeches or situations that require preliminary preparation. As a result, the speech appears cohesive and well-organized. The language register tends to be higher and more formal. • Dialogic speech involves two or more people in a conversation, characterized by exchanges of messages and information. It is thus configured as a communicative act with a dynamic structure. Unlike monologic speech, dialogic speech does not involve prior preparation; therefore, the speech tends to be simpler from a syntactic point of view, the articulation of words tends to be less precise (hypoarticulation), and it is also characterized by greater conciseness of expression. • In Read speech, the speaker reads a written text aloud (as in the case of audiobooks), therefore this type of speech is characterized by clear pronunciation (there is a tendency towards hyperarticulation), complete syntax and greater coherence and cohesion of the text. Furthermore, another fea- [3] R. Prabhavalkar, T. Hori, T. N. Sainath, R. Schlüter, ture is given by the modulation of reading speed S. Watanabe, End-to-end speech recognition: A and the use of strategic pauses and intonations survey, IEEE/ACM Transactions on Audio, Speech, to improve communicative efectiveness. and Language Processing (2023).
[4] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, S. Kumar, Transformer transducer: A MTartaenrsicarlibTeydpe SpMeoencohloTgyipce 50M0 iMnuintuetses streamable speech recognition model with transTranscribed Dialogic 400 Minutes former encoders and rnn-t loss, in: ICASSP 2020Transcribed Read 120 Minutes 2020 IEEE International Conference on Acoustics, Untranscribed Monologic 10000 Minutes Speech and Signal Processing (ICASSP), IEEE, 2020, Untranscribed Dialogic 500 Minutes pp. 7829–7833.
Untranscribed Read 2200 Minutes [5] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, et al., TLaisbtleof2material collected and verified for the evaluation Conformer: Convolution-augmented transformer and training of E2E-ASR systems (both supervised and self- for speech recognition (2020). supervised) in specific contexts for the Italian language. [6] A. Baevski, Y. Zhou, A. Mohamed, M. Auli, wav2vec 2.0: A framework for self-supervised learning of speech representations, Advances in neural infor
Behind ASR-related aspects, the consortium’s pur- mation processing systems 33 (2020) 12449–12460. poses also extend to other voice-related tasks, which [7] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhoinclude, but are not limited to, Text-To-Speech (TTS), tia, R. Salakhutdinov, A. Mohamed, Hubert: Speaker Identification (SI), Speaker Verification (SV), and Self-supervised speech representation learning by others. masked prediction of hidden units, IEEE/ACM Transactions on Audio, Speech, and Language Pro5. Conclusion cessing 29 (2021) 3451–3460. [8] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, In this work, we present the current panorama related R. Pang, Y. Wu, W2v-bert: Combining contrastive to E2E ASR systems, how their data usage evolved along learning and masked language modeling for selfwith technological improvements and the current issues supervised speech pre-training, in: 2021 IEEE Authat these improvements solved or introduced. Firstly, tomatic Speech Recognition and Understanding we observe the significant improvement in models’ per- Workshop (ASRU), IEEE, 2021, pp. 244–250. formances while pointing out issues connected to the [9] S. Karpagavalli, E. Chandra, A review on automatic models’ capacity assessment related to specific communi- speech recognition architecture and approaches, cation styles and domains. We observe the shift in model International Journal of Signal Processing, Image training costs, moving away from data becoming cheaper Processing and Pattern Recognition 9 (2016) 393– and easier to collect towards computing resources grow- 404. ing in quantity and costs. Then, we observed how the [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, advantages introduced by modern E2E (supervised and L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Atself-supervised) ASRs come at the expense of an increase tention is all you need, Advances in neural inforin their complexity, which consequently reduces their mation processing systems 30 (2017). interpretability. Finally, we propose a voluntary, high- [11] A. Graves, S. Fernández, F. Gomez, J. Schmidhuquality data collection initiative to evaluate and train ber, Connectionist temporal classification: labelling systems related to various speech communication styles unsegmented sequence data with recurrent neural to enable more informed use and greater accessibility of networks, in: Proceedings of the 23rd international E2E-ASR systems. conference on Machine learning, 2006, pp. 369–376. [12] A. Graves, Sequence transduction with recurrent neural networks, arXiv preprint arXiv:1211.3711 References (2012).
[13] L. M. Giordano Orsini, V. N. Vitale, F. Cutugno, [1] M. Malik, M. K. Malik, K. Mehmood, I. Makhdoom, Large scale acoustic models: A new perspective, Automatic speech recognition: a survey, Multime- Sistemi intelligenti (2023). doi:10.1422/108137. dia Tools and Applications 80 (2021) 9411–9457. [14] G. Coro, F. V. Massoli, A. Origlia, F. Cutugno, [2] J. Li, et al., Recent advances in end-to-end automatic Psycho-acoustics inspired automatic speech recogspeech recognition, APSIPA Transactions on Signal nition, Computers & Electrical Engineering 93 and Information Processing 11 (2022). (2021) 107238. [15] D. Gunning, Explainable artificial intelligence (xai), Defense advanced research projects agency (DARPA), nd Web 2 (2017) 1. [16] A. Prasad, P. Jyothi, How accents confound: Probing for accent information in end-to-end speech recognition systems, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 3739–3753. [17] D. Ma, N. Ryant, M. Liberman, Probing acoustic representations for phonetic properties, in: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, pp. 311–315. [18] A. Pasad, J.-C. Chou, K. Livescu, Layer-wise analysis of a self-supervised speech representation model, in: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2021, pp.
914–921. [19] V. N. Vitale, F. Cutugno, A. Origlia, G. Coro, Exploring emergent syllables in end-to-end automatic speech recognizers through model explainability technique, Neural Computing and Applications (2024) 1–27. [20] M. Nakamura, K. Iwano, S. Furui, Diferences between acoustic characteristics of spontaneous and read speech and their efects on speech recognition performance, Computer Speech & Language 22 (2008) 171–184. [21] P. Azizova, Linguistic analysis and learning of dialogical speech in literary texts, JETT 14 (2023) 86–94.