Modern Automatic Speech Recognition (ASR) systems, based on Deep Neural Networks (DNN), have achieved remarkable performance modelling huge quantity of speech data. However, recent studies have shown that fine-tuning pre-trained models, despite providing a powerful solution in low-resource settings, lacks robustness across diferent speech styles, and this is not just related to the amount of training data, but to substantial diferences in phonetic-prosodic characteristics. Therefore, this study aims to explore how modern E2E ASR systems' performance is afected by the amount of training data and the type of speech data and which acoustic-phonetic features most markedly exert an influence. To this aim, a k-fold cross-validation was performed by fine-tuning a pre-trained FastConformer model with datasets varying in type of speech data and size. Then we performed a correlation analysis between the values of the acoustic characteristics of the data and the recognition scores. The analyses allow the identification of an optimal combination of speech data type and amount of training data. Also, results show that using both more spontaneous speech or more controlled speech can be beneficial, provided that the speech rate is contained.
ation of the data considered. In fact, while most benchmarks consist of read or rather controlled speech producSpoken language is intrinsically variable. Speech pro- tions, the interest in ASR applications in real contexts, duced to convey a message can vary widely depending such as human-machine-interactions or transcription of on several internal and external factors, such as the com- spontaneous conversation, led to the evaluation of ASR municative and contextual situation, the formality of performance in diferent, less controlled and more spontathe exchange, the speaker’s disposition and individual neous scenarios, which resulted in diferent performance choices of the forms and phonetic realisation deemed as values for other types of data, e.g, lower for more sponmost appropriate and functioning to convey the intended taneous datasets [4]. In particular, a recent study on the message given the specific condition of production and evaluation of ASR systems, based on state-of-the-art sureception [1]. Thus, speech variability can be described pervised, self-supervised, and weakly supervised End-toas the synergetic contribution of linguistic, contextual, End models, on Italian speech [5, 6], showed consistent and social factors [2], which results in diferent types of performance diferences across speech types: dialogic, speech, often referred to as speech style, characterized monologic, and read speech. Namely, increasing perforby varying levels of spontaneity, fluency, speaking rate, mance from dialogic speech to monologic speech and prosodic variation, degree of phonic specification [3, 1]. from the latter to read speech.
Modern ASR systems, based on Deep Neural Networks Eforts devoted to overcoming this issue often consist
(DNNs), have achieved remarkable performance by mod- of building complex and costly models that require large
elling the linguistic and acoustic features of spoken lan- amounts of data and computational resources. However,
guage. However, these systems implicitly learn to model this can be problematic, especially when working with
soonly a small proportion of the possible variation that called “low-resource languages”. Diferent studies have
characterises spoken language. As a result, error rates provided evidence that a powerful solution is provided
increase with the degree of linguistic and phonetic vari- by fine-tuning pre-trained models (see [ 7]). However, [
Based on this body of research, this work aims to con- speech data, can we identify the optimal combination of tribute to the study of how diferent types of speech data speech data and amount of training data? are modelled and how this afects the robustness of the RQ2. What acoustic-phonetic characteristics afect the model toward the definition of an optimal dataset to ob- most modern E2E ASR performance? To what extent? tain robust recognition systems.
To investigate how data characterised by diferent
feaEspecially, but not exclusively, within the context of low- tures (data type) and varying amounts of training data
resource studies, the need to develop less resource-greedy (training data time) can afect the fine-tuning of
modASR systems emerges. To this end, diferent data efi- ern ASR models, our method includes a K-fold
crossciency techniques, e.g., learning or data augmentation validation procedure [
Another way to help achieve high performance with • fold splitting according to training-specific criteminimal data may consist in relying on less but more ria, i.e., speech type and training fold size (mininformative data by investigating how diferent types of utes); speech data are modelled and afect the robustness of the • selection of a pre-trained model for fine-tuning; model, and which combination of diferent speech types • evaluating model performance for the selected and amount of data optimises the informativeness and datasets; eficiency of a sample to fine-tune pre-trained models. • fine-tuning the pre-trained model by training it
To this end, a better understanding of the aspects of on the diferent folds;
speech that challenge ASR architectures the most is re- • comparison of the performance of the fine-tuned
quired. In the last 20 years, various studies have inves- models;
tigated which phonetic features afect automatic recog- • Word Error Rate - acoustic features correlation
nition the most (see [7] for an overview). In particular, analysis.
issues were observed to mostly concern features of
conversational speech such as grammatical inconsistencies, 3.1. Data
self-interruptions, backchannels, lexical and non-lexical
disfluencies, and the degree of pronunciation variation
[
Given the methodological focus of this study, we decided
to work with a well-known, restricted dataset to gain
clearer insights into the efectiveness of the method and
the findings. Hence, we selected data from a corpus
that was the object of previous phonetic studies [
Using the text annotation in TextGrid format [20], the dataset was split in Inter-Pausal Units based on pauses longer than 250 ms. This resulted in utterances with a mean duration of 4,81 seconds (standard deviation = makes it particularly suitable for real-time speech recog2,88, max length = 30 ms). The text was normalised by nition tasks. Furthermore, the architecture is highly scalremoving special characters, but leaving annotation of able, and indeed, FastConformer is at the core of topsegmental phenomena such as fillers (eeh, ehm, mh) and performing Nvidia ASR systems like Canary and Paraprolongations (e.g., laaa). The final considered dataset keet. consists of slightly more than 3h and 27881 tokens for The Group K-fold is a variation of k-fold crossG01, about 3h and a half and 39145 tokens for G02, and validation intended for scenarios where the data has a preabout 3 h and 29341 tokens for G03. G02 shows a higher defined group structure. The key constraint is to ensure speech rate than both G01 and G02. See Table 1 for total that the same group is not represented within the same duration, tokens and speech rate (SR), and mean (m) and splits, namely training, validation and test sets. In our standard deviation (sd) of utterance duration and tokens. case, samples from the same speaker will be grouped in the same split. This method prevents data leakage by ensuring that the model generalises to new, unseen groups, 3.3. Modelling not just new samples from existing groups. The corpus is Selecting an appropriate pre-trained model is a criti- split into three folds, one per speaker and idiosyncratic cal decision that influences the success of subsequent speech style (data set type), and these were further split downstream tasks. While many high-performing models into five sub-folds of diferent sizes (split size), resulting are available, such as Whisper or Phi-4, our selection in 15 diferent fold combinations described in Figure 1. was guided by several practical requirements: languagespecific support for Italian, computational eficiency, and 3.4. Evaluation and correlation analysis public availability to ensure experimental reproducibility and democratic access. Accordingly, we chose the The model performance across the diferent folds was FastConformer model pre-trained on Italian by Nvidia evaluated considering the Word Error Rate (WER) com[21]. The FastConformer is an eficient variant of the puted at the utterance level. Model comparison was conConformer architecture, designed to significantly reduce ducted based on WER mean and distribution values per the computational cost and latency of the standard Con- fold to observe which model performed better across the former model while maintaining high accuracy. This considered folds.
Then, correlation analysis between data characteristics and WER was performed to examine the influence of acoustic features on the performance of diferent time folds. Feature values were automatically extracted for each utterance employing the OpenSmile toolkit [22]. The Geneva Minimalistic Acoustic Parameter Set (eGeMAPSv02) [23], i.e., a restricted set of features based on interdisciplinary evidence and theoretical significance, was selected as the feature set. The study focuses, in particular, on the features that could be considered as the most relevant, as reported in previous literature [7] and inspection of the data.
The analysis starts by evaluating the model’s baseline performance on the defined datasets before applying kfold cross-validation to establish a reference for comparison. The selected model performs less for the G01 dataset (mWER = 0.51, sd = 0.32) than for the G03 dataset (mWER = 0.40, sd = 0.26) and the G02 dataset (mWER = 0.39, sd = 0.26), see the first three rows of Table 2. The overall mean WER across diferent data type sets is 0,43 (sd = 0.26).
Then, we observe the model’s performance on each fold. Figure 2 and Table 2 show the mean WERs per train set data type and size. The mean WERs across the data type sets (purple line) reach lower values than the baseline (red dashed line) already after fine-tuning with the smallest 15’ sets (mWER_15 = 0.32, mWER_30 = 0.22, mWER_60 = 0.18, mWER_120 = 0.16, mWER_all = 0.16). The values decrease as the size of the training set increases. However, the magnitude of the WER diference between subsequent size groups progressively diminishes until it becomes trivial between the models trained on 60’ speech and those trained on the entire datasets (about 3h).
We then consider the mean WER values grouped by train set data type. Although models trained on G01, as well as G02 and G03 data, perform better than the baseline, we observe that the models trained on G02 data perform worse than the others, with WERs closer to the overall baseline. In particular, the models trained on G02 are tested on G03 and are closer to the G03 baseline (mWER = 0.4). Instead, the models trained on G03 and tested on G01 for the speech rate values, the latter correlate with WERs show a larger diference with the G01 baseline (mWER = positively and increasingly along the train set size. How0.51) than the diference between models trained on G01 ever, this trend is considerably stronger for the models and tested on G02 and the G02 baseline (mWER = 0.39). trained on data from the G01 dataset (and tested on G02
Considering both the contribution of the train set data dataset). Weaker correlations are observed for the mean type and the size to the model performance improvement, values of F0, especially for the G02 and G03 models, with the optimal fold is G03_120. the strength slightly increasing with the size of the training set. Rather constantly weak correlations can be ob4.2. Features Correlation with WER served for median loudness, MFCC4 in voiced regions and WER values. Still rather constant but slightly stronger is the correlation between loudness peaks per second and WERs for the models trained on the G02 dataset.
To explore how diferent datasets afect model performance, we observe which features correlate with the trained models. The heatmap in Figure 4.2 shows the Pearson coeficients resulting from the correlation between a selection of relevant acoustic features and the WER for each model. The colour of each tile represents the direction of the correlation, while its intensity indicates the strength of the correlation. Red denotes a positive correlation, meaning higher feature values correspond to higher WER, whereas blue indicates a negative correlation, where higher feature values align with lower WER. White represents a weak or no correlation.
We observe negative correlations between the WER values and both the utterance duration and tokens. The correlation becomes weaker, but still noticeable, with increasing train set data size, and the same trend is observed for each dataset. An opposite trend is observed
This study contributes to investigations on how the performance of modern E2E ASR models is afected by the type and amount of speech data used for training and aims to define a way to identify an optimal combination of type and amount of speech data. The investigation is supported by observation of how diferent speech acoustic features contribute to the model performance.
The Fast-Conformer WER on the selected
semimonologic, semi-sponetanous data presents overall lower
values than the evaluation provided by a previous study
on Italian monologic data, i.e., 12.8 WER [6]. More
specifically, lower recognition scores are reported for This study provides evidence corroborating the idea
G01 speech, characterised by a more spontaneous speech that less but more informative data can be used to
finestyle, including more features such as non-lexical fillers tune pre-trained models, which could be useful for
fineand prolongations than the other speakers, which is in tuning in low-resource scenarios. Furthermore, the use
line with the literature [
The cross-fold evaluation shows that the models’ per- that ofer a favorable trade-of between performance and formance improves with train set size; however, the mag- computational resources. These models present a vinitude of the improvement gradually decreases until be- able alternative for deployment on resource-constrained, coming trivial between models trained on 120 minutes privacy-oriented devices. At the same time, they can and about 3 hours of speech. This finding supports the be quickly adapted to diferent low-resourced contexts, claim that simply increasing the size of the training set standing in practical contrast to larger-scale yet resourceis not always beneficial and not always enough to guar- demanding models. antee better performance. Although this trend stands In this study, we prioritised methodological soundness across all datasets, variation can still be observed. and understanding over immediate broad applicability.
The models trained on speech produced by the second We selected a known dataset restricted in size and speaker guide (G02) perform worse than the others, with recog- diversity to enhance the interpretability of the results, nition scores closer to the overall baseline. In particular, verify the method’s core efectiveness and establish a the models trained on G02 speech, that is characterised solid foundation for scaling to larger, more diverse corby higher speech rate and fewer pauses, are tested on pora. Future work will be devoted to further exploring G03 speech and achieve smaller improvement over the this direction by considering larger datasets that maxG03 baseline as compared to the models trained on G03 imise diferences in acoustic-phonetic features that were speech, showing a more controlled speech style, and G01 observed to be relevant for the modelling. speech, defined by a more spontaneous speech style. It is particularly worth noticing that the models trained on G03 and tested on G01 show the best recognition scores References over all size folds, thus overcoming the G01 baseline disadvantage. This seems to indicate that some speech data [1] B. V. Tucker, Y. Mukai, Spontaneous speech, Camare more informative than others and may even over- bridge University Press, 2023. come recognition issues related to more spontaneous [2] A. Vietti, Il ruolo della variabilità acustica nella and conversational speech styles; however, studies in costruzione del dato linguistico, in: Superare this direction should be further developed. l’evanescenza del parlato: un vademecum per il
Considering both the contribution of the train set data trattamento digitale di dati linguistici, Bergamo Unitype and size to the model performance improvement, versity Press, 2021, pp. 45–70. the dataset that optimises the combination of data type [3] P. Wagner, J. Trouvain, F. Zimmerer, In defense and amount is the one containing 120 minutes, i.e., two- of stylistic diversity in speech research, Journal of thirds of the available dataset, of the more controlled, but Phonetics 48 (2015) 1–12. still spontaneous, speech produced by G03 (RQ1). [4] P. Gabler, B. C. Geiger, B. Schuppler, R. Kern, Recon
In line with the literature [7], correlations between sidering read and spontaneous speech: Causal perrecognition scores and utterance durational features spectives on the generation of training data for auemerge. More specifically, higher length values (in terms tomatic speech recognition, Information 14 (2023) of utterance tokens and duration) correlate with lower 137. recognition errors, which indicates that providing a wider [5] N. Vitale, E. Tanda, F. Cutugno, Towards a responcontext enhances recognition. Conversely, higher speech sible usage of ai-based large acoustic models for rates hinder recognition. However, this efect is more automatic speech recognition: On the importance or less mitigated according to the speech type in the of data in the selfsupervised era, in: Atti quarto Contraining set (RQ2). This finding, as well as the constant vegno Nazionale CINI sull’Intelligenza Artificiale– and weak correlations observed for the other acoustic fea- Ital-IA 2024, 2024. tures, deserves further attention and needs to be explored [6] T. Cimmino, E. Tanda, V. N. Vitale, F. Cutugno, Evalin future works. uating asr performance in italian speech, in: STUDI
Overall, these findings show that using both more AISV, Milano: Oficinaventuno, under review. spontaneous speech and more controlled speech can be [7] J. Linke, B. C. Geiger, G. Kubin, B. Schuppler, What’s beneficial to fine-tune a pre-trained model, provided that so complex about conversational speech? A comthe speech rate is not too high. More detailed analyses parison of HMM-based and transformer-based ASR will be performed considering the values of the acoustic architectures, Computer Speech & Language 90 characteristics and their variation to gain deeper insight. (2025) 101738. Declaration on Generative AI During the preparation of this work, the author(s) used Grammarly in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.