This paper presents work in progress for the creation of a Large Vocabulary Automatic Speech Recogniser for Italian using NVIDIA NeMo. Thanks to this package, we were able to build a reliable recogniser for adults' speech by fine tuning the English model provided by NVIDIA and rescoring it with powerful neural language models, obtaining very good performances. The lack of a standard, reliable and publicy available baseline for Italian motivated this work.
The advent of the “Deep Learning Revolution” introduced astonishing changes also in the field of speech processing allowing for the development of brand new tools and devices able to recognise and synthesise speech exhibiting performances never seen before. It is sufficient to think to the new virtual assistants that populates our houses and mobile phones for getting an immediate idea about the improvements in this research field.
Most big IT companies developed, in the past 3/4 years, solutions well integrated with various devices that include high performance tools for speech processing. However, these solutions very often are not released freely, sometimes they require registrations and fees and, in the best situations, codes are free, but the models for a specific language are not available. A notable exception regards NVIDIA NeMo1, a conversational AI toolkit built for researchers working on Automatic Speech Recognition (ASR), Natural Language Processing (NLP), and Text-To-Speech synthesis (TTS). The primary objective of NeMo is
Copyright © 2021 for this paper by its author. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
1Other exceptions providing also multilingual models including Italian are Facebook Wav2Vec and SpeechBrain. to help researchers from industry and academia to reuse prior work, namely code and pretrained models for various languages, and make it easier to create new conversational AI models, maybe adapting tools and models to specific languages or particular domains.
This paper reports an attempt to build a high performance Large Vocabulary ASR system for Italian adults’ speech by exploiting all the features available in NeMo and most of the largest Italian spoken corpora available to the community.
Section 2 describes the various speech datasets used for developing the model, followed by Section 3 that describes the state of the art; in Section 4 we will describe the NeMo ASR model used in the experiments and Section 5 will discuss the experiments and the obtained results. Section 6 draws some provisional conclusions about our work.
This section describes the datasets we used for the creation of the Italian ASR model. We have to say that, of course, these are not the only spoken corpora available, but they are the biggest corpora commonly used for setting up an ASR system for Italian. They are typically very big, already organised and structured exactly for training ASR systems or specifically designed to maximise their impact and usefulness for ASR. We have also to say that, as far as we know, this is the first attempt to use all of them for ASR training in a single project. 2.1
Common Voice
With regard to the Italian subcorpus, they currently2 released version 7.0 (MCV7), containing 6,407 speakers for a total of 160,570 utterances with the correct transcriptions. In the standard splitting provided with the dataset the training set contains 131,041 utterances corresponding to 189.50 hours of speech, the validation set 14,764 utterances for 24.41 hours and the test set 14,765 utterances corresponding to 25.74 hours.
These splitting are very important for our experiments, as discussed in Section 5. 2.2
Multilingual LibriSpeech3 (MLS) dataset
VoxForge4 is an open speech dataset that was set up to collect transcribed speech for use with Free and Open Source Speech Recognition Engines. The Italian portion of VoxForge contains 10,633 utterances totalling 20.16 hours of transcribed speech. 2.4
APASCI
In order to properly describe the state of the art, we should first define the typical metrics used for evaluating ASR systems. Given the system transcription for an utterance and the correct transcription extracted from the gold standard, the most important metric is certainly the Word Error Rate (WER) defined as W ER = (InsertioGnosl+dNSuubmstbietrutoifonWs o+rdDseletions) , typically expressed in percentage. It compares the two transcriptions counting all the differences at word level using the edit distance between them. We can also define the Phone Error Rate (PER) and the Character Error Rate (CER) that use the same principle but applied, respectively, at phone or character level.
Examining the literature for the construction of
ASR models for Italian we immediately recognise
a lack of works devoted to the building of a general
Large Vocabulary ASR for adults’ speech. The
only work we found on that was
An exception to what we said before is
represented by the wo
More recent studies employing neural
models were able to build other quite reliable
systems. Weibin (2019) trained a system based on
DeepSpeech
Traditional speech recognition takes a generative approach, modelling the recognition process of speech sounds acoustics (O) as W = argmaxW P (O|W )P (W ) where W is a possible transcription as sequence of words. The actors of the game include a language model P (W ) that allows to estimate the most likely orderings of words in a given language (e.g. an n-gram model), a pronunciation model for each word in that sequence (e.g. a lexicon of phonetically transcribed words) and an acoustic model P (O|W ) that allows to estimate the probability of an input sequence of acoustic observations given each possible words sequence W . When we receive some spoken input, our goal would be to find the most likely sequence of text that maximises the words probability given a speech-acoustic input.
Over time, neural nets advanced to the point where each component of the traditional speech recognition model could be replaced by a neural model that had better performance and that had a greater potential for generalisation. For example, we could replace an n-gram model with a neural language model, and replace a pronunciation table with a neural pronunciation model, and so on. However, each of these neural models need to be trained individually on different tasks, and errors in any model in the pipeline could throw off the 5http://www.clips.unina.it 6https://librivox.org/ 7https://github.com/flashlight/wav2le tter whole prediction.
Nowadays, end-to-end ASR discriminative architectures models that simply take a sequence of audio inputs and give a sequence of textual outputs, and in which all components of the architecture are trained jointly towards the same goal, largely dominate the field. The model’s encoder would be akin to an acoustic model for extracting speech features, which can then be directly piped to a decoder which directly outputs text, as a sequence of characters, in a given language. If desired, we could still integrate a language model that would improve our predictions, piping it after the decoder8.
Grasping information from NeMo github site9,
we learn that the base ASR model provided by
NVIDIA is Jasper (”Just Another Spee
Most state-of-the-art ASR models are extremely large; they tend to have on the order of a few hundred million parameters. This makes them hard to deploy on a large scale given current limitations of devices on the edge. Another model is included into NeMo, QuartzNet (Kriman et al., 2020), a version of Jasper with separable convolutions and larger filters. It can achieve performance similar to Jasper but with an order of magnitude fewer parameters. Similarly to Jasper, the QuartzNet family of models are denoted as “QuartzNet [BxR]”, where B is the number of blocks and R is the number of convolutional sub-blocks within a block, and do not use the computationally costly recurrent layers in favour of more efficient convolutional layers. Each sub-block contains a 1-D separable convolution, batch normalisation, ReLU, and dropout (see Figure 1 for a complete diagram describing the QuartzNet internal structure). Both models described before optimise the Connectionist Temporal Classification (CTC) loss.
NVIDIA provided also a large number of pre8Partially taken from, https://docs.nvidia.co m/deeplearning/nemo/user-guide/docs/en/m ain/asr/intro.html 9https://github.com/NVIDIA/NeMo trained models10 for various languages. The two models for English “STT en Quartznet15x5” and Italian “STT it Quartznet15x5” (both at version 1.0.0rc1 published the 30th June 2021) are relevant for our work. The Quartznet 15x5 model family consists of 79 layers and has a total of 18.9 million parameters, with vfie blocks that repeat fifteen times plus four additional convolutional layers.
QuartzNet15x5 Encoder and Decoder English neural module’s checkpoints from NVIDIA were trained using Multilingual LibriSpeech and Mozilla’s English Common Voice 6.1 ”validated” set (a huge amount of data containing more than 3,300 hours of speech) with two types of data augmentation techniques: speed perturbation and Cutout. Speed perturbation means that additional training samples were created by slowing down or speeding up the original audio data by 10%. Cutout refers to randomly masking out small rectangles out of the spectrogram input as a regularization technique. NVIDIA’s Apex/Amp O1 optimization level was used for training achieveing 4.19% WER on LibriSpeech test-clean.
NeMo documentation also describes a procedure for fine-tuning the English model to adapt it to other languages, keeping the acoustic encoder 10https://ngc.nvidia.com/catalog/colle ctions/nvidia:nemo asr frozen and fine-tuning the decoder for producing transcriptions for a different language (Huang et al., 2020). In the cited paper they also get the relevant conclusion that it is much better, in terms of performance, to fine-tune the English model than to retrain from scratch a new model for a specific language. The Italian model provided by NVIDIA has been produced following the suggested procedure, in particular by retraining the QuartzNet decoder using the training portion of MCV version 6.1. We will consider this Italian model as a baseline for our experiments. 5
The STT it QuartzNet model provided by NVIDIA was trained using a reduced set of data and applying an output dictionary that includes some characters that do not belong to the Italian alphabet. For these reasons we preferred to restart the fine-tuning process directly from the original STT en Quartznet15x5 English model.
The training set we used to fine tune the NVIDIA STT en model to Italian is composed by joining the training portion of MCV7 and all files from MLS, VoxForge and APASCI, and contains 186,778 utterances/speech files totalling 372.62 hours of transcribed speech. 19,199 utterance/files were filtered out from the training set totalling 97.77 hours of removed speech. This is due to the fact that in some dataset, mainly in MLS and VoxForge, there were some utterances longer than 16.7 seconds, a time limit hard coded into NeMo in order to keep the model computationally tractable. We checked also that transcriptions contain only the 34 standard characters from the Italian alphabet (26 lowercase letters plus six accented characters, the apostrophe and the space) as it is a standard practice in ASR to lowercase transcriptions and to remove any punctuation mark not strictly useful or relevant to help the recognition.
With regard to decoding and rescoring, NeMo
offers various possibilities:
• Greedy Decoding. This method simply
computes the most likely sequence of characters,
also called as the “best-path decoder”, given the
audio input.
• Beam Search Decoding. Beam Search
Decoding (BSD) is another way of decoding model
prediction that leads to better results than the
greedy search. BSD, instead of choosing
always the best prediction at each step,
considers the top-K hypothesis having the
highest probabilities, where K is the so called
beam size. For all the subsequent experiments
we used beam size=1024, beam alpha=1.0 and
beam beta=0.5 (see NeMo documentation).
Language Models (LM) have shown to help the
accuracy of ASR models when combined to
BSD. NeMo currently supports the following
two approaches to incorporate language models
into the ASR models through BSD:
– N-gram Rescoring. In this approach, an
Ngram Language Model is trained on text data,
then it is used in fusion with beam search
decoding to find the best candidates. The beam
search decoders in NeMo support language
models trained with the KenLM library
Given such possibilities, we fine tuned the STT en model on a single V100 GPU using our joined dataset described above and the MCV7 validation and test set respectively for early stopping the training process and to evaluate all models. The hyperparameters we modified w.r.t. the original English model, and contained in the model itself, are listed in Table 1.
As notable exception to the NVIDIA suggested procedure for fine tuning a model, we have to re11Corresponding to the 2021 brand new update.
Par. train ds.batch size validation ds.batch size optim.lr optim.betas optim.weight decay optim.warmup steps optim.sched.min lr trainer.precision trainer.amp level port that we obtained the best results by unfreezing the encoder and letting it to slightly adapt the extracted speech features to the new language, namely Italian, that certainly share most of the sounds with the starting English model STT en, but contains also specific sounds (e.g. [ñ] and [L]) that may require small adaptations.
Table 2 outlines our results after a complete fine tuning of the end-to-end ASR model using the Italian dataset described before and applying different decoding and rescoring schemas. The improvement obtained with the fine-tuning process, when compared to the original model delivered by NVIDIA is relevant, but not so big, while when applying the BSD with the two rescoring algorithms the WER metric improve of 40% w.r.t. the greedy decoding schema.
Baseline (NVIDIA STT it) Greedy Decoding 15.64/4.00 BSD & 3-gram Resc. 10.79/3.18 BSD & 6-gram Resc. 10.77/3.17 BSD & Neural Resc. 9.54/ NVIDIA STT en + Our Retraining Greedy Decoding 14.86/3.78 BSD & 3-gram Resc. 10.41/2.97 BSD & 6-gram Resc. 10.36/2.95 BSD & Neural Resc. 9.04/ Test 16.90/4.46 11.59/3.54 11.57/3.53 10.51/ 15.82/4.14 10.96/3.27 10.94/3.26 9.67/
This paper presented work in progress for the construction of a reliable and performing ASR system for Italian adults’ speech. Thanks to the NVIDIA NeMo package, we were able to produce a very strong baseline reaching a WER = 9.67% over the MCV7 test set.
This is only the beginning of our work, as any change in the kind of speech used to train the system could degrade the whole performance, but, having used a collection of four different datasets containing thousands of different speakers and speech utterances for setting up such ASR system, we believe that the result should be robust enough. Unfortunately, the lack of a standardised benchmark for Italian does not allow for a quantitative and objective evaluation of this statement.
End-to-end character ASR model, and its improvement on WER, is only part of the game: the work on decoding and rescoring procedures produced much more improvements. Thus, the most important “take home lesson” is certainly to focus on the development of high performance LM specifically tuned for ASR.
All the models presented in this paper as well as the scripts and additional codes for using NeMo and generating the results will be made available12.
We acknowledge the CINECA13 award no. HP10C7XVUO (project QT4CLML) under the ISCRA initiative, for the availability of HPC resources and support.