This paper is focused on Natural Language Processing (NLP) and speech area, describes the most prominent approaches and techniques, provides requirements to datasets for text and speech model training, compares major toolkits and techniques, and describes trends for NLP and speech domain. Significant advances in Deep Learning (DL) of the last decade uncover new possibilities and demands for businesses, governments, and citizens. Such advances in Natural Language Technology (NLT) allow businesses to automate most routine and boring tasks in communication with customers and direct people's minds to more exciting and creative tasks [001]. To fully leverage NLT, two central stacks of technologies should be combined: Speech technologies-to translate speech into text and vice versa. NLP-to understand, interpret, and generate information in a text. This work reviews existing knowledge, directions, and avenues for future research in this increasingly important domain/area NLT/NLP. NLP is a field of artificial intelligence that helps the computer to understand and generate text. NLP is broadly used in many tasks: dialogue systems, sentiment analysis, machine translation, information retrieval, summarization, question answering, etc. During the last decade, there were few breakthroughs in DL fields, first for image recognition and later for natural language, which attracted researchers and businesses' colossal interest. We will review the most prominent and fundamental techniques, which significantly improve machine skills in natural language: Recurrent Neural Networks (RNNs), embedding concept, the concept of decoder and encoder, and shortly attention and transformers. Without this technique, it is hard to imagine such interest in the NLP field. Automatic Speech Recognition (ASR) and speech generation are techniques to convert human speech to text and back. After a structured communication system called language evolved, speech is the main instrument of any communication between human beings. For machines, such language is digits, and it was many iterations to present the human speech to machine understandable language. Sect. 2 will review the latest and most promising techniques, like the hybrid Hidden Markov Model (HMM) and end-to-end systems, combined with Deep Neural Networks (DNNs). Additionally, we will review data requirements in Sect. 3. As with currently available frameworks, input data quality and relevance contribute a major if not overwhelming part of the resulting model quality. In Sect. 4, a
comparative analysis of approaches and frameworks is presented. Evaluation metrics to measure NLP systems and ASR presented in Sect. 5, end-to-end training approaches in Sect. 6, and trends in Sect. 7.
Below we will review the main breakthrough points in the NLP area of the last decade. We will start with RNNs as the central concept in NLP (recurrent and combining information from previous iterations), then will present more advanced techniques to do feature engineering (not just a one-hot encoding of words in the dataset, but complicated vector representation with context and additional information related to word).
As the main starting point of the NLP, the area was machine translation. It is natural that the concept of encoder-decoder and sequence-to-sequence evolved, which will be covered as well. Attention and transformer will be reviewed as the last achievements in the NLP area.
RNNs were the primary building block for NLP tasks for an extended period. The main difference of RNNs from other DL architectures is the ability to remember data for sequence, not only for the last cell (word/token).
The network takes X as the input vector (usually encoded-word representations) and produces Y as the output vector. Each RNN cell takes current input xt and previous hidden state (activation) ht – 1, which stores information extracted during previous iterations. The network learns weights (parameters) Wh, Wx, and bias ba through the weights learning process. At each iteration of forwarding propagation, non-linear activation function g such as tanh (or similar) is applied to calculate output hidden state (activation) ht = g(Wh ht–1 + Wx xt + ba). (1)
Additionally, softmax (activation function g) may be applied in the end if output predictions needed by the task yt = g(Wy ht + by). (2)
Most important for tasks in the NLP area is that output includes information from previous, not only last. It is crucial mainly because of the nature of language. One last word (token) is not enough to understand the context of the sentence. Such a type of connection is called a recurrent connection (see Fig. 1).
This idea and concept of recurrent connection and context significantly affect the current
state of the NLP area. RNNs have many disadvantages, though, like unidirectionality, the
problem with capturing mid and long-term connections/dependencies inside a sequence. Today
it is rare to find RNN as the underlying architecture. More complicated architectures arrived
based on RNN, like Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), and
some architectures come to NLP from image recognition, like Convolutional Neural Networks
(CNNs) [
The primary purpose of embedding is to represent tokens (documents, phrases, context, a
piece of a word, or a character) as a numerical vector. Then neural networks can calculate and
use the probability distributions or likelihoods to separate semantically similar categories. So
that different tokens with similar meanings will have closer vectors and different by meaning
groups of tokens can be separated in vector space. [
The encoder-decoder approach became a breakthrough and led to a significant increase in the performance of language models. The input sequence [“What’s”|“up”|“?”] at the embedding layer gets numerical representation. Then numerical representation is sequentially fed to the RNN. After all, inputs get through RNN produces output. This part is called an encoder—it encodes input sequences (see Fig. 2).
The result of the encoder transfers to the decoder. The decoder generates predictions of a resulting sequence until it gets to the end of the sentence token.
The most outstanding achievement of encoder-decoder is the possibility to use it to create end-to-end models correctly and the possibility to handle input and output sequences of different lengths. The problem of inconsistent input and output lengths is especially actual in neural machine translation.
The encoder-decoder architecture is usually based on two RNNs or LSTM. The encoder encodes all input sequences and stores all information in the encoder vector, and the Decoder creates result predictions.
The main limitation of RNNs is dependencies tracking in long sentences. Long sentences (more than 20 words) just can’t be stored effectively in the output vector of RNN. That is why researchers come up with the attention mechanism.
The idea of attention is the same as attention from the human reading process. For humans,
a few words from a sentence are enough to understand the sentence well. During the translation
process, humans need just a few main words to translate, all other words simply out of attention.
The same for attention: the decoder focuses on some particular part of the source at each step.
Decoder focuses only on particular words at each step (increased saturation represents more
attention), not the whole input sequence. The attention mechanism uncovers such possibility
to a decoder by attention weights and context vector [
The transformer is one of the last breakthroughs which accelerates NLP significantly. The transformer is architecture builds on top of the encoder-decoder concept and is hugely based on the attention concept. The main breakthrough was parallelization by replacing sequential computation (RNNs or CNNs) with an attention-based network. The main components and concepts of this architecture will be presented and described below.
The encoder consists of multiple stacked self-attention and feed-forward layers with residual connections and a positional encoder. The embedding layer is usually applied at the bottom to convert the input sequence to numerical representation. The feed-forward network does not have dependencies and thus can be parallelized. That is an essential concept behind the Transformer’s possibility to learn on a vast amount of data that LSTM and GRU can’t afford.
The decoder also consists of multiple (equal to the encoder) stacked self-attention,
feedforward layers with residual connections, and additionally encoder-decoder attention layer in
the middle. In comparison to the encoder, the decoder’s self-attention layer differs. The main
idea here is masking future positions. In the encoder, each position can attend to all positions.
Still, in the decoder, to prevent leftward information flow to preserve the auto-regressive
property, each position can attend only to early positions in the output sequence [
The main goal of speech systems is to convert input audio wave sequence into text
representation in the case of an ASR system and vice versa in the case of speech generation
from the text (text-to-speech, TTS) [
End-to-end ASR systems.
To represent the input audio sequence in format machine understand, we have to ally some transformations. Research shows that it is not good enough to convert input waves into digits of corresponding amplitudes by sampling audio signals. Such features are very uninformative for the training process to squeeze as much information as possible to remember and generalize the audio signal.
The spectrogram was obtained during Fast Fourier Transformation (FFT) to represent time
(or similar non-linear), frequency, and energy at each point of time. Trans-formation represents
features in acoustic frames format (20–40 ms). Mel-Frequency Cepstral Coefficients (MFCCs)
or perceptual linear prediction is a common choice of non-linear transformation techniques for
feature extraction for ASR data [
(b)
To reconstruct the utterance that has just been said by putting the correct phonemes after HMM was used. It is done by using statistical probabilities that one phoneme follows the other.
In simplified words, HMM consists of three different layers: 1. The heart of the HMM model is an acoustic model that checks on the acoustic level the probability that the phoneme it recognizes is that phoneme.
2. After that lexicon (pronunciation) model is applied, checking the probability that recognized phonemes next to each other can stand next to each other.
3. In the end, the language model applied (usually in the way of n-grams) check on the word
level and whether words standing next to each other make sense. As an example here, the
model will choose “cat paws” instead of “cat pause” [
As can be seen from above—one of the significant limitations of HMM models is phoneme to grapheme mapping. Especially this problem arrived for a low-resource language where no one tried to prepare such a model. It might be very time-consuming to the prepared dataset for such mapping. That is one reason that simplified end-to-end models evolved. The main inspiration was to train the model with as few labeling and intermediate steps as possible. The model should learn by themselves map phonemes to grapheme directly or indirectly using the same input data used for current training. Another motivation is to jump into an area of unsupervised training, to exploit the vast amount of unlabeled audio data stored on the Internet.
There are few varieties of end-to-end ASR systems architectures. At the same time, all of them are built on two types: Connectionist Temporal Classification (CTC) and sequence-tosequence (encoder-decoder-based).
Before CTC main limitation of the end-to-end ASR system was that the model needed to have the whole sentence to start the translation. It means no streaming decoding possibility. CTC map input sequence X (MFCC) to output sequence Y (letters).
One of the CTC breakthroughs is that kind of local attention introduced, which splits the
continuous speech, and then the current modeling unit using attention runs on each split
segment. By doing this, the whole utterance is split into small segments, and local attention is
used to predict features (letter) [
Standard TTS systems consist of a few parts:
RNN modifications (LSTM, GRU, etc.) for recurrent sequence-to-sequence feature
prediction network that maps character embedding to MFCC spectrograms (description of
these components is familiar very similar to above-described components).
Vocoder system that synthesizes waveforms from those spectrograms. The relationship
between linguistic features and vocoder parameters that represent the vocal cord and vocal
tract characteristics learned by acoustic models. Vocoder parameters are generated (in the
synthesis stage) from the trained acoustic models, and a speech waveform is synthesized
using high-quality vocoder systems [
Data preparation is a significant step in any DL area, and NLP is not an exclusion. As DL techniques are remembering and generalization of training datasets, it is hard to overestimate the impact of a good and bad quality dataset. As in any other DL task, data is presented in features and relevant label form. 3.1.
Requirements for input data are the same as for other tasks in the DL area: input data should be as much as possible closer to the domain model will work in. In simple words, you can’t train a model for medical anamneses prediction using a financial data dataset. If the training process model didn’t see examples of token/word in the dataset, it will just not react to it. Over the past years, huge progress has been made to overcome such limitations, and embedding techniques with pre-trained token embedding helps a lot. Still, it is hard to overemphasize how better the trained model would be if you use relevant training data.
The NLP area has some specific requirements: data should be split by sentences, which is a
big problem for ASR. For such specific NLP tasks as punctuation, the typical case is to generate
a synthetic dataset and label it in the most suitable task way [
Nowadays, there are many labeled and much more unlabeled datasets, which is a good starting point for significant tasks, so software engineers do not need to collect and label datasets on their own. 3.2.
Datasets for ASR are some audio files (usually, 3–20) and related text transcripts (labels). All digits should be denormalized to a text representation. For the model to get used to acoustic, were four and format sound almost the same, and if in training dataset it will be “4” and “format,” it will be much harder for the model to generalize acoustically. Such a denormalization task can be very complicated. Let’s imagine number “3,” which may represent “three,” “third,” etc. And such a task is even more complicated for non-English languages.
The above gives us some information on how hard to prepare a good dataset for ASR, especially for low resource languages. There are few directions to overcome such limitations, for example, (1) use non-labeled audio data (unsupervised learning), which arise new limitations of computational resources; (2) to generate such dataset iteratively using smaller to generate bigger where we have generated 2,500 hours dataset using just 100 hours as a starting point. As an example of language to test such automated ASR dataset generation pipeline, we took low resource Ukrainian language, which is very limited in terms of available ASR datasets (see Table 1).
One of the main difficulties in data preparation for ASR is that many audios are stored in mono format, with mixed channels and hence mixed speakers. There are a few techniques to separate such audios by using voice activity detection and several speaker identifications.
ASR systems are pretty demanding for data. You need approximately the following amount of data to train the good model: Five thousand hours for a hybrid approach. Ten thousand hours for an end-to-end approach.
Thirty thousand hours for an unsupervised learning approach.
There are a few main divisions of approaches used to solve NLP tasks. Main are either to use pre-trained models like (BERT [46], RoBERTa [47]) or to train a model from scratch based on Bidirectional RNN (BRNN), LSTM, CNN architecture (see Fig. 5).
Pre-trained models can be divided into encoder-based and decoder-based. We will not review non Transformer based pre-trained models as with distilled pre-trained Transformer based models, you can achieve almost the same model efficiency, like BRNN but with significantly higher accuracy.
If the task or domain is specific and you can’t exploit the pre-trained model, it is good to train the BRNN, LSTM model from scratch (see Fig. 6).
A comparative analysis of a conventional Speech Processing Toolkit (SPT) and tools to build an ASR model is presented in Table 2. We compared in a few dimensions, and the most important ones are demand in data amount to train or fine-tune your model and learning curve to start working with the toolkit.
Training type
Data amount Supervised Supervised Supervised Supervised Unsupervised
Medium-low Medium-low High-medium High-medium Medium-low
Learning curve Hard Hard Easy Easy Normal
As can be seen from comparative analysis, no one toolkit can handle all steps of ASR datapreprocessing (including collection, splitting, labeling, preparing for ASR expected format), and model training. That is why we believe more teams contribute to creating more frameworks and toolkits to lower the learning curve and demand in training data.
The results of speech recognition make it possible to automatically track the illegal activity of users of the information system using keywords [002].
Because the NLP area is quite broad and the number of tasks in NLP is vast—there are no standard metrics for all tasks. We can split metrics by clustering tasks and highlight next: Machine translation models: bilingual evaluation understudy is a performance metric to measure the performance of machine translation models. It evaluates how well a model translates from one language to another. Language understanding evaluation: general language understanding evaluation is a benchmark based on different types of tasks rather than evaluating a single task. The three major categories of tasks are single-sentence tasks, similarity and paraphrase tasks, and inference tasks.
At the same time, there are many fine-tuning tasks, like part of sentence tagging, named entity recognition, etc. In such tasks, the most common metric is to count accuracy through precision where Ntp is a true positive, Nfp is a false positive.
And recall the equality calculates the coefficient where Nfn is a false negative.
F1 Score calculates from (1) and (2) [62]:
P = Ntp / (Ntp + Nfp)
R = Ntp / (Ntp + Nfn) F1 = 2 · P · R / (P + R). (3) (4) (5) 5.2.
It is much easier to come up with a unified metric for ASR than for NLP tasks, as we only have to measure if the word is recognized correctly or not. Hence, Word Error Rate (WER) is the most common to see an ASR accuracy metric. The lower WER is better than the ASR system. WER can then be computed as:
WER = (NS + ND + NI) / (Ns + ND + NC) (6) where NS is the number of substitutions, ND is the number of deletions, NI is the number of insertions, NC is the number of correct words [63].
It is worth mentioning that WER is very sensitive to domain and acoustic. For example, low (good) WER of 5% for literature (trained on books) domain can have 20–30% WER for call center calls [64].
With the mentioned breakthrough in NLP and speech areas, more and more businesses see great opportunities for the implementation of NLT based systems. That drives demand for NLP / speech engineers. Such demand can’t be satisfied with the existing supply, so the learning curve should be lowered. And one of the significant steps to such supply increase is end-to-end systems, which is much simpler from the end-user (engineer perspective). We see a huge trend and opportunity in end-to-end training approaches. Yes, it still requires more data, is less effective, etc., but we believe it is the future of NLP and speech systems.
Data labeling is the most comprehensive, long, and expensive process—last year’s trends are to use unlabeled data. It is much easier to collect data than to collect labeled data specifically for your domain or low-resource language. However, even unlabeled data have to comply with requirements. Today’s unsupervised learning approaches for NLP became standard, and with approaches for distillation and pruning on top of base training, they become effective and practical. We are still looking forward to more effective unsupervised learning in the speech area that will require not 50,000 hours of speech and enormous resources to train the model but more practical.
To teach the model to generalize using unlabeled data demands a much more significant amount of data and is thus very demanding in computational resources (50,000 hours of unlabeled data to train a state-of-the-art ASR model). That is why the third trend is the pre-trained models. For NLP, it is common to train a model on a significant amount of data once and then fine-tune for underlying tasks. As an example mentioned before—using a pre-trained BERT model to mark punctuation after ASR for NLP processing. Not to mention how huge a lower learning curve and time investments are for an engineer to prepare a production-ready model. You do not need to collect vast datasets only to train, for example, tokenizer. For speech, pre-training is a common cause for hybrid models and still have to be evolved for end-to-end approaches (especially for unsupervised learning), as it is clear that researcher or software engineer will not have the possibility to spend hundreds of thousands of USD to train the model using unsupervised data.
As humans learn many languages to understand other humans (even within one country), we expect to see more advanced multilingual models that can accurately understand multilingual dialogues.
We believe all these trends are possible because great-specialized frameworks for NLP and speech
appeared, as a few examples: HuggingFace [65] for NLP, Kaldi [56], ESPnet [59], FairSeq [61] for
speech-to-text recognition, Tacotron [