This paper outlines structures of different automatic speech recognition systems, hybrid and end-to-end, pros and cons for each of them, including the comparison of training data and computational resources requirements. Three main approaches to speech recognition are considered: hybrid Hidden Markov Model - Deep Neural Network, end-to-end Connectionist Temporal Classification and Sequence-to-Sequence. The Listen, Attend, and Spell approach is chosen as an example for the Sequence-to-Sequence model. Automatic speech recognition, ASR, hidden Markov model, HMM, deep neural network, DNN, LAS, hybrid, end-to-end, sequence-to-sequence, speech recognition, speech-to-text.
The high-level idea behind the speech recognition is to convert captured audio signals into relevant textual representation (Fig. 1).
2021 Copyright for this paper by its authors. how a particular word is spoken. We would find it is written out as the sequence of phonemes, which are basic units of sound (sequence of tokens). A pronunciation model converts the sequence of text into a sequence of pronunciation tokens. After that, it is fed into an acoustic model, which produces a result of how a given token sounds like.
In this pipeline, each component has its statistical model. The final result of all the models working together is to infer the most probable textual sequence Yʹ = {y1, y2, …, yL} with given data (audio ′ = arg max ( | ) ( ). features) X = {x1, x2, …, xT} many separated components.
However, the introduction of end-to-end models changed the architecture so there is no need for In our work, we provide an overview of ASR systems that are based on hybrid HMM-DNN architecture and end-to-end models, paying attention to the achieved results in speech recognition accuracy. Specifically, we will analyze the paradigm of the Connectionist Temporal Classification (CTC) model, Listen, Attend, and Spell (LAS) model that is a sequence-to-sequence model, and the most recent architectures of sequence-to-sequence online models. We outline their advantages and drawbacks. Unfortunately, due to the high variety of training approaches, model architectures, languages involved, and training and test data diversity, there is no possibility to gather the objective performance results comparison even from a big number of papers published in this area. Therefore, we will provide examples of the highest currently published results, to the best knowledge of the authors, of different approaches and types of ASR systems that we observe in this work. Finally, we give the conclusion and ideas of future works.
Despite that fact of great results of hybrid systems and less demand for training data and resources still, we have to admit that future works are mostly focused on online end-to-end systems that will penetrate most of the spheres of ASR application in the future: IoT, voice assistants, human-machine communication, etc. Therefore, we have to pay separate attention to those and explore it in detail in our future work. (1)
Going through several papers devoted to online end-to-end systems we notice that authors focus on
solving problems of extensive data requirements, the Word Error Rate (WER) for languages without a
high-quality pronunciation lexicon as well as some other narrow tactics for better results [
One of the interesting ideas has been proposed by a team of researchers from Facebook and
Microsoft. In the work [
What we have found even more interesting is a direction of research to reduce the requirements for
training data in online end-to-end systems. The authors of [
HMM suits the modeling of time-varying spectral vector sequences as a very efficient framework
[
In 2011, Microsoft Research presented a hybrid system (CD-DNN-HMM) where the HMM was
combined with a context-based DNN [
Generally speaking, HMM-based speech recognition systems contain three parts, as described in Section 1: acoustic, pronunciation, language models. Each of these parts can be built on HMM in combination with neural networks. And having its statistical model each component provides its resulting hypotheses about their results. So altogether it gives the following statement:
arg max ∗ ( | ) ≈ arg max ∗ ∑ ( | ) ( | ) ( )
(2)
where S is HMM state sequence S = {st ∈ {1, 2, …, I} | t = {1, 2, …, T}}, and p(X|S), p(S|Y), and p(Y)
reflect acoustic model, pronunciation model, and language model [
Despite the high performance of end-to-end systems, hybrid HMM-DNN and HMM-(B)LSTM
(Bidirectional Long Short-Term Memory) systems are dominant in a lot of production environments.
Real-life problems have their demands, for example, in the vast majority of cases there is extensively
more text data than audio or the real task requires several separate language models, so in these
situations HMM-DNN/(B)LSTM is a logical choice [
An interesting comparison of end-to-end vs hybrid HMM-DNN was performed by [
HMM-DNN and end-to-end systems following architectures were chosen. The hybrid
DNN/HMM and attention-based systems both had BLSTMs for acoustic modeling/encoding. Also,
LSTM and Transformer-based architectures were used for language models. The end-to-end system
had an attention-based encoder-decoder design. The training set is LibriSpeech. The best WER achieved
for both systems were: 8.4% for end-to-end system and 4.5% for hybrid HMM-BLSTM
with
Transformer-based language model. The described in [
All of the observed above gives us ground to conclude that hybrid systems possess the following limits: Multi-module architecture makes it complex in design, training, and optimization. As each of the hybrid system’s components has its statistical model and own neural network, it produces independent errors that are not coherent among all of them so it makes it an even more complicated task to achieve great results. And the most influencing drawback is that application of neural networks is limited to the function of HMM states observation that gives us an idea of the system to face the ceiling in the development.
However, as was stated above, there still valuable qualities of the hybrid HMM-DNN/(B)LSTM systems: State-of-the-art results that outperform end-to-end systems in a variety of tasks. Much better results on a much lower volume of training data. Possibility to solve efficiently several real-life tasks with lower training and computational resources involved.
All the limitations of the above-mentioned hybrid multi-module systems became an inspiration for researchers to create a process when the entire model is trained as one single big model, which later got a classification as an end-to-end model. What it does is just digests data, or features X = {x1, x2, …, xT} and produces resulting sequence Y = {y1, y2, …, yL} with just one powerful probabilistic model Y = p(Y|X).
The first such model is called CTC and was introduced by [
The speech recognition with a CTC-based model goes through two important processes: path probability calculation and path aggregation. Path probability calculation goes as follows. The spectrogram (features X) is fed into a bidirectional Recurrent Neural Network (RNN). The vocabulary for CTC is the labels, it can be letters {a, b, c, …, z} and an extra token <b> called a ‘blank token’. Each frame of the prediction is a log probability for a different token class the according to time step. It is called a score s. And the complete equation is:
( , ) = log ( , | ) (3) where softmax at step t gives a score s(k,t), and it is a log probability of category k, at a time step t, given the data X. Several results of softmax function are produced by the RNN over the entire step. The result of the model work should be the probability of the transcript through these individual probabilities over time. So, the system can take a path through the entire space of softmax function results, and look at just the symbols that correspond to each of the time steps. Then aggregation comes into place. From the path probability calculation process, it can be figured out that the length of the output path is equal to the length of the input speech sequence, which does not match with the real data. In the majority of cases, the length of transcription is shorter than the length of the input speech sequence. A many-toone, long-to-short mapping is a necessity to aggregate multiple paths into a shorter label sequence. 5.2.
Although the CTC model is great, from a modeling perspective, you would find that the model makes predictions just based on the current data. And once it’s done with making those predictions for each frame, there's no way of adjusting that prediction. It has to the best it can with those predictions.
An alternative end-to-end architecture that does not need intermediate steps is the
sequence-tosequence model [
For instance, LAS is one of the sequence-to-sequence implementations for audio sequence. LAS produces multiple outputs with probabilities for input sequence (multimodal outputs). And this is why this one model can learn such complex functions, because the more mistaken outputs it produces, the more feedback it has. Moreover, the model can learn very specific train dataset patterns, which makes this model a good candidate for domain fine-tuning. Another strong part of LAS is causality, which means the model can predict for example digit instead of the word representation. LAS still can benefit from external Language model—it is no substitute for billions of words texts for language model, so it is still a better idea to have two datasets, one for the ASR model, and another one to train language model to use as an additional layer on top of the ASR model.
There are few limitations in sequence-to-sequence models for speech: As it is an encoder-decoder-based model, it is not an online model, which means it is the next token prediction and we have to have the full sequence to produce output, we cannot give just chunk by chunk like in Hybrid models. As a result of the previous, we cannot generate accurate start and end of words using the sequence-to-sequence model. Attention mechanisms is a computational bottleneck for audio sequence. Accuracy is much lower for short sequences, which makes current architecture hard to implement for speech conversational systems, where customers can utterance just “yes” as an answer to system questions.
In this paper, we reviewed the most prominent and actual speech recognition approaches and
architectures [
The current main limitations of end-to-end architectures are the amount of data and computational resources to achieve comparatively to HMM-DNN accuracy. It just requires a few times more input datasets and more than 10 times more computational resources. At the same time, we believe in the future end-to-end architectures will become more efficient and advanced which leads to broader production implementation of end-to-end speech recognition approaches. Such an approach significantly decreases required skills and will most probably make speech recognition more affordable do develop.
In the future, we will focus on the direct comparison of three architectures in terms of accuracy using the same input datasets and will continue to follow end-to-end improvements to catch the moment when they will be ready to provide comparative accuracy while providing comparative effectiveness.
The research team is grateful to Ender Turing OÜ for defining the business problem, comments, corrections, inspiration, and computational resources. 8. References