Automatic Speech Recognition (ASR) has made significant progress, nearing human-level accuracy. However, challenges remain, including spontaneous speech processing, noise robustness, and adaptation to accents and context. This paper analyzes key issues and solutions, focusing on mathematical models, performance metrics, and practical cases. ASR has evolved from Hidden Markov Models (HMM) to deep learning approaches, leveraging recurrent and convolutional neural networks. Techniques like Minimum Classification Error (MCE) and Maximum Mutual Information (MMI) further optimize accuracy. Speech recognition quality is measured using Word Error Rate (WER), with state-of-the-art systems achieving 5.8-6.8%, compared to 5.1% for human transcription. Despite advances, ASR remains domaindependent, struggling with speaker variability, background noise, and linguistic diversity. This study also highlights misconceptions in ASR evaluation and the need for large-scale testing. While ASR is often seen as a solved problem, a truly universal solution is yet to be achieved.
Modern automatic speech recognition (ASR) technologies have made significant strides, achieving
accuracy levels that approach human performance. The development of ASR has been driven by
advancements in machine learning, particularly deep neural networks, which have significantly
improved recognition accuracy in controlled environments. However, ASR systems still face
fundamental challenges when dealing with spontaneous speech, background noise, diverse accents,
and context-dependent variations [
One of the primary difficulties in ASR is its dependence on domain-specificdata. Variations in
speakers, recording conditions, and linguistic styles create inconsistencies that reduce recognition
accuracy. Additionally, ASR systems struggle with spontaneous speech, where hesitations, filler
words, and non- standard grammatical structures are common [
This paper explores key challenges in ASR development, analyzes mathematical models and
techniques used to improve recognition accuracy, and examines practical examples to highlight
existing limitations. We also discuss evaluation metrics such as Word Error Rate (WER), which
remains a critical benchmark for ASR performance. While ASR is often perceived as a solved problem
in controlled environments, this study emphasizes that achieving a truly universal solution remains
an open challenge [
Automatic Speech Recognition (ASR) has been a subject of extensive research for decades, evolving
from early rule-based systems to modern deep learning approaches. This section reviews key
developments in ASR technology, including traditional statistical models, deep neural networks, and
optimization techniques aimed at improving recognition accuracy [
Initial ASR systems relied on statistical models, particularly Hidden Markov Models (HMM) combined with Gaussian Mixture Models (GMM) (Rabiner, 1989). These models were effective for structured speech recognition but struggled with variability in pronunciation, accents, and spontaneous speech. Over time, discriminative training techniques, such as Maximum Mutual Information (MMI) and
Minimum Classification Error(MCE), were introduced to improve accuracy (Woodland & Povey,
2002) [
The emergence of deep learning significantly transformed ASR capabilities. Recurrent Neural
Networks (RNNs) and Long Short-Term Memory (LSTM) networks (Graves et al., 2013) improved
sequential data processing, enabling better recognition of speech patterns. Later, Convolutional
Neural Networks (CNNs) were applied to spectrogram-based speech recognition (Abdel-Hamid et al.,
2014), enhancing feature extraction [
The introduction of Transformer-based architectures, such as wav2vec 2.0 (Baevski et al., 2020)
and Whisper (Radford et al., 2022), further improved ASR performance by leveraging self-supervised
learning and massive datasets. These models demonstrated superior generalization across languages,
accents, and noisy environments [
Despite advancements, ASR systems still face challenges in real-world applications. Studies
highlight difficulties with noisy environments (Kim et al., 2017), accent adaptation (Sun et al., 2018),
and domain-specific speech recognition (Li et al., 2020). Researchers continue exploring novel
approaches, including Bayesian optimization for error reduction (Sak et al., 2015) and hybrid ASR
models combining statistical and neural network-based methods (Hinton et al., 2012) [
This review highlights that while ASR has reached near-human accuracy in controlled settings,
challenges persist in spontaneous speech processing, noise robustness, and adaptation to diverse
linguistic contexts. Future research aims to develop more robust, context-aware ASR systems capable
of real-time adaptation to varying speech conditions [
Automatic Speech Recognition (ASR) is a technology that converts spoken language into text. It relies
on complex mathematical models and machine learning algorithms to process and interpret audio
signals [
Traditional ASR systems use Hidden Markov Models (HMM), while modern approaches incorporate deep neural networks (DNNs) for higher accuracy. 5. Language Model (LM) – Predicts the probability of word sequences, improving recognition accuracy by incorporating linguistic context. Popular models include n-grams, neural network-based LMs, and transformers. 6. Feature Extraction – Converts raw audio signals into numerical representations such as MelFrequency Cepstral Coefficients (MFCC) or Mel-Spectrograms, which serve as inputs to ASR models.
Decoding and Post-Processing – Utilizes search algorithms like the Viterbi decoder to find the most probable word sequence based on the acoustic and language models. Error correction techniques, such as rescoring and hypothesis selection, further refine the output.
These fundamental concepts form the basis of modern ASR systems, enabling applications in
virtual assistants, transcription services, and voice-controlled interfaces [
Key Methods of Automatic Speech Recognition. The evolution of Automatic Speech Recognition (ASR) includes several key stages: Statistical Models: Early systems were based on dynamic programming methods and Hidden
Markov Models (HMM), enabling recognition of limited vocabularies. Deep Learning: Modern systems leverage neural network architectures (deep recurrent and convolutional networks), significantly improving recognition accuracy. Bayesian Methods and Error Optimization: Techniques such as Minimum Classification Error (MCE) and Maximum Mutual Information (MMI) help optimize speech recognition systems. WER is calculated using the formula:
WER= 100∗( Insertions + Substitutions + Deletions)
Totalword (1) where S = number of substitutions, D = number of deletions, I = number of insertions, N = total number of words in the reference transcription.
Example: Original text: “Hello, how are you?” Recognized text: “Hello how you are?” Errors: 1 insertion (“you”) Challenges in Speech Recognition
There is a common belief that speech recognition is a solved problem, but this is not entirely true. While the problem is resolved for specific scenarios, a universal solution does not yet exist due to several challenges:
Domain Dependence Different speakers “Acoustic” recording channel: codecs, distortions Various environments: phone noise, city noise, background speakers Different speech pace and preparedness Different speech styles and topics Large and "inconvenient" datasets
Quality metrics that are not always intuitive
Speech recognition quality is measured using the Word Error Rate (WER) metric.
Insertions – words that were added but do not exist in the original audio recording. Substitutions – words that were incorrectly replaced with other words.
Deletions – words that were not recognized, resulting in omissions.
Example Calculation fOriginal sentence: “The stationary (unintelligible speech) phone rang late at night.” Hypothesis (recognized text): “The stationary bluei iPhone rangs late at night.”
WER= =60 %∗(The error rate is 60 % .) (2)
Diacritical marks (e.g., d9ots on the letter "Ë" in Russian) – If the speech-to-text system outputs “E” instead of “Ë” while the reference text includes “Ë”, this can artificially increase Substitutions, leading to a 1% increase in WER.
Different spellings of the same word – For example, “hello” vs. “hallo” vs. “hallé.” Uppercase vs. lowercase letters.
Averaging WER across separate texts instead of calculating it globally – If one text has WER = 0.6, another has WER = 0.5, and a third has WER = 0.8, it is incorrect to average these values arithmetically. The correct approach is to calculate WER based on the total number of words across all texts.
WER varies across different test samples – Some solutions may perform better or worse on specific audio recordings compared to competitors. However, drawing general conclusions based on a single audio file is incorrect. A large dataset is needed for meaningful evaluation. Types of Speech Recognition Systems. Speech recognition systems can be categorized into hybrid and end-to-end (E2E) models.
End-to-end (E2E) systems directly convert a sequence of sounds into a sequence of letters. Hybrid systems consist of separate acoustic and language models, which operate independently. The Amvera Speech solution is built on a hybrid architecture.
Structure of a Hybrid Speech Recognition System The Hybrid Speech Recognition System Works: 4. A neural network classifieseach individual sound frame. 5. The Hidden Markov Model (HMM) models dynamics, lexicon, and word structure, based on posterior probabilities from the neural network. 6. The Viterbi algorithm (Viterbi decoder, beam search) searches the HMM for the optimal path, considering classifier posteriors. Processing Pipeline:
Incoming Sound → Preprocessing → Feature Extraction → Classification Model & Language Model→Prediction
The first step in a hybrid speech recognition model is feature extraction, typically using MFCC (Mel-Frequency Cepstral Coefficients).
The acoustic model then classifiesaudio frames.
The Viterbi decoder (beam search) predicts the most probable words by combining acoustic model predictions with statistical language model data (e.g., n-gram probabilities).
Finally, rescoring is applied to generate the most likely word output. 0.3 0 0.1 0 0 0.1 0.1 0.9
Imagine that in the first frame,the classifierdetects the phoneme "д", and this continues for 10 consecutive frames. The loop will keep running until the phoneme "а" is detected. If the word exists in the dictionary, it will be recorded. Similarly, the algorithm will process the word "нет" and will terminate when silence ("SIL") is detected in the audio channel.
Let's combine the visualization of the search graph with frames for better clarity.
The general principles of training an acoustic model classifier, the process of mapping frames to phonemes, the number of classes, and ways to improve the system.
Training an Acoustic Model Classifier 4. Start with graphemes.
Example: М, А, Ш, А 5. Convert them into phonemes.
Result: m i1 sh a0 4. 5. 6. 7.
Biphones: 57 phonemes² = 3,249 classes Two-state biphones: (2 states × 57 phonemes)² = 12,996 classes
Triphones: 57 phonemes³ = 185,193 classes Problem 2: Unbalanced Class Distribution Solution – Clustering: Groups similar classes (5,000–10,000 clusters). Balances class sizes. The clustered unit is called a senone (for phoneme-based models) or chenone (for grapheme-based models).
Using MMI (Maximum Mutual Information) or MPE/sMBR (Minimum Phone Error / State-level Minimum Bayes Risk):
Train a Cross-Entropy (CE) model.
Build a numerator – a set of recognition variants that lead to the correct answer.
Build a denominator – a set of incorrect recognition variants.
Compute Loss = f(numerator) / f(denominator) → “boost correct predictions, suppress incorrect ones.”
Each ASR method uses different mathematical models for speech recognition. Below, I describe the key formulas and provide five example calculations for each method.
HMM-GMM (Hidden Markov Model + Gaussian Mixture Model)
Q (O∨ λ)=∑ q 1 , q 2 , … qr P (q 1)∏ T P (Ot ∨qt ) P (qt ∨qt − 1) (3) = (1, 2, … , ) is the observed speech sequence.
(|) are hidden states representing phonemes.
Wav2Vec 2.0 (End-to-End Transformer-Based ASR)
L=− ∑ t ∑ c yt , c log P (Ot ∨qt , θ) (5) is the raw speech input. ( = |, θ) is the probability of predicting character ccc at timestep . , is the true label (1 if correct, 0 otherwise).
θ are the model parameters.
Sample Calculations:
Whisper (Transformer-Based ASR by OpenAI)
Q (Y ∨ X )=∏ T P ( yt ∨ y 1 : t − 1 , X , θ) (6) is the input speech signal. is the predicted text sequence. (|1:−1, , θ) is the probability of each token at time θ, given the previous tokens and audio features.
The model is trained using a sequence-to-sequence transformer approach.
Sample
Example 2
Sample
Example 2
4.9 4.8 5. Results and discussion The evaluation of modern Automatic Speech Recognition (ASR) systems demonstrates significant progress but also reveals persistent challenges. Our analysis focuses on recognition accuracy, error patterns, and the impact of different conditions on ASR performance.
Experimental results indicate that human transcription maintains a Word Error Rate (WER) of 5.1% on benchmark datasets such as Switchboard. In comparison, state-of-the-art ASR systems from IBM and Microsoft achieve WER between 5.8% and 6.8%, demonstrating near-human performance. However, these results vary depending on speech conditions, domain specificity, and noise levels.
Despite the advances in ASR, recognition accuracy heavily depends on domain factors, including: Speaker variability: Different accents and speaking styles impact recognition accuracy. Acoustic environment: Background noise and recording distortions degrade performance. Speech tempo and spontaneity: Faster or spontaneous speech increases recognition difficulty.
Contextual adaptation: Current models struggle with specialized terminology and new words.
Hybrid ASR systems, incorporating Hidden Markov Models (HMM) and Neural Networks (NN), achieve better generalization across different domains. They allow separate optimization of acoustic and language models.
End-to-End ASR models (such as Transformer-based architectures) directly map audio to text but require vast amounts of labeled data for effective performance.
The accuracy of WER as a metric is often debated due to: Ambiguities in word normalization (e.g., variations in capitalization and punctuation). Incorrect phoneme-to-text alignments affecting error rates. Unfair comparisons across datasets due to varying speech complexity.
While ASR systems continue to advance, they still struggle with domain dependency, spontaneous speech, and error correction. Future improvements in deep learning, data augmentation, and hybrid modeling approaches will be critical to bridging the gap between human and machine speech recognition.
Despite significant advancements,automatic speech recognition (ASR) is still an evolving field with numerous challenges. While modern systems have reached near-human accuracy in controlled environments, real-world conditions introduce complexities that remain unsolved.
One of the biggest obstacles is spontaneous speech, which includes hesitations, disfluencies, interruptions, and informal expressions. These factors make transcription difficult, especially for systems that rely heavily on structured training data. Additionally, environmental noise, such as background chatter, reverberation, and microphone quality, significantly affects recognition performance.
Another major challenge is speaker variability. Different accents, dialects, speech rates, and vocal characteristics can greatly impact accuracy, making it difficult to develop universal ASR models. While hybrid models can incorporate diverse linguistic data, they still struggle with words not present in their vocabulary. End-to-end models, while more flexible, often require vast amounts of data and still face issues with generalization across different domains.
The author(s) have not employed any Generative AI tools. RESULTS. Herald of the Kazakh-British technical university. 2024;21(3):78-89. (In Kazakh) https://doi.org/10.55452/1998-6688-2024-21-3-78-89. [15] Alpar, S., Faizulin, R., Tokmukhamedova, F., & Daineko, Y. (2024). Applications of SymmetryEnhanced Physics-Informed Neural Networks in High-Pressure Gas Flow Simulations in Pipelines. Symmetry, 16(5), 538. https://doi.org/10.3390/sym16050538. [16] Nuralin M.; Daineko Y.; Aljawarneh S.; Tsoy D.; Ipalakova M. The real-time hand and object recognition for virtual interaction. 2024. PeerJ Computer Science.