This paper presents a deep learning based model to detect the completeness and correctness of a sentence. It's designed specifically for detecting errors in speech recognition systems and takes several typical recognition errors into account, including false sentence boundary, missing words, repeating words and false word recognition. The model can be applied to evaluate the quality of the recognized transcripts, and the optimal model reports over 90.5% accuracy on detecting whether ∗Corresponding author. †These authors contributed equally.
the system completely and correctly recognizes a sentence.
correct sentence boundaries in whole transcripts [
Automatic Speech Recognition (ASR) systems develop
technologies to recognize and translate spoken language
into text by machines [
In prior works, research focused mainly on
grammatical error detection [
Recently, transformer models have shown state-of-art
performance in generating word embeddings and
extracting intrinsic features of word sequences. In specific,
Bidirectional Encoder Representations from Transformers
(BERT) [
The paper is structured as follows: In Sec. 2, we explain the models and experimental design. In Sec. 3, we describe how the dataset is generated. We discuss the 2.
In this section, we use three state-of-art transformer
models BERT [
Besides, we also test the performance of using BERT
embedding plus a downstream text classification
network. For the classification networks, we use either a
bi-direction LSTM and a TextCNN. We use a one-layer
TextCNN with kernels sizes to be 2, 3 and 4. For LSTM,
we use a one-layer bi-directional LSTM network [
We ensemble the five trained classifiers with random forest. Configuration and the final classification performance are shown in Sec. 4.2.
For the model to have better generalizing capacity, a training set from diverse sources covering diverse topics and occasions is necessary. The following corpora are included in our proposed dataset:
News reports [
Ted 2020 Parallel Sentences Corpus [
Wikipedia corpus [13]: over 10 million topics Topical-Chat [14]: nearly 10 thousand human dialog conversations spanning 8 broad topics
To make the selected datasets suit our speech recognition model, we remove some non-English tokens, sentence ending symbols (‘.’, ‘!’, ‘?’), duplicated sentences and also short sentences (less or equal to 5 words) to avoid some recognition errors. After pre-processing on the data from the sources, we create the following two datasets:
Standard Dataset: contains 0.3 million sentences from News reports, 0.3 million sentences from Ted corpus, 0.3 million sentences from Wikipedia corpus, 0.2 million sentences from Topical-Chat, in total 1.1 million sentences. We split the Standard Dataset randomly over all data sources into train set, ablation set and test set, with a proportion of 8:1:1.
Large Dataset: contains 2.3 million sentences from News reports, 0.4 million sentences from Ted corpus, 2 million sentences from Wikipedia corpus, 0.2 million sentences from Topical-Chat; in total 5 million sentences. We split it into train and test set, with a proportion of 19:1.
We train and compare performances of various models on the Standard Dataset. As a comparison, we evaluate the performance of BERT trained on the large dataset to see how an enlarged training set afects generalization ability for this task.
For creating positive samples, punctuation is removed (except abbreviations such as it’s, Mr., I’ve, etc.) and words are converted to lower case.
For creating negative samples, we mimic typical errors of the speak recognition system, which are detailed in the following, and we propose corresponding methods to create negative samples with respect to typical errors.
False sentence boundary: When a speech recognition system fails to correctly separate two sentences, the ifrst sentence would be cut of in the middle and part of the sentence would be assigned to the next sentence (illustrated in Fig. 1 (a)). For such negative samples, we group the sentences by three, and randomly separate the three sentences into 2-4 sentences (so that on average negative samples created in this way would have equal length with positive samples). While choosing random separating points, the genuine sentence separations points, punctuation and typical words for starting subsentences (e.g. that, which, because, etc.) are avoided, and thus reduce the probability that a generated sample is still a complete sentence by chance (e.g. ‘I like you because you are beautiful’ to ‘I like you’.)
Missing words: A speech recognition system can fail to recognize one or several words from a sentence, and as a result some words may be missing in the produced transcripts (Fig. 1 (b)). For such negative samples, we randomly remove 1 word for sentences up to 3 words, and 2-4 words from longer sentences.
Repeating words: The system can record speakers’ unintended repeated words (Fig. 1 (c)). For such negative samples, we randomly repeat 1 word for sentences within 3 words, and 1-3 words from longer sentences.
False word recognition: The system can mistakenly recognize one word as another word (Fig. 1 (d)). For such negative samples, we randomly replace 1 word for sentences within 3 words, and 1-3 words from a longer sentences, by random words from another sentence.
Finally, the punctuation is removed and words are converted to lower case.
After creating the positive and negative samples, the sentences longer than 100 words are removed, for they are too long to appear in speech recognition. We create the same number of negative samples as that of positive samples, so that we have a balanced dataset. The ratio between diferent types of negative samples is 2:1:1:1. The type False Sentence Boundary corresponds to two times In this section, we report the results of our experiments. We describe below the setup, and then evaluate the different models in Sec. 4.1. In Sec. 4.2, based on the models, we train a Random Forest classifier to further aggregate the models and improve the performance. In Sec. 4.3, we compare the performance of BERT trained on Standard and Large Dataset. Finally, we show the result of BERT trained on a Multi-Labeled Dataset in Sec. 4.4.
Training details: We train each model for 5 epochs with batch size 64 using Adam optimizer. The initial learning rate is set as 3 − 5 for fine-tuning transformer models and 1 −3 for downstream classification networks. To prevent overfitting, we only save the model with optimal performance on test set after each epoch.
the number of other negative sample types since False forest classifier can generate a final classification through Sentence Boundary contains two types of false sentences, a majority vote mechanism. those which are cut of and those which are assigned To prevent random forest from overfitting the train with extra words. set, we use a separate ablation set, instead of the train set which the models are trained on. The best parameters after 10-fold cross-validation are 100 decision trees, and 4. Experiments and Discussion a maximum depth of 3. The test accuracy of the random forest reaches 90.51%, higher than the optimal accuracy among the individual models (90.26%), but not to a large extent. This is probably since the transformers (along with their embedding) share similar structures and do not diverge much on decisions.
In this section, we train BERT on the large dataset (5 times the size of the Standard Dataset) with less epochs (1 epoch in contrast to 5 epochs). Overall, the model is trained with the same iterations as with Standard Dataset. With the same training details described before (but only for one epoch), results show that training with Large Dataset provides a higher test accuracy (90.36%), compared with the accuracy trained with Standard Dataset (89.27%).
The results suggest that, provided with enough computational capacity, we can further improve our model’s generalization ability by training on a larger dataset. with missing words, even though in most of the cases more than one word is missing in the erroneous sentences. This is understandable because in most cases, not every word is indispensable, even we lose some words, and maybe the meaning is not exactly the same but the sentence still makes sense grammatically.
Finally we test our trained multi-modal BERT model on the real-world ASR outputs from CEASR corpus [15]. The predictions are shown in Fig. 3, where we can see the model is able to capture real-world ASR errors correctly, while we also provide an example where the model fails.
In this paper, a dataset for detecting speech recognition errors was created, where four diferent types of typical speech recognition errors were taken into account. Experimental results show that transformer models are capable of providing good performance on classification of the constructed dataset for speech recognition error, reporting approximately 90% accuracy for BERT, GPT2 and BIG-BIRD. A Random Forest was trained based on the five models, and further improved the test accuracy to over 90.51%. Overall, the results suggest that using state-of-art transformer models can provide good quality for detecting the errors in speech recognition systems, and provide feedback on further improvements of speech recognition systems. In our future works, special adjustments might be needed to better cope with identifying missing words in recognized sentences. [13] W. Foundation, Wikimedia downloads, ???? URL: https://dumps.wikimedia.org. [14] K. Gopalakrishnan, B. Hedayatnia, Q. Chen, A. Gottardi, S. Kwatra, A. Venkatesh, R. Gabriel, D. Hakkani-Tür, A. A. AI, Topical-chat: Towards knowledge-grounded open-domain conversations., in: INTERSPEECH, 2019, pp. 1891–1895. [15] M. A. Ulasik, M. Hürlimann, F. Germann, E. Gedik, F. Benites de Azevedo e Souza, M. Cieliebak, Ceasr: a corpus for evaluating automatic speech recognition, in: 12th Language Resources and Evaluation Conference (LREC) 2020, European Language Resources Association, 2020, pp. 6477–6485.