Most speech-driven systems on the first step convert audio to text through an automatic speech recognition (ASR) model and then pass the text to any downstream natural language processing (NLP) modules. However, these ASR models can lead to system failure or undesirable output when being exposed to natural language perturbation or variation in practice. In this paper, we introduce a simple yet eficient model for improving the understanding of the semantics of the input speeches and error correction by processing multi-hypothesis ASR systems.
• VLDB 2021[
• Stacked DeBert [
We use JiWER toolkit2 to clean up our datasets and calculate WER (in this case WAcc)[
( ℎ( )) (, ) = 1 − (, ) (1) (2) Where is original text and is text with ASR noise. ℎ is a function to transform text to phoneme. We use CMU Pronouncing Dictionary3 to transform our texts.
The PER metric allows us to more accurately measure the accuracy of our models. Table 2 shows an example of estimating results using PER and WER metrics. We can see that in some cases the WER metric shows no change in quality: the last two rows show the same WER result, while PER shows the diference between these rows of text. In other cases WER metric shows worse result than it actually is. In the first two rows of table 2 the diference between the predicted result and the correct answer is one apostroph. WER shows one whole word error, while PER shows only one phoneme error, which is much more accurate.
3.1. Models In this work we use several models and baseline.
• Baseline. As a simple baseline we use majority vote: If some text occurs times in a corpus, that text is considered correct, otherwise a random text is selected. • Advanced baseline. For better baseline we use two algorithms: ROVER[9] and HRRASA[10]. • T5.[11] The T5 model is trained on several datasets for 18 diferent tasks which majorly fall into 8 categories: text summarization, question answering, translation etc. In experiments we use 3 diferent sizes: t5-small, t5-base, t5-large. • PEGASUS.[12] PEGASUS model pretraining task is intentionally similar to summarization: important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We use PEGASUS model trained on Xsum dataset [13] We use HuggingFace Transformers 4 for model training and prediction. Each model is trained with following parameteres: encoder length 512, decoder length 64, batch size 3, 8 epochs, learning rate 5e-05, after each 1000 steps we evaluate our models with beam size 12. 3.2. Data
model baseline (N>1) baseline (N>2) HRRASA ROVER T5-small T5-small T5-base T5-base T5-large T5-large PEGASUS-xsum PEGASUS-xsum finetuned + + + +
The first problem we had with models for summarization was the limitation on the output of text. We can partly control text generation. All models have been pretrained on the tasks of generating from a paragraph to few sentences, while our task requires only one sentence as the output. Therefore, in some cases, the model generated multiple sentences, which had a negative impact on quality. We tried to counter this by replacing the "." token with the "|" token in the T5 model.
This paper presents our approach to noisy text sequence aggregation, which is ranked second place in the VLDB 2021 Crowd Science Challenge. Our paper shows the efectiveness of the method. The error analysis also shows that the proposed approach can perform better with additional datasets.
[9] J. Fiscus, A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover), in: 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, 1997, pp. 347–354. doi:10.1109/ASRU.1997.659110. [10] J. Li, Crowdsourced Text Sequence Aggregation Based on Hybrid Reliability and Representation, Association for Computing Machinery, New York, NY, USA, 2020, p. 1761–1764.
URL: https://doi.org/10.1145/3397271.3401239. [11] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu,
Machine Learning Research 21 (2020) 1–67. URL: http://jmlr.org/papers/v21/20-074.html. [12] J. Zhang, Y. Zhao, M. Saleh, P. J. Liu, Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2020. arXiv:1912.08777. [13] S. Narayan, S. B. Cohen, M. Lapata, Don’t give me the details, just the summary! Topicaware convolutional neural networks for extreme summarization, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018.