The sound of silence: how traditional and deep learning based voice activity detection in uences speech quality monitoring

Abstract

Real-time speech quality assessment is important for VoIP applications such as Google Hangouts, Microsoft Skype, and Apple FaceTime. Conventionally, subjective listening tests are used to quantify speech quality but are impractical for real-time monitoring scenarios. Objective speech quality assessment metrics can predict human judgement of perceived speech quality. Originally designed for narrow-band telephony applications, ITU-T P.563 is a single-ended or non-intrusive speech quality assessment that predicts speech quality without access to a reference signal. This paper investigates the suitability of P.563 in Voice over Internet Protocol (VoIP) scenarios and speci cally the in uence of silences on the predicted speech quality. The performance of P.563 was evaluated using TCD-VoIP dataset, containing speech with degradations commonly experienced with VoIP. The predictive capability of P.563 was established by comparing with subjective listening test results. The e ect of pre-processing the signal to remove silences using Voice Activity Detection (VAD) was evaluated for ve acoustic feature-based VAD algorithms: energy, energy and spectral centroid, Mahalanobis distance, weighted energy, weighted spectral centroid and four Deep learning model-based VAD algorithms: Deep Neural Network, Boosted Deep Neural Network, Long Short-Term Memory and Adaptive context attention model. Analysis shows P.563 prediction accuracy improves for di erent speech conditions of VoIP when the silences were removed by a VAD. The improvements varied with input content highlighting a potential to switch the VAD used based on the input to create a content aware speech quality monitoring system.

Keywords

• Speech Quality
• VoIP
• P
• 563
• Voice Activity Detection (VAD)

-

The availability of high speed mobile and xed broadband connection is driving the adoption of Voice over Internet protocol (VoIP), which exploits packetswitching techniques and become both an alternative and a replacement for Public switched networks (PSTN) [ 2 ]. Real-time monitoring is essential to provide predictions of the actual speech quality experienced by users of the VoIP applications such as Google Hangouts, Microsoft Skype, and Apple FaceTime. Speech quality is measured using subjective testing such as Absolute Category Rating (ACR) [ 1 ] and considered the most reliable method, but it is time consuming, impractical and inappropriate for real-time quality assessment. As an alternative approach, objective metrics are useful to the application developers and network system operators to ensure that the changes to the platform do not have a negative impact on user's quality of experience (QoE). This technique is faster, practical and appropriate for real-time quality assessment.

Single-ended objective metrics also referred to as no-reference metrics predict speech quality based on an evaluation of the degraded signal only, and could be deployed at the end point of VoIP channels to monitor quality [ 14 ]. Moller [ 14 ] reports that two non-intrusive estimation of narrow-band speech quality algorithms are standardized, the International Telecommunication Union (ITU) recommended P.563 [ 16 ] and the second, American National Standard Institute (ANSI) standard, referred to as `Auditory non-intrusive quality estimation plus' (ANIQUE+) [ 10 ]. An implementation of P.563 is publicly available while ANIQUE+ is commercially available. Other examples of non-standardized speech quality prediction metrics are Low complexity speech quality assessment (LCQA) [ 3 ] and a recently presented DNN-based quality prediction metric [ 15 ]. Reported results indicated that the DNN-based metric performed poorly in competing speaker scenarios. This paper evaluates the ITU standard P.563 [ 16 ] noreference metric to investigate the impact of silence in real-time speech quality monitoring. In a real-time monitoring scenario, longer silences than those expected by P.563 will distort the quality prediction. For example, P.563 speci es 25-75% speech activity as a valid input. If speech activity in a sample is less than 25%, the sample quality will be predicted as bad, as the input is invalid.

Early techniques used characteristics of the acoustic wave such as energy, spectral centroid, spectral entropy and zero-crossing rate to recognise the human voice based on threshold. These techniques rst make assumptions on the distribution of speech and background noises (usually in the spectral domain), and then design statistical algorithms to dynamically estimate the model parameters, making them exible in dealing with non-stationary noises [ 25 ]. But these methods have limitations e.g. model assumptions may not fully capture data distributions due to few parameters and they may not be exible enough in fusing multiple acoustic features, which results in partial utilization of information in speech. Recently, data driven deep learning approaches have been adopted which have the potential to overcome the limitations of those modelbased methods. The deep learning-based VADs can be integrated to the speech quality prediction systems to improve their performance. Also, they can fuse multiple features much better than traditional VADs e.g. deep neural networks (DNN), long short-term memory (LSTM). These models extract acoustic features during the course of training and uses di erent classi ers to classify speech and non-speech.

In this paper the suitability of applying an acoustic feature-based VAD and Deep learning model-based VAD to remove silences before presentation to P.563 is investigated. The paper is laid out as follows: Section 2 introduces the ITU no-reference objective metric P.563, Voice activity detectors (VADs), TCD-VoIP database, and Section 3 describes the evaluation method. Section 4 presents and discusses the results before concluding remarks and future directions are made in section 5. 2 2.1

Background

Single-ended Speech Quality Model: P.563

Single-ended, non-intrusive speech quality models are sometimes referred to as no-reference models. They are useful for real-time monitoring of speech quality and scenarios where a reference source is unavailable. One such model P.563 [ 13 ] was standardized by ITU in may 2004. It was designed for assessing samples of active speech and is used for narrow-band speech signals. The P.563 model is based on three principles [ 13 ] namely: a physical model of vocal tract; a reconstruction of the reference signal to assess unmasked distortions; and focusing on speci c distortions: e.g. temporal clipping, robotization, and noise. Quality prediction by P.563 involves several stages: pre-processing, dominant distortion classi cation, and perceptual mapping. The degraded signal is pre-processed which involves reverse ltering, speech level adjustment, identifying speech portions and calculating speech and noise levels via a VAD [ 16 ]. Distortions classes include unnaturalness of speech, robotic voice, beeps, background noise, signal to noise ratio (SNR), mutes, interruptions are extracted from the voiced signal parts. A dominant distortion class is determined and mapped to a single mean opinion score (MOS) which describes speech quality on a scale from 1 (bad) to 5 (excellent). The pre-processing stage in P.563 includes a VAD which is based on adaptive power threshold, using an iterative approach. The internal VAD is used to compute a range of features and the active and inactive portions of speech activity as shown in Fig. 1 and does not pre-process to remove silence which can be seen in full details [ 16 ]. The performance of P.563 has been shown to be poor on samples containing silences [ 7 ]. The importance of silence detection is investigated here, using two VADs in system, the rst VAD removes silences and the second VAD (inside P.563) is estimating speech activity. 2.2

Voice Activity Detector

Applications like speech and speaker recognition, speech enhancement, speech coding and speech synthesis need e cient feature extraction techniques where most of the voiced part contains speech attributes. Silence removal is a technique which is adopted for this purpose that facilitates the system to be computationally more e cient. The events in speech are classi ed as (i) silence; (ii) unvoiced; and (iii) voiced.

VAD detects presence and absence of human speech like silence, voiced and unvoiced based on speech features. It avoids unnecessary coding and transmis

Speech Signal

P.563 VAD

Spectral

Clarity

High

Frequency

Variation

Active

SNR

Inactive

Mutes

sion of silence packets in Voice over Internet Protocol (VoIP) applications, saving computation and network bandwidth. For continuous monitoring of conversation, an aggressive VAD may yield better predictive performance from the P.563 metric [ 13 ]. Five di erent acoustic feature-based VAD algorithms and four Deep-learning model-based VAD algorithms are presented below and evaluated. Ground Truth (GT): This is the ideal binary mask that a VAD would create. It was computed using the known speech-silence structure of the speech samples in TCD-VoIP dataset (See Table 1). This mask is created as a representation of a VAD with perfect voiced activity detection. The speech samples are divided into overlapping 25 ms frame size with 10 ms shift. Energy (E) [ 21 ]: The simplest method of voice activity detection is analysis of short time energy E of speech samples xi(n), given by equation (1) for each ith frame having samples n, with frame length N . The silences are identi ed by nding frames with maximum energy (peak-to-peak) less than 0.003 joule.

E(i) =

N n=1 1 XN jxi(n)j2

Energy and Spectral Centroid (ES) [5]: The speech samples are broken

into overlapping frames of size 25 ms with 10 ms shift and for each frame, two speech features namely; short time energy, given by equation (1) and spectral centroid, given by equation (2) are extracted. The spectral centroid, Ci, of ith frame is de ned as the centre of gravity of its spectrum, given by equation (2), (1) (2) Ci =

PN k=1(k + 1)Xi(k) PN

k=1 Xi(k) where Xi(k); k = 1; 2; :::N , is Discrete Fourier Transform (DFT) coe cients of ith short time frame of frame length N . For each speech features, a simple threshold is calculated separately. As long as the two thresholds are estimated, a common threshold is computed and applied to extract the voiced segments [ 5 ].

Uni-dimensional Mahalanobis Distance (UMD) [ 19 ]: This VAD detects voiced segments from the speech samples using uni-dimensional Mahalanobis distance function which is a linear pattern classi er [ 22 ]. It assumes that background noise present in the utterances are Gaussian in nature and uses statistical properties (mean and standard deviation) of background noise to make a sample as voiced or silence. It considers that for a 1D Gaussian distribution, 99.7 % of its probability mass is in the range of mean j j 3 i.e. P r[jx j 3] = 0:997.

The speech samples are divided into overlapping frames of size 25 ms with 10 ms shift and the Mahalanobis distance [ 19 ] \r" from random variable x to mean measured in units of standard deviation is de ned by equation (3). The steps of algorithm are detailed in [ 19 ].

r = jx j (3)

Weighted Energy and Weighted Spectral Centroid (WE) [5]: The

energy and spectral centroid VAD has two speech features namely; short time energy and short time spectral centroid and the threshold is calculated as the weighted average between the two rst local maxima of histogram for each feature individually [ 5 ]. Therefore, this VAD has been broken in two sub-parts, namely; weighted energy VAD and weighted centroid VAD.

Deep Neural Network (DNN) [ 23 ]: This DNN-based VAD implementation uses two classi er classes, namely: speech class and silence class. The input vector of the DNN is constructed for every frame of the speech signal based on multi-resolution cochleagram (MRCG) feature. MRCG is a concatenation of four cochleagram features with di erent window sizes and frame lengths. The cochleagram features are generated from gammatone lterbanks. The output vector is the posterior probabilities of these two classes, which is computed for each input vector based on threshold respectively. A DNN model is trained for an input feature sequence xm = [x1; x2; :::xM ] to an output sequence ym = [y1; y2; :::yM ], where m = 1; :::; M is the time of frame. Cross-entropy criterion is used to optimize the VAD.

Boosted Deep Neural Network (bDNN) [ 26 ]: A Boosted DNN VAD that is an extended version of the DNN VAD. The main di erence between training bDNN and DNN VAD is that for bDNN VAD, each speech frame is further expanded i.e. xm to xm = [xm W ; xm W +1; :::; xm; :::; xm+W 1; xm+W ] and similar for ym, where m = 1; :::; M is time of frame and \W " is user de ned half-window size [ 26 ]. It has (2W +1)d input and 2W +1 output feature sequence.

Long Short-Term Memory (LSTM) [ 24 ]: The LSTM VAD architecture [ 20 ] is unidirectional with \k" hidden layers and \n" hidden units per layer. The input vector to the LSTM is a single 40-dimensional multi-resolution cochleagram (MRCG) features. The output vector is the posterior probabilities of silence and speech class, which is computed for each input vector based on threshold respectively. Cross-entropy is used as the optimisation criterion and truncated back propagation through time (BPTT) learning algorithm as the minimisation criteria [ 24 ].

Adaptive context attention model (ACAM) [11]: An ACAM-based

VAD that is a frame-based speech or silence classi er. The input speech signal is divided into overlapping 25 ms frames with 10 ms shifts. 768-dimensional multiresolution cochleagram (MRCG) feature vectors are extracted and trained using an algorithm called Adam for stochastic optimization [ 12 ]. 2.3

Experimental Corpus

In order to investigate the in uence of silence in VoIP applications, a dataset with silences and speech samples was used. TCD-VoIP [ 6 ] is a publicly available dataset of degraded speech samples with corresponding subjective opinion scores, generated by ACR test described in ITU-T Rec. P.800 [ 1 ]. Table 1 shows the format of speech samples. Each sample has a leading and trailing silence with 2 sec silence in middle (50% silences). The average length of each sentence is 2.4 sec. The speech samples are 16 bit WAV les, sampled at 48 kHz. The utterances have been taken from Harvard test sentence list. There were 24 listeners in all experiments and each condition was tested with 4 speakers (2 male and 2 female). It contains ve types of platform independent degradation conditions (independent on codec, network or hardware), common to VoIP. These are: background noise, intelligible competing speakers, echo e ects, amplitude clipping, and choppy speech. The competing speakers is a separate case of large-crowed babble as the speaker is intelligible and this is a common VoIP call scenario. The echo e ects in a voice call usually occur due to transmitted speech being picked up in the receiving unit's microphone, creating a feedback loop like the scenario of microphone misplacement. Clipping occurs, when the amplitude of samples in a signal is set above the maximum permitted value e.g. microphone loudness. Choppy speech refers to speech, which is a ected by missing samples due to packet loss in the VoIP network. The background noise sourced the AURORA database, developed by Hirsch et al. [ 9 ]. The summary of conditions and parameters are presented in [ 6 ]. 3

Evaluation Method

The VAD mask, which is the binary mask, has been used to get the percentage of voiced segments of each VADs and compared to the ground truth (GT) mask to get the true positive/negative and false positive/negative which measures the performance of a particular VAD in terms of precision, recall and F-score [ 17 ]. (f) How voice activity detection in uences speech quality monitoring 7 (a) (b) (c)

No VAD (Baseline)

Deep learning model-based VADs

F-score measures the accuracy and it's maximum value is 1 (precision = recall = 1) which means that voice activity decision of a certain VAD algorithm is equal to the reference transcription. The performance of a VAD deteriorates when the portion of active speech is mis-classi ed as non-active giving temporal clipping [ 8 ].

The pre-processed samples of TCD-VoIP via each VADs are used as the input to P.563 to get the objective predictions. To investigate the performance of P.563 in measuring speech quality, the per-condition objective predictions (MOS-LQO) of pre-processed samples are calculated to compare with the percondition subjective mean opinion scores of samples (MOS-LQS). The degraded samples are re-sampled from 48 kHz to 8 kHz for testing. The test evaluates 384 speech samples. For each condition, 4 samples (2 male and 2 female speakers) are tested giving 96 conditions. The samples are processed to remove the WAV headers and evaluates with P.563 as a baseline. The Deep learning model-based VADs were trained using speech samples from the TIMIT corpus [ 4 ]. The TIMIT utterances have considerably shorter silences than speech which could create a class imbalance problem. To address this, the model developers added 2 second silence segments were before and after each utterance [ 11 ]. Multi-resolution cochleagram (MRCG) features were extracted as the input vector to these VADs. TCD-VoIP dataset are used as the test dataset. The samples are pre-processed by the VADs described in Section 2.2 to remove the silences and the resulting signals are evaluated with P.563 as illustrated in Fig. 2. The performance comparison between subjective listener test and objective metric prediction quality scores are quanti ed in terms of Pearson correlation coe cients ( p), Spearman rank order coe cients ( s) and root mean squared error (RM SE) [ 18 ]. Scatter plots of results classi ed by degradation condition are presented to visualise the statistics.

The statistics of TCD-VoIP dataset, broken down by condition type are presented in Table 4. It can be seen from Table 4 that the Pearson correlation of noise and all samples of WS VAD gives good correlation among all VADs and higher than No VAD (baseline). For clip conditions, UMD VAD and for chop conditions, boosted DNN VAD is good. WS VAD has strong Spearman correlation for noise conditions as compared to No VAD (baseline). LSTM VAD is good for competing speaker conditions and ES VAD for clip conditions. WS VAD has the lowest RMSE for noise conditions and little increase for other conditions with respect to baseline. WS VAD exhibited better performance than the other VADs tested, when used as a pre-processing unit to P.563 and has higher correlation with subjective MOS score i.e. estimating better speech quality.

To explore the impact using a VAD as a pre-processing unit to P.563, the N o VAD (baseline) and WS VAD plots for objective metric P.563 (denoted by MOS-LQO) on the TCD-VoIP dataset have been compared to subjective results (denoted by MOS-LQS) per-condition for each type of degradation in Fig. 4 (a) (b). The improvement for the noise condition (denoted with J) in particular is visually evident.

Fig. 4 (c) - (d) show the best traditional and best deep learning-based VAD performance with correlation between subjective and objective speech quality. While comparing the best deep learning-based VAD with best traditional VAD, it can be seen that bDNN VAD has a better f-score than weighted spectral centroid (WS) VAD for the CHOP speech condition in particular with little di erence for other degradation conditions. 5

Conclusions and Future Directions

P.563 was designed as an objective no-reference metric but provides inaccurate predictions when input samples contain long silence segments. This paper reports benchmarking results of nine di erent voice activity detectors used as a pre-processing unit to P.563 tested using the TCD-VoIP database. The results show that for the VoIP conditions tested, a weighted spectral centroid preprocessing VAD improves prediction of speech quality for all conditions tested compared to baseline. The boosted DNN VAD exhibited better performance for the CHOP condition in particular. Pre-processing with a VAD demonstrated that P.563 could be deployed for real-time monitoring VoIP speech quality even in situations where there are longer periods without voice activity. Deep learning VADs, trained and optimised for particular degradation conditions, i.e. content 5.0 4.5 B 1.0

C sensitive, could have a signi cant e ect on speech quality prediction accuracy as classifying a speech signal by condition and switching VAD to an condition optimal VAD may be easier and more robust than trying to tune traditional signal based VAD threshold parameters. Substituting the two VADs with a single VAD that is aware of content may yield further improvement in speech quality estimation. This provides an opportunity for future work. 6

Acknowledgements

This publication has emanated from research supported in part by a research grant from Science Foundation Ireland (SFI) and is co-funded under the European Regional Development Fund under Grant Number 13/RC/2077.

References

1. ITU-T recommendation P. 800 : Methods for subjective determination of transmission quality ( 1996 )

2. Alcatel-Lucent : PSTN industry analysis and service provider strategies: Synopsis . Alcatel-Lucent, Paris, France, Tech. Rep. Bell Labs Analysis for BT ( 2013 )

3. Bruhn , S. , Grancharov , V. , Kleijn , W.B.: Low-complexity, non-intrusive speech quality assessment ( June 2012 ), US Patent 8 , 195 , 449

4. Garofolo , J.S.: Timit acoustic phonetic continuous speech corpus . Linguistic Data Consortium ( 1993 )

5. Giannakopoulos , T.: A method for silence removal and segmentation of speech signals . University of Athens, Athens 2 ( 2009 )

6. Harte , N. , Gillen , E. , Hines , A. : TCD-VoIP, a research database of degraded speech for assessing quality in VoIP applications . In: Quality of Multimedia Experience (QoMEX) , Seventh International Workshop . pp. 1 { 6 ( 2015 )

7. Hines , A. , Gillen , E. , Harte , N.: Measuring and monitoring speech quality for Voice over IP with POLQA, ViSQOL and P.563. INTERSPEECH, Dresden, Germany ( 2015 )

8. Hines , A. , Skoglund , J. , Kokaram , A. , Harte , N.: Monitoring the e ects of temporal clipping on VoIP speech quality . In: INTERSPEECH ( 2013 )

9. Hirsch , H.G. , Pearce , D. : The Aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions . In: ASR2000 - Automatic Speech Recognition: Challenges for the new Millenium ISCA Tutorial and Research Workshop (ITRW) ( 2000 )

10. Kim , D.S. , Tarraf , A. : ANIQUE+: A new American national standard for nonintrusive estimation of narrow-band speech quality . Bell Labs Technical Journal 12 ( 1 ), 221 { 236 ( 2007 )

11. Kim , J. , Hahn , M. : Voice activity detection using an adaptive context attention model . IEEE Signal Processing Letters ( 2018 )

12. Kingma , D. , Ba , J.: Adam: A method for stochastic optimization . arxiv 1412.6980 ( 2014 )

13. Malfait , L. , Berger , J. , Kastner , M. : P. 563 - The ITU- T standard for single-ended speech quality assessment . IEEE Transactions on Audio, Speech, and Language Processing 14 ( 6 ), 1924 { 1934 ( 2006 )

14. Moller, S. , Chan , W.Y. , Co^te, N., Falk , T.H. , Raake , A. , Waltermann, M.: Speech quality estimation: Models and trends . IEEE Signal Processing Magazine 28 ( 6 ), 18 { 28 ( 2011 )

15. Ooster , J. , Huber , R., Meyer, B.T.: Prediction of perceived speech quality using deep machine listening . Proc. Interspeech 2018 pp. 976 { 980 ( 2018 )

16. P.563, I.T.R. : Single-ended method for objective speech quality assessment in narrow-band telephony applications . International Telecommunication Union , Geneva, Switzerland ( 2004 )

17. Pham , T.V. , Tang , C.T. , Stadtschnitzer , M.: Using arti cial neural network for robust voice activity detection under adverse conditions . In: Computing and Communication Technologies. International Conference . pp. 1 { 8 ( 2009 )

18. Pocta , P. , Melvin , H. , Hines , A. : An analysis of the impact of playout delay adjustments introduced by VoIP jitter bu ers on listening speech quality . Acta Acustica united with Acustica 101 ( 3 ), 616 { 631 ( 2015 )

19. Saha , G. , Chakroborty , S. , Senapati , S.: A new silence removal and endpoint detection algorithm for speech and speaker recognition applications . In: Proceedings of the 11th national conference on communications (NCC) . pp. 291 { 295 ( 2005 )

20. Sak , H. , Senior , A. , Beaufays , F. : Long short-term memory recurrent neural network architectures for large scale acoustic modeling . In: Fifteenth annual conference of the international speech communication association ( 2014 )

21. Sakhnov , K. , Verteletskaya , E. , Simak , B. : Approach for energy-based voice detector with adaptive scaling factor . IAENG International Journal of Computer Science 36 ( 4 ) ( 2009 )

22. Stork , D.G. , Duda , R.O. , Hart , P.E. , Stork , D. : Pattern classi cation . A WileyInterscience Publication ( 2001 )

23. Vesperini , F. , Vecchiotti , P. , Principi , E. , Squartini , S. , Piazza , F. : Deep neural networks for multi-room voice activity detection: Advancements and comparative evaluation . In: Neural Networks (IJCNN) , International Joint Conference . pp. 3391 { 3398 ( 2016 )

24. Zazo Candil , R. , Sainath , T.N. , Simko , G. , Parada , C. : Feature learning with rawwaveform cldnns for voice activity detection ( 2016 )

25. Zhang , X.L. , Wang , D. : Boosted deep neural networks and multi-resolution cochleagram features for voice activity detection . In: Fifteenth Annual Conference of the International Speech Communication Association ( 2014 )

26. Zhang , X.L. , Wang , D. : Boosting contextual information for deep neural network based voice activity detection . IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP) 24(2) , 252 { 264 ( 2016 )