Speech and Language Impairments (SLI) afect a large and heterogeneous group of people. With our work, we propose a novel, easy, and immediate detection tool to help diagnose people who sufer from SLI using speech audio signals, along with a new dataset containing English speakers afected by SLI. In this work, we experiment with feature extraction methods such as Mel Spectrogram and wav2vec 2.0, as well as classification methods such as SVM, CNN, and linear neural networks. We also work on data audio augmentation trying to overcome the very common limitations imposed by data scarcity in the medical field. The overall results indicate that the wav2vec 2.0 feature extractor, paired with a linear classifier, provides the best performance with a reasonably high accuracy of over 96%.
sification systems. Automatic classification technologies
are widely applied in voice assistants [
This study proposes an analysis of a novel, yet simple, as interpolation and nonlinear mixing on the spectrum.
approach of using exclusively audio recordings for SLI We will now list and briefly explain the most used audio
detection. Specifically, in Section 2 we start by exploring augmentation techniques.
the current literature, and then we will talk about the Pitch Shifting. The tone of each audio signal in the
problems faced in collecting our data and how we handled dataset is lowered or raised by a factor preserving its
them in section 3. After that, in Section 4, we will go duration.
through an analysis of the techniques and models used to Time Stretching. The audio sample is slowed down or
perform the detection, the trials and results we obtained sped up by a ratio without altering the pitch drastically.
from them in Section 5, and then we will discuss the Time Shifting. Time is shifted to the left or to the right
limits of our approach in Section 6. We will finally draw by a random factor or by a predetermined amount.
our conclusions in Section 7. Volume Adjustment. The volume of the audio file is
altered, there is a change in loudness, or sometimes a
dynamic range compression is applied.
2. Related Works Noise addition. Noise is introduced into the samples,
other than a simple random Gaussian noise there are
In the ever-evolving landscape of computer science and many types of noises such as white noise [
One of the most important challenges in developing an This topic is so important that researchers also develeficient and efective audio classification system is ac- oped and designed methods that generate entirely new cessing a large and well-annotated dataset. One of the samples, for example with the aid of a Generative Advermain obstacles in developing sound classifications is a sarial Network (GAN) in [40] people created new variants lack of a suficient quantity of labeled data. This is due of the audio samples that already existed in their dataset to the following main reasons: class imbalance, data pri- and then utilized an evolutionary algorithm to search vacy issues, time constraints involved in data collection, the input domain to select the best-generated samples, in high dependency on expertise for efective annotation, this way they were able to generate audio in a controlled etc. [28, 29, 30] Data Augmentation (DA) is defined as the manner that contributed to an improvement in classificacreation of new data by adding deformations to increase tion performance of the original task. One very recent the variety of the data so that these deformations do not DA method proposed by Google is SpecAugment [41], change their semantic value. It is well known that DA can in this method, the two-dimensional spectrum diagram improve the algorithm’s performance, tackle the issue is treated as an image with time on the horizontal axis of overfitting [ 31, 32], and improve the generalization and frequency on the vertical axis. Encoder-decoder netability of Deep Neural Networks (DNN); this happens be- works are becoming very popular in fields diferent from cause DA averages over the orbits of the group that keeps NLP, this is because they can convert a high-dimensional the data distribution invariant, which leads to variance input into a lower-dimensional vector in latent space, researchers in [42] have experimented with a Long Short Term Memory (LSTM) based auto-encoder to produce artificial data. 2.2. Audio Feature Extraction and Models trons (MLP) were very useful in person identification using speech and breath sounds [53], Hidden Markov Models (HMM) [54], logistic regression and linear discriminant analysis [55] and others. Some studies exploited the efectiveness of multiple simpler methods with ensemble methods such as random forests [56, 51], XgBoost [57], and so on. Unfortunately, considering the complexity of sound and the need to sometimes train an extremely sensitive classifier that can identify diferent representations of sound features, traditional ML still sufers in these kinds of tasks from having less complex models. In this case, the choice of DL methods has been proven to be more eficient. DL methods difer from traditional ones because they can extract meaningful features from data through the application of a hierarchical structure [58] CNNs were able to achieve significant and more accurate training results [59]. People tried to combine the best of these two worlds by implementing hybrid methods, for example, researchers merged an SVM and a GRU-RNN in [60].
It should be noted that data augmentation is not the only way to reduce overfitting and improve the generalization ability of DL models. Model structure optimization, transfer learning, and One-shot and Zero-shot learning are also known strategies that deal with overfitting from different aspects. We will now focus on the most common processing flow of audio classification: preprocessing the original audio data, feature extraction, and feeding the features into the DL model. Audio signals have very high dimensionality, so thousands of floating point values are required to represent a short audio signal, raising the need for exploring dimensionality reduction and feature extraction methods. The degree of how great or poor a model performs is also determined by the choice of features used feature representation is crucial to improve the performance of learning algorithms in the sound classification task. One of the first features that comes to mind when thinking of an audio signal is the spectrogram, its 3. Dataset characteristics have been widely used by previous researchers in diferent domains of sound classification, In the medical field, in particular, regarding specific probsuch as heartbeat sounds to detect heart diseases [43]. lems such as the one presented in this paper, data is not Another method used to extract features implements the always freely available or available at all. This is mostly Mel-Frequency Cepstrum (MFC), which is a representa- due to privacy concerns [61, 62]. Another important reation of the short-term power spectrum of a sound, based son, which is also related in some ways to privacy [62], on a linear cosine transform of a log power spectrum lies in the overall low level of digitization of healthcare on a nonlinear mel scale of frequency, where the Mel- information [63]; in fact, according to Gopal G. et al. Frequency Cepstral Coeficients (MFCC) were successful [64], healthcare has the lowest level of digital innovain representing sounds for the detection of respiratory tion compared to other industries, such as media, finance, diseases [44]. Some methods that also use the MFC are insurance, and retail, contributing to limited growth of the long-mel [45], mel filter bank energy [ 46], inverted labor productivity. In addition to this, it is also worth MFCC [47], and many more. Although mel spectrogram noting that not every dataset containing the desired medand MFCC are commonly used, people also implement ical information is also in the desired format, in which bag of audio words [48], Discrete Gabor Transform (DGT) case the only remaining option is to create an entirely audio image representation [49], ZCR, entropy of energy, new dataset from scratch, that is what we did. spectral centroid, spectral spread, spectral entropy [50], and so on. 3.1. Data Collection
Classification is a common task in ML and pattern recognition. DL methods applied in these tasks, such as The process of collecting audio data is a pivotal phase CNN models, often do not perform as well as more tradi- in this research. For our dataset, we aimed to collect a tional ML methods such as random forest, Adaboost, etc., suficient amount of pure, non-multimodal, audio data in especially in small data [51]. On the other hand, typical a waveform representation. Audio data can be stored in ML algorithms, such as ensemble classifiers have been various formats, each with its characteristics, trade-ofs, shown to learn features better and adapt more with im- and use cases. Common audio formats include Waveproved generalization abilities even in the case of small form Audio File Format (WAV), MPEG-1 Audio Layer and imbalanced datasets. Over the past years, difer- 3 (MP3), Free Lossless Audio Codec (FLAC), and more. ent ML algorithms have been used for detecting sound These formats difer in terms of compression, quality, and events and medical sounds, and the achieved results were compatibility. For this study, we opt for the WAV format of great significance. Classifiers, such as Support Vec- [65], which is an uncompressed audio file format, develtor Machine (SVM), have shown to be very efective in oped by IBM and Microsoft, that eficiently stores audio sound classification tasks [ 52], also MultiLayer Percep- data in a waveform representation without any loss of information from them, keeping just human speech sounds (with or without background noise). After that, we analyzed the time windows by dividing each one of them into smaller ones containing the speech of one single Figure 2: Audio sample with noise from our dataset person each. Even if there exist diferent tools available to detect human speech, considering the scarcity of data we sufer, we decided to perform this step manually to formation. Thanks to its characteristics, which guarantee be sure that the quality of our dataset is not afected. the highest amount of information for an audio signal, Secondly we split the time windows that we obtained WAV is the audio format used as input by wav2vec 2.0 in 3-second clips. We chose this length as a trade-of be[66], a state-of-the-art speech model developed by the tween a suficient length, to capture fluency information Facebook AI Research group (FAIR) that is one of the and a brief duration. Our decision was also based on the models used in this work. standard approach used in the state-of-the-art working
The data collection process began with the identifi- with wav2vec 2.0 in these kinds of tasks [68, 69]. Then cation of audio samples containing English speakers af- these clips were saved in two diferent subsets, creatfected by Speech and Language Impairment (SLI) orig- ing the Train and the Test set, ensuring that the same inating from diferent conditions. This diverse dataset speakers do not overlap in both datasets. was intentionally curated to optimize the performance Finally the acquired data was augmented to increase of SLI detection. By including speakers with a range of its dimension. We applied the following audio augmenimpairments, the model is exposed to a broad spectrum tations techniques: Time shifting, Time stretching, Pitch of speech patterns and anomalies, thereby enhancing its shifting, and Noise addition, using Gaussian noise. To do ability to accurately detect SLI in real-world applications. so, we used the python library audiomentations [70]. For To source such data, we turned to YouTube, a vast and Time shifting we resampled the time windows shifting user-friendly repository of video and audio content. The the starting time further by 1.5 seconds; For Time stretchvideos found were then converted into audio files in WAV ing we slowed down the speed of the audios by a ratio format using an online converter. of 0.8; For the Pitch shifting we both lowered and raised
We finally paired the collected data with a subset of the pitch tone by a value of 3, obtaining for each clip two the LibriSpeech dataset [67] containing healthy English additional ones; Finally for the Noise addition, we added speakers only. a 0.01 amplitude Gaussian noise. Audio waveforms before and after noise addition are shown in Fig. 1 and 3.2. Data Preprocessing Fig. 2. All the augmentation techniques were applied on the original audio; Time shifting was directly applied on To feed the waveform signals to the model, we needed the time windows, while the other ones on the initial 3 to ensure that they were appropriately prepared and pro- seconds clips. cessed. Efective data preprocessing is fundamental to The number of samples in the created dataset is shown enhancing the model’s performance, as it directly im- in Table 1, while in Table 2 we collect the audio data pacts the model’s ability to extract meaningful patterns augmentation techniques used and their respective paand insights from raw input data. This was performed rameters. in diferent steps. Firstly we identified diferent time windows from each audio file to cut out unnecessary in
The dataset contains audio files in the WAV format, its data is afected not only by its advantages but also by its drawbacks. The complete dataset, which comprehends both original and augmented data, was too large to be loaded in an online manner using the original files. To overcome this problem we loaded the data in batches and concatenated them in subsets that were saved in the .arrow format [71], a columnar memory format for lfat and hierarchical data, organized for eficient analytic operations. In this way, large data can be saved, loaded, and processed avoiding memory usage problems. The best way to approach a problem is to know deeply every factor that influences it and how the key components work, after that, one can tackle it and try to capture its essence with the maximum capabilities. In the following subsections, we present a brief description of the techniques we used and the models we implemented.
4.2. Wav2vec 2.0 The way humans hear frequencies in sound is known as pitch, it is a subjective impression of the frequency. They do not perceive frequencies linearly, on the contrary, humans are more sensitive to diferences between lower frequencies than higher ones. For example, the diference between audios of frequency 100 and 200 is way bigger than 1000 and 1100, even though the absolute diference is the same amount. Humans perceive sounds on a logarithmic scale rather than a linear scale. The Mel Scale [72] was developed to take this into account by conducting experiments with a large number of listeners. It is a scale of pitches, such that each unit is judged by listeners to be equal in pitch distance from the next. The human perception of the amplitude of a sound is called loudness, similarly to frequency, also loudness is heard logarithmically rather than linearly. The Decibel scale is used to measure the loudness of a sound, for example, a sound with an amplitude of 20 is 10 times louder than one with an amplitude of 10. We can see that, to deal with sound realistically, we need to use a logarithmic scale via the Mel Scale and the Decibel Scale when dealing with Frequencies and Amplitudes in our data. Spectrograms are generated from sound signals using Fourier Transforms. A Fourier Transform (FT) [73] is a mathematical formula that allows us to decompose the signal into its constituent frequencies and displays the amplitude of each frequency present in the signal. Spectrograms are generated from sound signals using FTs. In other words, an FT converts the signal from the time domain into the frequency domain, and the result is called a spectrum. A spectrogram consists in dividing the sound signal into smaller time segments, then applying the FT to each segment, and finally, the combination of these segments in a single plot is called spectrogram. A Mel Spectrogram makes two important changes relative to a regular spectrogram that plots frequency vs time: it uses the Mel scale instead of frequency on the y-axis and uses the Decibel scale instead of amplitude to indicate color. In Fig. 3 we can see a normalized version of the Mel spectrogram of one of the audios present in the dataset. Wav2vec 2.0 [66] is an exceptional tool that learns powerful representations from speech mimicking the human learning experience. People start, in fact, since the early stages of their lives comprehending language without labeled data, i.e. kids learn from listening to adults around them. It is also able to outperform state-of-the-art models while using 100 times less labeled data, thus demonstrating the feasibility of training without huge amounts of labeled data which is very hard to achieve in a field dealing with a complex medium such as audio.
Classification is the part that stands out the most in an entire model because it outputs the labels that are used to compute the evaluation metrics, even though it is the most noticeable part of a model, in our case they are just the final piece of the puzzle since most of the work is done in the previous steps of the pipeline; still, we want to pay some attention to the type of classifiers we used in our work.
Support Vector Machine (SVM) [74] is one of the Figure 4: Wav2vec 2.0 pipeline ifrst algorithms learned by every ML expert, it is simple yet it can achieve excellent results, especially with small amounts of data where other ML algorithms tend to have some dificulties. The objective of the support
The model can be visualized in Fig. 4 and next, we will vector machine algorithm is to find a hyperplane in an describe its components. N-dimensional space ( − the number of features) that
Multi-layer convolutional feature encoder. It distinctly classifies the data points. To separate the two consists of several blocks containing a temporal con- classes of data points, many possible hyperplanes could volution followed by layer normalization and a GELU be chosen. SVM finds a plane that has the maximum maractivation function. gin, i.e. the maximum distance between data points of
Context network. It follows the Transformer ar- both classes. Maximizing the margin distance provides chitecture, diferently from a normal Transformer that some reinforcement so that future data points can be uses fixed positional embeddings, a convolutional layer classified with more confidence. The biggest dificulty is used instead, and it acts as a relative positional embed- encountered when testing the SVM is that even with low ding. The output of the convolution followed by a GELU amounts of data the model had memory issues, since auis added to the inputs and then a layer normalization is dio features are extremely large and with multiple classes, applied. while SVM excels with data that has fewer classes, thus
Quantization module. It discretizes the output of making it hard to fully exploit SVM’s strengths. the feature encoder to a finite set of speech represen- One of the best and most eficient methods to generate tations via product quantization. Product quantization labels from an ML model is adding a linear layer at the amounts to choosing quantized representations from mul- end of the pipeline, that is what we did with our wav2vec tiple codebooks and concatenating them. The Gumbel 2.0 feature extractor, we have included a linear classifier softmax enables choosing discrete codebook entries in a (, , ) = · + and we trained its weights to fully diferentiable way. output two types of labels, one for people afected by a
The feature encoder : → takes as input the raw SLI and one for the others. waveform and outputs the latent speech representa- Resnet34 is a very famous residual neural network tions 1, ..., for time steps, then they are fed to the that was pre-trained on ImageNet-1k and was released transformer : → that captures information from by Microsoft [75], thanks to residual learning and skip the entire sequence and outputs context representations. connections this type of model can be much deeper than The output of the feature encoder is also discretized to normal convolutional neural networks. We decided to with a quantization module to represent the targets ifne-tune this model with the features extracted with the in the self-supervised objective. During the model’s pre- log mel spectrogram from our dataset. training a part of the latent speech representations that are generated from the feature encoder are masked, and then the model learns the representations of speech au- 5. Results dio by solving a contrastive task, which requires identifying the true quantized latent speech representation In this section, we will describe the diferent architectures for a masked time step within a set of distractors. After that we tested in detail and then we will comment on the pre-training on unlabeled speech, the model is fine-tuned obtained results. on labeled data with a Connectionist Temporal Classification (CTC) loss.
in particular the pre-trained wav2vec2-base model from HuggingFace [76], to perform Feature Extraction on the pre-processed non-augmented dataset and then use a Table 3 SVM, the Support Vector Classifier (SVC) model from Parameters used to compute the Spectrogram scikit learn [77], to perform the classification process taking the extracted features in input. As it was explained in Log Mel Spectrum Parameters the previous section 4, wav2vec2.0 takes a raw waveform Sample rate 22050 signal as input, 3 seconds clips in WAV format in our Windows length 2048 case, then extracts audio features from them following Hop length 512 what it had learned in its previous training. The extracted N mels 128 features were then standardized using the StandardScaler from scikit learn, removing the mean and scaling them to Table 4 unit variance. The standardization of a dataset is a com- Architectures Accuracy mon requirement for many ML estimators: they might behave badly if the individual features do not more or less Models Accuracy look like standard normally distributed data (e.g. Gaus- LASSO (Full Model) [78] 0.84 sian with 0 mean and unit variance). Finally, we fitted 1NN CHI Strategy [79] 0.8832 the SVM using a linear kernel. LMT BL Strategy [79] 0.9269
Using the SVM model as a classifier was our first at- MLP BL Strategy [79] 0.9013 tempt to cope with the limited number of samples at our NB BL Strategy [79] 0.9269 disposal. Once the dataset was augmented we ceased CNN [80] 0.8421 to use the SVM due to its intrinsic limitations at work- Our Models Accuracy ing with large datasets; so we opted for a complete DL Wav2vec2.0 + SVM 0.6627 approach. Wav2vec2.0 + FC 0.9661
For our second architecture, we substituted the classi- Log Mel Spectrogram + CNN 0.9362 ifer head with a simple Fully Connected ( FC), or linear, layer, keeping the wav2vec 2.0 model to perform the Feature Extraction, this time, on the augmented dataset. probably due to the magnitude of the feature space exWe trained the model for 5 epochs through the Trainer tracted by the wav2vec 2.0 model. class by HuggingFace on a batch of 32 samples each, set- Using, instead, an augmented dataset together within ting the learning rate to 2 − 5 after a warm-up period a DL approach we manage to reach a very high value at a ratio of 0.1 and decreasing its value linearly till the of accuracy, the highest of our models. The wav2vec end of the training. 2.0 feature extractor, having enough data to work with,
The last architecture tested was a CNN, more precisely managed to extract the key features and information resnet34, that received as input the log mel spectrogram needed to correctly identify which voice belongs to a of the audios and generated as output the labels of the healthy speaker or an impaired one. given audio. All the procedures to extract the spectro- The CNN model that was fine-tuned with Log Mel gram were carried on with the librosa library, firstly the Spectrum features achieved great accuracy in labeling sample was resampled with a new rate of 22050, then samples, unfortunately, through a more accurate analysis the mel spectrogram extracted was normalized and fi- of the confusion matrices shown in 5, 6, and 7, we disnally scaled. Regarding the CNN, only the last layer was covered that the number of false negatives is extremely modified, it was replaced with a linear layer that had two high compared to the false positives. In the medical field, output channels and the whole model was fine-tuned especially for tools helping with diagnosis, it is crucial without freezing the previous layers. Training was car- to have the smallest number of false negatives, since an ried out for 50 epochs, the learning rate started at 2 − 4 undetected disease is much worse than a false positive, and decayed by a factor of 10 every 10 epochs; the loss medical operators could be missing a lot of vital anomafunction used was the CrossEntropyLoss. All parameters lies and in time they will lose trust in the system. In used to compute the spectrum are shown in Table 3. our case recall is way more important than the precision score, from Table 5 we can see that the CNN model 5.2. Evaluations reaches only a recall score of 0.85, on the other hand wav2vec 2.0 achieves a better recall and F1 Score.
In Table 4 we show the accuracy of our architectures, compared with others architectures [78] As we can see, the first model is the one with the lowest score. This means that, despite the ability of the SVM to avoid overiftting on the poor quantity of data provided, it cannot accurately detect the speakers afected by SLI. This is
y h lt a e H I L S 93 52
constraints. The problem of data scarcity has already Future works should focus on the creation of a new been addressed in section 3 so we will now talk about dataset comprising people speaking diferent languages, quality. since it is not yet known, to our knowledge, whether flu
In the realm of ML and DL, it has been well docu- ency problems can be generalized in all languages and a mented that the issue of low-quality data and disparities wide age range, knowing that the features and the overall in data collection methodologies exacerbate the inherent characteristics of the voice between children and adults biases within the data when utilized for training algo- change in general, due to their anatomical diferences rithms, a clear example is given by the societal or political [86]. biases reflected in word embeddings or large language Given the technological advancement in the field of models [81, 82]. This concern arises when the data col- generative audio with astonishing tools such as the aulected for training purposes exhibits significant varia- dio manipulation software produced by ElevenLabs [87], tions in quality and collection techniques, resulting in which can clone voices, generate new ones, translate a heightened vulnerability to intrinsic biases within the them into other languages, and make them read texts, data. Such biases can subsequently propagate through new kinds of audio enhancement can be experimented the training process, influencing the performance and with, and although they cannot be used now, because fairness of ML and DL algorithms leading to further dis- they cannot replicate stuttering or other kinds of fluency parities and discrimination in the real world, due to the features that characterize people afected with SLI yet, accessibility to such tools [83, 84]. Particularly, in our they are promising tools to take into consideration for work, the collection of English speakers afected by SLI the near future. presents the limitation of containing mostly speakers with American accents. In real-world applications this 7. Conclusions can have negative efects on the model performance, for example, the algorithm could achieve higher and better This work proposes a novel approach to Speech and Lanresults with American people rather than with Mexican guage Impairment (SLI) detection, based solely on audio ones, or other English-speaking minority ethnic groups and AI audio-based techniques, together within an enof people whose accent diefrs from the standard Ameri- tirely new dataset composed of English speakers afected can one [84]. by SLI. The results show that, even with some limitations
Another limitation of our dataset is that it does not related to the scarcity of data available, Deep Learning contain children speakers. This is because finding such methods can achieve accurate estimations on healthy materials on the web is often dificult, and it is more or impaired speakers. In particular, wav2vec 2.0, with a dificult to create them from scratch due to the small Fully Connected layer as the classification head, reaches number of certified children afected by SLI and, since an accuracy of over 96% on our test set. Our findings also they are minors, due to more strict privacy concerns. confirm that data audio augmentation techniques are funThe most used dataset in this field [ 85] consists of one damental to training Deep Learning models adequately. second clips of Czech speaking children, both healthy or afected by SLI. Although this dataset could be useful for the detection of SLI, it is limited to the Czech language and children speakers. This kind of limitation is common in the healthcare field, especially in SLI detection.