The phonemes recognition is one of the fundamental problems in automatic speech recognition. Despite the great progress in speech recognition, discrimination of isolated phonemes is still challenging task due to coarticulation, and great variability in speaking style. The aim of this work is to develop a system for classification of isolated English vowels from the TIMIT dataset. In the paper, the following conventional methods are compared: a) k-Nearest Neighbours approach as a simple nonlinear instance-based classifier b) Gaussian Mixture Model, which belongs to the class of probabilistic acoustical modelling techniques. As a front-end, we applied standard mel-frequency cepstral coefficients with their time derivates. Various experimental methods such as trimming of audio data and cross-validation were used to increase recognition precision and reliability of system evaluation. The developed system will be used as a baseline for comparison with other newer state-of-the-art approaches.
1.1
Sha and Saul [
The above-mentioned works are focused on different type
of phone set from the TIMIT database. Several studies
regarding the vowels classification have also been made.
Weenink [
An empirical comparison of five classifiers was presented
in [
Amami et al. [
Palaz et al. [
Although it is not always possible to achieve exactly the same comparison of existing systems, Table 1 summarizes some of the most important systems in the field of TIMIT phonemes recognition over the last twenty years.
Subsequently, the presented survey is ranked according to the system accuracy, including the used methods and the sets of features.
The
The TIMIT speech corpora contains read speech and is primarily designed for studying acoustic-phonetic phenomena and for testing automatic speech recognition systems. 630 people participated in creating of this database, each contributing by reading 10 phonetically rich sentences. The recordings are in the eight main dialects of American
2.2
The extraction of appropriate features is one of the basic task of objects recognition. In the conventional ASR frontend, speech is represented by a sequence of feature vectors retaining particularly useful information from the signal. There are a large number of approaches and features extraction methods in ASR techniques. The features that have been used in our algorithm will be described in the following section.
Mel Frequency Cepstral Coefficients - are the most
commonly used acoustic features in ASR. MFCCs are
designed to respect non-linear sound perception by human
ear [
In our system, the MFCCs are computed as follows (Fig.
1): The pre-emphasis is applied to the speech signal in order
to emphasize its high-frequency components. The next step
is to divide the signal into 16 ms long frames with an overlap
of 1/2 of the frame length. The given frame length was
selected based on previous studies on isolated phonemes
recognition [
The triangular filter bank has a linear frequency distribution in the Mel frequency range: ( ) = 1125 ∗ ln(1 +
700 ) (1) where f [Hz] is the frequency in the linear scale and mel (f) [mel] corresponds to the frequency in the mel scale.
The last step is to calculate the coefficients using the discrete cosine transformation DCT.
Fig. 1. Block diagram of the MFCC computation An important parameter is also the energy of the frame. Log energy is usually added as the 13th feature to MFCC. The short-term energy function is defined by: ∞ =−∞ =
∑ [ ( ) ( − )]2 where s(k) is signal sample in time k and w(n) is the corresponding window type. It is then possible to obtain an average energy value for each frame. The disadvantage of this characteristic is the high sensitivity to rapid changes in the signal level. Values of this characteristic can be also used to separate silence segments from speech segments.
Static features, which are obtained using the procedure
above, do not capture inter-frame changes along time index.
Therefore, dynamic (or delta) features are commonly
appended to the feature vectors. Usually delta features are
the estimates of the time derivatives of static features and are
a computed as follows [
In the developed system, total features consist of 39 elements per frame: - 12 MFCC, - 12 delta (ΔMFCC), - 12 delta-delta (ΔΔMFCC), - 3 log energy. 2.3
The classification process can be divided into a learning and testing phase. Thus, data set needs to be divided into two subsets. Because of 10-fold cross-validation evaluation process (2.4), we selected the same number of vowels from each class.
Once the data were split, models of selected vowels were trained and tested according to the chosen method. The general classification scheme can be seen in Fig. 2. their easy implementation and good classification properties.
modelling of audio features in the feature
space. GMM is defined as the probability density function
formed by a linear superposition of K Gaussian components
[
( | , ∑ ) where, the probability density function of the multivariate Gaussian distribution for n-dimensional vector x is given by: 1 (2 )2|∑|1/2 1 2 ( |µ, ∑) = exp(− ( − µ) ∑−1( − µ)) (3) with mean vector µ ∈ Rn and covariance matrix ∑ ∈ Rn x n. πk are mixing coefficients, which must satisfy the following conditions 0 ≤ ≤ 1and
∑ =1 = 1 The classification function for the proposed GMM classifier has the following form:
We recall a description of these methods in the following (2) section. (4) (5) (6) (7) (8) (9) ( ) = max( ( ))
where Cp(x) is GMM of the class C.
Thus, we are looking for the maximal probability over all C classes.
The training algorithm, which returns a set of parameters Θ = {µ, and π} for each class, is based on the Maximum Likelihood (ML) criterion. Given the model p (x, Θ) with the unknow parameters, the aim is to derive its parameters based the training data – set of the feature vectors {x1, x2, …, xm}. The ML method uses Fisher likelihood function, which is defined as: ( 1, 2, . . . . , | ) = ∏ ( | ) =1
The maximum of this function with respect to unknown parameters Θ can be formalized as follows: =1 ̂ = arg max ∑ log ( | )
The maximization defined by (9) is a complicated task
that does not have an explicit solution. The
expectationmaximization (EM) algorithm [
Training the GMM statistical model for each single vowel is challenging for both computing power and memory. Fitting the model also suffers from lack of a sufficient amount of training data. It is therefore advisable to train a universal generic model (so called Universal Background Model UBM), which represents the possible distribution of the features for a wide group of sounds, and then derive from it the class-specific model for an individual vowel. The used for UBM adaptation to the vowel model (i.e. classspecific GMM). In the presented experiments, only vectors of mean values of UBM were adjusted to obtain individual models.
Given a sequence of features vectors = { 1, 2 ,. . . , } from one class of vowels, the score is expressed by (10), where θv and θ
denote the actual vowel model and universal model respectively. According to (10), the greater the probability p( |θv) against background model for as many feature vectors as possible, the more will be supported the hypothesis that the recognized audio sample belongs to the given vowel class.
=
∑ log 1 =1
( | ) ( | ) (10)
The k-Nearest Neighbours (k-NN) is a simple nonlinear
instance-based classification method and is one of the most
popular classical approaches of cluster analysis. It classifies
an unknown sample based on the known classification of its
neighbours [
The model itself is essentially made up of a training set, and the learning process consists in storing of patterns from all training samples in one model. Given an unknown sample, the distances between the unknown sample and all the samples in the training set can be computed. Input attributes must be numeric so that their distance can be calculated for each of the two patterns. Samples from the training set have number attributes, and each one sample represents a point in the -dimensional space. If a classifier wants to determine the target attribute of an unknown sample, it searches in the sample space of the training set for those that are closest to that unknown sample. Training set can be defined as: { , } =1,…, , ∈ {1,2, … , } (11) where xi is a sample with its corresponding label C and K is the size of the whole training set, L is a number of classes (i.e. number of vowels). Given unknow sample x, we are looking for sample k according to following formula: ‖ − ‖ = ‖ − ‖ =1,…., (12)
Subsequently, the sample x is placed to the same class that belongs to.
In the proposed system we used the Euclidean distance, which is the most commonly used metric for distance determination, as well as the city-block, Chebyshev and cosine distance metrics. They are defined as follows: important factors play a role in the successful classification: • • the choice of distance function the choice of the value for the parameter k (i.e. number of neighbours)
It is advised to choose an odd number for k to avoid the scenario when two classes labels achieve the same score. Some issues need to be considered during the selection of k value. Classes
with a great number of samples can
overwhelm small ones and the results will be biased, so it is
not recommended to set large k value. The advantage of
using many samples in the training set is not exploited if k is
too small [
The disadvantage of this classifier is the calculation of all distances for each classification, which can considerably slow
down the process and it can be computationally expensive if the training set or the number of unknown samples is large.
If there is not a sufficient number of observations, an
appropriate approach to determine the optimal solution for
training/testing is the so-called cross-validation technique.
[
The data set is divided into k parts, with one part always
being used for testing, and the remaining k-1 parts being
used for training. The process is repeated so that each part is
used for testing just once (Fig. 3). The advantage of
validation
is
a
relatively
accurate
estimate
of the
classification success. The disadvantage of validation is that
it requires more computer memory and consumes more time
because a lot of calculations are needed at every step.
on isolated vowels extracted from the TIMIT data set. Two
sets of vowels were created. The first set consists of the 5
classes aa, eh, iy, ow, uh. This subset correlates with the
common vowels of the most European languages (e.g.
‘a’,’e’,’i’,’o’,’u’ in Slovak) [
Proposed algorithms were implemented in MATLAB
2018b with support of the Voicebox [
Classifier training and testing was performed by 10-fold cross-validation. Data was initially randomly divided into 10 equally large subsets. Each of them contained approximately the same number of vowels represented by the feature vectors. Nine of them were used to train the model and the rest one to test it. This was repeated 10 times, so that all 10 subsets were tested. All data were parameterized by 39 MFCCs (incl. deltas and delta-deltas) per 16 ms frame with 8 ms overlap. The features matrix dimension for each vowel was 10800x39 (frames x features).
The results of the experiments with 5 vowels classification using k-NN and simple trained GMM are shown in Tables 2 and 3 respectively. There are shown the results achieved for various k-NN setup (type of metric and number of neighbours) and GMMs (number of gaussians and covariance matrix types) settings. An effort has been made to achieve a better classification accuracy by editing the data. Therefore, the entire database was mixed so that the speech dialects are evenly distributed between the training and the test part. Another data modification was vowel trimming by omitting the first and last frames for each vowel recording. So that silent parts as well as parts affected by coarticulation or unprecise vowel border detection were not taken into account. In addition, the middle frames are known to contain the most important information about the vowel. Such modified data are referred as D2, D1 indicates original data. GMM achieved the best success rate of 91.1% at n = 32 gaussians and full covariance matrix. The comparison of the best results for 5 vowels achieved by the above-mentioned methods is shown in Fig. 4.
Fig. 4. The comparison of classification of 5 selected vowels
In the last experiment, testing was performed on a larger set of classes - 18 vowels of American English were selected. Data needed for UBM training were selected from other recordings available in the database. A total of 4600 recordings from 510 speakers in a total length of approximately 3 hours and 54 minutes were used to train the UBM model. The front-end with data manipulation is the same as in experiments with the recognition of 5 vowels (referred as D2 in the text above). The experiments with GMM-UBM training/classification approach is also added. Fig. 5 shows the best results achieved Interestingly, the k-NN algorithm outperformed both GMM and GMM-UBM approaches. It achieved 84.2% vowel recognition accuracy, at setting k = 5 neighbours and cityblock metric. The second most successful system was GMM-UBM, which achieved success rate of 78.1% at n = 256 gaussians and full covariance matrix. The worst performance had the GMM classifier, probably due to insufficient amount of training data It achieved a system success rate of 75.5% at n = 16 gaussians and full covariance matrix.
Table 4 shows the classification of the individual vowels for the best k-NN model settings in form of confusion matrix. The data in table indicates the performance of the algorithm as well as the false recognized vowels. This is the best way to see how the system works when recognizing individual vowels. The diagonal shows the correctly classified vowels. The lines specify incorrectly identified vowels. The final success rate in percentage is also stated.
Metric
Significant improvement can be seen for both methods of classification if only stationary middle part of the vowels is analysed (D2). At k-NN method, a success rate of 95.08% with k = 3 neighbours and cityblock metric, was achieved. The total number of correctly classified vowels was 4548 out of 5400 and the success rate of 84.2% was achieved. As seen from Fig. 5 and Table 4, in the case of k-NN, the vowels: aa, ae, ao, aw, and ux were recognized best, while for the vowels ax, eh, and ix, a considerable number of samples were misclassified. Note that using GMM-UBM classifier, largest recognition errors occurred in other group of vowels (see Fig. 5). The largest difference in recognition rate between k-NN and GMM-UBM is in the case of the vowels aa, ux, ix. From Fig 5, also disbalance between simple GMM and GMM-UBM can be seen (theoretically, GMM-UBM should outperform GMM in all cases). Probably, further optimization of GMM-UBM is required.
Phoneme recognition task on the TIMIT database consists of several years of intensive research. There exists a number of systems and their classification success has naturally improved over time. Results presented in this paper are comparable to the existing research reported in the literature (see section 1.1). However, it is not possible to compare these works directly with our system because of different parameters and experimental settings that have been used. 4
This work deals with the design of a system for recognition of isolated vowels extracted from the TIMIT dataset and subsequent optimization of the training algorithm. Three different approaches for phoneme classification were k-NN, GMM, and GMM-UBM. The kNN method achieved the best results with overall accuracy of 95.08% for 5 vowels and 84.2% for 18 vowels recognition. GMM-UBM gave comparable results for 18 vowels recognition but classification error was distributed differently among vowel classes than in the case of k-NN. This recognition disbalance issue between k-NN and GMM approaches needs further investigation.
This publication is the result of the project implementation: Centre of excellence for systems and services of intelligent transport II, ITMS 26220120050 supported by the Research & Development Operational Programme funded by the ERDF.