Models for the classification of vowels are of continuing interest in both phonetics and for the development of automatic speech recognition (ASR) systems. The phonetics researchers favor linear classifiers based on formants, whereas ASR systems have adopted deep neural networks in recent years. In our work we compare the performance of several kinds of convolutional neural networks (CNN) with the linear classifier on the task of classifying short vowel frames from TIMIT corpus. Our primary hypothesis was that the CNN models would prove significantly more precise than linear models, including during consonant-vowel and vowel-consonant transitions, while obviating the inherent difficulties of formant tracking. We confirmed the hypothesis, although the improvement was modest. Our secondary goal was to investigate the possibility to mitigate loudness sensitivity of CNN models, and determine whether the mitigation would have deleterious effect on the classification performance of CNN models. Our experiments indicate that the loudness invariant CNN architecture performs equally well to traditional spectrum based convolutional models.
The difference in a single English vowel may convey
nearly a dozen different meanings (e.g. hVd examples in
[
First, one may idealize a vowel as a stationary sound.
This is a reasonable approximation since speakers can
stretch almost any vowel to many times its ordinary
duration without much effort. This view was adopted for
instance in classical studies of English vowels in [
Second, one may view a vowel as a dynamic entity
evolving in time to both accommodate its acoustic
surroundings as well as to express inherent vowel dynamic.
This view fits much better to acoustics observed in
conversational speech, where higher articulation rates lead
speakers to shorten stationary vowel centers to the point
of nearly eliminating them. This view was already hinted
on in [
A model belonging to the first paradigm is sometimes called a frame recognition model, since it can be trained based on extracted vowel windows (frames), and it can categorize a vowel given a previously unseen sound window (frame). We will call a model belonging to the second paradigm a segmental model.
There are two principal methodologies that are used to
create frame recognition models. For phonetics, the
typical approach is to create a linear model based on the first
three formants, which are known to be key determinants
of a vowel quality. The second methodology is a powerful
class of classification models that has garnered much
attention in automated speech recognition systems, namely
convolutional neural networks (CNN) [
The goal of this work is to compare, both qualitatively
and quantitatively, classification results obtained by the
classical phonetic approach based on formants and the
newer method based on convolutional neural networks
with the eye on suitability of adoption of CNN for
phonetic analyses. A key potential problem is the known
dependence of deep neural network models on large amount
of data [
The performance of classifiers is easiest to investigate in
cases when they are faced with a difficult recognition
problem. A well-known pair of vowels that repeatedly shows
up in listening experiments as difficult to distinguish is the
aa/ao pair [
We opted to use the TIMIT corpus [
We used vowel segments of male speakers from SA1
and SA2 sentences of the TIMIT corpus (we exluded
female speakers due to the well-known problem of F1
determination for voices with high fundamental frequency,
as well as due to likely separate models required for
different sexes [
In this work we used classes implied by the words,
rather than the classes provided by TIMIT annotators.
The primary reason is that those classes are less
ambiguous, and possibly more reliable [
The analysis window was set to be 24ms (384
samples). For each vowel altogether 21 frames were extracted
equally spaced within the segment duration as indicated
by TIMIT segmentation boundaries. Thus there were
altogether 109410 samples divided among TRAIN and TEST
datasets approximately in ratio 2:1. Extraction of formants
and LPC spectra was performed using phonTools R
package [
Neural network models Currently, search for neural network architectures is an active research field. We used ad hoc designs as shown in Figure 2.
For convolutional networks we used architectures shown in the upper part of Figure 2. All convolutional layers were 1-D with kernel of length 3; we used 5 neurons on the first layer and 10 on the second. ReLU nonlinearity was used both in convolutional and dense layers. The dense layer had 32 neurons. Finally, we distinguish several variations of the model based on provided spectral input: CNN-A. The input was 192 point Fourier logperiodogram.
CNN-B. The input was LPC spectrum of order 19, sampled at 192 equally spaced frequencies. SPECTRUM AT 5%
FINAL SPECTRUM
The purpose of the variations was to try providing data to CNN which is closer to data that formant models use (e.g. the third formant F3 almost never exceeds value 3,5kHz).
Finally, let us explain the rationale behind intensityinvariant models CNN-C and CNN-D. If the sound source is further away from the listener, obviously the sound is quieter. In the context of frame classification, there should be no change in prediction if the sound envelope is uniformly shifted by an equal constant. Admittedly, the prediction may change, if more context were available, because then a decrease of intensity may constitute a phonological contrast (e.g. syllabic stress). Let us note that any model based on formants satisfies this invariance property, (1) (2) by virtue of ignoring formant amplitudes. To achieve an equivalent property with CNN, one can transform the input tensor. Thus instead of supplying input vector s representing log-magnitudes of spectral components s = (s1; s2; : : : ; sM) we provide as input the vector N(s) defined as follows: N(s) := (s2 s1; s3 s1; : : : ; sM s1) The key feature of this transform is that the sounds differing purely in intensity, say by d decibels, map to the same input vector since
N(s1 + d; s2 + d; : : : ; sM + d) = N(s1; s2; : : : ; sM) (3) Moreover, the topographic distribution of input vector components, which is important for CNN, is maintained, and only a very low frequency component is missing, which should not be relevant for vowel identification. 3.2
We considered two kinds of segmental models. First, we
supplied frame information from two frames: the onset at
10% and the offset at 90% position. This is inspired by
research of Nearey and Morrison [
Second, we trained LSTM (long short-term memory)
network [
We have opted for pairwise classification paradigm, where
we classify frames (or segments) from a pair of words at
a time. This decision is motivated primarily by our desire
to elucidate strengths and weaknesses of various models,
which may be confounded in the multiclass setting. We
note that the paradigm has analogues and applications in
phonetics (concept of minimal word pairs [
Interpretability for classification models is highly desirable for recognition models in phonetics. For frame classification models it is possible to analyze their performance based on the position within vowel. Summary analysis is presented in Figure 3, from which we draw the following conclusions:
Formant models (GLM as well as k-NN) in several cases closely follow convolutional neural networks (e.g. all-dark, or dark-wash, or dark-oily). This indicates that very often formant positions are sufficient for classification. 4.2
We trained also segmental models in order to gauge the effect of restricting classification to frames rather than segments. The results are summarized in Table 3.
Clearly, segmental models can make good use of extra information, since all six segments are well classified. The most likely explanation is the ability of segmental models to discriminate consonant-vowel/vowel-consonant transitions. 4.3
For ASR applications, especially for the systems based
on hidden Markov models [
It is hard to argue which model predicted likelihoods that are more correct, since there is no ground truth. Subjectively, one may feel that CNN models are too certain about their predictions, especially since the predictions are considerably less smooth than that of the formants based GLM model. The small size of the dataset may be a possible cause. 5
Our experiments indicate that it is feasible to construct
CNN models for frame classification for medium size
phonetic corpora. The resulting models showed superior
performance to formant-based models even without extensive
parameter optimization, which is in accordance with
findings of a recent study on Korean vowels [
Work on this paper was partially supported by grant VEGA 2/0144/18. We are thankful to Peter Tarábek for his suggestion to use the stacked input architecture for onset+offset CNN models, which proved much more accurate than the previously investigated branched architecture. 1.0 0.8 0.6 0.4 1.0 0.8 0.6 0.4
GLM CNN−A
CNN−B CNN−C
CNN−D k−NN wash
water dark wash
water dark 1.0 0.8 0.6 0.4 0.2 0.0 1.0 0.8 0.6 0.4 0.2 0.0 oily all oily all oily all 0.0 0.4 0.8
0.0 0.4 0.8 internal percentage wash dark water all dark dont oily all dark dont oily d o o h li ike1.0 l 0.8 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 internal percentage