We address the challenge of machine interpretation of subtle speech intonations that convey complex meanings. We assume that emotions and interrogative statements follow regular prosodic patterns, allowing us to create an unsupervised intonation template dictionary. These templates can then serve as encoding mechanisms for higher-level labels. We use piecewise interpolation of syllable-level formant features to create intonation templates and evaluate their efectiveness on three speech emotion recognition datasets and declarative-interrogative utterances. The results indicate that individual syllables can be detected for basic emotions with nearly double the accuracy of chance. Additionally, certain intonation templates exhibit a correlation with interrogative implications.
CEUR ceur-ws.org
Paralingual intonations can alter sentence meanings without changing the words, such as adding
sarcasm or conveying politeness. These nuances are often challenging for machine speech
recognition due to the ambiguity of implications [
This research aims to simplify the mel-spectrum into a few explainable variables that capture
paralingual cues in temporal sequences. Traditional sequence learning methods like RNN have
limitations due to dataset variations and domain-specific issues [
There have been few pieces of research done on the comparison between the applicability of
prosodic and lexical features for emotional clue extraction, they point out the higher importance
of prosodic features as compared to their lexical dependence [
https://staffprofiles.bournemouth.ac.uk/display/arehman (A. Rehman); https://jzhang.bournemouth.ac.uk/ (J. Zhang); https://staffprofiles.bournemouth.ac.uk/display/xyang (X. Yang) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings
Training clips
Test syllable m a r g o r t e p S le M c
A
o
ti
n
e
tt
t
n
a
m
r
o
F
n
o
leab titan
ll e
yS gm
e
S
o
ti
a
intonation [
The proposed method is based on the syllable formant attention mechanism, which relies on the
ifrst two formants of vowel sounds as efective descriptors of speech [
The final output of the interpolation process is the residual errors and correlations with the feature templates that can be used to estimate useful paralingual cues. For a syllable template of a feature ℎ (out of 0, 1, equation that creates a template model:
) there are five polynomials ,ℎ in a 4-degree polynomial ,ℎ ( ) = ,ℎ,0 + ,ℎ,1 + ,ℎ,2 2 + ,ℎ,3 3 + ,ℎ,4 4 (1) where is the index of the syllable being fitted, ,ℎ is the interpolated value of the formant feature ℎ at frame index , and is the incremental index of the equation in the feature template dictionary. We want ,ℎ to be as close to the actual value of that feature as possible. A loss minimization method is used that fits the know formant features onto a polynomial curve in two stages. The first stage creates an estimate of best-fit for the polynomial matrix ,ℎ using matrix manipulations as (2) (3) ,ℎ = [ ⋅
]−1 ⋅ [ ⋅ ]
,
where is a matrix with frames index of all the syllables to be fitted as rows and the polynomial
coeficients ( ∈ {0, ..., 4} for quartic fitting) as columns where each element has a value of
= . Whereas is the single-column matrix of the expected formant feature values at each
frame of the syllable. Then at the second stage, a loss minimization by gradient descent is used
to improve the fitting of the polynomials. We used the simplex algorithm for unconstrained
minimization of the polynomial regression loss [
A sum-squared error is used to estimate ,ℎ during the unconstrained minimization as =0 =0 ,ℎ = ∑ ∑ [ ,ℎ ( ) − ,ℎ ( )]2 where ,ℎ is the calculated values by the polynomial equation, and ,ℎ is the actual value of the formant feature ℎ of syllable at frame , is the total number of syllables in being fitted for this template, and
is the total frame length of an individual syllable . The initial coeficients ,ℎ produced in the first stage of coeficients are only a rough estimate to help reach the minima in less time. The actual unconstrained minimization of the polynomial curve is performed using the simplex algorithm. The algorithm moves towards the function adjusting the parameters of the worst points. The simplex algorithm usually converges in 5 to 20 iterations because of the initialization, otherwise, it takes many more iterations. Figure 3 shows the regression values of quartic curves of the first formants of each syllable laid over the ,ℎ minimum by actual frequencies.
The sequences of the first two formants; 0 and 1 , and the magnitude are fitted using audio speech recordings to create 3 separate sets of curve templates using the curve fitting method described above. The 3 sets are further divided into 10 (for each of the 3 features) categories by length because we assume that the duration of a syllable is also one of the key discriminating features. Feature sequences of various durations have their own sets of template curves. The number of total templates for a feature-duration class depends on the variety in the recording and on the coeficient of determination 2 threshold set for a match to be considered or to create a new template if the match score is lower than the threshold.
Once all the template coeficients have been estimated, they can be used as a match predictor model for various paralingual tasks such as emotion recognition or interrogative speech detection. For example, if there are a total of 100 templates of various shapes and sizes for 3 formant features, the 2 scores for all of the 100 templates can be used as a feature vector to train a classifier to predict any paralingual label for a syllable.
We evaluated the proposed method on the tasks of speech emotion recognition and interrogative intonation analysis. We make 2 assumptions to evaluate our proposed approach: • Basic emotions can be recognized from individual syllables without needing a huge dataset for regularization. We test this in two ways: 1) By training speech emotion recognition models on one dataset and testing on another, and 2) by decreasing the size of data used for training to check if the proposed method can be trained with a small sample. • Interrogative intonation can be distinguished from individual syllables when the same statement is said with a declarative vs interrogative tone. Due to the lack of a dataset that can be used for machine learning, we used a small dataset that was only big enough for observations.
For speech emotion recognition, we used 3 widely used databases that are recorded in a
scripted or improvised scenario in English. IEMOCAP database [
For interrogative intonation analysis, we used a small set of 72 utterances that were originally
collected for the purpose of controlled stimuli by Xie et al. [
Using the method described in Section 2, a set of feature templates was derived from the training dataset. The templates’ polynomial coeficients are used to estimate correlations 2 of each formant feature ( 0, 1, ) with their respective sets of templates. The template that matches the test syllable’s feature sequence the most would have the highest correlation among all the template curves. The number of template curves is not fixed and depends on the error threshold set during training. In relative terms, the error threshold was set to 0.08, i.e., if the nearest match has 2 ≤ 0.92 for a new syllable with the already known templates then a new template is created using the new syllable. Otherwise, the new syllable data is appended to the data series of the nearest matching template, which is then refitted again using the simplex algorithm.
The encoded features vectors extracted from the proposed method (the match scores with the templates) were then used to train a single hidden layer MLP (Multi-Layer Perceptron) with 8 units to perform the emotion classification task. For the cross-corpus validation tasks, the width of the syllable feature vector for training on RAVDESS was 133, 150 for IEMOCAP, and 146 for MSP-Improv.
The results in Table 1 show that the UA (Unweighted Accuracy) for the proposed method is
significantly better than the chance (25%) that reflects the amount of prosodic information
0.8
captured by templates. More importantly, the cross-corpus accuracies show that templates
learned from one corpus can predict the emotions in other corpora. For comparison, Alex et
al. [
We introduced a method for analyzing syllable-level intonation patterns by matching formant features in syllables with common patterns. Using template similarity scores, we predicted the emotional tone of each syllable and explored the correlation between match scores and questioning intonation levels. Our findings revealed that only a few templates efectively captured interrogative intonation in the sample recordings.
This research had limitations, including the absence of syllable-level annotated data, which afected learning precision due to variations within utterances. Future work involves collecting and annotating syllable-level data to enhance computational paralinguistics. Another challenge was the computational heaviness of polynomial optimization for large datasets. Future eforts will explore alternative approaches for syllable template extraction to improve eficiency.