Speech is a mainstay of communication across literally all human activities. Besides facts and statements speech carries substantial information regarding experiences, thoughts, and emotions, therefore adding significant context. Moreover, non-linguistic elements such as pauses add more to the message. The field of speech emotion recognition (SER) has been developed precisely to develop algorithms and tools performing what humans learn to do from early on. One promising line of research comes from applying deep learning techniques trained on numerous audio attributes to discern between various emotions as dictated by a given model of fundamental human emotions. Extreme learning machines (ELMs) are neural network architectures achieving eficiency through simplicity and can potentially operate akin to a sparse coder. When trained by a plethora of audio attributes, such as cepstral coeficients, zero crossing rate, and autocorrelation, then it can classify emotions in speech based on the established emotion wheel model. The evaluation, done with the Toronto emotional speech set (TESS) on an ELM implemented in Julia, is quite encouraging. extreme learning machine, speech emotion recognition, emotion classification, Plutchik model, higher order patterns, are fundamental, with interpretations ranging from so- The recent scientific literature regarding ELMs, SER, and ∗Corresponding author.
Language, whether oral or written, is among the major sources of human emotion and perhaps a mainstay of civilization itself. The field of speech emotion recognition (SER) almost since its formulation has been an essentially demanding field systematically garnering intense interdisciplinary interest since it aims to answer fundamental questions regarding human speech, which includes major elements such as intonation and pitch as well as latent and non-linguistic elements such as pauses and the length of sentences. Because of the complexity and volatility of human speech, SER relies heavily on machine learning (ML) and recently on deep learning (DL) techniques for performing tasks.
Human emotion models such as the emotion wheel
by Plutchik [
Among the various models proposed for the various SER tasks, extreme learning machines (ELMs) have shown considerable potential. The latter can be partially at least attributed to the ELM structure which has only a single but very long hidden layer. In turn, this allows for straightforward and easy to interpret training schemes, all of which eventually stem from a synaptic weight regularization property. This is aligned with the intuition that a certain optimality condition should hold in order for the weights to be uniquely derived.
The primary research contribution of this conference paper is the development of an ELM implemented in Julia and operating like a sparse encoder for the emotion classiifcation of sentences coming from the ubiquitous Toronto emotion speech set (TESS) collection, a benchmark for training ML and DL models for SER tasks.
The remainder of this work is structured as follows. graph mining is briefly reviewed in section 2. In section 3 the proposed methodology is described, whereas the results obtained using the TESS dataaset are analysed in section 4. Possible future research directions are given in section 5. Bold capital letters denote matrices, bold small vectors, and small scalars. Acronyms are explained the notation of this work is summarized in table 1.
CEUR htp:/ceur-ws.org ISN1613-073
CEUR
Workshop Proceedings (CEUR-WS.org)
the attention focus of a number of fields [
Emotion models have been developed in order to explain
how emotions work, their intensity and elicit conditions,
how they may be composed in case of emotion levels,
and possibly their evolutionary purpose. In this set of
models the one proposed by Plutchik has been among
the earliest and one of the most commonly used in
engineering applications. Additionally, it has an easy to
understand and intuitive-friendly visual interpretation,
which is shown in figure 1. Notice that this figure depicts
a two dimensional projection of a cone.
the inherent complexity of the SER tasks, ML approaches
such as ensemble learning [
ELMs have been used in ML because of the simplicity
of their architecture [14]. They have been used as part of
ML pipelines for wavelet transforms [
Graph mining is a field relying heavily on ML [ 21]
and graph signal processing techniques [22]. Regarding
the use of ML, self organizing maps (SOMs) for
recommending cultural content are presented in [23], exploiting
natural language attributes for finding linked
requirements between software artefacts is the focus of [ 24],
decompressing a sequence of Twitter graphs compressed Figure 1: Plutchik model (From Wikipedia).
with the two-dimensional discrete cosine transform
using a tensor stack network (TSN) is described in [
The individual output of the -th neuron can be computed from the nonlinear combination of equation (2).
Therein is the number of input neurons which is much smaller than that of the hidden neurons , namely ≪ , and equal to the dimensionality of each data point.
△ ℎ (x )= (∑ , x [ ]) = (w x )
The nonlinear activation function (⋅) may take a number of forms such as the logistic function or polynomial kernels. In this case it is the hyperbolic tangent function of (3). It has the advantage of being diferentiable and of being the Bayes estimator of a bipolar source under additive white Gaussian (AWGN) noise.
(; 0)=△ tanh (; 0)
The first derivative of can be expressed as a second order polynomial of the latter as shown in (4). This expression is that of Malthus population models.
(; 0)= △ (; 0)
= 0(1 − 2(; 0))
The column synaptic weight vector w is formed by stacking the weights w, connecting the -th hidden neuron with the -input one. Moreover, this is also the -th column of the synaptic weight matrix W. If the data points are stacked on top of each other, then the input matrix X is formed. Thus H of (1) can be rewritten as in (5), where the function of (3) is elementwise applied.
H =△ (WX )
In general ELMs, depending on their training formulation, can perform regularized least squares fitting in order to determine the optimal weights as in (6) where 0 is a hyperparameter. Therein the regularization term adds robustness to the algorithmic minimization process.
To this end, the nonlinear least squares problem of (6) was formulated, where the Frobenius matrix norm is used since it is diferentiable. Also
trix containing the one hot encoding of the eight primary emotions and W∗ is the solution. 2 W∗ =△ argmin [ ] = argmin [ 0‖W‖2 + ‖ (WX ) − Y‖ ]
Expanding (6) and taking into consideration the expansion of the Frobenius norm the objective function to be minimized can be recast as in (7). Because of the form Frobenius norm and that of the nonlinear activation function not only is diferentiable but it also has a single global minimum. Additionally, the regularization term ensures that synaptic weight sparsity also taken into consideration. Thus, minimizing translates into finding the 1–7 (2) (3) (4) (5) (6) weight set achieving a tradeof between fitting the ELM its central frequency. This allows the details of a speech response to the target response with the least possible signal to be more discernible. energy. The latter can be considered as the explanation The spectrogram of a signal is a function of time and (7) closest to that dictated by Occam’s razor. = 0 tr (W W) + tr (( (WX ) − Y) ( (WX ) − Y))
The minimization problem of (7) is a regularized nonlinear least squares problem. The hyperparameter 0 determines the relative weight of the synaptic weight matrix sparsity compared to how well the ELM response matches the target response. The problem of (7) can be solved by a plethora of methodologies including iterative ones such as fixed point methods. However, they should take into consideration the nonlinear term introduced by the activation function. This can be accomplished by utilizing methods such as the Gauss-Newton or a regularized version thereof. In this work the steepest descent iterative method was selected because of its simplicity and because of the single global minimum has, since the latter is essentially a sum of squares.
Furthermore, it can be argued that the proposed ELM, if properly trained, operates like a sparse coder with each activation neuron corresponding to a single emotion. This approach is clear it can be extended to an arbitrary number of emotions, provided of course that the appropriate attributes are available. However, the ELM proposed here can in fact discover the emotional direction and not the valence itself.
In this subsection the various features used to train the ELM described here, their primary properties, and their respective meaning are explained. Said attributes are also shown in table 3 along with a brief explanation.
The cepstral coeficients
[ ] express a modified short term power spectrum of a signal [ ]consisting of speech samples. They are derived by an algorithmic process which involves the following steps: • The sequence is pre-emphasized such that higher frequencies receive an energy boost. • The spectrum is smoothed with a window, usually a Hamming window of odd length. • The power spectrum is translated in the nonlinear
are the cepstral coeficients.
The natural meaning of the cepstral coeficients is that they represent a power spectrum where each frequency band has a resolution roughly inversely proportional to frequency and shows how its frequency content evolves in small time steps. Typically it can be obtained by the wavelet transform, by the short time Fourier transform (STFT), or by a bank of bandpass filters such as Gabor and shifted Chebyshev filters. In any case, the resulting heatmap has been transformed to a long column vector, which incurs some information loss as the spatial structure is lost. This is attributed to the fact that the proposed ELM is trained with data points with are real vectors. An architecture natively handling matrices may be more adept in this scenario.
The -th autocorrelation coeficient [ ] of any realvalued stationary sequence [ ] is defined as the expected value of the sequence multiplied by a shifted version of itself by positions. In practice these stochastic coefifcients are often approximated by the sample mean of equation (8) under the assumption of ergodicity. Autocorrelation coeficients are a measure of the self-similarity of the sequence under consideration and play a central role in discovering higher order patterns through the Wiener filter. It should be noted that the higher is, the less reliable the estimation of [ ] becomes as fewer term pairs are available. Therefore, in most engineering applications is small compared to the total length of the speech sample sequence. As a direct consequence of the Cauchy-Schwarz inequality, the maximum autocorrelation coeficient is the first one [0].
1 − −−1 =0 [ ] ≈ ∑ [ ] [ + ], 0 ≤ ≤ − 1 (8)
Finally, the zero crossing rate (ZCR) is an important feature which assumes that the mean value of the speech signals has been subtracted from it during a preprocessing phase. ZCR is closely tied with the primary mode of the Hilbert-Huang spectrum (HHS), which is built on fundamental signals inherent in the sequence. Thus, intuitively speaking, the HHS is very similar to the Fourier spectrum but it is composed of basis signals progressively extracted from the original signal itself and hence having irregular shapes instead of weighted complex exponentials. In this context ZCR plays a role analogous to that of the fundamental frequency in Fourier analysis.
The audio attributes used in this work are also shown in table 3 along with their interpretation.
The architecture of the proposed ELM is shown in 2. Notice that all hidden neurons belong to the same ELM layer and they are conceptually but not physically segmented to show they are an integer multiple of the neurons of the input layer. Hence the hidden layer can be thought of as comprising of segments, although in practice all hidden neurons are simultaneously trained.
The implementation language of choice was Julia. It is a rapidly emerging multiparadigm high level language aiming at computation-heavy tasks such as those frequently encountered in DL and ML scenarios, large database clustering, extensive and fine grained simulations, and graph signal processing.
The TESS dataset contains 200 target words spoken in the context of a carrier phrase by two actresses, a younger and an older one aged 26 and 64 respectively. Each recording contains 2000 data points, which are suficient for processing, and they represent the neutral state plus six of the primary emotions according to Plutchik’s model, namely these of anger, disgust, fear, happiness, pleasant surprise, and sadness. Therefore, from the emotions listed in table 2 anticipation and trust are absent. Consequently, from the four pairs of primary bipolar emotions only two are fully present in TESS.
In figure 3 is shown the heatmap resulting from the analysis of the ELM training. From it the following can be immediately inferred: • The neutral emotional state is the only one which can be accurately discovered in the context of this work. This can be attributed to the fact that compared to the other states there is no valence. In turn this allows its isolation from the rest of the states in the attribute space with a margin suficient for the ELM to discern it. • On the contrary anger is the most dificult to be discovered. A possible explanation is that its bipolar opposite emotion is also present in TESS and, thus, certain instances have been misattributed to it. Moreover, anger is also confused with surprise and sadness. The former is possibly due to valence, whereas the latter because of polarity. • Concerning the other bipolar pair of sadness and happiness, they are clearly distinguished from each other, but nevertheless there is a small probability they will be misclassified respectively as disgust and as pleasant surprise. This can be attributed to their valence as well as to the semantics of each emotion under consideration. • The remaining emotions can be also be distinguished relatively easy from the others in the dataset. Still, the negative emotions tend to be classified with a lower level accuracy compared to the positive ones, with the single exception of sadness. This can be explained by their prevalence in TESS. tectures capable of natively handling two-dimensional attributes such as the class of graph neural networks.
This conference paper is part of Project 451, a long term research initiative with a primary objective of developing novel, scalable, numerically stable, and interpretable higher order analytics.