Nowadays studying a mechanism of human emotional responses attracts much attention. Information about a human personality and condition, which is expressed in a manner of speech, is just as important as his statements. However, computer synthesis and speech recognition systems do not currently use this information. It is possible to numerically assess certain physiological characteristics related to emotions (cardiogram, muscle curves, EEG, speech). In order to assess an emotion objectively, it is necessary to use a complex approach including testee's self-evaluation and recording characteristics of certain body functional systems. There are widely distributed databases containing examples of such characteristics. Previous bases contain recordings of scenic speech with imitated emotions. Modern researchers prefer working with natural emotions caused by irritants - incentives. The paper specifies a multi-channel bioengineering system for studying emotions “EEG-Speech+”, which is created in TSTU, and how to work with it. It also describes two series of experiments. The first one includes searching for signs of emotion valence by a speech signal attractor. The second one includes investigating the emotion dynamics by an EEG signal. The authors describe the structure of an extended multimodal emotion base, which stores the results of all experiments. They also consider its open online version.
Studying human emotional response mechanism refers to an interdisciplinary field of knowledge, which attracts more and more attention nowadays. Research on the works of Anokhin P.K., Simonov P.V., Leontyev, Ilyin E.P., Danilova N.N., Izard K., Rusalova M.N., Ukhtomsky A.A., Fress P., Chomskaya E.D., Everly G., Rosenfeld R., Hebb D., etc., allows identifying several basic conclusions, which are not disputed by the scientific community at this stage: emotions are inherent not only in a human, but in all intelligent representatives of mammals; the emotional response mechanism is innate, some emotions are shown at the earliest stages of life; emotions are most often a reaction in response to an external or internal irritants – an incentive; the system of human emotional reactions is developing; it is formed in the process of accumulation of his personal experience and formation of cognitive functions.
The mechanism of emotional responses is a further development of reflex systems of the mammalian organism. It solves two sets of tasks: improves the means of adapting an organism to changes in external conditions and creates an apparatus for implementation of communicative processes and maintenance of socially significant contacts. Human emotional responses are related to brain activity and are revealed in functioning peculiarities of certain body functional systems.
In a colloquial human interaction, extralinguistic information about speaker’s personality and state, which is expressed in his manner of speech, is as important as the text of a statement. However, computer synthesis and speech recognition systems do not currently use information about emotions, which is a very important factor in communication.
Systems that are capable of generating emotionally colored speech and recognizing human emotional state will be in demand in virtual learning, for studying brain dysfunction, identifying network content and interactive entertainment. In addition, they will be useful for people who have different speech deviations. Modern speech synthesizers do not model emotional speech. The algorithms for recognizing human emotional state are only being developed.
Nowadays there are no objective means of measuring quantitative characteristics of emotions. However, there are opportunities for quantitative assessment of certain physiological characteristics related to them (cardiogram, galvanic skin response, muscle curves, electroencephalograms, and speech patterns).
Considering testee’s subjective assessments in an emotion, objective emotion evaluation requires an integrated complex approach including both testee’s self-assessment and recording characteristics of certain functional systems.
Successful development of emotion recognition modules by various signals recorded in a person, who is experiencing an emotion, is possible when there is a big volume of such signals. Geographical and ethnic studies show that an emotional expression is formed and changes with the course of the history of linguistics. Consequently, the sources of emotional responses should be carriers of an appropriate language.
Initially, the bases with the records of emotionally colored speech have become widespread. They are gradually expanded. Other biomedical signals (cardiogram, galvanic skin response, heart rate, muscle curves, electroencephalograms, etc.) taken at the moment when a testee demonstrates an emotional response are added to speech samples.
Early studies of emotional responses are based on the records of scenic speech with imitated emotions [1, 3, 6, 12, 18, and 19]. Usually, exterior listeners recognize such emotions correctly. The analysis of acoustic characteristics is based on the records of identic texts. Nevertheless, it is not known how well an actor is able to represent all speech characteristics that ordinary people show when they experience similar emotions. Imitated emotions are reproduced on assignment and do not need incentives.
In studies, the difference between experienced and expressed emotions is minimal. In everyday social interactions, it is often appropriate to suppress emotions. Moreover, it is preferable to express emotions that people do not really experience at the moment. A computer synthesizer of an emotional speech, which is created based on studying only simulated emotions, might deform user intentions.
Therefore, the majority of modern researchers work
with stimulated emotions (Table 1) instead of using
emotion imitations. Such emotions are natural and are
triggered by specially prepared emotiogenic incentives.
Information support of the bases includes these
incentives or their descriptions. There are some papers
that pay attention to classification, evaluation or
marking of incentives [
The need to confirm the desired emotion in a testee
leads to expanding a list of types of biomedical signals
stored in databases [7, 13, and 17]. In experiments,
testees are usually instructed not to restrain their
emotions, but in real social interactions personal
feelings are not expressed so openly. For this reason,
some researchers use other people as sources of
emotionogenic incentives [
A specialized bioengineering system “EEG-Speech+”
has been created and developing at the Department of
Automation of Technological Processes of the Tver
State Technical University [
Human
Computer
Interaction
Lab, 2011,
Eng. [
Databаse,
2013,
French.
[
EEG; physiologic measuring; face video; assessments assessments EEG EEG; physiologic measuring; face and body video; speech; position of the pupil; assessments EEG; ECG; speech; face video; assessments multichannel scheme for recording testee's responses to external emotiogenic incentives. Simultaneous recording of several types of biomedical signals allows confirming changes in testee’s emotions according to the scenario of the experiment.
Fig. 1 shows the composition and interaction scheme of the components of the bioengineering system “EEG-Speech+”. By now, the system has been expanded to five channels for recording emotional response (Ch1 - Ch5 in fig. 1): video, sound, electroencephalogram (EEG), muscle curve (EMG) and information (testee's report).
A personal computer B serves to present visual or acoustic incentives and contains a base of incentives, as well as all software necessary for their reproduction. A special device [4, p. 78] delivers olfactory incentives to a testee. The main workstation A controls the process of presenting olfactory incentives.
Each experiment session has a specially prepared
scenario (Table 2). The workstation A receives
biomedical signals from all channels used in the current
experiment. The signals are stored in the appropriate
database of testees. The received signals are processed
and cleared of interference and artifacts. The
bioengineering system software includes three groups
of modules (Modules of groups I, II and III in Fig. 1)
[
The software modules are implemented in MATLAB in C# language. The bioengineering system software is installed on the main workstation A, but can be used on any personal computer, so that processing of experimental results can be remote and in a distributed mode.
There are a lot of experiments with the bioengineering system “EEG-Speech+”. The studies include several directions: search for signs of biomedical signals related to an emotional response; determining the direction of emotion development (growth, fading).
Most biomedical signals are not stationary and irregular, i.e. a probability distribution of signal parameters is random. Therefore, the methods of nonlinear dynamics become relevant for their processing.
In particular, in order to identify individual characteristics of emotions based on the initial biomedical signal, there is a reconstruction of an attractor, which becomes an object of research later. placing electrodes for long-term recording of biomedical signals; tuning
the experiment; start recording biomedical signals.
channels selected for the
demonstration
demonstration
demonstration A short survey of a testee: confirmation of the expected emotional response neutral positive fading of positive, transition to neutral neutral negative fading of negative, transition to neutral 1 min.
stop recording biomedical signals; cutting off the channels; detaching electrodes.
Detailed survey of a testee: playback of incentives and their marking
So, a number of authors use an index of a restored
attractor correlation dimension [
The paper [
The author notes a significant increase in the correlation dimension under conditions of emotional experience comparing with a neutral state.
One of other signs of emotion recognition is the
Lyapunov exponent. The paper [
Studying of attractors reconstructed from Russian speech patterns showed that when a testee experiences positive emotions, the attractor form expands, in the case of negative one it gets narrow. Consequently, the number of points in the center changes. Thus, we can assume that a correlate of a sign of emotions can be the point density indicator of attractor trajectories.
The hypothesis was checked through the research
that involved students and postgraduates of the Tver
State Technical University at the age of 18–25. The
testees were offered to watch videos of up to 3 minutes,
which can be conditionally divided into three groups:
1. a positive incentive (k+);
2. a negative incentive (k-);
3. a neutral incentive (N).
challenge phrase.
used the indicator [
After each video the participants had to say a As a measure of the attractor density in the center, we = ⁄ , = ℎ + ⁄2,
Which is the ratio of the number of attractor points related to one of the cells of an orthogonal grid covering an attractor projection, ( ) to the cell area ( ). ℎ is the number of points inside each j-th cell. The number of points ( ) on the boundary of the j-th and j+1-th cells is divided equally between boundary cells.
The autocorrelation function determines the optimal value of the time delay τ, which varies depending on a (1) testee.
Attractor properties were analyzed using the first projection of the attractor, or rather the area of the greatest cluster of points localized near the origin of coordinates (Fig. 2).
Fig. 2. A projection of an attractor, which was reconstructed from a speech signal, with a selected area of the greatest cluster of points
The duration of each received speech record for analysis was 20,000 readings (≈1 seconds). The records went through auto-normalization with the removal of artifacts.
It has been experimentally established that the
presence of a noise component does not affect the
classifying ability of the parameter [
In total, we analyzed 74 speech signal fragments from 8 testees (3 incentives for each sign of emotions).
It should be noted that a negative incentive causes an increase of the index in relation to a neutral state (from 2 to 55%) almost in all testees. On the contrary, with a positive video incentive, this parameter tends to decrease (from 5 to 38%). The obtained result confirms the hypothesis about the interrelation between an emotional impact sign and an attractor density.
Similar experiments were performed with samples of
voice recordings from the international database
EmoDB [
Unlike the samples of Russian speech, German speech is characterized by an increase (on average by 20%) of the number of points in the attractor center affected by positive incentives in relation to a neutral state. Negative incentives also cause an increase in density (on average by 10%).
Conclusion. It is established that the sign of emotions significantly affects the number of points of the reconstructed attractor in the center. This is true both for Russian speech samples and for studying phrases in German. The density parameter available from experiments can be used to construct a classifier.
In a series of experiments, a testee was consistently presented with several negative incentives (-E), and then several positive ones (+E). Before changing an incentive sign, a testee was presented with neutral frames with a green background. Each experiment lasted no less than 20 and not more than 25 minutes.
While watching incentives, testee’s EEG was continuously recorded. His speech was recorded after each incentive. The processing of the experimental results had two stages.
The first stage included creating fragments of biomedical signals free from noise (for speech signals) and artifacts (for EEG signals).
Perception of incentives of the same sign (-E or +E) resulted in sequences of EEG fragments. Their characteristics contain information on changes in testee’s emotional responses.
The second stage of processing the experimental results included identification and quantitative evaluation of these latent characteristics. The bioengineering system “EEG-Speech+” provides calculation of signal power spectral analysis (EEG or speech signals), as well as attractor reconstruction based on them.
Figure 4 shows a projection of an attractor constructed from an EEG fragment (lead C4-A2), which is correlated with the terminal part of the first negative incentive. 4d0e0n0sity, ρj 3500 3000 2500 2000 1500 1000 500
0 9d0e0n0sity, ρj 8500 8000 7500 7000 6500 6000 5500 5000 4500
Negative
Neutral Positive a Positive b Negative Neutral
A series of experiments included using 2–4-minute video clips with sound as emotiogenic incentives. The testees were TSTU students and postgraduates aged 18–25.
Each video incentive was pre-marked by a testee according to a sign of an emotional response.
Fig. 4. A projection of an attractor constructed from an EEG signal (lead C4-A2)
The experiments showed that leads F7-A1 and F8-A2 had the strongest changes in power spectra when a testee was watching positive and negative incentives.
However, the reproducibility of this result was not
high. Therefore, each lead had reconstructed attractors
with their properties depending on the sign of testee’s
emotional response, as shown in previous studies [
To characterize attractors, we used the features
proposed in [
Observation of changes in the signs of and k0 showed that in most experiments there is their correlation with a sign of an emotional response.
When a testee experiences positive emotions, k0 decreases. It increases during experiencing negative emotions.
Conclusion. Preliminary results show the possibility of using an attractor density as a sign of EEG signals, illustrating the development of an emotional state at a certain time interval. The observation interval does not have imposed limitations.
The experimental results are a basis for a multimodal
emotion database, which contains examples of signals
with a bright and slightly expressed emotional color. At
the first stage, the database has speech patterns and
associated EEG patterns [
The examples of emotional responses in the database are not labeled with the names of emotions (“anger”, “fear”, “joy”, etc.). We use only natural emotional responses, so we determine the valence of an emotion (positive, negative or neutral) and its level (strong, weak, etc.).
The multimodal emotion database includes: 266 patterns of a challenge phrase lasting 2–6 seconds, pronounced by different speakers who are not actors in response to a presentation of a video incentive; 2660 vowel phonemes lasting 0.025–0.25 seconds segmented from challenge phrases; 240 EEG patterns cleared from artifacts lasting for 12 seconds.
Since 2016, there is a public database containing
examples of emotional responses [
There are up to 10 samples for certain testees. Emotiogenic incentives are specially prepared videos with sound, which cause positive, negative and neutral emotional states. 2. Recordings of speech signals (in .wav) and EEG signals (in .txt) of 9 testees. There are several recording sessions for certain testees. Registration of speech and EEG was parallel. Emotiogenic incentives were also videos with sound. Parallel recording of speech signals and EEG allowed objectively fixing the presence of positive, negative and neutral responses of testees to incentives.
Acknowledgments. The reported study has been funded by RFBR according to the research projects: № 17-0100742, № 18-37-00225.