=Paper=
{{Paper
|id=Vol-2590/short34
|storemode=property
|title=Data Collection And Processing Problems in Automatic EEG Emotion Recognition
|pdfUrl=https://ceur-ws.org/Vol-2590/short34.pdf
|volume=Vol-2590
|authors=Alexander Sergeev,Andrey Bilyi
|dblpUrl=https://dblp.org/rec/conf/micsecs/SergeevB19
}}
==Data Collection And Processing Problems in Automatic EEG Emotion Recognition==
https://ceur-ws.org/Vol-2590/short34.pdf
Data Collection And Processing Problems in
Automatic EEG Emotion Recognition
Alexander Sergeev[0000−0001−5864−494X] sergeevalxndr@yandex.ru and
Andrey Bilyi[0000−0002−6133−4368] bilyi andrei@mail.ru
ITMO University, 49 Kronverksky Pr., St. Petersburg, 197101, Russia
Abstract. Research in the field of automatic recognition of emotions is
quite important nowadays and can be used in a various field from mili-
tary affairs (polygraphs) to medicine (psycho-correction) and entertain-
ment. With the development of computer technology, it became possible
to create a system that will automatically detect emotions in real time.
However, many of the modern techniques of emotion detection strongly
rely on a huge amount of valid and suitable data to process. Thus, the
study of methods for obtaining data of sufficient quality is necessary.
The article Is devoted to an analysis of the methods and problems of de-
termining the emotional state of a person using electroencephalography.
The problems of calling and evaluating emotions, as well as data pro-
cessing when developing a software module for determining emotions,
are considered. As a result of the analysis, recommendations for data
collection were made, key features for machine learning algorithms were
determined and a program of experiments for data collection was devel-
oped.
Keywords: Emotional state · Emotion recognition · Electroencephalog-
raphy · Psycho-correction
1 Introduction
Undoubtedly, emotions play a big role in everyday life. Their influence extends
to various aspects of human life, from communication and interaction with peo-
ple to decision making. Information about the experienced emotions is widely
used in various fields of society. For example, determining a response to a given
stimulus can help correct a psycho-physiological state, allow an assessment of an
attitude to the environment (workplace, equipment used) and draw appropri-
ate conclusions, or contribute to improving the content recommendation system
through more detailed feedback from users.
Copyright c 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0).
2 A. Sergeev, A. Bilyi
Thereby, automatic and accurate determination of an emotional state is im-
portant. With the development of information technology and the current avail-
ability of medical and computer equipment, research in this area is also becoming
more accessible and attractive.
2 Subject area
Achievements in the field of neurophysiological studies of human brain activ-
ity are significantly lagging behind in depth and completeness of data from the
same animal studies. Also, the need for the practical application of these methods
(medicine, optimization of operator activity, the creation of neurocomputer in-
terfaces) necessitates the continued of fundamental research on the human brain.
Methods and technologies for determining emotions can be applied in various
areas of society, from law enforcement agencies (various polygraphs) to enter-
tainment field (environmental regulation in games based on current emotional
state) and medical institutions (psychocorrection). However, the most relevant
area of application of this study is the field of study of the functional state of a
person.
Recognition of emotions can be carried out on the basis of audiovisual meth-
ods, such as recognition and analysis of facial expressions, speech, body language
and others, but these methods do not always allow us to give a reliable assess-
ment of a person’s current emotional state. Facial expressions, speech, body
movements, and other external physical characteristics can be easily altered if
desired, and can also be misinterpreted according to the true emotion experi-
enced. The mental state is more difficult for a person to control. Therefore, the
determination of the emotional state by measuring the bioelectric activity of the
brain is of great scientific and practical value.
The information theory of emotions by Simonov P.V. was chosen as the main
theoretical base of this study. According to this theory, emotion is a reflection
by the human or animal brain of any actual need (its quality and magnitude)
and the likelihood (possibility) of its satisfaction, which the brain evaluates on
the basis of genetic and previously acquired individual experience [1]. In general,
the rule for the emergence of emotions can be represented in the formula:
E = f [N, (Ir − Ih), ...], (1)
where E is emotion; N = strength and quality of an actual need; (Ir-Ih) - assess-
ment of the ability to meet needs based on innate and ontogenetic experience, Ir
- information about means required to achive the need; Ih - information about
means to achive the goal that subject has at the current moment.
3 Problems of determining emotions using EEG
The main problems in the field of determining emotions using EEG are:
– The complexity of invoking a specific emotion
Title Suppressed Due to Excessive Length 3
– The complexity of an objective and unambiguous assessment
– Data processing
EEG-based emotion recognition imposes certain restrictions on the available
methods. Working with electrical signals involves invoking a specific emotion. In
the case of facial recognition, emotion can be reproduced using Paul Ekkman’s
[FACS] system [3], however, such methods are not applicable here. In this case,
to extract pure emotion, it is necessary to immerse a person in a situation where
he can experience it, to make him feel the need for something that can cause
emotion. For example, when displaying a video in which any injustice is shown,
a person will be empathic for the events in the video and is likely to experi-
ence positive emotions if the injustice is eliminated, or negative emotions if the
situation is not resolved well.
Before conducting experiments, it is recommended to conduct an initial di-
agnosis of the test subjects: to determine the interests, personal preferences and
other psychological characteristics of the subjects. Based on this information, it
will be possible to select the stimuli that are best suited for a particular subject.
At this stage, you can also determine the temperament of the subject using the
test, which can help in the future with training machine learning model. Some
other parameters such as gender, age and dominant hand can also complete the
feature set. The last parameter can greatly affect on the outcome result, since
the work of the cerebral hemispheres is different for each right- and left-handed
people.
To invoke a specific emotion, it is necessary to identify stimuli. It can be
presented in various forms of media content (music, films, pictures, computer
games), as well as in some actions with the test subject. For example, a compli-
ment can cause embarrassment or joy, as well as offering the test subject a cup
of tea. It is worth noting that different stimuli have different levels of immersion
and have different effects on the emotional state. For example, a funny or sad
music video can cause negative emotions instead of joy or sadness for the reason
that the subject does not like this musical genre. Computer games provide the
highest level of immersion, but may require the participation of the test person
in them, and as a result, interference and artifacts on the EEG signal may occur.
Video materials do not imply the active participation of the test subject, and
involve only observation, and therefore are the most appropriate stimulus. An
important point is also the entry barrier of the stimulus. Funny or disgusting pic-
tures are more effective and productive for invoking emotion than a ten-minute
video with some story.
The big problem is the correct assessment of experienced emotions. To train
the machine learning model, manual control and markup of data according to
the ”presentation of the stimulus - assessment” model is assumed. Control can
be carried out by various methods:
– Survey of the test subject (which of the following emotional states have you
experienced? Rate the strength and sign of the experienced emotion on a
scale of 1-10)
4 A. Sergeev, A. Bilyi
– Data on heart rate, HRV, ECG, etc. (subject claims he had experienced
a sheer emotion, while his physical indicators tell otherwise - it is worth
considering the correctness of the answer)
– Subject’s face expressions (subject with an emotionless face claims he had
experienced a sheer emotion, or states with obvious facial expressions that
the stimulus did not cause him any emotions)
At this stage, it is necessary to reject possible incorrect reactions, and in some
cases make a decision on the basis of subject’s physical condition, rather than on
the basis of his answers. An important role in determining the reaction is played
by the fact that the subject knows about the conducting of the experiment and
can change his answers willing to help the researcher or, vice versa, hide his
emotions for some personal reasons.
During the tests, it is possible to oversaturate the subject’s emotional state,
worsen the emotional reaction and, as a result, receive incorrect data. This can
happen with the continuous presentation of several stimuli of the same type in a
row or with insufficient rest time of the subject. Thus, it is necessary to alternate
stimuli of different types in random order, as well as give the subject time to
“reboot” and relax. The recommended relaxation time is one minute, however,
the time may vary depending on the state of the subject. Relaxation should
occur with closed eyes and clean thoughts, however, this process is fully depends
on the test subject.
4 Data collection and processing
When developing a software module for emotion recognition, the methods of
data processing are important. The most popular processing method is building
and training a machine learning model. Some researchers use K-nearest neigh-
bours[6] and support vector machine methods[5]. Tools for data gathering are
also important. So, for example, different EEG devices will have a different num-
ber of electrodes and different locations for it, which makes it difficult to develop
a universal tool that can work with different types of equipment.
The feature selection is very important. First of all, it is necessary to choose
those electrodes that are closest to the centers of emotion processing, namely:
electrodes near the thalamus, hypothalamus, frontal and middle lobes of the
brain. The selection of all available electrodes may be redundant and lead to
retraining of the ML model, so the least important electrodes can be excluded.
Of the additional features, you can take such parameters as a person’s tempera-
ment. For this, the subjects will need to undergo appropriate testing in advance.
Additional ECG electrodes, heart rate or HRV sensors are also can be selected
as a features. However, this also imposes a limitation on the equipment used.
There are several models for classifying emotions that can be used in this
study:
– There is a circumplex model that compares a qualitative assessment of emo-
tion with its arousal and valence.
Title Suppressed Due to Excessive Length 5
– There is a vector model, that consists of vectors that point in two directions,
representing a ”boomerang” shape. The model assumes that there is always
an underlying arousal dimension, and that valence determines the direction
in which a particular emotion lies.
– Positive activation - negative activation (PANA) model suggests that posi-
tive affect and negative affect are two separate systems. Similar to the vector
model, states of higher arousal tend to be defined by their valence, and states
of lower arousal tend to be more neutral in terms of valence in the PANA
model, the vertical axis represents low to high positive affect and the hori-
zontal axis represents low to high negative affect.
Lets take the circumplex model as example (shown in Fig 1). Arousal and Valence
can be used as features for the ML model. Since Beta rhythms are associated with
an activity, and alpha rhythms with a relaxed state, arousal state of the emotion
can be characterized by a large activity of beta rhythms and low activity of
alpha rhythms [2]. Thus, the ratio of beta / alpha rhythms can be an objective
indicator of arousal. The calculation of the arousal parameter is presented in
formula 2.
sum(β)
Arousal = , (2)
sum(α)
Fig. 1. Circumplex model α4 α3
V alence = − , (3)
β4 β3
Studies by psychophysiologists also show that activity in the right and left
hemispheres is associated with sensory and logical thinking, these types of think-
ing lead to appropriate behavior, and this is already associated with a feeling
of positive and negative emotions, respectively[4]. By determining the ratio of
beta / alpha rhythms, you can calculate the inactivity of the right hemisphere,
and from it to subtract the inactivity of the left hemisphere, getting the value
of valence in result. The calculation of the valence parameter is presented in
formula 3.
Many factors influence the EEG data. So, for example, physical activity is
more reflected on the EEG than mental and emotional ones, so the movement
of a person (blinking, movement of the body and limbs, swallowing, etc.) dur-
ing experiments can lead to artifacts in a certain time period, therefore, random
6 A. Sergeev, A. Bilyi
unwanted movements on the part of the subject should be kept to a minimum. In-
ternal defects in the EEG equipment and the electrical background from nearby
third-party equipment (mobile phones, laptops) can also distort the signal, so the
readings must additionally go through several stages of preliminary processing.
For instance, the bioelectrical activity of the brain is defined in the range from 1
to 40 hertz among Delta, Theta, Alpha and Beta rhythms. Therefore, all signals
that are not in this range can be taken as noise and artifacts and be excluded
from the processing. It can be achieved by using low and highpass filters. Among
the signal processing methods, some researchers use the Fast Fourier Transform
and Discrete Wavelet Transform [5] to divide the signal to the specific bands.
Additionally, these bands can be divided into a several sub-bands, and its sum-
mary power can be calculated and used as a feature.
Thus, to obtain the correct data set, it is necessary to take into account:
– Experiment program;
– Selection of subjects;
– Set of stimuls;
– Preliminary manual data evaluation;
– Feature selection;
– Processing interference and artifacts;
5 Experiment program
During the analysis, some key points for data collection and analysis were identi-
fied, and an experiment program and some requirements for its implementation
were created. The experiment for a single subject is the following sequence of
actions:
1. Connecting a test subject to an EEG device, connecting electrodes, checking
equipment.
2. Recording background activity of the brain. The subject remains motionless
with his eyes closed for several minutes.
3. Analysis of the condition and behavior of the subject. Needed to determine
the most necessary emotional reactions at the moment.
4. Presentation of stimuli. It is carried out using the display of a computer/
laptop. The subject concentrates on the screen and prepares to perceive the
information.
5. Assessment of the induced reaction.
6. Subject recovery, similar to pt.2.
7. Repeat paragraphs 3-6.
Thus, taking all notes into account, the correct data will be collected for
training the machine learning model. After this, it is necessary to conduct an
initial analysis of the data, to classify the results according to the responses of
the researcher and the subjects.
Title Suppressed Due to Excessive Length 7
6 Conclusion
The article identified and analyzed the main problems in determining emotion
using EEG signals and an experiment program was created. Based on the in-
formation obtained during this study, it is planned to conduct experiments to
collect data and develop a software module for determining emotions using EEG.
References
1. Simonov P.V.: The emotional brain. Physiology. Neuroanatomy Psychology of emo-
tions, Science, St. Petersburg, 1981
2. Simonov P.V.: The highest nervous activity of man. Motivational and emotional
aspects. Science, 1975
3. Facial Action Coding System - Paul Ekman Group
https://www.paulekman.com/facial-action-coding-system. Last accessed 01
Nov 2019
4. Ekman, P.: Psychology of emotions. St. Petersburg, Peter, 2011
5. Matlovic T., Gaspar P., Simko Y., Bielikova M., Moro R.: Emo-
tions Detection Using Facial Expressions Recognition and EEG, 2016,
http://www2.fiit.stuba.sk/ bielik/publ/abstracts/2016/smap2016-matlovic-et-
al.pdf. Last accessed 02 Nov 2019
6. Bombatkar, A., Bhoyar, G., Morjani, K., Gautam, S., Gupta, V.: Emotion recog-
nition using Speech Processing Using k-nearest neighbor algorithm, IJERA, 2014
https://pdfs.semanticscholar.org/a1f5/c39aece58f6504e0334d96eaede32c7329cf.pdf.
Last accessed 02 Nov 2019