=Paper=
{{Paper
|id=Vol-2790/paper35
|storemode=property
|title=
Data Augmentation for Domain-Adversarial Training in EEG-based Emotion Recognition
|pdfUrl=https://ceur-ws.org/Vol-2790/paper35.pdf
|volume=Vol-2790
|authors=Ekaterina I. Lebedeva
|dblpUrl=https://dblp.org/rec/conf/rcdl/Lebedeva20
}}
==
Data Augmentation for Domain-Adversarial Training in EEG-based Emotion Recognition
==
https://ceur-ws.org/Vol-2790/paper35.pdf
Data Augmentation for Domain-Adversarial
Training in EEG-based Emotion Recognition
Ekaterina Lebedeva
Moscow State University, Moscow, Russia,
kate.1ebedeva@yandex.com
Abstract. Emotion Recognition is an important and challenging task
of modern affective computing systems. Neuronal action potentials mea-
sured by the Electroencephalography (EEG) provide an important data
source with a high temporal resolution and direct relevance to a human
brain activity. EEG-based evaluation of the emotional state is compli-
cated due to the lack of labeled training data and to a strong presence of
subject- and session-dependencies. Various adaptation techniques can be
applied to train a model that would be robust to a domain mismatch in
EEG data but the amount of available training data is still insufficient.
In this work we propose a new approach based on the domain adversarial
training and combining available training corpus with much larger unla-
beled dataset in a semi-supervised training framework. A detailed analy-
sis of available datasets and existing methods for the emotion recognition
task is presented. The effect of emotion recognition performance degra-
dation caused by the subject- and session-dependencies was measured
on DEAP dataset proving the need to develop approaches that would
utilize larger datasets in order to obtain a better generalized model.
Keywords: Electroencephalography (EEG) · Emotion recognition · Sig-
nal processing · Deep learning · Domain adaptation.
1 Introduction
Recently, there has been growing interest in using the EEG signal to analyze
the functioning of the human brain. The results of EEG processing began to
be used in the creation of brain-computer interfaces (BCIs) and in neurophys-
iology studies. Emotion recognition is one of the essential tasks in these fields.
Works on affective disorders report that analysing EEG signal during emotion
task manipulations could provide an assessment of risk for major depressive
disorder [1]. There are many works on the subject of affective brain-computer
interactions. The authors of these works believe that recognizing emotions from
EEG signal will allow robots and machines to read people’s interactive intentions
and states and respond to human emotions [2–4]. Moreover, solving the prob-
lem of recognizing emotions may contribute the development of neuromarketing
to determine consumer preferences [5]. And another area of task application is
workload estimation [6] and driving fatigue detection [7].
Copyright © 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
397
1.1 Electroencephalography
Electroencephalography is a multichannel continuous signal recorded with elec-
trodes that measures differences between electric potentials that are registered
in two areas of the brain. During this recording, electrodes are placed on the
surface of the scalp. To improve the conductivity of the skin, a gel is applied
to the contact surface of the electrodes. Elastic helmets are used to fixate the
electrodes on the head. In recent years a number of accessible consumer-level
brain-computer interfaces (BCI) became available on the market [8–10]. These
devices usually include a fewer number of electrodes that often are used with-
out conductive/adhesive gel. It makes BCI technology cheaper and more afford-
able. Due to this, there is a trend that more data is available. EEG record-
ing is always contaminated with artifacts, such as EOG(ocular), ECG(cardiac),
EMG(muscle), and noise. Therefore, work pipeline should contain signal prepro-
cessing to automatically handle this problem. Different processes are reflected
in different frequency bands of the electrical activity of the brain. For example,
alpha rhythm (8 to 12 Hz) reflects to attentional demands and beta activity (16
to 24 Hz) reflects to emotional and cognitive processes in brain [11].
As possible variants of the experimental protocol the following systems for
recording EEG signals are used:
1. Resting states with eyes open (REO) or with eyes closed (REC). The patient
is relaxed state and does not think about anything. This procedure is used
to analyze the general condition of the patient. And it is suitable for anyone,
including people with disabilities.
2. Event-related potentials (ERPs) [12]. In such experiments, a signal is sent
from a computer representing a stimulus to a computer recording an EEG
whenever a stimulus or response occurs. Such stimuli may be periodic light
exposure at different values of the frequency of exposure. Segments of EEG
data that are time-locked to the event signals are extracted from the overall
EEG and averaged.
3. Task-related. Neural activity is recorded under various cognitive tasks. The
patient should also be relaxed and his attention should be focused only on
the implementation of the task. These can be tasks such as counting in the
mind or reading.
4. Somnography [13]. EEG is recorded during sleep stage. The sleep electroen-
cephalogram (EEG) can be recorded for analyzing the stages of sleep or the
causes of sleep deprivation.
1.2 Emotions
Emotion is a mental state and an affective reaction towards an event based on
a subjective experience. It is hard to measure because it is a subjective feeling.
Emotions can be evaluated in terms of ”positive”, ”negative” or ”like”, ”dis-
like” [5]. It is also possible to distinguish a set of basic emotions such as anger,
fear, sadness, disgust, happiness, surprise [34] and try to solve the classifica-
tion problem. Researchers often use a two- or three-dimensional space to model
398
emotions [14, 15], where different emotion points can be plotted on a 2D plane
consisting of a Valence axis and Arousal axis (Fig. 1a) or on a 3D area with
addition Dominance axis (Fig. 1b).
(a) 2D (b) 3D
Fig. 1: Emotion space models [16].
1.3 Methods
One of the earliest works on emotion recognition from EEG was presented in
1997 [17]. Machine-learning approach is one of the classic ways for solving the
problem of emotion recognition. In this method it is necessary to extract reliable
informative features closely related to the emotional state of the subject. The sig-
nal is divided into components by Independent Component Analysis (ICA) [19,
20] to separate artifacts. The main method used for extracting spectral fea-
tures is the Fourier transform [32]. A detailed description of popular features
for analysing EEG signal is presented in the article [18]. In machine-learning
approach, discriminant analysis [35] or Bayesian analysis [34] can be used for
classification.
In addition to obtaining ML features, deep learning methods can be used.
And that is a more modern way of solving the problem. This approach is often
used as conjunction machine learning feature extraction with neural network
classification. For classification such neural networks as SAE [21], LSTM [22] are
often configured [37, 39, 23]. A fully neural network solution has recently been
proposed: SAE+LSTM method [23] on DEAP dataset. In this work Stacked
AutoEncoder (SAE) is used for solving ICA problem and the emotion timing
modeling is based on the Long Short-Term Memory Recurrent Neural Network
(LSTM-RNN).
1.4 Data Augmentation
Neural networks require a big amount of training data. Usually EEG datasets
contain data from a small number of subjects. This is due to the fact that special
399
devices and the correct experimental conditions are required to collect the data.
Several datasets could be combined to increase the amount of training data.
But each dataset was collected by different devices, with different experimental
protocol and different stimuli. Therefore, it is difficult to conduct training on
data from several sources. Another problem is the low accuracy of prediction
for subjects whose data were not available in the training set. Various domain
adaptation techniques are used to reduce data variability [40].
The volume of the union of datasets labeled by emotions is still not large
enough. The solution for expanding data with other EEG datasets is proposed
in this work. It is possible to use EEG datasets without emotional labels if they
contain video recordings of the experiment. Data can be marked by emotions
detected from the video, the similar approach was suggested for problem of
emotion recognition from speech [24]. It increases the amount of work, but helps
to expand the training set.
2 Datasets
There are several datasets for EEG-based emotion recognition task. Every corpus
was collected according to unique protocol. Available datasets for solving the
problem are described below.
2.1 DEAP
DEAP dataset (A Database for Emotion Analysis Using Physiological Signals) [25]
is a widely used in EEG-based emotion recognition area [39, 23, 40]. This dataset
was collected as a part of an adaptive music video recommendation system de-
velopment. The experiment was attended by 32 people. Data was collected from
subjects while watching 40 one-minute music videos stimuli. During the exper-
iment, participants performed self-assessment of their levels of arousal, valence
and dominance (Fig. 2). As a result, 32-channel electroencephalogram and pe-
ripheral physiological signals were recorded. For 22 of the 32 participants, frontal
face video was also recorded. Dataset is convenient, as it contains not only orig-
inal data in BDF (Biosemi Data Format), but also preprocessed data in MAT-
LAB and Python formats. Dataset is open only for academic research and it is
available for download after signing the EULA (End User License Agreement).
2.2 eNTERFACE-2006
Another popular dataset was made as a part of eNTERFACE-2006 project [26].
The purpose of the project is to collect a sufficient data to build an integrated
framework for multi-modal emotion recognition. Data collection was carried out
for 5 male subjects in 3 sessions. Stimuli are images from the IAPS (Interna-
tional Affective Picture System) [27] which consists of 1196 pictures evaluated in
arousal-valence dimensions. For experiment 3 groups of images were selected: 106
calm, 71 positive exciting, 150 negative exciting. Each session lasted 15 minutes
400
Fig. 2: The Self-Assessment Manikin(SAM) for rating the affective dimensions
of valence, arousal, and dominance levels [30].
and consisted of 30 blocks, each block is succession of 5 images corresponding
to a single emotion. EEG and fNIRS signals with peripheral information were
recorded in ..bdf format. Eventually the data were marked not only with a pre-
liminary evaluation of the images, but also with participants self-assessment.
2.3 SEED, SEED-IV
SEED (SJTU Emotion EEG Dataset) [28, 38] contains data from 15 subjects
in 3 sessions with an interval of about one week. As stimuli, 15 video clips
lasting 4 minutes were selected. During the experiment, subjects conducted self-
assessments based on “positive,” “negative,” or “neutral” terms for evaluating
emotions. Dataset contains preprocessed EEG in 45 .mat (Matlab) files. EEG
data is downsampled, preprocessed and segmented. In addition, the dataset com-
prises files with extracted features. It contains the features of differential entropy
(DE) of the EEG signals, which is convenient for testing the classifiers.
SEED-IV [29] is another dataset collected later. In this experiment, a differ-
ent system of emotion classification was used: happy, sad, neutral, fear. And in
addition to EEG, eye movement information was recorded with the eye track-
ing glasses, that makes SEED-IV multi-modal dataset for emotion recognition.
Dataset contains EEG raw data, extracted features from EEG (differential en-
tropy and power spectral density) and raw data and extracted features of eye
movements, all in .mat format. Both of these datasets can be downloaded after
signing the license agreement.
2.4 Neuromarketing
Neuromarketing is the field of marketing research that helps to determine con-
sumers’ preferences and predict their behavior using unconscious processes, which
ensures effective utilization of the product. In [5] The Neuromarketing dataset
was created for building predictive modeling framework to better understand
consumer choice. This corpus of data was made by recording an EEG signal
401
from 40 subjects while viewing consumer products. During the experiment, par-
ticipants marked E-commerce products in terms of “likes” and “dislikes”. The
resulting dataset is publicity available and can be used in scientific works and
marketing researches.
2.5 Imagined Emotions
A different experiment design that included cue-based emotion stimuli was pre-
sented in [31]. Each participant listened to a sample of voice recording that
suggested a specific emotional state. A participant had to imagine a correspond-
ing emotional scenario or to recall a related emotional experience. The presented
dataset consists of EEG signals collected from 32 subjects who have experienced
15 emotional states, and participants’ assessments of the authenticity and inten-
sity of the tested emotions on a scale of 1 to 9.
3 Related Works
Emotion recognition is an analysis of multi-channel samples of EEG data. Each
sample is considered to have a single emotional state that is supposed to be
constant during the recording. Depending on the system of classification of
emotions that was used in the experiment design, either the emotion must be
determined from a preassigned set, or an assessment should be given on the
Arousal-Valence(-Dominance) scales. Thus, the emotion recognition task can be
considered a classification or a regression problem.
3.1 Preprocessing and Feature Extraction
Electroencephalogram data consists not only of the recordings of brain activity
but also of a number of artifact components of various origins. Therefore the ex-
tensive filtering and artifact removal procedures must be included as a necessary
part of the analysis pipeline. Deletion of recording sections with artifacts can
be performed by specialists but that requires a thorough and expensive analysis
of each sample. After the initial cleaning step, the multi-channel signal can be
decomposed into quasi-independent components by solving a blind source sep-
aration task. This can be achieved with the Independent Component Analysis
(ICA) or with more recent autoencoder-based approaches.
During the feature extraction step, EEG signal is divided into short time
frames. The EEG features are extracted from each frame and combined into a
feature sequence. The signal is represented as a set of overlap frames using the
window function. It can be a rectangular window, but usually a smoothing win-
dow, such as the Hanning window, is used. For spectral analysis of the EEG data,
the Fourier transform [32] is used to obtain a frequency domain representation
of each window. Then, feature extraction can be performed independently for
each frequency band. Following metrics and statistics can be utilized as informa-
tive features: max, min, average amplitude and Power spectral density (PSD).
Following cross-channel features can be calculated:
402
1. Root Mean Square
v
u n
u1 X
RM S = t S2 (1)
N n=1 n
where Si — ith channel amplitude
2. Pearson Correlation Coefficient between 2 channels
PN
i=1 (xi − x)(yi − y)
P CC = qP qP (2)
N 2 N 2
i=1 (xi − x) i=1 (yi − y)
3. Magnitude Squared Coherence Estimate
|Pi j|
M SCE = (3)
Pi · Pj
where Pij — cross-PSD i, j th channels, Pi — PSD ith channel
A more detailed review of feature extraction methods can be found in [18].
3.2 Model Training
Emotion recognition problem in feature space can be approached with one of
the machine learning methods for classification. In [34] an emotion recognition
method using a Naive Bayes model was proposed. The classification problem
under the maximum likelihood framework was formulated as:
yb = arg max P (X|y) (4)
y
where y is label and X is feature vector. The Naive Bayes framework assumes
that the features in X are independent of each other conditioned upon the class
label. This paper compares two model distribution assumptions. It is shown that
the Cauchy distribution assumption typically provides better results than the
Gaussian distribution assumption.
In [35] a comparison of K Nearest Neighbours classifier and Linear Discrimi-
nant Analysis is presented. The experiment was conducted on a private dataset
and showed the maximum average classification rate of 83.26% using KNN and
75.21% using LDA. These solutions are suitable for the classification problem
when it is necessary to recognize emotion from a given set. If affective labeling
is presented as a vector of real values (such as Arousal-Valence scale), this ap-
proach can also be applied with regression methods instead of classification [36].
Despite this, labels are often made binary when evaluating the accuracy of an
algorithm.
403
3.3 Deep Learning Approach
Today, neural network algorithms are used everywhere, since they can recognize
deeper, sometimes unexpected patterns in data. And in the studied area, deep
neural network-based feature extraction and emotion recognition began to be
intensively applied.
In [38] Deep Belief Network (DBN) was trained with differential entropy fea-
tures. The experiment performed classification for three emotional categories on
the SEED dataset. The results show that the DBN models obtain higher accu-
racy than previously considered models such as kNN, LR and SVM approaches.
An emotion recognition system that uses deep learning models at two stages
of work pipeline was introduced in [23]. Stacked autoencoder was used for decom-
position of source signal (as a substitute for Independent Component Analysis)
and extracting EEG channel correlations. LSTM-RNN network is used for emo-
tion classification based on Frequency Band Power Features extracted from the
SAE output. The mean accuracy of emotion recognition, calculated by binarized
labels, achieved 81.10% in valence and 74.38% in arousal on the DEAP dataset.
3.4 Domain Adaptation
Training an accurate model requires an approach that would be robust to vari-
ations in individual characteristics of participants and recording devices, since
EEG data suffers from an intense dependence on the device and the subject.
It is important to apply a domain adaptation technique to a model that would
compensate the subject variability or heterogeneity in various technical specifi-
cations.
The paper [40] compares different domain adaptation techniques on two
datasets: DEAP and SEED. Transfer Component Analysis (TCA) [42] and Max-
imum Independence Domain Adaptation (MIDA) [41] performed the best results
for subject within-dataset domain adaptation. It is shown that applying these
techniques lead to an improvement gain up to 20.66% over the baseline accuracy
where no domain adaptation technique was used. A research of these techniques
application for cross-dataset domain adaptation was also conducted. The article
concluded that TCA and MIDA can effectivly improve the accuracy by 7.25%
– 13.40% compared to the baseline accuracy where no domain adaptation tech-
nique was used.
In [43] another approach to a domain adaptation was considered based on
neural networks that are trained to solve emotion and domain recognition prob-
lems. Samples of feature vectors from two domains in the same quantity are
fed to the model, producing emotion label for each EEG sample. Several first
layers of neural network act as a feature extractor, producing a fixed-dimension
representation of EEG samples in a latent space. These representations are used
to solve two different tasks: emotion label classification and domain recognition.
A gradient reversal layer is applied to the domain predictor [44] leading to the
adversarial training scheme during which the parameters of feature extractor
404
layers are updated to make embedding distributions of different domains statis-
tically similar. Fully connected layers make representations for label predictor,
which estimates emotion class for each sample. During training, samples from
one domain contains labels, whereas the second domain is unlabeled. The label
predictor is optimized to minimize the classification error on the first domain.
During test, the model inputs are unlabeled data only. This method was com-
pared with multiple domain adaptation algorithms on benchmark SEED and
DEAP and proved to be superior in both cross-subject and cross-session adap-
tation.
4 A Proposed Approach
4.1 Domain-Adversarial Training
The problem of domain adaptation is crucial in the emotion recognition task. The
architectures of the proposed approach are presented on Fig. 3, 4. This approach
combines the ideas presented in works [43] and [44]. In fig. 3 the domain classifier
predicts which domain the data belongs to and the set of feature extractor
parameters is updated by adversarial training to make the distribution of data
representations of different domains more similar. In fig. 4 the input is data from
two domains: labeled and unlabeled. Data representations of labeled domain are
sent to label predictor and domain discriminator, representations of unlabeled
data are transmitted only to domain discriminator, which determines whether
these domains match or not.
These architectures differ in that in the first case, the classification of domains
occurs independently for input samples, and in the second, pairwise comparisons
are performed. In the future work, a comparison of these two approaches will be
carried out and the best approach will be defined.
4.2 Data Augmentation
In order to improve the performance of the model, a large number of EEG
datasets without affective labels [45] can be utilized emotion recognition task.
To train a domain classifier (for subject identity recognition), more data can be
used, since there is no need for labeled data. Since a domain predictor is trained
on a much larger number of domains, and a larger amount of training samples,
it potentially can be more robust to various specific channel characteristic vari-
ability. The emotion classifier is still trained on the same amount of data, but
the performance can be improved since the latent representations are trained to
be domain-independent. DEAP dataset includes data of only 32 subjects, as well
as other datasets for EEG-based emotion recognition also contain a limited vari-
ability of subjects. At the same time, EEG datasets without affective labeling
are much larger. For example, in the Temple University Hospital (TUH) EEG
data corpus [47] there is EGG data of more than 10000 participants. It is more
efficient to train neural networks on such data volume, therefore, as the solution
405
Fig. 3: The model architecture of the domain-adversarial training with domain
classifier.
Fig. 4: The model architecture of the domain-adversarial training with domain
discriminator.
406
it is proposed to use unlabeled data. Thus, the neural network will be trained
on a larger set of subjects, and therefore, will provide a better generalized model
for new subjects.
4.3 Auto-labeling of EEG Datasets
Another possible solution is to enrich the sample for training using the multi-
modal emotion recognition. For this purpose, EEG datasets without labels, which
contain other modalities such as video recordings of a subject’s face, can be
used, for example SEED-VIG [46]. Then the data can be automatically labeled,
recognizing the emotions experienced by the participants from the video. Un-
fortunately, the EEG datasets with the recordings of such modalities are rare,
so this approach probably will not be allow to significantly expand the training
data.
4.4 A Preliminary Motivation Study
Below is an illustration of the fact that the problem of cross-subject adaptation
really requires a solution. An experiment was conducted demonstrating a de-
crease in the accuracy of emotions recognition in the absence of subject data in
the training sample. The preprocessed data from DEAP dataset was used. PSD
for five frequency bands were extracted as features. The following ML classi-
fiers were trained: SVM, Random Forest Regression. The data were divided into
training, validation and test samples in a ratio of 6 : 1 : 1 respectively. In the
first experiment, the data of each subject was divided between the samples. In
the second experiment, the data of each subject entirely relate to one or another
sample. The table 1 shows the differences in the accuracy of determining emo-
tions for these two experiments. According to the results the presence of learning
problems on isolated subjects is shown.
Table 1: Experiment results
(a) For SVM classifier
Rating scale 1st experiment 2nd experiment
Valence 68.4% 52.2%
Arousal 65.1% 57.7%
Dominance 68.9% 52.4%
(b) For Random Forest Regression
Rating scale 1st experiment 2nd experiment
Valence 83.2% 47.6%
Arousal 82.6% 60.3%
Dominance 81.8% 53.1%
407
5 Conclusion and Future Work
This paper discribes the EEG-based emotion recognition task and its existing
solution methods. There was formulated the problem of domain mismatch and
insufficient data amount for training neural networks. As a solution, there was
proposed the application of existing domain adaptation techniques with data
augmentation due to datasets without emotional labels.
In the future, it is planned to conduct testing on DEAP dataset, using TUH
EEG data corpus, to evaluate how emotion classification would be robust to
subjects and session and channel differences. It is also planned to use the SEED
dataset and perform the same analysis to study the task of training a dataset-
independent emotion recognition model. A detailed validation study will be per-
formed to compare the results with existing methods of domain adaptation.
References
1. Stewart, J. L., Coan, J. A., Towers, D. N., Allen, J. J. B.: Frontal EEG asymme-
try during emotional challenge differentiates individuals with and without lifetime
major depressive disorder. Journal of Affective Disorders 129(1-3), 167–174 (2011).
https://doi.org/10.1016/j.jad.2010.08.029
2. Yin, Z., Wang, Y., Liu, L., Zhang, W., Zhang, J.: Cross-Subject EEG Feature Selec-
tion for Emotion Recognition Using Transfer Recursive Feature Elimination. Fron-
tiers in Neurorobotics 11 (2017). https://doi.org/10.3389/fnbot.2017.00019
3. Urgen, B., Plank, M., Ishiguro, H., Poizner, H., Saygin, A.: Eeg theta and mu oscil-
lations during perception of human and robot actions. Frontiers in Neurorobotics 7
(2013). https://doi.org/10.3389/fnbot.2013.00019
4. Calvo, R. A., D’Mello, S.: Affect detection: An interdisciplinary review of models,
methods, and their applications. IEEE Transactions on affective computing, 1(1),
18–37 (2010).
5. Yadava, M., Kumar, P., Saini, R. et al: Analysis of EEG signals and its application
to neuromarketing. Multimedia Tools and Applications 76, 19087–19111 (2017).
https://doi.org/10.1007/s11042-017-4580-6
6. Kothe, C. A., Makeig, S.: Estimation of task workload from EEG data: new and
current tools and perspectives. In: Annual International Conference of the IEEE
Engineering in Medicine and Biology Society, pp. 6547–6551. EMBC, IEEE (2011).
7. Shi, L. C., Lu, B. L.: EEG-based vigilance estimation using extreme learning ma-
chines. Neurocomputing 102, 135–143 (2013).
8. Muse Homepage, https://choosemuse.com. Last accessed 30 May 2020
9. Emotiv Homepage, https://www.emotiv.com. Last accessed 30 May 2020
10. Neurosky Homepage, http://neurosky.com. Last accessed 30 May 2020
11. Ray W. J., Cole H. W.: EEG alpha activity reflects attentional demands, and
beta activity reflects emotional and cognitive processes. Science 228(4700), 750–
752 (1985).
12. Lopez-Calderon, J., Luck, S. J.: ERPLAB: an open-source toolbox for the
analysis of event-related potentials. Frontiers in Human Neuroscience 8 (2014).
https://doi.org/10.3389/fnhum.2014.00213
13. Brunner, D. P., Münch, M., Biedermann, K., Huch, R., Huch, A., Borbély, A. A.:
Changes in Sleep and Sleep Electroencephalogram During Pregnancy. Sleep 17(7),
576–582 (1994). https://doi.org/10.1093/sleep/17.7.576
408
14. Lang, P. J.: The emotion probe: studies of motivation and attention. American
psychologist 50(5), 372 (1995).
15. Scherer, K. R.: What Are Emotions? And How Can They
Be Measured? Social Science Information 44(4), 695–729 (2005).
https://doi.org/10.1177/0539018405058216
16. Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Yang, X.: A review of emotion
recognition using physiological signals. Sensors 18(7), 2074 (2018).
17. Musha, T., Terasaki, Y., Haque, H. A., Ivamitsky, G. A.: Feature extraction from
EEGs associated with emotions. Artificial Life and Robotics 1(1), 15–19 (1997).
18. Jenke, R., Peer, A., Buss, M.: Feature extraction and selection for emotion recog-
nition from EEG. IEEE Transactions on Affective computing 5(3), 327–339 (2014).
19. Hyvärinen, A., Oja, E.: Independent component analysis: A tutorial. LCIS, Helsinki
University of Technology, Finland (1999).
20. Vinther, M.: Independent Component Analysis of Evoked Potentials in EEG.
Orsted, DTU (2002).
21. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P. A., Bottou, L.:
Stacked denoising autoencoders: Learning useful representations in a deep network
with a local denoising criterion. Journal of machine learning research 11(12), 3371–
3408 (2010).
22. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computing 9(8),
1735–1780 (1997).
23. Xing, X., Li, Z., Xu, T., Shu, L., Hu, B., Xu, X.: SAE+LSTM: A New Frame-
work for Emotion Recognition From Multi-Channel EEG. Frontiers in Neurorobotics
13(37) (2019). https://doi.org/10.3389/fnbot.2019.00037
24. Albanie, S., Nagrani, A., Vedaldi, A., Zisserman, A.: Emotion recognition in speech
using cross-modal transfer in the wild. In: Proceedings of the 26th ACM interna-
tional conference on Multimedia, pp. 292–301. ACM Press (2018).
25. Koelstra, S., Muhl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., Pa-
tras, I.: Deap: A database for emotion analysis; using physiological signals. IEEE
transactions on affective computing 3(1), 18–31 (2011).
26. Savran, A., Ciftci, K., Chanel, G., Mota, J., Hong Viet, L., Sankur, B., Rom-
baut, M.: Emotion detection in the loop from brain signals and facial images. In:
Proceedings of the eNTERFACE 2006 Workshop (2006).
27. Lang, P. J.: International affective picture system (IAPS): Digitized photographs,
instruction manual and affective ratings. Technical report (2005).
28. Duan, R. N., Zhu, J. Y., Lu, B. L.: Differential entropy feature for EEG-based
emotion classification. In: 6th International IEEE/EMBS Conference on Neural En-
gineering (NER), pp. 81–84. IEEE (2013).
29. Zheng, W. L., Liu, W., Lu, Y., Lu, B. L., Cichocki, A.: Emotionmeter: A multi-
modal framework for recognizing human emotions. IEEE transactions on cybernetics
49(3), 1110–1122 (2018). https://doi.org/10.1109/TCYB.2018.2797176
30. Bradley, M. M., Lang, P. J.: Measuring emotion: the self-assessment manikin and
the semantic differential. Journal of behavior therapy and experimental psychiatry
25(1), 49–59 (1994).
31. Onton, J. A., Makeig, S.: High-frequency broadband modulation of elec-
troencephalographic spectra. Frontiers in human neuroscience 3, 61 (2009).
https://doi.org/10.3389/neuro.09.061.2009
32. Grass, A. M., Gibbs, F. A.: A Fourier transform of the elec-
troencephalogram. Journal of Neurophysiology 1(6), 521–526 (1938).
https://doi.org/10.1152/jn.1938.1.6.521
409
33. Korats, G., Le Cam, S., Ranta, R., Hamid, M.: Applying ica in eeg: choice of
the window length and of the decorrelation method. In: International Joint Confer-
ence on Biomedical Engineering Systems and Technologies, pp. 269–286. Vilamoura,
Springer (2012).
34. Sebe, N., Lew, M. S., Cohen, I., Garg, A., Huang, T. S.: Emotion recognition using
a cauchy naive bayes classifier. In: Object recognition supported by user interaction
for service robots 1, pp. 17-20. IEEE (2002).
35. Murugappan, M., Ramachandran, N., Sazali, Y.: Classification of human emotion
from EEG using discrete wavelet transform. Journal of biomedical science and en-
gineering 3(04), 390 (2010).
36. Soleymani, M., Asghari-Esfeden, S., Fu, Y., Pantic, M.: Analysis of EEG signals
and facial expressions for continuous emotion detection. IEEE Transactions on Af-
fective Computing 7(1), 17–28 (2015).
37. Jirayucharoensak, S., Pan-Ngum, S., Israsena, P.: EEG-based emotion recognition
using deep learning network with principal component based covariate shift adap-
tation. The Scientific World Journal (2014).
38. Zheng, W. L., Lu, B. L.: Investigating critical frequency bands and
channels for EEG-based emotion recognition with deep neural networks.
IEEE Transactions on Autonomous Mental Development 7(3), 162–175 (2015).
https://doi.org/10.1109/tamd.2015.2431497
39. Yan, J., Chen, S., Deng, S.: A EEG-based emotion recognition model with rhythm
and time characteristics. Brain informatics 6(1), 7 (2019).
40. Lan, Z., Sourina, O., Wang, L., Scherer, R., Müller-Putz, G. R.: Domain adaptation
techniques for EEG-based emotion recognition: a comparative study on two public
datasets. IEEE Transactions on Cognitive and Developmental Systems 11(1), 85–94
(2018).
41. Yan, K., Kou, L., Zhang, D.: Learning domain-invariant subspace using domain
features and independence maximization. IEEE transactions on cybernetics 48(1),
288–299 (2017).
42. Pan, S. J., Tsang, I. W., Kwok, J. T., Yang, Q.: Domain adaptation via transfer
component analysis. IEEE Transactions on Neural Networks 22(2), 199–210 (2010).
43. Li, J., Qiu, S., Du, C., Wang, Y., He, H.: Domain Adaptation for
EEG Emotion Recognition Based on Latent Representation Similar-
ity. IEEE Transactions on Cognitive and Developmental Systems (2019).
https://doi.org/10.1109/TCDS.2019.2949306
44. Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation.
In: International conference on machine learning, pp. 1180-1189. PMLR (2015).
45. The Institute for Signal and Information Processing, Temple University EEG
Corpus, https://www.isip.piconepress.com/projects/tuh eeg. Last accessed 31 May
2020
46. Zheng, W. L., Lu, B. L.: A multimodal approach to estimating vigilance using
EEG and forehead EOG. Journal of neural engineering 14(2), 026017 (2017).
47. Obeid, I., Picone, J.: The temple university hospital EEG data corpus. Frontiers
in neuroscience 10, 196 (2016). https://doi.org/10.3389/fnins.2016.00196
410