Data Augmentation for Domain-Adversarial Training in EEG-based Emotion Recognition

Abstract

Emotion Recognition is an important and challenging task of modern a ective computing systems. Neuronal action potentials measured by the Electroencephalography (EEG) provide an important data source with a high temporal resolution and direct relevance to a human brain activity. EEG-based evaluation of the emotional state is complicated due to the lack of labeled training data and to a strong presence of subject- and session-dependencies. Various adaptation techniques can be applied to train a model that would be robust to a domain mismatch in EEG data but the amount of available training data is still insu cient. In this work we propose a new approach based on the domain adversarial training and combining available training corpus with much larger unlabeled dataset in a semi-supervised training framework. A detailed analysis of available datasets and existing methods for the emotion recognition task is presented. The e ect of emotion recognition performance degradation caused by the subject- and session-dependencies was measured on DEAP dataset proving the need to develop approaches that would utilize larger datasets in order to obtain a better generalized model.

Keywords

• Electroencephalography (EEG)
• Emotion recognition
• Signal processing
• Deep learning
• Domain adaptation

-

Recently, there has been growing interest in using the EEG signal to analyze the functioning of the human brain. The results of EEG processing began to be used in the creation of brain-computer interfaces (BCIs) and in neurophysiology studies. Emotion recognition is one of the essential tasks in these elds. Works on a ective disorders report that analysing EEG signal during emotion task manipulations could provide an assessment of risk for major depressive disorder [ 1 ]. There are many works on the subject of a ective brain-computer interactions. The authors of these works believe that recognizing emotions from EEG signal will allow robots and machines to read people's interactive intentions and states and respond to human emotions [2{4]. Moreover, solving the problem of recognizing emotions may contribute the development of neuromarketing to determine consumer preferences [ 5 ]. And another area of task application is workload estimation [ 6 ] and driving fatigue detection [ 7 ]. 1.1

Electroencephalography

Electroencephalography is a multichannel continuous signal recorded with electrodes that measures di erences between electric potentials that are registered in two areas of the brain. During this recording, electrodes are placed on the surface of the scalp. To improve the conductivity of the skin, a gel is applied to the contact surface of the electrodes. Elastic helmets are used to xate the electrodes on the head. In recent years a number of accessible consumer-level brain-computer interfaces (BCI) became available on the market [8{10]. These devices usually include a fewer number of electrodes that often are used without conductive/adhesive gel. It makes BCI technology cheaper and more a ordable. Due to this, there is a trend that more data is available. EEG recording is always contaminated with artifacts, such as EOG(ocular), ECG(cardiac), EMG(muscle), and noise. Therefore, work pipeline should contain signal preprocessing to automatically handle this problem. Di erent processes are re ected in di erent frequency bands of the electrical activity of the brain. For example, alpha rhythm (8 to 12 Hz) re ects to attentional demands and beta activity (16 to 24 Hz) re ects to emotional and cognitive processes in brain [ 11 ].

As possible variants of the experimental protocol the following systems for recording EEG signals are used: 1. Resting states with eyes open (REO) or with eyes closed (REC). The patient is relaxed state and does not think about anything. This procedure is used to analyze the general condition of the patient. And it is suitable for anyone, including people with disabilities. 2. Event-related potentials (ERPs) [ 12 ]. In such experiments, a signal is sent from a computer representing a stimulus to a computer recording an EEG whenever a stimulus or response occurs. Such stimuli may be periodic light exposure at di erent values of the frequency of exposure. Segments of EEG data that are time-locked to the event signals are extracted from the overall EEG and averaged. 3. Task-related. Neural activity is recorded under various cognitive tasks. The patient should also be relaxed and his attention should be focused only on the implementation of the task. These can be tasks such as counting in the mind or reading. 4. Somnography [ 13 ]. EEG is recorded during sleep stage. The sleep electroencephalogram (EEG) can be recorded for analyzing the stages of sleep or the causes of sleep deprivation. 1.2

Emotions

Emotion is a mental state and an a ective reaction towards an event based on a subjective experience. It is hard to measure because it is a subjective feeling. Emotions can be evaluated in terms of "positive", "negative" or "like", "dislike" [ 5 ]. It is also possible to distinguish a set of basic emotions such as anger, fear, sadness, disgust, happiness, surprise [ 34 ] and try to solve the classi cation problem. Researchers often use a two- or three-dimensional space to model emotions [ 14, 15 ], where di erent emotion points can be plotted on a 2D plane consisting of a Valence axis and Arousal axis (Fig. 1a) or on a 3D area with addition Dominance axis (Fig. 1b).

(a) 2D

(b) 3D One of the earliest works on emotion recognition from EEG was presented in 1997 [ 17 ]. Machine-learning approach is one of the classic ways for solving the problem of emotion recognition. In this method it is necessary to extract reliable informative features closely related to the emotional state of the subject. The signal is divided into components by Independent Component Analysis (ICA) [ 19, 20 ] to separate artifacts. The main method used for extracting spectral features is the Fourier transform [ 32 ]. A detailed description of popular features for analysing EEG signal is presented in the article [ 18 ]. In machine-learning approach, discriminant analysis [ 35 ] or Bayesian analysis [ 34 ] can be used for classi cation.

In addition to obtaining ML features, deep learning methods can be used. And that is a more modern way of solving the problem. This approach is often used as conjunction machine learning feature extraction with neural network classi cation. For classi cation such neural networks as SAE [ 21 ], LSTM [ 22 ] are often con gured [ 37, 39, 23 ]. A fully neural network solution has recently been proposed: SAE+LSTM method [ 23 ] on DEAP dataset. In this work Stacked AutoEncoder (SAE) is used for solving ICA problem and the emotion timing modeling is based on the Long Short-Term Memory Recurrent Neural Network (LSTM-RNN). 1.4

Data Augmentation

Neural networks require a big amount of training data. Usually EEG datasets contain data from a small number of subjects. This is due to the fact that special devices and the correct experimental conditions are required to collect the data. Several datasets could be combined to increase the amount of training data. But each dataset was collected by di erent devices, with di erent experimental protocol and di erent stimuli. Therefore, it is di cult to conduct training on data from several sources. Another problem is the low accuracy of prediction for subjects whose data were not available in the training set. Various domain adaptation techniques are used to reduce data variability [ 40 ].

The volume of the union of datasets labeled by emotions is still not large enough. The solution for expanding data with other EEG datasets is proposed in this work. It is possible to use EEG datasets without emotional labels if they contain video recordings of the experiment. Data can be marked by emotions detected from the video, the similar approach was suggested for problem of emotion recognition from speech [ 24 ]. It increases the amount of work, but helps to expand the training set. 2

Datasets

2.1

DEAP

There are several datasets for EEG-based emotion recognition task. Every corpus was collected according to unique protocol. Available datasets for solving the problem are described below.

DEAP dataset (A Database for Emotion Analysis Using Physiological Signals) [ 25 ] is a widely used in EEG-based emotion recognition area [ 39, 23, 40 ]. This dataset was collected as a part of an adaptive music video recommendation system development. The experiment was attended by 32 people. Data was collected from subjects while watching 40 one-minute music videos stimuli. During the experiment, participants performed self-assessment of their levels of arousal, valence and dominance (Fig. 2). As a result, 32-channel electroencephalogram and peripheral physiological signals were recorded. For 22 of the 32 participants, frontal face video was also recorded. Dataset is convenient, as it contains not only original data in BDF (Biosemi Data Format), but also preprocessed data in MATLAB and Python formats. Dataset is open only for academic research and it is available for download after signing the EULA (End User License Agreement). 2.2

eNTERFACE-2006 Another popular dataset was made as a part of eNTERFACE-2006 project [ 26 ]. The purpose of the project is to collect a su cient data to build an integrated framework for multi-modal emotion recognition. Data collection was carried out for 5 male subjects in 3 sessions. Stimuli are images from the IAPS (International A ective Picture System) [ 27 ] which consists of 1196 pictures evaluated in arousal-valence dimensions. For experiment 3 groups of images were selected: 106 calm, 71 positive exciting, 150 negative exciting. Each session lasted 15 minutes and consisted of 30 blocks, each block is succession of 5 images corresponding to a single emotion. EEG and fNIRS signals with peripheral information were recorded in ..bdf format. Eventually the data were marked not only with a preliminary evaluation of the images, but also with participants self-assessment. 2.3

SEED, SEED-IV

SEED (SJTU Emotion EEG Dataset) [ 28, 38 ] contains data from 15 subjects in 3 sessions with an interval of about one week. As stimuli, 15 video clips lasting 4 minutes were selected. During the experiment, subjects conducted selfassessments based on \positive," \negative," or \neutral" terms for evaluating emotions. Dataset contains preprocessed EEG in 45 .mat (Matlab) les. EEG data is downsampled, preprocessed and segmented. In addition, the dataset comprises les with extracted features. It contains the features of di erential entropy (DE) of the EEG signals, which is convenient for testing the classi ers.

SEED-IV [ 29 ] is another dataset collected later. In this experiment, a di erent system of emotion classi cation was used: happy, sad, neutral, fear. And in addition to EEG, eye movement information was recorded with the eye tracking glasses, that makes SEED-IV multi-modal dataset for emotion recognition. Dataset contains EEG raw data, extracted features from EEG (di erential entropy and power spectral density) and raw data and extracted features of eye movements, all in .mat format. Both of these datasets can be downloaded after signing the license agreement. 2.4

Neuromarketing

Neuromarketing is the eld of marketing research that helps to determine consumers' preferences and predict their behavior using unconscious processes, which ensures e ective utilization of the product. In [ 5 ] The Neuromarketing dataset was created for building predictive modeling framework to better understand consumer choice. This corpus of data was made by recording an EEG signal from 40 subjects while viewing consumer products. During the experiment, participants marked E-commerce products in terms of \likes" and \dislikes". The resulting dataset is publicity available and can be used in scienti c works and marketing researches. 2.5

Imagined Emotions

A di erent experiment design that included cue-based emotion stimuli was presented in [ 31 ]. Each participant listened to a sample of voice recording that suggested a speci c emotional state. A participant had to imagine a corresponding emotional scenario or to recall a related emotional experience. The presented dataset consists of EEG signals collected from 32 subjects who have experienced 15 emotional states, and participants' assessments of the authenticity and intensity of the tested emotions on a scale of 1 to 9. 3

Related Works

Emotion recognition is an analysis of multi-channel samples of EEG data. Each sample is considered to have a single emotional state that is supposed to be constant during the recording. Depending on the system of classi cation of emotions that was used in the experiment design, either the emotion must be determined from a preassigned set, or an assessment should be given on the Arousal-Valence(-Dominance) scales. Thus, the emotion recognition task can be considered a classi cation or a regression problem. 3.1

Preprocessing and Feature Extraction

Electroencephalogram data consists not only of the recordings of brain activity but also of a number of artifact components of various origins. Therefore the extensive ltering and artifact removal procedures must be included as a necessary part of the analysis pipeline. Deletion of recording sections with artifacts can be performed by specialists but that requires a thorough and expensive analysis of each sample. After the initial cleaning step, the multi-channel signal can be decomposed into quasi-independent components by solving a blind source separation task. This can be achieved with the Independent Component Analysis (ICA) or with more recent autoencoder-based approaches.

During the feature extraction step, EEG signal is divided into short time frames. The EEG features are extracted from each frame and combined into a feature sequence. The signal is represented as a set of overlap frames using the window function. It can be a rectangular window, but usually a smoothing window, such as the Hanning window, is used. For spectral analysis of the EEG data, the Fourier transform [ 32 ] is used to obtain a frequency domain representation of each window. Then, feature extraction can be performed independently for each frequency band. Following metrics and statistics can be utilized as informative features: max, min, average amplitude and Power spectral density (PSD). Following cross-channel features can be calculated: 1. Root Mean Square

where Si | ith channel amplitude 2. Pearson Correlation Coe cient between 2 channels v

n RM S = tuu N1 n=1

X Sn2 P CC = qPiN=1(xi

PiN=1(xi x)(yi

y) x)2qPiN=1(yi

y)2 M SCE = jPijj

Pi Pj 3. Magnitude Squared Coherence Estimate (1) (2) (3) where Pij | cross-PSD i; jth channels, Pi | PSD ith channel A more detailed review of feature extraction methods can be found in [ 18 ]. 3.2

Model Training

Emotion recognition problem in feature space can be approached with one of the machine learning methods for classi cation. In [ 34 ] an emotion recognition method using a Naive Bayes model was proposed. The classi cation problem under the maximum likelihood framework was formulated as: y = arg max P (Xjy) b y (4) where y is label and X is feature vector. The Naive Bayes framework assumes that the features in X are independent of each other conditioned upon the class label. This paper compares two model distribution assumptions. It is shown that the Cauchy distribution assumption typically provides better results than the Gaussian distribution assumption.

In [ 35 ] a comparison of K Nearest Neighbours classi er and Linear Discriminant Analysis is presented. The experiment was conducted on a private dataset and showed the maximum average classi cation rate of 83.26% using KNN and 75.21% using LDA. These solutions are suitable for the classi cation problem when it is necessary to recognize emotion from a given set. If a ective labeling is presented as a vector of real values (such as Arousal-Valence scale), this approach can also be applied with regression methods instead of classi cation [ 36 ]. Despite this, labels are often made binary when evaluating the accuracy of an algorithm. 3.3

Deep Learning Approach

Today, neural network algorithms are used everywhere, since they can recognize deeper, sometimes unexpected patterns in data. And in the studied area, deep neural network-based feature extraction and emotion recognition began to be intensively applied.

In [ 38 ] Deep Belief Network (DBN) was trained with di erential entropy features. The experiment performed classi cation for three emotional categories on the SEED dataset. The results show that the DBN models obtain higher accuracy than previously considered models such as kNN, LR and SVM approaches.

An emotion recognition system that uses deep learning models at two stages of work pipeline was introduced in [ 23 ]. Stacked autoencoder was used for decomposition of source signal (as a substitute for Independent Component Analysis) and extracting EEG channel correlations. LSTM-RNN network is used for emotion classi cation based on Frequency Band Power Features extracted from the SAE output. The mean accuracy of emotion recognition, calculated by binarized labels, achieved 81.10% in valence and 74.38% in arousal on the DEAP dataset. 3.4

Domain Adaptation

Training an accurate model requires an approach that would be robust to variations in individual characteristics of participants and recording devices, since EEG data su ers from an intense dependence on the device and the subject. It is important to apply a domain adaptation technique to a model that would compensate the subject variability or heterogeneity in various technical speci cations.

The paper [ 40 ] compares di erent domain adaptation techniques on two datasets: DEAP and SEED. Transfer Component Analysis (TCA) [ 42 ] and Maximum Independence Domain Adaptation (MIDA) [ 41 ] performed the best results for subject within-dataset domain adaptation. It is shown that applying these techniques lead to an improvement gain up to 20.66% over the baseline accuracy where no domain adaptation technique was used. A research of these techniques application for cross-dataset domain adaptation was also conducted. The article concluded that TCA and MIDA can e ectivly improve the accuracy by 7.25% { 13.40% compared to the baseline accuracy where no domain adaptation technique was used.

In [ 43 ] another approach to a domain adaptation was considered based on neural networks that are trained to solve emotion and domain recognition problems. Samples of feature vectors from two domains in the same quantity are fed to the model, producing emotion label for each EEG sample. Several rst layers of neural network act as a feature extractor, producing a xed-dimension representation of EEG samples in a latent space. These representations are used to solve two di erent tasks: emotion label classi cation and domain recognition. A gradient reversal layer is applied to the domain predictor [ 44 ] leading to the adversarial training scheme during which the parameters of feature extractor layers are updated to make embedding distributions of di erent domains statistically similar. Fully connected layers make representations for label predictor, which estimates emotion class for each sample. During training, samples from one domain contains labels, whereas the second domain is unlabeled. The label predictor is optimized to minimize the classi cation error on the rst domain. During test, the model inputs are unlabeled data only. This method was compared with multiple domain adaptation algorithms on benchmark SEED and DEAP and proved to be superior in both cross-subject and cross-session adaptation. 4 4.1

A Proposed Approach

Domain-Adversarial Training

The problem of domain adaptation is crucial in the emotion recognition task. The architectures of the proposed approach are presented on Fig. 3, 4. This approach combines the ideas presented in works [ 43 ] and [ 44 ]. In g. 3 the domain classi er predicts which domain the data belongs to and the set of feature extractor parameters is updated by adversarial training to make the distribution of data representations of di erent domains more similar. In g. 4 the input is data from two domains: labeled and unlabeled. Data representations of labeled domain are sent to label predictor and domain discriminator, representations of unlabeled data are transmitted only to domain discriminator, which determines whether these domains match or not.

These architectures di er in that in the rst case, the classi cation of domains occurs independently for input samples, and in the second, pairwise comparisons are performed. In the future work, a comparison of these two approaches will be carried out and the best approach will be de ned. 4.2

Data Augmentation

In order to improve the performance of the model, a large number of EEG datasets without a ective labels [ 45 ] can be utilized emotion recognition task. To train a domain classi er (for subject identity recognition), more data can be used, since there is no need for labeled data. Since a domain predictor is trained on a much larger number of domains, and a larger amount of training samples, it potentially can be more robust to various speci c channel characteristic variability. The emotion classi er is still trained on the same amount of data, but the performance can be improved since the latent representations are trained to be domain-independent. DEAP dataset includes data of only 32 subjects, as well as other datasets for EEG-based emotion recognition also contain a limited variability of subjects. At the same time, EEG datasets without a ective labeling are much larger. For example, in the Temple University Hospital (TUH) EEG data corpus [ 47 ] there is EGG data of more than 10000 participants. It is more e cient to train neural networks on such data volume, therefore, as the solution it is proposed to use unlabeled data. Thus, the neural network will be trained on a larger set of subjects, and therefore, will provide a better generalized model for new subjects. 4.3

Auto-labeling of EEG Datasets

Another possible solution is to enrich the sample for training using the multimodal emotion recognition. For this purpose, EEG datasets without labels, which contain other modalities such as video recordings of a subject's face, can be used, for example SEED-VIG [ 46 ]. Then the data can be automatically labeled, recognizing the emotions experienced by the participants from the video. Unfortunately, the EEG datasets with the recordings of such modalities are rare, so this approach probably will not be allow to signi cantly expand the training data. 4.4

A Preliminary Motivation Study

Below is an illustration of the fact that the problem of cross-subject adaptation really requires a solution. An experiment was conducted demonstrating a decrease in the accuracy of emotions recognition in the absence of subject data in the training sample. The preprocessed data from DEAP dataset was used. PSD for ve frequency bands were extracted as features. The following ML classiers were trained: SVM, Random Forest Regression. The data were divided into training, validation and test samples in a ratio of 6 : 1 : 1 respectively. In the rst experiment, the data of each subject was divided between the samples. In the second experiment, the data of each subject entirely relate to one or another sample. The table 1 shows the di erences in the accuracy of determining emotions for these two experiments. According to the results the presence of learning problems on isolated subjects is shown.

(b) For Random Forest Regression Rating scale 1st experiment 2nd experiment Valence 83.2% 47.6% Arousal 82.6% 60.3% Dominance 81.8% 53.1%

Conclusion and Future Work

This paper discribes the EEG-based emotion recognition task and its existing solution methods. There was formulated the problem of domain mismatch and insu cient data amount for training neural networks. As a solution, there was proposed the application of existing domain adaptation techniques with data augmentation due to datasets without emotional labels.

In the future, it is planned to conduct testing on DEAP dataset, using TUH EEG data corpus, to evaluate how emotion classi cation would be robust to subjects and session and channel di erences. It is also planned to use the SEED dataset and perform the same analysis to study the task of training a datasetindependent emotion recognition model. A detailed validation study will be performed to compare the results with existing methods of domain adaptation.

References

1. Stewart , J. L. , Coan , J. A. , Towers , D. N. , Allen , J. J. B. : Frontal EEG asymmetry during emotional challenge di erentiates individuals with and without lifetime major depressive disorder . Journal of A ective Disorders 129 ( 1-3 ), 167 { 174 ( 2011 ). https://doi.org/10.1016/j.jad. 2010 . 08 .029

2. Yin , Z. , Wang , Y. , Liu , L. , Zhang , W. , Zhang, J.: Cross-Subject EEG Feature Selection for Emotion Recognition Using Transfer Recursive Feature Elimination . Frontiers in Neurorobotics 11 ( 2017 ). https://doi.org/10.3389/fnbot. 2017 .00019

3. Urgen , B. , Plank , M. , Ishiguro , H. , Poizner , H. , Saygin , A. : Eeg theta and mu oscillations during perception of human and robot actions . Frontiers in Neurorobotics 7 ( 2013 ). https://doi.org/10.3389/fnbot. 2013 .00019

4. Calvo , R. A. , D'Mello , S.: A ect detection: An interdisciplinary review of models, methods, and their applications . IEEE Transactions on a ective computing , 1 ( 1 ), 18 { 37 ( 2010 ).

5. Yadava , M. , Kumar , P. , Saini , R. et al: Analysis of EEG signals and its application to neuromarketing . Multimedia Tools and Applications 76 , 19087{ 19111 ( 2017 ). https://doi.org/10.1007/s11042-017-4580-6

6. Kothe , C. A. , Makeig , S. : Estimation of task workload from EEG data: new and current tools and perspectives . In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society , pp. 6547 { 6551 . EMBC, IEEE ( 2011 ).

7. Shi , L. C. , Lu , B. L. : EEG-based vigilance estimation using extreme learning machines . Neurocomputing 102 , 135 { 143 ( 2013 ).

8. Muse Homepage , https://choosemuse.com. Last accessed 30 May 2020

9. Emotiv Homepage , https://www.emotiv.com. Last accessed 30 May 2020

10. Neurosky Homepage, http://neurosky.com. Last accessed 30 May 2020

11. Ray W. J. , Cole H. W.: EEG alpha activity re ects attentional demands, and beta activity re ects emotional and cognitive processes . Science 228 ( 4700 ), 750 { 752 ( 1985 ).

12. Lopez-Calderon , J. , Luck , S. J.: ERPLAB: an open-source toolbox for the analysis of event-related potentials . Frontiers in Human Neuroscience 8 ( 2014 ). https://doi.org/10.3389/fnhum. 2014 .00213

13. Brunner , D. P. , Munch, M. , Biedermann , K. , Huch , R. , Huch , A. , Borbely , A. A. : Changes in Sleep and Sleep Electroencephalogram During Pregnancy . Sleep 17 ( 7 ), 576 { 582 ( 1994 ). https://doi.org/10.1093/sleep/17.7. 576

14. Lang , P. J.: The emotion probe: studies of motivation and attention . American psychologist 50(5) , 372 ( 1995 ).

15. Scherer , K. R. : What Are Emotions? And How Can They Be Measured? Social Science Information 44 ( 4 ), 695 { 729 ( 2005 ). https://doi.org/10.1177/0539018405058216

16. Shu , L. , Xie , J. , Yang , M. , Li , Z. , Li , Z. , Liao , D. , Yang , X.: A review of emotion recognition using physiological signals . Sensors 18 ( 7 ), 2074 ( 2018 ).

17. Musha , T. , Terasaki , Y. , Haque , H. A. , Ivamitsky , G. A. : Feature extraction from EEGs associated with emotions . Arti cial Life and Robotics 1 ( 1 ), 15 { 19 ( 1997 ).

18. Jenke , R. , Peer , A. , Buss , M. : Feature extraction and selection for emotion recognition from EEG . IEEE Transactions on A ective computing 5 ( 3 ), 327 { 339 ( 2014 ).

19. Hyvarinen, A. , Oja , E.: Independent component analysis: A tutorial . LCIS , Helsinki University of Technology, Finland ( 1999 ).

20. Vinther , M. : Independent Component Analysis of Evoked Potentials in EEG . Orsted, DTU ( 2002 ).

21. Vincent , P. , Larochelle , H. , Lajoie , I. , Bengio , Y. , Manzagol , P. A. , Bottou , L. : Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion . Journal of machine learning research 11(12) , 3371 { 3408 ( 2010 ).

22. Hochreiter , S. , Schmidhuber , J.: Long short-term memory . Neural Computing 9 ( 8 ), 1735 { 1780 ( 1997 ).

23. Xing , X. , Li , Z. , Xu , T. , Shu , L. , Hu , B. , Xu , X. : SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG . Frontiers in Neurorobotics 13 ( 37 ) ( 2019 ). https://doi.org/10.3389/fnbot. 2019 .00037

24. Albanie , S. , Nagrani , A. , Vedaldi , A. , Zisserman , A. : Emotion recognition in speech using cross-modal transfer in the wild . In: Proceedings of the 26th ACM international conference on Multimedia , pp. 292 { 301 . ACM Press ( 2018 ).

25. Koelstra , S. , Muhl , C. , Soleymani , M. , Lee , J. S. , Yazdani , A. , Ebrahimi , T. , Patras , I. : Deap: A database for emotion analysis; using physiological signals . IEEE transactions on a ective computing 3 ( 1 ), 18 { 31 ( 2011 ).

26. Savran , A. , Ciftci , K. , Chanel , G. , Mota , J. , Hong Viet , L. , Sankur , B. , Rombaut , M. : Emotion detection in the loop from brain signals and facial images . In: Proceedings of the eNTERFACE 2006 Workshop ( 2006 ).

27. Lang , P. J.: International a ective picture system (IAPS): Digitized photographs, instruction manual and a ective ratings . Technical report ( 2005 ).

28. Duan , R. N. , Zhu , J. Y. , Lu , B. L. : Di erential entropy feature for EEG-based emotion classi cation . In: 6th International IEEE/EMBS Conference on Neural Engineering (NER) , pp. 81 { 84 . IEEE ( 2013 ).

29. Zheng , W. L. , Liu , W. , Lu , Y. , Lu , B. L. , Cichocki , A. : Emotionmeter: A multimodal framework for recognizing human emotions . IEEE transactions on cybernetics 49(3) , 1110 { 1122 ( 2018 ). https://doi.org/10.1109/TCYB. 2018 .2797176

30. Bradley , M. M. , Lang , P. J. : Measuring emotion: the self-assessment manikin and the semantic di erential . Journal of behavior therapy and experimental psychiatry 25(1) , 49 { 59 ( 1994 ).

31. Onton , J. A. , Makeig , S. : High-frequency broadband modulation of electroencephalographic spectra . Frontiers in human neuroscience 3 , 61 ( 2009 ). https://doi.org/10.3389/neuro.09.061.2009

32. Grass , A. M. , Gibbs , F. A. : A Fourier transform of the electroencephalogram . Journal of Neurophysiology 1 ( 6 ), 521 { 526 ( 1938 ). https://doi.org/10.1152/jn. 1938 . 1 .6. 521

33. Korats , G. , Le Cam , S. , Ranta , R. , Hamid , M. : Applying ica in eeg: choice of the window length and of the decorrelation method . In: International Joint Conference on Biomedical Engineering Systems and Technologies , pp. 269 { 286 . Vilamoura, Springer ( 2012 ).

34. Sebe , N. , Lew , M. S. , Cohen , I. , Garg , A. , Huang , T. S.: Emotion recognition using a cauchy naive bayes classi er . In: Object recognition supported by user interaction for service robots 1 , pp. 17 - 20 . IEEE ( 2002 ).

35. Murugappan , M. , Ramachandran , N. , Sazali , Y. : Classi cation of human emotion from EEG using discrete wavelet transform . Journal of biomedical science and engineering 3 ( 04 ), 390 ( 2010 ).

36. Soleymani , M. , Asghari-Esfeden , S. , Fu , Y. , Pantic , M.: Analysis of EEG signals and facial expressions for continuous emotion detection . IEEE Transactions on Affective Computing 7 ( 1 ), 17 { 28 ( 2015 ).

37. Jirayucharoensak , S. , Pan-Ngum , S. , Israsena , P. : EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation . The Scienti c World Journal ( 2014 ).

38. Zheng , W. L. , Lu , B. L. : Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks . IEEE Transactions on Autonomous Mental Development 7 ( 3 ), 162 { 175 ( 2015 ). https://doi.org/10.1109/tamd. 2015 .2431497

39. Yan , J. , Chen , S. , Deng , S.: A EEG-based emotion recognition model with rhythm and time characteristics . Brain informatics 6(1) , 7 ( 2019 ).

40. Lan , Z. , Sourina , O. , Wang , L. , Scherer , R. , Mu ller -Putz, G. R. : Domain adaptation techniques for EEG-based emotion recognition: a comparative study on two public datasets . IEEE Transactions on Cognitive and Developmental Systems 11 ( 1 ), 85 { 94 ( 2018 ).

41. Yan , K. , Kou , L. , Zhang , D.: Learning domain-invariant subspace using domain features and independence maximization . IEEE transactions on cybernetics 48(1) , 288 { 299 ( 2017 ).

42. Pan , S. J. , Tsang , I. W. , Kwok , J. T. , Yang , Q. : Domain adaptation via transfer component analysis . IEEE Transactions on Neural Networks 22 ( 2 ), 199 { 210 ( 2010 ).

43. Li , J. , Qiu , S. , Du , C. , Wang , Y. , He , H.: Domain Adaptation for EEG Emotion Recognition Based on Latent Representation Similarity . IEEE Transactions on Cognitive and Developmental Systems ( 2019 ). https://doi.org/10.1109/TCDS. 2019 .2949306

44. Ganin , Y. , Lempitsky , V. : Unsupervised domain adaptation by backpropagation . In: International conference on machine learning , pp. 1180 - 1189 . PMLR ( 2015 ).

45. The Institute for Signal and Information Processing, Temple University EEG Corpus, https://www.isip.piconepress.com/projects/tuh eeg. Last accessed 31 May 2020

46. Zheng , W. L. , Lu , B. L.: A multimodal approach to estimating vigilance using EEG and forehead EOG . Journal of neural engineering 14(2) , 026017 ( 2017 ).

47. Obeid , I. , Picone , J.: The temple university hospital EEG data corpus . Frontiers in neuroscience 10 , 196 ( 2016 ). https://doi.org/10.3389/fnins. 2016 .00196