Recently, there has been growing interest in using the EEG signal to analyze the functioning of the human brain. The results of EEG processing began to be used in the creation of brain-computer interfaces (BCIs) and in neurophysiology studies. Emotion recognition is one of the essential tasks in these fields. Works on affective disorders report that analysing EEG signal during emotion task manipulations could provide an assessment of risk for major depressive disorder [1]. There are many works on the subject of affective brain-computer interactions. The authors of these works believe that recognizing emotions from EEG signal will allow robots and machines to read people's interactive intentions and states and respond to human emotions [2][3][4]. Moreover, solving the problem of recognizing emotions may contribute the development of neuromarketing to determine consumer preferences [5]. And another area of task application is workload estimation [6] and driving fatigue detection [7].
Electroencephalography is a multichannel continuous signal recorded with electrodes that measures differences between electric potentials that are registered in two areas of the brain. During this recording, electrodes are placed on the surface of the scalp. To improve the conductivity of the skin, a gel is applied to the contact surface of the electrodes. Elastic helmets are used to fixate the electrodes on the head. In recent years a number of accessible consumer-level brain-computer interfaces (BCI) became available on the market [8][9][10]. These devices usually include a fewer number of electrodes that often are used without conductive/adhesive gel. It makes BCI technology cheaper and more affordable. Due to this, there is a trend that more data is available. EEG recording is always contaminated with artifacts, such as EOG(ocular), ECG(cardiac), EMG(muscle), and noise. Therefore, work pipeline should contain signal preprocessing to automatically handle this problem. Different processes are reflected in different frequency bands of the electrical activity of the brain. For example, alpha rhythm (8 to 12 Hz) reflects to attentional demands and beta activity (16 to 24 Hz) reflects to emotional and cognitive processes in brain [11].
As possible variants of the experimental protocol the following systems for recording EEG signals are used:
1. Resting states with eyes open (REO) or with eyes closed (REC). The patient is relaxed state and does not think about anything. This procedure is used to analyze the general condition of the patient. And it is suitable for anyone, including people with disabilities. 2. Event-related potentials (ERPs) [12]. In such experiments, a signal is sent from a computer representing a stimulus to a computer recording an EEG whenever a stimulus or response occurs. Such stimuli may be periodic light exposure at different values of the frequency of exposure. Segments of EEG data that are time-locked to the event signals are extracted from the overall EEG and averaged. 3. Task-related. Neural activity is recorded under various cognitive tasks. The patient should also be relaxed and his attention should be focused only on the implementation of the task. These can be tasks such as counting in the mind or reading. 4. Somnography [13]. EEG is recorded during sleep stage. The sleep electroencephalogram (EEG) can be recorded for analyzing the stages of sleep or the causes of sleep deprivation.
Emotion is a mental state and an affective reaction towards an event based on a subjective experience. It is hard to measure because it is a subjective feeling.
Emotions can be evaluated in terms of "positive", "negative" or "like", "dislike" [5]. It is also possible to distinguish a set of basic emotions such as anger, fear, sadness, disgust, happiness, surprise [34] and try to solve the classification problem. Researchers often use a two-or three-dimensional space to model emotions [14,15], where different emotion points can be plotted on a 2D plane consisting of a Valence axis and Arousal axis (Fig. 1a) or on a 3D area with addition Dominance axis (Fig. 1b). Fig. 1: Emotion space models [16].
One of the earliest works on emotion recognition from EEG was presented in 1997 [17]. Machine-learning approach is one of the classic ways for solving the problem of emotion recognition. In this method it is necessary to extract reliable informative features closely related to the emotional state of the subject. The signal is divided into components by Independent Component Analysis (ICA) [19,20] to separate artifacts. The main method used for extracting spectral features is the Fourier transform [32]. A detailed description of popular features for analysing EEG signal is presented in the article [18]. In machine-learning approach, discriminant analysis [35] or Bayesian analysis [34] can be used for classification.
In addition to obtaining ML features, deep learning methods can be used. And that is a more modern way of solving the problem. This approach is often used as conjunction machine learning feature extraction with neural network classification. For classification such neural networks as SAE [21], LSTM [22] are often configured [37,39,23]. A fully neural network solution has recently been proposed: SAE+LSTM method [23] on DEAP dataset. In this work Stacked AutoEncoder (SAE) is used for solving ICA problem and the emotion timing modeling is based on the Long Short-Term Memory Recurrent Neural Network (LSTM-RNN).
Neural networks require a big amount of training data. Usually EEG datasets contain data from a small number of subjects. This is due to the fact that special devices and the correct experimental conditions are required to collect the data. Several datasets could be combined to increase the amount of training data. But each dataset was collected by different devices, with different experimental protocol and different stimuli. Therefore, it is difficult to conduct training on data from several sources. Another problem is the low accuracy of prediction for subjects whose data were not available in the training set. Various domain adaptation techniques are used to reduce data variability [40].
The volume of the union of datasets labeled by emotions is still not large enough. The solution for expanding data with other EEG datasets is proposed in this work. It is possible to use EEG datasets without emotional labels if they contain video recordings of the experiment. Data can be marked by emotions detected from the video, the similar approach was suggested for problem of emotion recognition from speech [24]. It increases the amount of work, but helps to expand the training set.
There are several datasets for EEG-based emotion recognition task. Every corpus was collected according to unique protocol. Available datasets for solving the problem are described below.
DEAP dataset (A Database for Emotion Analysis Using Physiological Signals) [25] is a widely used in EEG-based emotion recognition area [39,23,40]. This dataset was collected as a part of an adaptive music video recommendation system development. The experiment was attended by 32 people. Data was collected from subjects while watching 40 one-minute music videos stimuli. During the experiment, participants performed self-assessment of their levels of arousal, valence and dominance (Fig. 2). As a result, 32-channel electroencephalogram and peripheral physiological signals were recorded. For 22 of the 32 participants, frontal face video was also recorded. Dataset is convenient, as it contains not only original data in BDF (Biosemi Data Format), but also preprocessed data in MAT-LAB and Python formats. Dataset is open only for academic research and it is available for download after signing the EULA (End User License Agreement).
Another popular dataset was made as a part of eNTERFACE-2006 project [26]. The purpose of the project is to collect a sufficient data to build an integrated framework for multi-modal emotion recognition. Data collection was carried out for 5 male subjects in 3 sessions. Stimuli are images from the IAPS (International Affective Picture System) [27] which consists of 1196 pictures evaluated in arousal-valence dimensions. For experiment 3 groups of images were selected: 106 calm, 71 positive exciting, 150 negative exciting. Each session lasted 15 minutes Fig. 2: The Self-Assessment Manikin(SAM) for rating the affective dimensions of valence, arousal, and dominance levels [30]. and consisted of 30 blocks, each block is succession of 5 images corresponding to a single emotion. EEG and fNIRS signals with peripheral information were recorded in ..bdf format. Eventually the data were marked not only with a preliminary evaluation of the images, but also with participants self-assessment.
SEED (SJTU Emotion EEG Dataset) [28,38] contains data from 15 subjects in 3 sessions with an interval of about one week. As stimuli, 15 video clips lasting 4 minutes were selected. During the experiment, subjects conducted selfassessments based on "positive," "negative," or "neutral" terms for evaluating emotions. Dataset contains preprocessed EEG in 45 .mat (Matlab) files. EEG data is downsampled, preprocessed and segmented. In addition, the dataset comprises files with extracted features. It contains the features of differential entropy (DE) of the EEG signals, which is convenient for testing the classifiers. SEED-IV [29] is another dataset collected later. In this experiment, a different system of emotion classification was used: happy, sad, neutral, fear. And in addition to EEG, eye movement information was recorded with the eye tracking glasses, that makes SEED-IV multi-modal dataset for emotion recognition. Dataset contains EEG raw data, extracted features from EEG (differential entropy and power spectral density) and raw data and extracted features of eye movements, all in .mat format. Both of these datasets can be downloaded after signing the license agreement.
Neuromarketing is the field of marketing research that helps to determine consumers' preferences and predict their behavior using unconscious processes, which ensures effective utilization of the product. In [5] The Neuromarketing dataset was created for building predictive modeling framework to better understand consumer choice. This corpus of data was made by recording an EEG signal from 40 subjects while viewing consumer products. During the experiment, participants marked E-commerce products in terms of "likes" and "dislikes". The resulting dataset is publicity available and can be used in scientific works and marketing researches.
A different experiment design that included cue-based emotion stimuli was presented in [31]. Each participant listened to a sample of voice recording that suggested a specific emotional state. A participant had to imagine a corresponding emotional scenario or to recall a related emotional experience. The presented dataset consists of EEG signals collected from 32 subjects who have experienced 15 emotional states, and participants' assessments of the authenticity and intensity of the tested emotions on a scale of 1 to 9.
Emotion recognition is an analysis of multi-channel samples of EEG data. Each sample is considered to have a single emotional state that is supposed to be constant during the recording. Depending on the system of classification of emotions that was used in the experiment design, either the emotion must be determined from a preassigned set, or an assessment should be given on the Arousal-Valence(-Dominance) scales. Thus, the emotion recognition task can be considered a classification or a regression problem.
Electroencephalogram data consists not only of the recordings of brain activity but also of a number of artifact components of various origins. Therefore the extensive filtering and artifact removal procedures must be included as a necessary part of the analysis pipeline. Deletion of recording sections with artifacts can be performed by specialists but that requires a thorough and expensive analysis of each sample. After the initial cleaning step, the multi-channel signal can be decomposed into quasi-independent components by solving a blind source separation task. This can be achieved with the Independent Component Analysis (ICA) or with more recent autoencoder-based approaches.
During the feature extraction step, EEG signal is divided into short time frames. The EEG features are extracted from each frame and combined into a feature sequence. The signal is represented as a set of overlap frames using the window function. It can be a rectangular window, but usually a smoothing window, such as the Hanning window, is used. For spectral analysis of the EEG data, the Fourier transform [32] is used to obtain a frequency domain representation of each window. Then, feature extraction can be performed independently for each frequency band. Following metrics and statistics can be utilized as informative features: max, min, average amplitude and Power spectral density (PSD). Following cross-channel features can be calculated:
where S i -i th channel amplitude 2. Pearson Correlation Coefficient between 2 channels
where P ij -cross-PSD i, j th channels, P i -PSD i th channel A more detailed review of feature extraction methods can be found in [18].
Emotion recognition problem in feature space can be approached with one of the machine learning methods for classification. In [34] an emotion recognition method using a Naive Bayes model was proposed. The classification problem under the maximum likelihood framework was formulated as:
where y is label and X is feature vector. The Naive Bayes framework assumes that the features in X are independent of each other conditioned upon the class label. This paper compares two model distribution assumptions. It is shown that the Cauchy distribution assumption typically provides better results than the Gaussian distribution assumption.
In [35] a comparison of K Nearest Neighbours classifier and Linear Discriminant Analysis is presented. The experiment was conducted on a private dataset and showed the maximum average classification rate of 83.26% using KNN and 75.21% using LDA. These solutions are suitable for the classification problem when it is necessary to recognize emotion from a given set. If affective labeling is presented as a vector of real values (such as Arousal-Valence scale), this approach can also be applied with regression methods instead of classification [36]. Despite this, labels are often made binary when evaluating the accuracy of an algorithm.
Today, neural network are used everywhere, since they can recognize deeper, sometimes unexpected patterns in data. And in the studied area, deep neural network-based feature extraction and emotion recognition began to be intensively applied.
In [38] Deep Belief Network (DBN) was trained with differential entropy features. The experiment performed classification for three emotional categories on the SEED dataset. The results show that the DBN models obtain higher accuracy than previously considered models such as kNN, LR and SVM approaches.
An emotion recognition system that uses deep learning models at two stages of work pipeline was introduced in [23]. Stacked autoencoder was used for decomposition of source signal (as a substitute for Independent Component Analysis) and extracting EEG channel correlations. LSTM-RNN network is used for emotion classification based on Frequency Band Power Features extracted from the SAE output. The mean accuracy of emotion recognition, calculated by binarized labels, achieved 81.10% in valence and 74.38% in arousal on the DEAP dataset.
Training an accurate model requires an approach that would be robust to variations in individual characteristics of participants and recording devices, since EEG data suffers from an intense dependence on the device and the subject. It is important to apply a domain adaptation technique to a model that would compensate the subject variability or heterogeneity in various technical specifications.
The paper [40] compares different domain adaptation techniques on two datasets: DEAP and SEED. Transfer Component Analysis (TCA) [42] and Maximum Independence Domain Adaptation (MIDA) [41] performed the best results for subject within-dataset domain adaptation. It is shown that applying these techniques lead to an improvement gain up to 20.66% over the baseline accuracy where no domain adaptation technique was used. A research of these techniques application for cross-dataset domain adaptation was also conducted. The article concluded that TCA and MIDA can effectivly improve the accuracy by 7.25% -13.40% compared to the baseline accuracy where no domain adaptation technique was used.
In [43] another approach to a domain adaptation was considered based on neural networks that are trained to solve emotion and domain recognition problems. Samples of feature vectors from two domains in the same quantity are fed to the model, producing emotion label for each EEG sample. Several first layers of neural network act as a feature extractor, producing a fixed-dimension representation of EEG samples in a latent space. These representations are used to solve two different tasks: emotion label classification and domain recognition. A gradient reversal layer is applied to the domain predictor [44] leading to the adversarial training scheme during which the parameters of feature extractor layers are updated to make embedding distributions different domains statistically similar. Fully connected layers make representations for label predictor, which estimates emotion class for each sample. During training, samples from one domain contains labels, whereas the second domain is unlabeled. The label predictor is optimized to minimize the classification error on the first domain.
During test, the model inputs are unlabeled data only. This method was compared with multiple domain adaptation algorithms on benchmark SEED and DEAP and proved to be superior in both cross-subject and cross-session adaptation.
The problem of domain adaptation is crucial in the emotion recognition task. The architectures of the proposed approach are presented on Fig. 3, 4. This approach combines the ideas presented in works [43] and [44]. In fig. 3 the domain classifier predicts which domain the data belongs to and the set of feature extractor parameters is updated by adversarial training to make the distribution of data representations of different domains more similar. In fig. 4 the input is data from two domains: labeled and unlabeled. Data representations of labeled domain are sent to label predictor and domain discriminator, representations of unlabeled data are transmitted only to domain discriminator, which determines whether these domains match or not.
These architectures differ in that in the first case, the classification of domains occurs independently for input samples, and in the second, pairwise comparisons are performed. In the future work, a comparison of these two approaches will be carried out and the best approach will be defined.
In order to improve the performance of the model, a large number of EEG datasets without affective labels [45] can be utilized emotion recognition task. To train a domain classifier (for subject identity recognition), more data can be used, since there is no need for labeled data. Since a domain predictor is trained on a much larger number of domains, and a larger amount of training samples, it potentially can be more robust to various specific channel characteristic variability. The emotion classifier is still trained on the same amount of data, but the performance can be improved since the latent representations are trained to be domain-independent. DEAP dataset includes data of only 32 subjects, as well as other datasets for EEG-based emotion recognition also contain a limited variability of subjects. At the same time, EEG datasets without affective labeling are much larger. For example, in the Temple University Hospital (TUH) EEG data corpus [47] there is EGG data of more than 10000 participants. It is more efficient to train neural networks on such data volume, therefore, as the solution it is proposed to use unlabeled data. Thus, the neural network will be trained on a larger set of subjects, and therefore, will provide a better generalized model for new subjects.
Another possible solution is to enrich the sample for training using the multimodal emotion recognition. For this purpose, EEG datasets without labels, which contain other modalities such as video recordings of a subject's face, can be used, for example SEED-VIG [46]. Then the data can be automatically labeled, recognizing the emotions experienced by the participants from the video. Unfortunately, the EEG datasets with the recordings of such modalities are rare, so this approach probably will not be allow to significantly expand the training data.
Below is an illustration of the fact that the problem of cross-subject adaptation really requires a solution. An experiment was conducted demonstrating a decrease in the accuracy of emotions recognition in the absence of subject data in the training sample. The preprocessed data from DEAP dataset was used. PSD for five frequency bands were extracted as features. The following ML classifiers were trained: SVM, Random Forest Regression. The data were divided into training, validation and test samples in a ratio of 6 : 1 : 1 respectively. In the first experiment, the data of each subject was divided between the samples. In the second experiment, the data of each subject entirely relate to one or another sample. The table 1 shows the differences in the accuracy of determining emotions for these two experiments. According to the results the presence of learning problems on isolated subjects is shown.
This paper discribes the EEG-based emotion recognition task its existing solution methods. There was formulated the problem of domain mismatch and insufficient data amount for training neural networks. As a solution, there was proposed the application of existing domain adaptation techniques with data augmentation due to datasets without emotional labels.
In the future, it is planned to conduct testing on DEAP dataset, using TUH EEG data corpus, to evaluate how emotion classification would be robust to subjects and session and channel differences. It is also planned to use the SEED dataset and perform the same analysis to study the task of training a datasetindependent emotion recognition model. A detailed validation study will be performed to compare the results with existing methods of domain adaptation.