EEG-based brain-computer interface (BCI) devices have proved to be powerful tools for predicting human emotions. Although Deep learning (DL) techniques have been extensively used to build emotion recognition architectures using EEG-based BCI, they lack interpretability. We propose a prototype of an EEG-based emotion recognition system that can detect the user's emotional state using a deep learning model embedded into an interpretable framework to analyze the decisions of the model and the contributions of the features. The proposed model achieves high performance while showing relevant information on the impact of frequency and spatial features used to predict the emotional states.
to the interest in automatic clinical EEG diagnosis. According to physiological studies, the cerebral cortex is The research field of afective computing has achieved the primary controller of humans’ higher emotional cogremarkable results enabling the integration of emotion nitive capabilities. Hence, it would be advantageous to recognition algorithms in diferent clinical settings. On identify brain areas that are strongly associated with the one hand, the availability of increasingly low-cost de- emotions using EEG-based emotion detection, mainly in vice, and considerable advances in artificial intelligence clinical trials for neuromotor rehabilitation or psychologalgorithms have triggered the rapid development of ap- ical therapies. plications for emotional recognition via brain waves. In particular, EEG-based brain-computer interface (BCI) devices have proved to be remarkably powerful tools for 2. Modeling the emotions brainwave acquisition, both due to their rapid deployment and their wide application in diferent scenarios and contexts.
EEG signals are primarily used to diagnose and treat
various brain disorders, including epilepsy, tremor,
concussions, strokes, and sleep disorders. Machine learning
(ML) as an analysis method has been used in recent EEG
applications. ML Methods for automated EEG analysis
have attracted great interest, especially in clinical
diagnostics. For example, ML enables the automation of the
process of EEG-based sleep stages [
The most common use is for Russell’s 2D emotion
model [
The design of a machine learning emotion recognition
model requires the careful planning of several steps, i.e., 4.1. Dataset Description
collecting EEG data, preprocessing phase, retrieving
features, choosing or reducing them, and classifying emo- The Dreamer dataset consists of 23 users’ EEG signals
tions. The selection of the best classification tool which during emotional elicitation. The emotional elicitation
is capable of accurately classifying individual emotions protocol was performed using audio/video clips.
Eighhas been one of the most important elements in devel- teen video clips were used and classified into nine
baoping an efective emotion classification system. Most sic emotions such as amusement, excitement, happiness,
existing techniques consider emotion recognition as a calmness, anger, disgust, fear, sadness, and surprise. Each
problem of classification and attempt to identify between user was required to watch all video clips ranging from
category emotions or across diferent areas in Russell’s 65 to 393 seconds. After each video clip ended, users
pro2D emotional model. Several ML models have been devel- vided a self-assessment based on a 5-point Likert scale for
oped in recent years to handle the categorization of EEG Valence, Arousal, and Dominance. To carry out this task,
data for human emotion identification. Among these ap- participants were asked to complete the Self-Assessment
proaches are the commonly used classification methods Manikin questionnaire at the end of each experiment [
Although most works in the literature contribute more lished the dataset in Matlab format 1.
to the classification of emotions than to the regression
of emotional dimensions, the regression approach
aiming to predict continuous values in the emotional plane 5. Methods
could be suggested in clinical contexts where it is
important to track the evolution of a patient’s emotional 5.1. Preprocessing approach
state during treatment. Deep learning (DL) techniques, One of the significant problems with EEG signals is the
such as autoencoder, deep belief network (DBN), convo- strong presence of artifacts (noise) or faulty EEG
chanlutional neural network, and recurrent neural networks, nels that can impair data analysis. An essential aspect
have been extensively used to build emotion recognition of our study was automated preprocessing to create a
architectures that outperform other standard machine user-friendly routine for acquiring real-time EEG signals.
learning approaches [
The preprocessing flow is critical because our prototype and a last to 64 neurons. A BatchNormalization was
peraims to provide a real-time detection system of the user’s formed at the end of the first two layers of filters. Each
emotional state. Our Preprocessing approach is also ap- layer was then condensed with the MaxPooling-1D in
orplied to the Dreamer dataset for training deep learning der to extract the most relevant correlation of engineered
models. Afterward, we checked the eficiency of the auto- features. The kernel size is kept at 3, and the activation
matic preprocessing technique by visually inspecting all functions are Relu for convolutional layers. At the end of
EEG trials. More specifically 414 samples were selected the convolutional layers, a Flatten operation is performed
containing information related to Valence, Arousal, and to create the input arrays for the next neural network.
Dominance values. The first pre-processing step was The neural network useful in order to predict regression
the removal of the DC ofset (DC ofset). Then the data values is a Fully Connected Layer composed of four
layformat conformed to the MNE framework respecting the ers, one of which is 128 neurons input, a second hidden at
10-20 standard. A notch filter calibrated to the cutof 128 neurons, and a third hidden layer at 32 neurons. The
frequency of 50hz was used to remove noise due to the activation functions are relatively Tanh for the first two
commercial electric current. The trial was normalized layers and Relu for the 32-neuron layer. Then a Dropout
in the frequency range 1-40Hz. Epochs of length equal operation of 0.2 was performed in order to regularize
to 1 second were created from the continuous EEG sig- learning to avoid overfitting. Finally, there is the last
nal. independent component analysis (ICA), was used three neurons’ output layer with linear output function
to remove all noisy epochs and for the identification of useful for the purpose of the regression task. The three
EEG signal components. All artifacts in the signal were classes we want to predict are Valence, Arousal, and
Domcorrectly removed. identification and interpolation of inance. During the learning, the mean_absolute_error
defective channels and epochs were performed using the was monitored as a loss function, and a callback was
pyprep framework 2. However, epochs exceeding a cer- set to stop learning if the loss did not improve after ten
tain noise threshold are removed and not interpolated. iterations. (Patience = 10). The optimizer chosen is the
Finally, the continuous EEG signal is reconstructed by Adam algorithm [
The main aim of our study is to provide an emotion 5.2. Training dataset recognition system that can provide real-time feedback on the user’s emotional condition. In order to achieve this After obtaining the preprocessed EEG signal, the data set goal, the minimum length in terms of seconds was sought is structured as follows: (i) 4 seconds epochs are extracted in relation to the greater level of accuracy of the R2 metfrom continuous EEG trial; (ii) Overlapp epochs are ob- ric. In practice, the minimum time that maintains the tained every 1280 samples. The new generated epochs levels of accuracy above 0.9% of R2 was sought, achieving have the same label as the originals; (iii) Theta, Alpha, periods not less than 4 seconds. In the same way, the optiBeta1, Beta2, Beta3 bands are extracted with neurokit2 mal overlap coeficient was chosen to maintain the value framework3. The band range considered are 4-8Hz in of the metric R2 not less than 0.9%. This optimization of Theta, 8-13Hz in Alpha, 13-16 Hz in Beta1, 16-20Hz in the hyperparameters was carried out experimentally to Beta2, 20-30Hz in Beta3. directly find the best solution that could avoid considering the eras of the EEG signal not too long but neither 5.3. DL Model Description too short. Assuming to use epochs of 1-second length, or 128 samples, is not representative of an emotional state. Increasing the length of the epoch, the value R2 increases, but consequently, it creates a problem relative to the time of scan of the signal EEG during the acquisition in real-time. 5.4. Interpretable model
predictions of the DL model for the independent test
set. SHAP represents the marginal contribution of each
input variable in the model’s decision-making process.
This algorithm is based on game theory and in particular
Shapley’s approach for evaluating the contribution of
each player in a cooperative game. SHAP introduces a
variant of Shapley’s approach through the use of a local
All the obtained features were initially split into train,
validation, and test with the sklearn train_test_split
library in proportion 80% for training and the remaining
20% for testing. The training dataset was then split into
75% for train and 25% for validation. After this
operation, the normalization was performed with the MinMax
scaler of sklearn [
contribution function, which calculates the contribution of each variable for each input instance of the test set.
For each output variable, we provide a waterfall plot in SHAP representing the contribution of each feature towards the final output of the DL model for a particular instance of data. The plot shows the base value, which is the expected output of the model when no features are observed, and the sum of the contributions of each feature to the final output for the given instance of data. Each feature is represented as a horizontal bar, and the length of the bar represents the magnitude of its contribution. Features that increase the output are shown in blue, while those that decrease the output are shown in red. The plot shows how each feature contributes to the ifnal prediction, highlighting which features are driving the model for the given instance of data.
tive accuracy: 2 = 0.93, Mean Absolute Error = 0.08, Mean Absolute Percent Error = 0.07. All metrics are calculated with sklearn.metrics. Fig. 3 shows the course of the function of loss during the learning phase. It can be observed that no overfitting occurs during the training of the model. The proposed architecture has been trained to predict valence, arousal, and dominance levels simultaneously, thus classifying human emotions dynamically over time. From this point of view the model could be used to track the emotional history of a subject in real-time.
The waterfall plots for each outcome shown in Figure 4 suggest that the selected features for the four channels have comparable impacts for all test samples, except for the feature AF3 Beta 3, whose impact is higher than the others for the predictions of the three outputs. This ifnding encourages further exploration of the impact of frequency features in the 20-30 Hz band and the use of frontal electrodes for human emotion recognition.
The proposed DL model for the three-dimensional regression of valence, arousal and dominance, constitutes a prototype for the continuous tracking of human emotional states and for explaining the impact of spatial and frequency features. It could provide efective information in clinical settings and be used as a tool to support diagnosis. Future developments include training with larger populations and the use of non-linear and complex features to complement the frequency features used in this work.
P.O. Puglia FESR 2014 – 2020 (project code 6ESURE5) ‘SECURE SAFE APULIA’, Fincons CdP3, PASSPARTOUT, Servizi Locali 2.0, ERP4.0. Also this work has been carried out while Tommaso Colafiglio was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome in collaboration with Politecnico di Bari.