In this paper it is proposed a deep learning technique for emotion classification using spectrogram images produced from the related electroencephalogram (EEG) signals. The idea is to use the classical, state-of-the-art deep learning techniques for image recognition applied to the spectrogram of the EEG signal. The goal is to detect and recognize the level of valence, arousal, dominance, and likability. Extensive experiments are carried on with diferent convolutional neural network architectures on the publicly available DEAP dataset in order to find the best possible model with respect to the accuracy of the prediction. A new data augmentation technique on EEG signals has been experimented with and validated. The model has been developed and evaluated by taking a random permutation of the dataset and partitioning it in 80% training, 10% validation, and 10% test. Doing so allowed us to assess the model's ability to recognize an individual's emotion based on the EEG signals of other individuals. Results show that the models can learn and detect emotions with high accuracy at the same level as the state of the art of analogous models already presented in the literature.
CEUR ceur-ws.org
The use of physiological signals for emotion recognition and detection has been the subject of several studies in the recent past. An emotion can be detected and recognized by a selfassessment of the subject or can be inferred from physiological signals. Subjective self-reports, while valuable, could raise validity issues. On the other hand, physiological signals can be used to evaluate emotions objectively. The ability to objectively assess the emotional state of a person could be employed in a wide variety of fields like (but not limited to) education, medicine, and entertainment with applications ranging from medical diagnostic, robotic and automatic assistance of motion impaired persons, validation of recommendation algorithms and, generally speaking, for brain-computer interface (BCI).
An emotion can be defined as a complex psychological state that involves three distinct
components: a subjective experience, a physiological response, and a behavioral or expressive
response. Various descriptions of these states have been proposed: In one approach, a discrete
categorization of emotions is devised as proposed by Ekman et al. [
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
both proposed several universal basic emotions such as anger, fear, sadness, disgust, surprise,
and joy. In this perspective, the emotions could be represented as a graph whose nodes are the
emotions and the edges are the links that connect similar emotions, and various algorithms can
be employed to explore the properties such graphs[
A second perspective of describing emotions is called dimensional approach (see Figure 1). In
this case, the emotions are described along three dimensions: valence, arousal, and dominance.
Valence goes from pleasant (very positive) to unpleasant (very negative). Arousal indicates how
much excitement is involved in the feeling of the emotion: from very excited and active, to
very calm, bored, and/or sleepy. Dominance indicates the degree of control one feels: from
being helpless and weak or without control to feeling in control of everything. We will refer to
them in the following as emotion dimensions. The dimensional approach allows us to assess
emotions quantitatively, and it is a common approach adopted in much of the literature on
emotion detection and, therefore, is the approach used in this paper. There can be many ways to
asses an emotional dimension: the subject self-assessment, physiological signals, and external
signals like facial expression. Physiological signals may range from Galvanic Skin Resistance,
Hearth Beat Rate, Respiration Rate, and Electroencephalogram (EEG). The last method attracted
much interest from the researchers because it directly obtains signals from where the emotions
start to form -the brain- (although the paper [
In this work, we developed deep learning models using the DEAP EEG recording dataset [
The spectrogram is a 2-dimensional array obtained from the EEG signal; each column of
represents the signal’s Power Spectral Densities (PSD) in the time interval . In other words,
(, ) is the power of the sinusoid of the signal at frequency in the time interval . We obtain a
spectrogram image or a 3-dimensional tensor by stacking all EEG channels together. Then, we
use the state-of-the-art convolutional neural network (CNN) architectures [
The work is organized as follows. In Section 2, we briefly cover the most significant related literature. In Section 3, we describe the dataset and the pre-processing algorithms we employed to obtain the tensors used as input to the models. In Section 4, we report the experiments’ results and the models’ evaluation. In Section 5, we give final remarks and directions for future works.
Emotion detection and recognition have been the subject of several studies in literature [
It is worth noting that the vast majority (if not all) of the works published in literature employ a 10-fold cross-validation scheme (see [19]). In this work, we also used a more classical evaluation scheme based on the partition of the dataset into three subsets: training, validation, and test sets. In the evaluation scheme containing a validation set, the hyperparameters of the model, the principal component analysis, and all the design choices about the model’s efectiveness are chosen using the validation set. During the fine-tuning of the hyperparameters, the best model is chosen based on its performance on the validation set. Once the choice is made, it is tested on a test set to assess the model’s performance and objectively evaluate its efectiveness. It is also worth noting that, since the datasets present in literature albeit contain a fair amount of data, their quantity is orders of magnitude less than the quantity of data contained in the analogous dataset commonly employed in the field of image detection [ 20] and object detection [21, 22] where billions of (labeled) images are easily provided. Therefore, the performances of the deep learning models developed in the literature for emotion detection from EEG signals are less impressive than the analogous performance in image detection and recognition.
We built and tested the prediction models using one of the best-known online datasets, the DEAP dataset. This dataset contains the EEG signals of 32 individuals that were collected while the subjects watched and listened to music videos taken from YouTube. Each subject was invited to view 40 one-minute videos and then asked to express her emotions on the dimensional model described above. In the following we refer to the EEG signals of one participant watching one video as an experiment or trial. Additionally, a parameter called likability was used to quantify how much the participant liked the stimulus. In the following we treat likability as another emotion dimension. For each emotion dimension, the participant was asked to rate its intensity on a continuous scale between 1 and 9, where 1 stands for minimum intensity, and 9 for maximum intensity. We transformed the continuous scale of each emotion dimension into a binary value () ∈ {0, 1} so that () = 0 if < 5 , () = 1 if ≥ 5 .
The EEG signal consists of 32 channels, each corresponding to an electrode that measures the diference in electric potential in the skull area where it is positioned.
We used for all the experimentation ResNet101 and VGG11 architectures, referred in the following as the CNN architectures. The input tensor to the CNN architecture has three dimensions denoted by the depth , the height and the width . Each data point of the input tensor will be denoted as (, , ) . In all subsequent configurations the width of the input tensor is the time.
We divided the raw signal of each of the 32 EEG channels in 34 overlapping segments, called time segments, each of which is composed by 256 data points and the amount of overlapping is 32 data points1. For each time segment the Discrete Fourier Transform (DFT) is computed with a Tukey window and from it we obtained the PSD (for more details on the definition of PSD and spectrogram please refer to [23, 24, 25]). Since each window is composed by 256 data points, the DFT returns a total of 129 diferent frequencies. These frequencies are between 0 and 64Hz, since the signal available in the DEAP dataset has been (down)sampled at 128Hz. Since the DEAP dataset’s preprocessed data have a bandpass frequency filter from 4.0-45.0Hz, we excluded all the frequencies outside such interval (which are 0), eventually obtain 2(45 − 4) + 1 = 83 diferent frequencies. Figure 2 shows: in (a) an example of the original signal and in (b) the PSD of one window.
Therefore in its basic configuration the input tensor has dimensions = 32 , = 83 and = 34
(see Figure 3 for a heat map of the channels’ spectrograms stacked together). We called it the
normal, configuration. However we developed other two configurations of the input tensor.
The second configuration is obtained by the first one by subtracting to each time segment the
PSDs of the first 3 seconds of recording, in which the subject has been exposed to a neutral
1The amount of overlapping has been chosen by following the best practices of the digital signal processing
community.
stimulus. We called it as the delta configuration. A third configuration has been obtained in the
following manner. Three frequency intervals have been chosen: theta (4-7 Hz), alpha (7-13 Hz)
and beta (13-30 Hz) according to the paper [
We developed three diferent way of data normalization. In the first, called ifne normalization, for each time slot , for each frequency and for each EEG channel we computed the maximum 1(, , ) for all the trials. We first compute The second type of data normalization, called coarse normalization, is obtained by computing the maximum 2() , along all the trials for each EEG channel . In this case we first computed 2(, , ) = (, , )
2() 1(, , ) = (, , ) 1(, , ) 1(, , ) = 1(, , ) − 1(, , ) 1(, , ) and after this we computed the average 1(, , ) and the standard deviation 1(, , ) for all the trials using the values of 1. Then each data point of the normalized tensor 1 is given as and after this we computed the average 2() and the standard deviation 2() along all the experiments for each EEG channel using the values of 2. Then each data point of the normalized tensor 2 is obtained as 2(, , ) = 2(, , ) −
2() 2() in the third type of data normalization, called all normalization, we computed the maximum of all data points of the input tensor. Then,
3(, , ) = 3(, , ) = 3(, , ) − (, , ) and after this we computed the average and the standard deviation is of all data points of the tensor 3. Then each data point of the normalized tensor 3 is obtained as
Other type of configurations has been obtained by swapping the height axis (frequencies) and the depth axis (channels) of the first two configurations. We call these configurations as obtained by the swap of frequencies.
We employed a data augmentation scheme which consists in a random shift of the input tensor, after normalization but before the swap of frequencies, by an amount of up to 15% or 20% or 22% of the width of the tensor. This is motivated by the fact that a particular emotion may be felt by one subject at certain time while other subjects may feel the same emotion slightly before or slightly later. We used, in the backpropagation algorithm, the cross entropy (CE) loss function. Generally speaking, the CE gives a measure of the distance between two probability distributions and over the same domain = {1, 2 … , } , i.e. a set of classes. Its definition is (, ) = ∑ () log =1
1 () (1) Let = ()
be the -dimensional output of the CNN architecture where is the input tensor and is the number of classes of the model. Let be the ground truth (GT) class corresponding to . Applying the softmax function to we obtain a -dimensional vector such that 0 ≤ () ≤ 1 and ∑
=1 () = 1 . The predicted class ̂ , is the class for which ( ̂ ) ≥ () for ∈ . If we interpret as a probability function, the best model should produce a that minimizes the CE between the GT probability distribution () . It is common to assume that () = 0 if ≠ and () = 1 if = . Suppose, for the sake of simplicity that = 2 , as in all of our models, and suppose that to the input , with GT class corresponds a probability () = 0.49 if ≠ and () = 0.51 if = . Clearly, in this case, the model produces a which is quite distant to . Nevertheless the model’s predicted class is correct. In other words, any probability function such that ( ) > () for all ≠ is a correct output of the model regardless its distance from the GT probability . Given these observations, we applied in our experiments the label smoothing [26] scheme. In other words we relaxed the requirement that () = 1 if = and () = 0 otherwise, and assume only that ( ) > () for all ≠ . We tested the label smoothing scheme by choosing ( ) to be equal to 0.75 or 0.9 or 1.0.
Counting all the possible combinations of hyperparameters we obtain a total of 864 diferent configurations, for each CNN architecture, denoted in the following as grid points.
We used Pytorch framework [27] for the CNN models and the package Scipy [28] for computing spectrograms and DFTs of the EEG signal. We made publicly available the software we developed in a Colaboratory notebook2. In order to asses which configuration gives out the best model we conducted a grid search on the 864 grid points described in the previous section. Both Resnet101 and VGG11 architectures were modified in order to accept in input a tensor that has depth not necessarily equal to 3, which is the fixed default in these architectures since they were originally developed for RGB image detection and recognition. Every model outputs a binary value which represents the high/low value of the corresponding emotion dimension.
We applied a backpropagation algorithm based on SGD, with a batch size of 32, momentum=0.9, weight_decay=0.0005. We employed a warm up scheme in which the initial learning rate is first decreased by a factor of 10−3 and then linearly increased in the first 32 epochs. After this warm up, the learning rate for all grid points without a pretrained model is 3.0 × 10−3 and for the pretrained models is 1.0 × 10−3. In both cases and after warm up, the learning rate is decreased every 20 epochs by a factor of 0.85. The total number of epochs for all grid points is 440.
In order to test the efectiveness of the models we conducted a completely randomized experiments. To do so we computed a random permutation of the 1280 experiments of the dataset. Of the first 256 experiments of the permutation 128 has been used for the validation and the other 128 for test set. The remaining 1024 are used for the training set. In Table 1 are reported for each emotion dimension and each set the quantity and percentage of experiments labeled 1.
We also conducted several -fold analysis. In the first one we computed a random permutation of the 1280 trials and used a 10-fold analysis by considering 128 trials for the test and the other for the training. Furthermore we conducted a 32-fold analysis by considering 40 trials of a 2https://colab.research.google.com/drive/1g3uV0uI7cFlOOew3rzfRz6BzsINgkgRO?usp=sharing single participant as a test and the others as a training. This last experiment has been conducted because most of the works in the literature report using this kind of evaluation analysis.
The experiment setup is as follow. We first conducted a grid search over all the grid points described above using only the ResNet101 architecture. Based on the results of the grid search we selected the best hyperparameters. Then we train again both ResNet101 and VGG11 on these hyperparameters. During this last training we saved the models which achieved the best accuracy on the validation set in the last 340 epochs. We repeated this training for 30 times overall collecting 30 ResNet101 and 30 VGG11 models for each of the four emotion dimensions. We observed a lot of variability during the training phase even when the learning rate becomes very small and the model overfits the training set. For example, in Figure 4 we have the accuracy plot for both the training and validation set for the best grid point of the model predicting Arousal. This suggested utilizing an ensemble of the models. We tested an ensemble of 30 and 15 of these models all from the same architectures. We also used an ensemble of 60 and 30 models from both CNN architectures.
In all the grid points the training average accuracy is over 0.750 and can easily reach 0.999, a clear sign that the models overfits quite efectively. Furthermore we observed that even when the model overfits and the learning rate is very small, there is still quite a lot of variability in the accuracy on the validation set. For each diferent type of emotion dimension and for each grid point, we computed the average, standard deviation and the maximum of the accuracy over the last 340 epochs for the validation set. Eventually we selected the hyperparameters reported in Table 2.
In Table 3 we report the results of the ensemble tests. In the ensemble test we run a number of models and the output is defined by the majority. The values reported are the accuracy obtained using diferent ensemble configurations. We tested single architecture (ResNet101 and VGG11) with 15 and 30 models ensemble. We also mixed 30 ResNet101 and 30 VGG11 (resp. 15). The row called average is the average of the accuracy of the four emotion dimensions. The row Avg. single models represents the average of the accuracy over all ∈ {15, 30} single models. The row diferences is the diference in percentage between the two previous rows. We can see that in all cases the ensemble of models improve the accuracy of both the validation and the test set. By mixing both architectures we obtain a gain of more than 4% in accuracy.
In Table 4 are reported the results of the -fold experiments. In the -fold experiment the dataset is partitioned in disjoint parts. One part is the test set and the other − 1 will comprise the training set. We did two kind of experiments. In the first one a random partition of the trials is determined; this partition is composed by 10 elements each of which contains 128 trials. In the second case the partition is composed by 32 elements each of which contains the 40 trials of one participant. In all cases we record the maximum accuracy of the model over the test set and averaged this number over all the tests. In the columns containing the label . > 100 , we took the maximum accuracy on the test set restricted over the last 340 epochs of the training because after the epoch 100 the model is more stable. When the column contains the label . > 0 we took the maximum accuracy on the test set over all the epochs. We see that in the case when = 32 participants and we take the maximum accuracy over all the epochs, we obtain the best accuracy results which are quite comparable to the best already achieved in literature.
In this work we developed a new technique for emotion detection using the EEG signal. We
stacked together the spectrogram of each EEG channel in order to obtain a 3-dimensional tensor
which encloses frequencies, temporal and spatial information. In this way we transformed the
emotion detection problem in an image recognition problem. Therefore we were able to use the
state-of-the-art convolutional architectures which proved quite efective on image detection.
Future works aims to apply the same methodology to the other dataset already available in
literature. Another line of research is to integrate to the EEG signals other types of biological
signals (ECG, GSR, etc.) since, as reported in literature [
URL: https://dx.doi.org/10.21227/barx-we60. doi:10.21227/barx- we60. [17] M. Soleymani, J. Lichtenauer, T. Pun, M. Pantic, A multimodal database for afect recognition and implicit tagging, IEEE Transactions on Afective Computing 3 (2012) 42–55. doi:10.1109/T- AFFC.2011.25. [18] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255. doi:10.1109/CVPR.2009.5206848. [19] A. Topic, M. Russo, Emotion recognition based on eeg feature maps through deep learning network, Engineering Science and Technology, an International Journal (2021). [20] P. Goyal, M. Caron, B. Lefaudeux, M. Xu, P. Wang, V. Pai, M. Singh, V. Liptchinsky, I. Misra, A. Joulin, P. Bojanowski, Self-supervised pretraining of visual features in the wild, CoRR abs/2103.01988 (2021). URL: https://arxiv.org/abs/2103.01988. arXiv:2103.01988. [21] R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, M. Goldblum, A cookbook of self-supervised learning, 2023. arXiv:2304.12210. [22] A. Ferrato, C. Limongelli, M. Mezzini, G. Sansonetti, Using deep learning for collecting data about museum visitor behavior, Applied Sciences (Switzerland) 12 (2022). URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85122372298&doi=10.3390% 2fapp12020533&partnerID=40&md5=34cbc3ee04afb84f4911d6172acac2f7. doi:10.3390/ app12020533, cited by: 11; All Open Access, Gold Open Access, Green Open Access. [23] M. X. Cohen, Analyzing neural time series data: theory and practice, MIT press, 2014. [24] P. Welch, The use of fast fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms, IEEE Transactions on Audio and Electroacoustics 15 (1967) 70–73. doi:10.1109/TAU.1967.1161901. [25] W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipes 3rd Edition:
The Art of Scientific Computing, 3 ed., Cambridge University Press, USA, 2007. [26] M. Mezzini, Empirical study on label smoothing in neural networks, Computer Science
Research Notes 2802 (2018) 200–205. URL: www.scopus.com, cited By :1. [27] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035. URL: http://papers.neurips.cc/ paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf. [28] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, SciPy 1.0 Contributors, SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods 17 (2020) 261–272. doi:10.1038/s41592- 019- 0686- 2.