In the eld of A ective Computing, one of the most important issues is the identi cation of the emotional state of a subject. There are a plethora of research works in emotion identi cation, works that have their foundations in other elds such as philosophy, psychology, neuroscience, and cognitive sciences. Nowadays, with the emergence of wearable devices and DIY electronics kits, the interest in developing emotion identi cation systems with these low-cost devices has gained more attention. The use of low-cost devices came out with new challenges related to the low quality of the signals acquired due to less noise-tolerant sensors which are used in real-life environments. In this context, the main objective of this work is to present a methodology, based on machine learning techniques for time series forecasting, to build models able to identify emotional states, from signals acquired from low-cost devices, as accurately as a professional medical device can do. To this end, we proposed the use of two devices: Nexus-10 MKII, a biofeedback and neurofeedback system from MindMedia, used to obtain reference measure, and BiTalino (r)evoltuion Boar Kit (BiTalino hereinafter), a low-cost physiological signals acquisition device from PLUX Wireless Biosignals S.A. In this work,11 Machine Learning models have been developed to predict the emotional state, identi ed by Nexus-10, with the signals provided by BiTalino. Our experiments show that the best model was a Random Forest which can predict the emotional state in the test set with a RM SE of 0:172 and a R2 of 0:858.
Emotions are a fundamental part of human behaviour and, as pointed out in
[
From its beginnings, AfC research has been focused on developing systems
able of 1) human emotion identi cation, 2) expressing emotions and 3) \feeling"
emotions [
Emotion identi cation requires representational models in which identi ed
emotional states could be measured. Multiple models have been proposed by
researches of a wide range of elds, ranging from psychology and philosophy
to neuroscience and cognitive science (see [
In the eld of neuroscience, relevant studies have revealed a correlation
between the response of the Autonomous Nervous Systems (ANS) to human
emotions and the valence-arousal plane [
Although, huge number of medical devices are available for acquiring these
signals from the medical community, currently there is a growing interest in
developing emotion recognition systems using low-cost devices, such as wristbands
and electronic DIY kits [
To obtain the required data, an experiment was performed with the collaboration of students of the University of Murcia. Students between 18 and 28 years old have been studied. The volunteers have been contacted individually, following a methodology for the experiment, giving them an appointment with exact date and time. The experiment consists of the visualisation of a collection of 40 well known paintings, arranged randomly for each participant. While the subjects visualise the stimuli, the necessary physiological signals are acquired through the sensors of the mentioned devices. The data have been treated with the utmost con dentiality, in accordance with Spanish Law 3/2018 of 5 December on the Protection of Personal Data and Guarantee of Digital Rights.
After accommodating the subject as best as possible with the sensors in place, the phases of the experiment are remembered. It's also reminded that is crucial that, for the duration of the entire experiment, the subject must look at the screen. 1 Nexus-1 is Medical CE certi ed and FDA registered again. This phase allows us to know the rhythms and frequencies of the subject when they are in a state of minimal activity. { Basal Phase. This phase lasts 60 seconds. During this phase, the subject has to look directly at a black screen. This phase makes possible to know what is the \normal" state of the subject when they are active without receiving any stimuli. { Stimulus Phase. In this phase, the subject will observe a collection of forty well-known paintings randomly arranged on the screen. These paintings are the visual stimuli that are projected individually one after the other, with a duration of 8 seconds. { Basal Phase. Another basal phase exactly the same as the rst one.
Once the experiment is nished, all recordings are stopped and the corresponding les, with the collected data, are saved. Sensors are then removed from the subject and cleaned. 3
During the experiment, two di erent devices have been used for physiological signals acquisition. To obtain a reliable emotional index for training the machine learning model, NeXus-10 has been used. Nexus-10 is capable of acquiring multiple physiological signals: EEG (2 derivations), EOG (electrooculography), GSR, BVP and temperature. For the objective of this work, GSR and BVP (Blood volume Pulse) signals have been considered. During the experiment, EEG (2 derivations), EOG and eye-tracking information have also been acquired for other purposes beyond the scope of this work.
The other device used isBITalino, from Plux Wireless Biosignals S.A. BiTalino is a physiological signal acquisition device which is based on similar projects, such as Arduino and Raspberry Pi. It is a low-cost, modular, multi-purpose, easily accessible and con gurable acquisition device capable of capturing multiple physiological signals in real-time: Accelerometer, ECG, EDA (Electrodermal Activity), EEG (1 derivation), EGG (Electrogastrography), EMG (Electromyography), EOG, temperature and light. For acquiring the pulse signal, a pulse sensor, connected to one of the analogical channels has been used. Its cost and its open hardware and software philosophy make BiTalino a very interesting tool for developing projects.
In this work, the following physiological signals have been acquired: { NeXus-10 MKII : GSR and BVP, both at a sample rate of 32Hz. GSR sensor is placed in the proximal phalanges II and III of the left hand. BVP sensor is placed in the distal phalanx II of the right hand. { Bitalino: GSR and Pulse both at a sample rate of 1000Hz. The GSR sensor is placed in the middle phalanges II and III of the left hand. The BVP sensor is placed in the distal phalanx I of the left hand.
These sample rates produces signals of 17440 samples for the Nexus-10 and 545000 samples for signals for the BiTalino.
Despite the quality of signals acquired with NeXus-10 MKII, some
processing is needed. As we are interested in the Skin Conductance Level (SCL), the
tonic component of the GSR, a Continuous Decomposition Analysis using
Nonnegative Deconvolution have been applied to the GSR signal using Ledalab
Software [
Signal acquired through BiTalino required some processing to remove both noise and artefacts. First, a Butterworth lter of order 3 and cuto frequency of 2.5 Hz has been applied. The ltered signal is then processed by a Savitzky-Golay lter of order 1 and a frame length of 75. No prepossessing has been done on the EDA signal. The gure 2 shows a comparison between the signal BVP obtained by the NeXus-10 MKII and the BITalino after ltering.
As BiTalino and Nexus-10 acquired signals at di erent sample rates, a downsampling process was applied to BiTalino signals to equal the number of samples and synchronise the timestamps. Then, the signals acquired were processed and segmented according to the stimuli presented. Finally, for each painting, four time series, each one composed of 364 samples, have been obtained.
In this work, the Emotional Index (EI) is calculated as proposed in [
Using this approach, the emotional state of a subject can be de ne as: (1) (2) (3) where and
EI = 1 = 32 + 2 # # if SCLz otherwise 0; HRz
0 # = arctan(HRz; SCLz)
SCLz and HRz represent the Z-score variables of the SCL and HR, acquired from Nexus-10, respectively. The and required for the transformation are calculated from the corresponding signals acquired during the 2 baselines phases (at the beginning and the end of the experiment). The EI, obtain through t1, 2 and 3 equations, varies between [ 1; 1], where positives values are associated with positive emotions and negative values to negative emotions. Once EI has been calculated, all the signals are downsampled to produce one sample per second. At the end, a dataset with 13832 samples is obtained. 6
Once signals have been processed and EI has been calculated, for each stimulus three temporal series, BITalino EDA and pulse signals together with the EI, are used to create a multivariate time series. Therefore, the problem for building a model for emotion identi cation from BITalino can be approached as a multivariate time series forecasting problem using Machine Learning Techniques.
To build the model, di erent data set con gurations, with a di erent number of lagged variables, have been tested: { Forma 1. No lagged variables considered, to predict EIti only values of
GSRti and HRti are taken into account as predictors. { Forma 2. Two lagged version of predictors has been added to the previous data set producing two new datasets: Forma 2 v2 and Forma 2 v3 with one and two lagged versions of GSR and HR respectively. { Forma 3. Two lagged version of the EI has been added to Forma 2 datasets generating two new datasets: Forma 3 v2 and Forma 3 v3 with one and two lagged versions of EI respectively. { Forma 4. Two new datasets have been created: Forma 4 v2 and Forma 4 v2 with with one and two lagged versions of EI added to Forma 1 respectively.
From the original dataset, 20% of the samples have been reserved for testing. In this work, we have considered the following regression models: Linear, Knn, CART (Classi cation and Regression Trees), Random Forest, Bayesian Ridge, Lasso, Linear SVM (Support Vector Machines), -SVM, -SVM, SGD (Stochastic Gradient Descent) and Multilayer Perceptron. All the models have been trained over the seven datasets previously generated using 10 folds strati ed cross-validation with a grid hyperparameter search. RSM E and R2 have been chosen as performance measures. At the end of the process, 77 models were generating (11 regression models 7 datasets).
First of all, in order to reduce the number of models to be analysed, for each model, the best pair (model, dataset), according to their evaluation in the test set, has been chosen (Table 1).
Model Dataset RMSE (train) RMSE (test) R2(train) R2(test) LR forma3 v3 0:393 0:011 0:387 0:092 0:502 0:024 0:410 0:183 KNN forma4 v3 0:367 0:009 0:443 0:078 0:570 0:017 0:164 0:485 CART forma3 v3 0:000 0:000 0:562 0:084 1:000 0:000 0:259 0:303 RF forma3 v2 0:191 0:014 0:172 0:084 0:881 0:016 0:858 0:177 BRR forma3 v3 0:393 0:011 0:387 0:092 0:502 0:024 0:410 0:183 Lasso forma4 v3 0:395 0:011 0:384 0:100 0:498 0:026 0:426 0:196 Lin-SVM forma4 v3 0:176 0:020 0:179 0:079 0:433 0:070 0:305 0:250 -SVM forma1 0:563 0:006 0:561 0:055 0:021 0:013 0:266 0:318 -SVM forma3 v3 0:155 0:009 0:168 0:076 0:503 0:0243 0:350 0:248 SGD forma4 v3 0:396 0:013 0:390 0:090 0:495 0:032 0:400 0:184 MLP forma3 v3 0:394 0:014 0:412 0:090 0:501 0:033 0:385 0:366
In order to determine if the observed di erences in performance are statistically signi cant, statistical hypotheses tests have been applied. Due to the small number of sample in each group, if di cult to prove the parametric assumption (normality and sphericity), therefore the non-parametric Friedman's test has been conducted, rendering an 2 of 56.44 and 45.22 for RSM E in train and test data and 58.45 and 40.84 for R2 in train and test data, which are considered signi cant (p < 10 4).Additionally, Nemenyi's Post-Hoc Test tests were conducted and revealed that, in the case of RSM E in test data: { CART performs signi cantly di erent than lasso, RF , -SVM and linear
SVM, with p-values 0.029,0.001,0.001 and 0.029 respectively. { mlp performs signi cantly di erent than RF and -SVM with p-values 0.007 and 0.017 respectively. { -SVM performs signi cantly di erent than RF and -SVM with p-values 0.022 and 0.049 respectively.
After evaluating the results, two models stood out from the others: RandomF orest and the -SVM. Although -SVM has a slightly higher RM SE value, the Random Forest algorithm was chosen, as the RM SE di erence is approximately 0:005 while the Random F orest R2 value is approximately 0:508 (out of 1) higher than -SVM R2 value and also present less variability. Another conclusion is that CART is the worse model and the unique model presenting over tting. Summarising, Random F orest the best model for predicting EI from GSR and Pulse signals provided by BiTalino, with an RM SE of 0:172 and an R2 of 0:858 on test set. The gure 3 shows an example of algorithm prediction. In this work, a methodology, based on Machine Learning techniques, for building models for emotions detection with low-cost hardware. As low-cost hardware, BiTalino from PLUXWireless Biosignals, S.A. has been chosen, and the results obtained show a good performance of the models obtained, producing reliable predictions of the Emotional Index EI very close to those obtained by medical certi ed equipment as the Nesux-10-MRKII of MindMedia. Another important advantage is that, through the process described here, a big part of the signal processing stack could be avoided.
Another important conclusion, based on model performance measures, is that the use of lagged variables, in our case 6 (two for each time series) is a good approach to overcome problems due to noise in signal acquisition.
Among future works, we are working in real-time implementation of the generated models. To this end, a real-time version of the two lters considered are being implemented. Apart from this, new experiments are being scheduled to increase the size of data sets. Another line is focused on the implantation of the lters on hardware or rmware.