Unsupervised-based framework for aged worker’s stress detection Gabriele Rescioa, Alessandro Leonea and Pietro Sicilianoa a National Research Council of Italy, Institute for Microelectronics and Microsystems, Via Monteroni, c/o Campus Ecotekne, Palazzina A3, Lecce, Italy Abstract The new industry 4.0 paradigm provides for totally automated and interconnected industrial production processes that require greater human-machine interaction. This involves the onset of new problems related to the stress evaluation of the aged worker which is found to operate in new and more complex work contexts. In literature several works for human stress detection are presented, they use above supervised machine learning with high accuracy level detection, but needed a complicated training phase. Moreover, a relevant issue in the field of stress detection lies in the model validation, indeed the commonly questionnaires used to record perceived stress levels are prone to subjective inaccuracies. To reduce this limitation, in this paper an unsupervised machine learning based stress detection system, in which the labels from perceived stress levels are not needed, is presented. It analyses heart rate, galvanic skin response and electrooculogram signals, relevant for the detection of excessive stress and cognitive load. The developed architecture software has been experimented in laboratory contest and preliminary obtained results appear promising. Keywords 1 Stress monitoring, Unsupervised learning, Wearable sensors. 1. Introduction The Industry 4.0 represents a fundamental shift in manufacturing that will transform both processes and people. This emphasizes the need to integrate the human operator in a socially sustainable way into the new production paradigms [1]. Nowadays, it is common to design a workplace according to the requirements of the operator and the increase of automation (e.g., cobots, exoskeletons) is reducing the physical effort of workers. However, the activities of the operator 4.0 will entail an increased share of complex cognitive tasks. This is a challenge especially for aged workers who have to operate in this new context. Therefore, more attention should be paid to emerging stressors and risks to mental health [2]. Over the years, the most widespread technique to assess human stress concerns the use of psychological questionnaires. The main limitation of this type of survey is that it does not allow real- time and continuous monitoring. This makes it impossible to identify the causes and work tasks that are stressful for the worker. A potential solution to overcome this problem is to consider bodily responses, such as physiological signals (e.g. skin temperature, breathing rate, heart rate, …) in terms of stress index. These parameters can be measured accurately and continuously through ambient or wearable sensors-based devices. The ambient sensor-based monitoring technologies (e.g. Ultra-Wide Band systems, video cameras, …) are less intrusive with respect to the wearable devices, but they require a complex and ad hoc design of environments to allow accurate measurements. In the last years, through the development of miniaturized technologies, minimally invasive wearable sensors Workshop on Artificial Intelligence for an Ageing Society (AIxAS 2020), November 25–27, 2020 EMAIL: gabriele.rescio@cnr.it (G. Rescio); alessandro.leone@cnr.it (A. Leone); pietro.siciliano@le.imm.cnr.it (P. Siciliano) ORCID: 0000-0003-3374-2433 (G. Rescio); 0000-0002-8970-3313 (A. Leone); 0000-0002-1312-4593 (P. Siciliano) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) have been realized and this has favored their diffusion in the monitoring of health parameters and to prevent dangerous events in the healthcare field [3, 4]. In literature, several studies adopt supervised machine learning technique for the stress event classification. The main analysis physiological parameters are heart rate, breathing rate, blood pressure, skin temperature, galvanic skin response and recently electrooculogram. The accuracy values measured for these works is very high, up to 99% [5]. However, they required a complex training phase, based on labelled dataset of simulated events that it could be inaccurate and different from real-world data [6]. This issue could be reduced using an unsupervised approach. Unsupervised learning technique is particularly suitable for analyzing data that doesn’t need to be associated to a labelling information. It can also be used in many applications related to data mining, such as the analysis of electrophysiological signals. In literature few works deal with unsupervised detection of mental stress and to the best of our knowledge, deep learning framework based on unsupervised detection of acute mental stress, in industrial context, using heart rate (HR), electrical dermal activity (EDA) and electrooculogram (EOG) signals has not been investigated before. This paper proposes a stress assessment system based on an unsupervised learning technique to monitor the operator’s mental load with the aim to increase his wellbeing in the working environment. It exploits minimally wearable devices and combines HR, EDA and EOG measurements in order to automatize stress detection conditions. To analyze the framework performance, traditional mental stress test and simulations of manufacturing activities were performed. 2. Materials and methods In this section the hardware-software architecture and the procedure used for the physiological signal acquisition are described. 2.1. Multisensory platform The main computational steps of the software architecture are a) pre-processing, b) calibration, c) feature extraction and d) Clustering. The algorithmic framework has been developed using three low invasive commercial wearable devices: 1) the ES_R glasses produced by J!NS MEME [7] for EOG signals, 2) the E4 wristband produced by Empatica [8] for EDA signal, 3) the Bioharness 3.0 chest strap band produced by Zephyr [9] for HR signal. The HR signal is also acquired from the E4 device, but it was verified that its accuracy is lower than that of Bioharness. Therefore, it was preferred to use 3 devices (shown in Figure.1) instead of 2. Figure 1.Wearable devices. a) J!NS MEME ES_R glasses, b) Zephyr Bioharness 3.0 chest strap, c) Empatica E4 wristband. The J!NS MEME ES_R acquires the EOG data through three-point electrooculography sensors placed on the nose pads glasses for the evaluation of vertical and horizontal EOG components. The glasses send data through a low energy Bluetooth connection. The autonomy of the rechargeable lithium-ion battery is about 16 hours. The sample rate for the presented work is 50Hz. The Empatica E4 wristband acquires EDA signal to measure the sympathetic nervous system arousal through two metal electrodes placed on the skin. The sample rate of EDA signal is 4Hz. The wristband is equipped with a low energy Bluetooth connection and a proprietary mobile App, for the visualization and the storing of data. Moreover, it is possible to record the data using the on-board memory. The Bioharness 3.0 band measures the HR through electrode sensors housed within a chest trap and an electronic module integrated in the trap. The Bioharness 3.0 operates either in streaming mode for real-time data viewing on a mobile device using Bluetooth low energy or in recording mode using its internal memory. 2.2. Procedure In order to analyse the signals coming from the device and to test the framework software, a data collection has been defined in controlled (simulated) conditions by involving 7 female and 4 male volunteers (age 31.3±5.8). They were recruited at multidisciplinary laboratory of Engineering department of Ancona’s University within a funded project by the Italian ministry for Education and Research. All participants were asked to wear the aforementioned wearable devices and perform five tasks, separated by a rest period of two minutes. The experiment procedures are detailed in [10] and they have provided the simulation of manufacturing activities, by using LEGO bricks and a toolbox. Moreover, the volunteers accomplished mental arithmetic test, frequently used in the literature to induce stress [11]. At the end of the experiments, users were asked to judge the cognitive load generated by all tasks. The mental arithmetic tests and complex LEGO assembly, without instructions, resulted the most stressful unanimously. So, more than 90 stress and 130 rest events were identified and accordingly labelled for the performance evaluation. During the data collection each user wore the three devices: the data coming from the wristband and chest strap devices were stored on the memory integrated on board, while the glasses sent the data to an embedded PC. The data so acquired were synchronized and analyzed in off-line mode through the mathematical computing software Mathworks Matlab [12]. 2.3. Data Analysis In this section it is detailed the analysis of the raw data, acquired during the laboratory tests described in the previous paragraph. The main computational steps of the software framework architecture are listed below. 2.3.1 Pre-processing and calibration The aims of data pre-processing step are to reduce environmental electrical noise and the artefacts due to device movements. Regarding EDA signal, the technique implemented in Kocielnik et al [13] was used to remove signal artefacts. This technique eliminates the anomalous signals with respect to the shape characteristics of conductance spikes in normal skin. In particular, it was eliminated the signals that present changes more than 20% per second or decreased more than 10% per second. The EDA signal consists of phasic and tonic components. The tonic component is related to the slow changing baseline levels and to the individual background characteristics. While the phasic component shows the fast skin conductance response due to a stimulus measured over a short duration of time and it can be event related. To extract the phasic component a Butterworth band-pass filter with cut frequencies between 0.16Hz and 2Hz was implemented. Considering the EOG signals, the Empirical Mode Decomposition (EMD) technique [14] and the glasses acceleration data were examined in order to reduce the noise and the artifact due to the sudden head movements or to the contact losses between glasses electrodes and nose skin. Basically, it was eliminated the Vertical and Horizontal EOG peak signals in correspondence of excessive accelerometer module peaks, calculated by using the data coming from the tri-axial accelerometer integrated in the glasses. Ten EMD IMFs from the EOG data were extracted, and the IMFs 2 to 6 empirically were selected and added to obtain the filtered data. For HR signal the interval between successive heart beats (RR intervals) coming from the chest trap were considered and the artefacts were filtered from RR sequences acquired, analyzing relative trusted RR intervals. In order to correctly handle pre-processed data and to reduce the inter-individual variability between different users, a calibration procedure was accomplished by recovering the initial conditions after the device mounting. To obtain this, the baseline of EDA, HR and EOG signals was calculated as the mean of the data acquired for 10 seconds, while the user was in rest condition, in the first phase of each data collection trial. Then the baseline value was used to normalize the pre-processed acquired signals. 2.3.2 Feature extraction and selection For the purpose of stress events assessment, attention was focused on several time domain features, commonly used in the analysis of stress detection [15-19] and the most significant features were chosen. They were calculated within a sliding window of 120 seconds for HR signal and of 5 seconds for both EDA and EOG signals. For EDA signal the following features were extracted: 1) the peak amplitude sum, 2) peak energy sum, 3) peak rise time sum, 4) normalized root mean square, 5) the mean of the first signal derivative, 6) the standard deviation of the first signal derivative, 7) variance. Regarding EOG signal the calculated features are: 1) eye blink frequency, 2) the averages of EOG peak-to-peak voltage amplitude of vertical and horizontal signal components 3) Variance of vertical and horizontal EOG components. Finally, for the HR the heart rate variability was calculated and then the following features were extracted: 1) Root Mean Square of the Successive Differences, 2) Standard Deviation of NN intervals, 3) the proportion of interval differences of successive NN intervals greater than 50 ms, 4) mean heart rate. To reduce the complexity of the signal processing and to enhance the performance of the system, a feature selection technique was applied. For this phase, the Laplacian score was used, it is suitable for ranking features for unsupervised learning [20]. In the Figure 2 the ranking for the signals is reported, the x-axis shows the number equivalent to the features described above, while the y-axis represents the features score; for each EOG signal feature the vertical and horizontal components were considered. Figure 2. Feature rank for a) EDA signal, b) HR signal, c) EOG signal (V and H indicate the Vertical and Horizontal EOG components). The x-axis shows the numbers equivalent to the feature lists described in this paragraph. In this way it was possible to choose the most important features for each signal. In particular, the following features were selected: 3), 2) and 1) for EDA; 3) and 2) for EOG; 1) and 3) for HR (the numbers refer to the feature lists described above). 2.3.3 Clustering After the features were extracted for all three devices, the feature vectors were applied as input to the classifier, implemented to distinguish the rest condition from stress condition, for each signal. The algorithm chosen for this phase is the commonly used and effective K-means clustering [21]. It is an unsupervised algorithm that identifies k number of centroids, through an iterative procedure of positioning optimization, and then allocates each data sample to the nearest cluster, based on the distance between the sample and centroid. The algorithm uses specific types of distance metric (sqeuclidean, cityblock, cosine, correlation, ...), whose can be chosen to improve the clustering performance. 3 Results To evaluate the performance of the stress detection system, compared to different distance functions of k-means, the average sensitivity and specificity according to the equation (1) and (2) [10] were calculated. The sensitivity indicates the ability of the framework to recognize a human stress condition, while the specificity is the ability to reduce the false event detections. = (1) = (2) where TP (True Positive) represents that a stress phase occurs and the algorithm detects it; FP (False Positive) indicates that a stress phase does not occur and the algorithm activates an alarm; TN (True Negative) means that a stress phase does not occurs and the algorithm does not detect it; FN (False negative) implies that stress phase occurs but the algorithm does not detect it. Moreover, the average clustering accuracy was calculated in percentage according to the equation (3). Accuracy = (3) These metrics were calculated for each device, considering the squeclidean, cityblock and cosine distance metrics. The acquired data was split and labelled in “no-stress” and “stress” phases as described in “Procedure” section. In Table 1 the results obtained with the best k-means parameters setting were reported for each signal. Table 1 Detection performance in terms of Sensitivity, Specificity and Accuracy Signal Sensitivity (%) Specificity (%) Accuracy (%) HR 72.9 65.3 70.6 EDA 76.8 70.6 74.6 EOG 67.5 63.4 63.7 As it is possible to verify from the table, the best performance was achieved by the EDA signals. To increase the reliability and efficiency of the stress detection system, the final prediction is given by a combination of the three outputs classifiers by using a weighted voting approach in which the EDA classifier has higher coefficient value (according to its clustering performance on a validation set). So, summing up all weighted votes and by selecting the class with the highest aggregate the new values for sensitivity, specificity and accuracy are 83.3%, 74.3% and 79.7% respectively. The algorithm was also tested by using the other unselected features, but performance has not improved. These performances are slightly lower than those of existing approaches for stress detection [5], but the use of unsupervised approaches makes the proposed system suitable in a real industrial context. Moreover, minimally invasive devices have been tested that also allow to identify stress conditions due to a high cognitive load, through the EOG signal. 4 Conclusions The paper proposes a multisensory and unsupervised machine learning based system for the stressful condition detection in industrial 4.0 context. The system is mainly addressed to elderly workers, who have to operate in this new context. It could be useful to select the most suitable tasks for each worker, optimizing its performance and wellbeing. The heart rate, electrodermal activity and electrooculogram signals were considered for the evaluation of cognitive load and stress. The developed framework was tested on data acquired during laboratory tests, in which mental arithmetic test and manufacturing activities by using LEGO bricks and a toolbox were simulated. The unsupervised k-means approach was used, and the results show good performance in terms of specificity, sensitivity and accuracy (about 80%) and it does not need the training phase, that it is usually inaccurate and heavy. Future researches will be addressed to investigate other unsupervised machine learning classifiers in order to improve the clustering performance. Moreover, the system will be test in a real industrial case study 5 Acknowledgements This work has been carried out within the PON REACT project, with the co-financing of the European Union – FESR or FSE, PON Research and Innovation 2014-2020. Authors would like to thank the staff of Università Politecnica delle Marche, for supporting the data collection. 6 References [1] D. Romero, J. Stahre, T. Wuest, O. Noran, P. Bernus, Å. Fast-Berglund, and D. Gorecky, “Towards an operator 4.0 typology: a human-centric perspective on the fourth industrial revolution technologies” In International conference on computers & industrial engineering, pp. 1–11, 2016. [2] T. Arai, R. Kato, and M. Fujita, “Assessment of operator stress induced by robot collaboration in assembly” CIRP Annals - Manufacturing Technology vol. 59, pp. 5–8, 2010. [3] B. Andò, S. Baglio, C. O. Lombardo, V. Marletta, “A Multi-Sensor Data Fusion Approach for ADL and Fall classification”, IEEE Transactions on Instrumentation and Measurement, vol. 65, N.9, pp. 1960-1967, 2016. [4] A. Leone, G. Rescio, L. Giampetruzzi and P. Siciliano, "Smart EMG-based Socks for Leg Muscles Contraction Assessment," 2019 IEEE International Symposium on Measurements & Networking (M&N), 2019, pp. 1-6. [5] Y. S. Can, .t Arnrich, C. Ersoy, “Stress detection in daily life scenarios using smart phones and wearable sensors: A survey”, Journal of Biomedical Informatics, Volume 92, 2019, 103139. [6] E., Kallio J., Mäntyjärvi J., Kyllönen V., Lindholm M., Gimel’farb G. (2017) “Unsupervised Stress Detection Algorithm and Experiments with Real Life Data”. In: Oliveira E., Gama J., Vale Z., Lopes Cardoso H. (eds) Progress in Artificial Intelligence. EPIA 2017. Lecture Notes in Computer Science, vol 10423. Springer, Cham. [7] https://jins-meme.com/. [8] https://www.empatica.com/. [9] https://www.zephyranywhere.com/. [10] A. Leone, G. Rescio, P. Siciliano, A. Papetti, A. Brunzini and M. Germani, "Multi sensors platform for stress monitoring of workers in smart manufacturing context," 2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), Dubrovnik, Croatia, 2020, pp. 1-5. [11] S. Serino, A. Matic, D. Giakoumis, G. Lopez, P. Cipresso (eds) “Pervasive Computing Paradigms for Mental Health”. MindCare 2015. Communications in Computer and Information Science, vol 604. Springer, Cham. [12] https://www.mathworks.com [13] R. Kocielnik, N. Sidorova, F. M. Maggi, M. Ouwerkerk and J. H. D. M. Westerink, "Smart technologies for long-term stress monitoring at work," Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems, Porto, 2013, pp. 53-58. [14] A. Zeiler, R. Faltermeier, I. R. Keck, A. M. Tomé, C. G. Puntonet and E. W. Lang, "Empirical Mode Decomposition - an introduction," The 2010 International Joint Conference on Neural Networks (IJCNN), Barcelona, 2010, pp. 1-8. [15] R. R. Singh, S. Conjeti, R. Banerjee, “A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals”, Biomedical Signal Processing and Control, Vol 8, Issue 6, 2013, Pages 740-754. [16] T. Ogawa, M. Takahashi, and R. Kawashima, “Human cognitive control mode estimation using JINS MEME” IFAC-PapersOnLine vol. 49(19), pp. 331-336, 2016. [17] H. Ledger, “The effect cognitive load has on eye blinking” The Plymouth Student Scientist, vol. 6(1), pp. 206-223, 2013. [18] Kim, Hye-Geum et al. “Stress and Heart Rate Variability: A Meta-Analysis and Review of the Literature.” Psychiatry investigation vol. 15,3 (2018): 235-245. [19] R. Zangróniz, A. Martínez-Rodrigo, J.M. Pastor, M.T. López, A. Fernández-Caballero. “Electrodermal Activity Sensor for Classification of Calm/Distress Condition”. Sensors (Basel). 2017;17(10):2324. Published 2017 Oct 12. [20] Xiaofei He, Deng Cai, and Partha Niyogi. 2005. “Laplacian score for feature selection”. In Proceedings of the 18th International Conference on Neural Information Processing Systems (NIPS'05). MIT Press, Cambridge, MA, USA, 507–514. [21] K. P. Sinaga and M. Yang, "Unsupervised K-Means Clustering Algorithm," in IEEE Access, vol. 8, pp. 80716-80727, 2020.