Toward a technological oriented assessment in psychology: A proposal for the use of contactless devices for Heart Rate Variability and facial emotion recognition in psychological diagnosis Raffaele Sperandeo1, Alfonso Davide Di Sarno1, Teresa Longobardi1, Daniela Iennaco1, Lucia Luciana Mosca1, Nelson Mauro Maldonato2 1 SiPGI - Postgraduate School of Integrated Gestalt Psychotherapy, Torre Annunziata, Naples, Italy 2 Department of Neuroscience and Reproductive Sciences and Odontostomatology of the Uni- versity of Naples Federico II, Naples, Italy raffaele.sperandeo@gmail.com Abstract.Diagnosis is a complex cognitive process that takes shape within an interpersonal relationship. It aims at the evaluation of mental and affective processes that make the patient suffer, through their classification and identifi- cation of the mechanisms and psychological factors that originated them. This process can be made more effective thanks to the introduction in the diagnostic context of technological tools, non-intrusive and relatively simple to use for the detection of biomedical parameters. In the first section, this work highlights some of the critical issues related to psychological diagnosis; subsequently the methods of detecting physiological parameters, such as the Heart Rate Variabil- ity and facial expressions related to the patient's emotional fluctuations, are de- scribed. Finally, the concept of diagnosis will be introduced, assisted by compu- tational methods, aimed at supporting the work of the clinical psychologist in the complex procedure of diagnosis. Keywords:Psychological Diagnosis, Heart Rate Variability, Emotion Recogni- tion, Kinect v2 1 Introduction: Beyond the limits of diagnosis In 2003 APA [1] defined psychological diagnosis as the evaluation of abnormal behavior and mental and affective processes that are maladaptive and / or a source of suffering through their classification in a recognized diagnostic system and the identi- fication of the mechanisms and psychological factors that originated them and main- tain them [2]. Diagnosis is also a cognitive process that takes place within an interper- sonal relationship, which is its basis and influence it. This process allows the identifi- cation of a psychopathology, if it is present, and can provide data necessary for the structuring of an effective therapeutic plan [3, 4, 5]. In this perspective, the relation- Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 ship established between the psychologist who performs the psychological evaluation and the patient is of fundamental importance. In this paper, after discussing the limits of current diagnostic models, a computational approach to diagnosis in psychopathol- ogy is presented, which introduces the use of technologies to integrate mental rea- gents with objective biomedical parameters [6]. Since the results of the Rosenham experiment in the 1970s [7], psychiatric diagnoses have been extremely influenced by the subjectivity of the observer. However, the current nosographies (DSM and ICD10) have solved this problem by tightening the operative definitions of the syn- dromes and have generated nosographies with poor naturalistic adherence [8]. The debate is still strong and one of the possibilities presented by scholars for the defini- tion of naturalistic nosographic criteria that reflect modern knowledge in the biomedi- cal and neuroscientific fields, emphasize the use of objectivable biomedical parame- ters [9]. Without claiming to identify biological markers for mental disorders, the detection of psychophysiological signals can be clinically very useful [10]. The signals generated by the activity of the autonomic nervous system are suitable for integrating the description of the emotional state of the patients. One of the most significant physiological parameters is the Heart Rate Variability which is significantly correlated to individual emotional responses [11]. 2 A question of heart: the implication of the Heart Rate Variability in cognition and emotion The term "Heart Rate Variability" (HRV) indicates the time difference between two sequential heart beats. It is also called R-R variability since it is given by the measurement of the interval of two peaks "R" in the reading of the QSR complex of an electrocardiographic trace. Physiological, cognitive and affective events of differ- ent nature can cause HRV fluctuations. HRV is controlled by the autonomic nervous system. As is known, the latter consists substantially in the parasympathetic system, active when low levels of arousal are present (eg. rest, digestion) and in the sympa- thetic nervous system, conversely active when elevations of the arousal state are present, for example in stress conditions. The parasympathetic system decreases the heart rate, increasing HRV, the sympathetic system increases the heart rate by de- creasing the HRV. This type of mechanics necessarily also involves arousal fluctua- tions related to changes in the emotional state: low activation states, and therefore a high HRV, seems to be related to a condition of substantial well-being, whereas, in- stead, it is shown that in different psychopathological conditions such as anxiety[12], depression [13, 14], bipolar disorder [15, 16], phobic manifestations [17] and panic disorder [18] there is a fall in HRV. This makes it possible to define this value as an index of individual self-regulatory abilities [19]. Therefore, the HRV indicates the health state of the autonomic nervous system, as a high HRV is associated with great- er flexibility of physiological processes and the production of adaptive responses to environmental stimuli and changes, inversely to what happens to individuals with low HRV [20, 21]. 3 2.1 The need to switch from "contact" to "contactless The HRV can be measured by a high number of sensors of various degrees of complexity. The golden standard in HRV measurement is the electrocardiogram (ECG), which, although it is an effective and accurate detection, at the same time implies the use of a complex device directly connected to the subject's skin, potential- ly inconvenient and intrusive. In fact, a traditional ECG system requires that at least three bioelectrodes are positioned in different parts of the body to obtain an effective detection, significantly limiting the patient's mobility, making it inapplicable in psy- chopathological assessments [22]. This resulted in the need to implement equipment that could detect the heart rate in a non-intrusive manner, as an alternative to the ECG, such as less intrusive devices such as smartwatch or Heart rate chest strap [23] which, however, introduce a foreign element in the interaction between the patient and the psychodiagnostic. In order to eliminate any element of disturbance in data collection a series of methodologies have been developed; they are based on small changes in the color of the skin of the face, invisible to the human eye but visible through digital devices. Methods based on the photoplethysmographic(PPG) approach [24, 25, 26] are described in the literature, they allow to identify microvascular blood volume changes in tissues, through the micro variations of the cutaneous absorption of light [27], which is proportional to the variation of blood flow [28]. PPG technolo- gy has the advantages of being relatively simple as it is composed of a light source and a photodetector, comfortable for the patient and economically sustainable [29]. The PPG approach was implemented by other authors [30] through the Eulerian Vid- eo Magnification (EVM)[31], introduced to amplify the imperceptible variations in skin color. The EVM amplifies the color in a video sequence and deconstructs it in different temporal space bands by detecting the color change of the skin over time [32]. Gambi et al [30] propose using frames of a human face obtained from the input of a Kinect V2 as a RedGreenBlue(RGB) camera processing the area of the face through the EVM algorithm. Through a process of extraction of ROI (Region of inter- est), they limit the detection to the face and neck areas, so the Fast Fourier Transform algorithm is applied to the video signal, which converts the data collected into a col- lection of coefficients of a combination linear of sinusoids. The variations of the fre- quency of the sinusoidal curves allow to obtain as output the HRV [33]. Tools such as Kinect v2 was chosen because a single contactless device makes available a series of additional data such as the analysis of the subject's movements or facial expressions, detectable simultaneously with the images of the RGB camera. The Kinect v2, was built in 2014 by Microsoft, and is composed of two cameras, RGB and Infra Red (IR), allowing to obtain different information streams such as: stream of 2D color image frames, a stream of 3D depth image frames. These features allow it to function as a valid depth sensor. Through the Software Development Kit (SDK) [34], made availa- ble by the manufacturer, it provides a skeleton tracker that gives a stable tracking of the individual, providing 3D information on the position of 25 joints per person al- lowing the detection and recognition of complex movements [35, 36]. These methods are not without criticality, as pointed out by Wang et al.[37], which show that subtle color changes or head movements may not be recorded during detection or due to 4 camera distortions or changes in light conditions, an avoidable eventuality through a rigorous control of the setting in which this methodology is applied. 3 Beyond HRV: towards an integrated diagnosis In view of the functional integration of data to psychological diagnosis, it is ex- tremely important not only to detect the physiological change of emotion or the pa- tience’s experience, but also how this is expressed. Humans use different signals to express emotions, such as facial expressions, gesticulation and voice. It is known that non-verbal aspects give the most of the information. It has been es- timated that the expression of emotions is conveyed through facial expressions for at least 55% of the communication, while 7% is instead attributed to the expression through verbal language [38]. Seven basic human emotions are commonly recog- nized: joy, surprise, anger, sadness, fear, disgust and neutral; the recognition proce- dure of an emotional experience is extremely complex, above all because different emotional expressions share some salient expressive characteristics and an observer could recognize the emotion but not be able to identify the different nuances of that experience with clarity: for example a sad smile or a fear caused by disgust[39]. The researchers relied on different theoretical approaches to apply technological metho- dologies to the recognition of emotions. Therefore, some studies was been inspired by the detailed work developed by Ekman, the Facial Action Coding System (FACS)[40], a system based on the change of facial muscles characteristic of the in- dividual expression of human emotions. This system has coded the movement of spe- cific facial muscles called "Action Units" (AU), which reflect the continuous changes in facial expressions. Based on Ekman's studies, over 46 fundamental AUs, producing the facial expressions of emotions, have been codified [41]. Kinect face tracking is based on Active Appearance Model (AAM), one of the most popular methods for pattern recognition applied to deformable objects. It is an algorithm for matching the statistical model of the shape and appearance of the object to a new image, widely used in the localization of the features of the faces [42]. Although some authors con- sider it desirable to use different sensors for an effective expressive-emotional recog- nition, this method was implemented through the use of depths and RGB data of the Kinect camera [43]. Several studies have revealed the effectiveness of using Kinect in facial expression recognition [44, 45, 46]. 4 Conclusions Psychological diagnosis is a complex interactive relational process of fundamental importance for the preparation and management of the patient's therapeutic plan and also an important indication of the relational modalities that the therapist can follow during the treatment. This complex relationship can be enriched by the introduction in the diagnostic context of technological tools, non-intrusive, economic and relatively simple to use for the detection of fundamental biomedical parameters. The detection of both the 5 HRV and the expression of emotions through facial expression can be essential to obtain the most reliable and objectable psychological assessment model possible. For a near future we mean to conceive differently the psychopathological diagnosis, bas- ing it on new neuro-psychophysiological discoveries and introducing a diagnostic standard based on computational methods in order to support the work of the clinical psychologist in the complex definition of the treatment protocol [47]. References 1. APA (American PsychologicalAssociation), Parere sulla diagnosi. 2003. 2. Sperandeo, R., Esposito, A., Maldonato, M., & Dell’Orco, S. (2015, May). Analyzing cor- relations between personality disorders and frontal functions: a pilot study. In International Workshop on Neural Networks (pp. 293-302). Springer, Cham. 3. Cantone, D., Sperandeo, R., Maldonato, M. N., Cozzolino, P., &Perris, F. (2012). Disso- ciative phenomena in a sample of outpatients. Rivista di psichiatria, 47(3), 246-253. 4. Maldonato, N. M., Sperandeo, R., Moretto, E., & Dell'Orco, S. (2018). A non-linear pre- dictive model of borderline personality disorder based on multilayer perceptron. Frontiers in psychology, 9, 447. 5. Dazzi, N., Lingiardi, V., Gazzillo, F., La diagnosi in psicologia clinica. Personalità e psi- copatologia, Milano, Raffaello Cortina, 2014 6. Repetto, C., Serino, S., Maldonato, M., Longobardi, T., Sperandeo, R., Iennaco, D., & Ri- va, G. (2019, April). Immersive Episodic Memory Assessment with 360° Videos: The Pro- tocol and a Case Study. In International Symposium on Pervasive Computing Paradigms for Mental Health (pp. 117-128). Springer, Cham. 7. Rosenhan, D. L., On being sane in insane places. Science. 1973;179:250–258 8. Sperandeo, R., Maldonato, M., Moretto, E., & Dell’Orco, S. (2019). Executive Functions and Personality from a Systemic-Ecological Perspective. In Cognitive Infocommunica- tions, Theory and Applications (pp. 79-90). Springer, Cham. 9. Double, D. (2002). The limits of psychiatry. Bmj, 324(7342), 900-904. 10. Krystal, J. H., & State, M. W. (2014). Psychiatric disorders: diagnosis to therapy. Cell, 157(1), 201–214. doi:10.1016/j.cell.2014.02.042 11. Williams, D., Cash, C., Rankin, C., Bernardi, A., Koenig, J., & Thayer, J. (2015). Resting heart rate variability predicts self-reported difficulties in emotion regulation: a focus on different facets of emotion regulation. Frontiers In Psychology, 6. doi: 10.3389/fpsyg.2015.00261 12. Gorman, J. M., & Sloan, R. P. (2000). Heart rate variability in depressive and anxiety dis- orders. American Heart Journal, 140(4), S77–S83. doi:10.1067/mhj.2000.109981 13. Kidwell, M., &Ellenbroek, B. A. (2018). Heart and soul: heart rate variability and major depression. Behavioural pharmacology, 29(2), 152-164. 14. Dominique, L., Musselman, D. L., Charles, B. N., &Nemeroff, C. B. (1998). The relation- ship of depression to cardiovascular disease. Arch Gen Psychiatric, 55, 580-592. 15. Chang, H. A., Chang, C. C., Kuo, T. B., & Huang, S. Y. (2015). Distinguishing bipolar II depression from unipolar major depressive disorder: Differences in heart rate variability. The World Journal of Biological Psychiatry, 16(5), 351-360. 6 16. Voggt, A., Berger, M., Obermeier, M., Löw, A., Seemueller, F., Riedel, M., ... &Severus, E. (2015). Heart rate variability and Omega-3 Index in euthymicpatients with bipolardisor- ders. EuropeanPsychiatry, 30(2), 228-232. 17. Kawachi, I., Sparrow, D., Vokonas, P. S., & Weiss, S. T. (1995). Decreased heart rate va- riability in men with phobic anxiety (data from the Normative Aging Study). The Ameri- can journal of cardiology, 75(14), 882-885. 18. Klein, E., Cnaani, E., Harel, T., Braun, S., & Ben-Haim, S. A. (1995). Alteredheart rate va- riability in panicdisorderpatients. Biologicalpsychiatry, 37(1), 18-24. 19. Segerstrom, S. C., &Nes, L. S. (2007). Heart rate variability reflects self-regulatory strength, effort, and fatigue. Psychological science, 18(3), 275-281. 20. Meerwijk, E. L., Chelsa, C. A., Weiss, S. J. (2014). Psychological pain and reduction rest- ing-state heart rate variability in adults with a history of depression. Psychophysiology51 247–256. 10.1111/psyp.12175 21. Haiblum-Itskovitch, S., Czamanski-Cohen, J., &Galili, G. (2018). Emotional response and changes in heart rate variability following art-making with three different art materials. Frontiers in psychology, 9. 22. Zhang, Z.,Photoplethysmography-based heart rate monitoring in physical activities via joint sparse spectrum reconstruction. IEEE Trans Biomed Eng 2015;62(8): 1902–1910 23. Sartor, F., Gelissen, J., van Dinther, R., Roovers, D., Papini, G. B., & Coppola, G. (2018). Wrist-worn optical and chest strap heart rate comparison in a heterogeneous sample of healthy individuals and in coronary artery disease patients. BMC sports science, medicine & rehabilitation, 10, 10. doi:10.1186/s13102-018-0098-0 24. Verkruysse, W., Svaasand, L.O., Nelson, J.S. Remote plethysmographic imaging using ambient light. Opt. Express. 2008;16:21434–21445 25. Benezeth, Y., Li, P., Macwan, R., Nakamura, K., Gomez, R., & Yang, F. (2018, March). Remote heart rate variability for emotional state monitoring. In 2018 IEEE EMBS Interna- tional Conference on Biomedical & Health Informatics (BHI) (pp. 153-156). IEEE. 26. Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N., Self-Adaptive Matrix Completion for Heart Rate Estimation From Face Videos Under Realistic Condi- tions; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Seattle, WA, USA. 27–30 June 2016 27. Saquib, N., Papon, M. T. I., Ahmad, I., & Rahman, A. (2015, January). Measurement of heart rate using photoplethysmography. In 2015 International Conference on Networking Systems and Security (NSysS) (pp. 1-6). IEEE. 28. Wang, C., Li, Z., Wei, X., Monitoring heart and respiratory rates at radial artery based on PPG. Opt Int J Light Electron Opt 2013;124(4):3954–3956. 29. Sviridova, N., Sakai, K., Human photoplethysmogram: new insight into chaotic characte- ristics. Chaos Solitons & Fractals. 2015;77:53–63 30. Gambi, E., Agostinelli, A., Belli, A., Burattini, L., Cippitelli, E., Fioretti, S., ... Spinsante, S. (2017). Heart Rate Detection Using Microsoft Kinect: Validation and Comparison to Wearable Devices. Sensors (Basel, Switzerland), 17(8), 1776. doi:10.3390/s17081776 31. Wu, H. Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., & Freeman, W. (2012). Eule- rian video magnification for revealing subtle changes in the world. 32. Liu, Y. F., Vuong, C., Walker, P. C., Peterson, N. R., Inman, J. C., Filho, P. A., & Lee, S. C. (2016). Noninvasive Free Flap Monitoring Using Eulerian Video Magnification. Case reports in otolaryngology, 2016, 9471696. doi:10.1155/2016/9471696 33. Al-Naji, A., Gibson, K., &Chahl, J. (2017). Remote sensing of physiological signs using a machine vision system. Journal of medical engineering & technology, 41(5), 396-405. 34. Microsoft Corporation, Kinect for windows SDK programming guide, version 1.5, 2012. 7 35. Giancola, S., Corti, A., Molteni, F., & Sala, R. (2016, November). Motion capture: an evaluation of kinect V2 body tracking for upper limb motion analysis. In International Conference on Wireless Mobile Communication and Healthcare (pp. 302-309). Springer, Cham. 36. Hwang, S., Tsai, C. Y., & Koontz, A. M. (2017). Feasibility study of using a Microsoft Kinect for virtual coaching of wheelchair transfer techniques. Biomedical Engineer- ing/BiomedizinischeTechnik, 62(3), 307-313. 37. Wang, C., Pun, T., & Chanel, G. (2018). A Comparative Survey of Methods for Remote Heart Rate Detection From Frontal Face Videos. Frontiers in bioengineering and biotech- nology, 6, 33. doi:10.3389/fbioe.2018.00033 38. Mehrabian A. Communication without words. IOJT. 2008:193–200 39. Alabbasi, H. A., Moldoveanu, P., & Moldoveanu, A. (2015). Real time facial emotion rec- ognition using kinect V2 sensor. IOSR J. Comput. Eng. Ver. II, 17(3), 2278-2661. 40. Ekman, R. (1997). What the face reveals: Basic and applied studies of spontaneous expres- sion using the Facial Action Coding System (FACS). Oxford University Press, USA. 41. Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Pro- ceedings of the National Academy of Sciences of the United States of America, 111(15), E1454–E1462. doi:10.1073/pnas.1322355111 42. Coots, T. F., G. J. Edwards, C. J. Taylor. Active Appearance Models. – IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, 2001, No 6, pp. 484-498. 43. 43. Wang, Q., & Yu, Y. (2015). AAM Based Facial Feature Tracking with Kinect. Cybernet- ics and Information Technologies, 15(3), 127-139. 44. Gunawan, A.A.S., Face expression detection on Kinect using active appearance model and fuzzy logic. Procedia Comput. Sci. 2015;59:268–274. 45. Szwoch, M., Pieniążek, P., Facial emotion recognition using depth data; Proceedings of the 8th International Conference on Human System Interactions; Warsaw, Poland. 25–27 June 2015; pp. 271–277 46. Wei, W., Jia, Q., Chen, G., Real-time facial expression recognition for affective computing based on Kinect; Proceedings of the IEEE 11th Conference on Industrial Electronics and Applications; Hefei, China. 5–7 June 2016; pp. 161–165. 47. Sperandeo, R., Esposito, A., Maldonato, M., & Dell’Orco, S. (2015, May). Analyzing cor- relations between personality disorders and frontal functions: a pilot study. In International Workshop on Neural Networks (pp. 293-302). Springer, Cham.