Toward a technological oriented assessment in psychology: A proposal for the use of contactless devices for Heart Rate Variability and facial emotion recognition in psychological diagnosis

Abstract

Diagnosis is a complex cognitive process that takes shape within an interpersonal relationship. It aims at the evaluation of mental and affective processes that make the patient suffer, through their classification and identification of the mechanisms and psychological factors that originated them. This process can be made more effective thanks to the introduction in the diagnostic context of technological tools, non-intrusive and relatively simple to use for the detection of biomedical parameters. In the first section, this work highlights some of the critical issues related to psychological diagnosis; subsequently the methods of detecting physiological parameters, such as the Heart Rate Variability and facial expressions related to the patient's emotional fluctuations, are described. Finally, the concept of diagnosis will be introduced, assisted by computational methods, aimed at supporting the work of the clinical psychologist in the complex procedure of diagnosis.

Keywords

• Psychological Diagnosis
• Heart Rate Variability
• Emotion Recognition
• Kinect v2

-

In 2003 APA [ 1 ] defined psychological diagnosis as the evaluation of abnormal behavior and mental and affective processes that are maladaptive and / or a source of suffering through their classification in a recognized diagnostic system and the identification of the mechanisms and psychological factors that originated them and maintain them [ 2 ]. Diagnosis is also a cognitive process that takes place within an interpersonal relationship, which is its basis and influence it. This process allows the identification of a psychopathology, if it is present, and can provide data necessary for the structuring of an effective therapeutic plan [ 3, 4, 5 ]. In this perspective, the relation

Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). ship established between the psychologist who performs the psychological evaluation and the patient is of fundamental importance. In this paper, after discussing the limits of current diagnostic models, a computational approach to diagnosis in psychopathology is presented, which introduces the use of technologies to integrate mental reagents with objective biomedical parameters [ 6 ]. Since the results of the Rosenham experiment in the 1970s [ 7 ], psychiatric diagnoses have been extremely influenced by the subjectivity of the observer. However, the current nosographies (DSM and ICD10) have solved this problem by tightening the operative definitions of the syndromes and have generated nosographies with poor naturalistic adherence [ 8 ]. The debate is still strong and one of the possibilities presented by scholars for the definition of naturalistic nosographic criteria that reflect modern knowledge in the biomedical and neuroscientific fields, emphasize the use of objectivable biomedical parameters [ 9 ].

Without claiming to identify biological markers for mental disorders, the detection of psychophysiological signals can be clinically very useful [ 10 ].

The signals generated by the activity of the autonomic nervous system are suitable for integrating the description of the emotional state of the patients.

One of the most significant physiological parameters is the Heart Rate Variability which is significantly correlated to individual emotional responses [ 11 ]. 2

A question of heart: the implication of the Heart Rate

Variability in cognition and emotion

The term "Heart Rate Variability" (HRV) indicates the time difference between two sequential heart beats. It is also called R-R variability since it is given by the measurement of the interval of two peaks "R" in the reading of the QSR complex of an electrocardiographic trace. Physiological, cognitive and affective events of different nature can cause HRV fluctuations. HRV is controlled by the autonomic nervous system. As is known, the latter consists substantially in the parasympathetic system, active when low levels of arousal are present (eg. rest, digestion) and in the sympathetic nervous system, conversely active when elevations of the arousal state are present, for example in stress conditions. The parasympathetic system decreases the heart rate, increasing HRV, the sympathetic system increases the heart rate by decreasing the HRV. This type of mechanics necessarily also involves arousal fluctuations related to changes in the emotional state: low activation states, and therefore a high HRV, seems to be related to a condition of substantial well-being, whereas, instead, it is shown that in different psychopathological conditions such as anxiety[ 12 ], depression [ 13, 14 ], bipolar disorder [ 15, 16 ], phobic manifestations [ 17 ] and panic disorder [ 18 ] there is a fall in HRV. This makes it possible to define this value as an index of individual self-regulatory abilities [ 19 ]. Therefore, the HRV indicates the health state of the autonomic nervous system, as a high HRV is associated with greater flexibility of physiological processes and the production of adaptive responses to environmental stimuli and changes, inversely to what happens to individuals with low HRV [ 20, 21 ].

The need to switch from "contact" to "contactless

The HRV can be measured by a high number of sensors of various degrees of complexity. The golden standard in HRV measurement is the electrocardiogram (ECG), which, although it is an effective and accurate detection, at the same time implies the use of a complex device directly connected to the subject's skin, potentially inconvenient and intrusive. In fact, a traditional ECG system requires that at least three bioelectrodes are positioned in different parts of the body to obtain an effective detection, significantly limiting the patient's mobility, making it inapplicable in psychopathological assessments [ 22 ]. This resulted in the need to implement equipment that could detect the heart rate in a non-intrusive manner, as an alternative to the ECG, such as less intrusive devices such as smartwatch or Heart rate chest strap [ 23 ] which, however, introduce a foreign element in the interaction between the patient and the psychodiagnostic. In order to eliminate any element of disturbance in data collection a series of methodologies have been developed; they are based on small changes in the color of the skin of the face, invisible to the human eye but visible through digital devices. Methods based on the photoplethysmographic(PPG) approach [ 24, 25, 26 ] are described in the literature, they allow to identify microvascular blood volume changes in tissues, through the micro variations of the cutaneous absorption of light [ 27 ], which is proportional to the variation of blood flow [ 28 ]. PPG technology has the advantages of being relatively simple as it is composed of a light source and a photodetector, comfortable for the patient and economically sustainable [ 29 ]. The PPG approach was implemented by other authors [ 30 ] through the Eulerian Video Magnification (EVM)[ 31 ], introduced to amplify the imperceptible variations in skin color. The EVM amplifies the color in a video sequence and deconstructs it in different temporal space bands by detecting the color change of the skin over time [ 32 ]. Gambi et al [ 30 ] propose using frames of a human face obtained from the input of a Kinect V2 as a RedGreenBlue(RGB) camera processing the area of the face through the EVM algorithm. Through a process of extraction of ROI (Region of interest), they limit the detection to the face and neck areas, so the Fast Fourier Transform algorithm is applied to the video signal, which converts the data collected into a collection of coefficients of a combination linear of sinusoids. The variations of the frequency of the sinusoidal curves allow to obtain as output the HRV [ 33 ]. Tools such as Kinect v2 was chosen because a single contactless device makes available a series of additional data such as the analysis of the subject's movements or facial expressions, detectable simultaneously with the images of the RGB camera. The Kinect v2, was built in 2014 by Microsoft, and is composed of two cameras, RGB and Infra Red (IR), allowing to obtain different information streams such as: stream of 2D color image frames, a stream of 3D depth image frames. These features allow it to function as a valid depth sensor. Through the Software Development Kit (SDK) [ 34 ], made available by the manufacturer, it provides a skeleton tracker that gives a stable tracking of the individual, providing 3D information on the position of 25 joints per person allowing the detection and recognition of complex movements [ 35, 36 ]. These methods are not without criticality, as pointed out by Wang et al.[ 37 ], which show that subtle color changes or head movements may not be recorded during detection or due to camera distortions or changes in light conditions, an avoidable eventuality through a rigorous control of the setting in which this methodology is applied. 3

Beyond HRV: towards an integrated diagnosis

In view of the functional integration of data to psychological diagnosis, it is extremely important not only to detect the physiological change of emotion or the patience’s experience, but also how this is expressed. Humans use different signals to express emotions, such as facial expressions, gesticulation and voice.

It is known that non-verbal aspects give the most of the information. It has been estimated that the expression of emotions is conveyed through facial expressions for at least 55% of the communication, while 7% is instead attributed to the expression through verbal language [ 38 ]. Seven basic human emotions are commonly recognized: joy, surprise, anger, sadness, fear, disgust and neutral; the recognition procedure of an emotional experience is extremely complex, above all because different emotional expressions share some salient expressive characteristics and an observer could recognize the emotion but not be able to identify the different nuances of that experience with clarity: for example a sad smile or a fear caused by disgust[ 39 ]. The researchers relied on different theoretical approaches to apply technological methodologies to the recognition of emotions. Therefore, some studies was been inspired by the detailed work developed by Ekman, the Facial Action Coding System (FACS)[ 40 ], a system based on the change of facial muscles characteristic of the individual expression of human emotions. This system has coded the movement of specific facial muscles called "Action Units" (AU), which reflect the continuous changes in facial expressions. Based on Ekman's studies, over 46 fundamental AUs, producing the facial expressions of emotions, have been codified [ 41 ]. Kinect face tracking is based on Active Appearance Model (AAM), one of the most popular methods for pattern recognition applied to deformable objects. It is an algorithm for matching the statistical model of the shape and appearance of the object to a new image, widely used in the localization of the features of the faces [ 42 ]. Although some authors consider it desirable to use different sensors for an effective expressive-emotional recognition, this method was implemented through the use of depths and RGB data of the Kinect camera [ 43 ]. Several studies have revealed the effectiveness of using Kinect in facial expression recognition [ 44, 45, 46 ]. 4

Conclusions

Psychological diagnosis is a complex interactive relational process of fundamental importance for the preparation and management of the patient's therapeutic plan and also an important indication of the relational modalities that the therapist can follow during the treatment.

This complex relationship can be enriched by the introduction in the diagnostic context of technological tools, non-intrusive, economic and relatively simple to use for the detection of fundamental biomedical parameters. The detection of both the HRV and the expression of emotions through facial expression can be essential to obtain the most reliable and objectable psychological assessment model possible. For a near future we mean to conceive differently the psychopathological diagnosis, basing it on new neuro-psychophysiological discoveries and introducing a diagnostic standard based on computational methods in order to support the work of the clinical psychologist in the complex definition of the treatment protocol [ 47 ].

References

1. APA (American PsychologicalAssociation) , Parere sulla diagnosi . 2003 .

2. Sperandeo , R. , Esposito , A. , Maldonato , M. , & Dell'Orco , S. ( 2015 , May). Analyzing correlations between personality disorders and frontal functions: a pilot study . In International Workshop on Neural Networks (pp. 293 - 302 ). Springer, Cham.

3. Cantone , D. , Sperandeo , R. , Maldonato , M. N. , Cozzolino , P. , & Perris , F. ( 2012 ). Dissociative phenomena in a sample of outpatients . Rivista di psichiatria , 47 ( 3 ), 246 - 253 .

4. Maldonato , N. M. , Sperandeo , R. , Moretto , E. , & Dell'Orco , S. ( 2018 ). A non-linear predictive model of borderline personality disorder based on multilayer perceptron . Frontiers in psychology, 9 , 447 .

5. Dazzi , N. , Lingiardi , V. , Gazzillo , F. , La diagnosi in psicologia clinica . Personalità e psicopatologia, Milano, Raffaello Cortina, 2014

6. Repetto , C. , Serino , S. , Maldonato , M. , Longobardi , T. , Sperandeo , R. , Iennaco , D. , & Riva , G. ( 2019 , April). Immersive Episodic Memory Assessment with 360° Videos: The Protocol and a Case Study . In International Symposium on Pervasive Computing Paradigms for Mental Health (pp. 117 - 128 ). Springer, Cham.

7. Rosenhan , D. L. , On being sane in insane places . Science . 1973 ; 179 : 250 - 258

8. Sperandeo , R. , Maldonato , M. , Moretto , E. , & Dell'Orco , S. ( 2019 ). Executive Functions and Personality from a Systemic-Ecological Perspective . In Cognitive Infocommunications, Theory and Applications (pp. 79 - 90 ). Springer, Cham.

9. Double , D. ( 2002 ). The limits of psychiatry . Bmj , 324 ( 7342 ), 900 - 904 .

10. Krystal , J. H. , & State , M. W. ( 2014 ). Psychiatric disorders: diagnosis to therapy . Cell , 157 ( 1 ), 201 - 214 . doi: 10 .1016/j.cell. 2014 . 02 .042

11. Williams , D. , Cash , C. , Rankin , C. , Bernardi , A. , Koenig , J. , & Thayer , J. ( 2015 ). Resting heart rate variability predicts self-reported difficulties in emotion regulation: a focus on different facets of emotion regulation . Frontiers In Psychology, 6 . doi: 10 .3389/fpsyg. 2015 .00261

12. Gorman , J. M. , & Sloan , R. P. ( 2000 ). Heart rate variability in depressive and anxiety disorders . American Heart Journal , 140 ( 4 ), S77 - S83 . doi: 10 .1067/mhj. 2000 .109981

13. Kidwell , M. , & Ellenbroek , B. A. ( 2018 ). Heart and soul: heart rate variability and major depression . Behavioural pharmacology , 29 ( 2 ), 152 - 164 .

14. Dominique , L. , Musselman , D. L. , Charles , B. N. , & Nemeroff , C. B. ( 1998 ). The relationship of depression to cardiovascular disease . Arch Gen Psychiatric , 55 , 580 - 592 .

15. Chang , H. A. , Chang , C. C. , Kuo , T. B. , & Huang , S. Y. ( 2015 ). Distinguishing bipolar II depression from unipolar major depressive disorder: Differences in heart rate variability . The World Journal of Biological Psychiatry , 16 ( 5 ), 351 - 360 .

16. Voggt , A. , Berger , M. , Obermeier , M. , Löw , A. , Seemueller , F. , Riedel , M. , ... & Severus , E. ( 2015 ). Heart rate variability and Omega-3 Index in euthymicpatients with bipolardisorders . EuropeanPsychiatry , 30 ( 2 ), 228 - 232 .

17. Kawachi , I. , Sparrow , D. , Vokonas , P. S. , & Weiss , S. T. ( 1995 ). Decreased heart rate variability in men with phobic anxiety (data from the Normative Aging Study) . The American journal of cardiology , 75 ( 14 ), 882 - 885 .

18. Klein , E. , Cnaani , E. , Harel , T. , Braun , S. , & Ben-Haim , S. A. ( 1995 ). Alteredheart rate variability in panicdisorderpatients . Biologicalpsychiatry , 37 ( 1 ), 18 - 24 .

19. Segerstrom , S. C. , & Nes , L. S. ( 2007 ). Heart rate variability reflects self-regulatory strength, effort, and fatigue . Psychological science , 18 ( 3 ), 275 - 281 .

20. Meerwijk , E. L. , Chelsa , C. A. , Weiss , S. J. ( 2014 ). Psychological pain and reduction resting-state heart rate variability in adults with a history of depression . Psychophysiology51 247-256 . 10 .1111/psyp.12175

21. Haiblum-Itskovitch , S. , Czamanski-Cohen, J. , & Galili , G. ( 2018 ). Emotional response and changes in heart rate variability following art-making with three different art materials . Frontiers in psychology, 9.

22. Zhang , Z. , Photoplethysmography-based heart rate monitoring in physical activities via joint sparse spectrum reconstruction . IEEE Trans Biomed Eng 2015 ; 62 ( 8 ): 1902 - 1910

23. Sartor , F. , Gelissen , J., van Dinther, R. , Roovers , D. , Papini , G. B. , & Coppola , G. ( 2018 ). Wrist-worn optical and chest strap heart rate comparison in a heterogeneous sample of healthy individuals and in coronary artery disease patients . BMC sports science, medicine & rehabilitation, 10 , 10. doi: 10 .1186/s13102-018-0098-0

24. Verkruysse , W. , Svaasand , L.O. , Nelson , J.S. Remote plethysmographic imaging using ambient light . Opt. Express . 2008 ; 16 : 21434 - 21445

25. Benezeth , Y. , Li , P. , Macwan , R. , Nakamura , K. , Gomez , R. , & Yang , F. ( 2018 , March). Remote heart rate variability for emotional state monitoring . In 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI) (pp. 153 - 156 ). IEEE.

26. Tulyakov , S. , Alameda-Pineda , X. , Ricci , E. , Yin , L. , Cohn , J.F. , Sebe , N. , Self-Adaptive Matrix Completion for Heart Rate Estimation From Face Videos Under Realistic Conditions; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Seattle , WA, USA. 27 - 30 June 2016

27. Saquib , N. , Papon , M. T. I. , Ahmad , I. , & Rahman , A. ( 2015 , January). Measurement of heart rate using photoplethysmography . In 2015 International Conference on Networking Systems and Security (NSysS) (pp. 1 - 6 ). IEEE.

28. Wang , C. , Li , Z. , Wei , X. , Monitoring heart and respiratory rates at radial artery based on PPG . Opt Int J Light Electron Opt 2013 ; 124 ( 4 ): 3954 - 3956 .

29. Sviridova , N. , Sakai , K. , Human photoplethysmogram: new insight into chaotic characteristics . Chaos Solitons & Fractals . 2015 ; 77 : 53 - 63

30. Gambi , E. , Agostinelli , A. , Belli , A. , Burattini , L. , Cippitelli , E. , Fioretti , S. , ... Spinsante , S. ( 2017 ). Heart Rate Detection Using Microsoft Kinect: Validation and Comparison to Wearable Devices . Sensors (Basel, Switzerland), 17 ( 8 ), 1776. doi: 10 .3390/s17081776

31. Wu , H. Y., Rubinstein , M. , Shih , E. , Guttag , J. , Durand , F. , & Freeman , W. ( 2012 ). Eulerian video magnification for revealing subtle changes in the world .

32. Liu , Y. F. , Vuong , C. , Walker , P. C. , Peterson , N. R. , Inman , J. C. , Filho , P. A. , & Lee , S. C. ( 2016 ). Noninvasive Free Flap Monitoring Using Eulerian Video Magnification . Case reports in otolaryngology, 2016 , 9471696 . doi: 10 .1155/ 2016 /9471696

33. Al-Naji , A. , Gibson , K. , & Chahl , J. ( 2017 ). Remote sensing of physiological signs using a machine vision system . Journal of medical engineering & technology, 41(5) , 396 - 405 .

34. Microsoft Corporation , Kinect for windows SDK programming guide , version 1.5 , 2012 .

35. Giancola , S. , Corti , A. , Molteni , F. , & Sala , R. ( 2016 , November). Motion capture: an evaluation of kinect V2 body tracking for upper limb motion analysis . In International Conference on Wireless Mobile Communication and Healthcare (pp. 302 - 309 ). Springer, Cham.

36. Hwang , S. , Tsai , C. Y. , & Koontz , A. M. ( 2017 ). Feasibility study of using a Microsoft Kinect for virtual coaching of wheelchair transfer techniques . Biomedical Engineering/BiomedizinischeTechnik, 62 ( 3 ), 307 - 313 .

37. Wang , C. , Pun , T. , & Chanel , G. ( 2018 ). A Comparative Survey of Methods for Remote Heart Rate Detection From Frontal Face Videos . Frontiers in bioengineering and biotechnology , 6 , 33. doi: 10 .3389/fbioe. 2018 .00033

38. Mehrabian A . Communication without words . IOJT . 2008 : 193 - 200

39. Alabbasi , H. A. , Moldoveanu , P. , & Moldoveanu , A. ( 2015 ). Real time facial emotion recognition using kinect V2 sensor . IOSR J. Comput. Eng. Ver . II, 17 ( 3 ), 2278 - 2661 .

40. Ekman , R. ( 1997 ). What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS) . Oxford University Press, USA.

41. Du , S. , Tao , Y. , & Martinez , A. M. ( 2014 ). Compound facial expressions of emotion . Proceedings of the National Academy of Sciences of the United States of America , 111 ( 15 ), E1454 - E1462 . doi: 10 .1073/pnas.1322355111

42. Coots , T. F. , G. J. Edwards , C. J. Taylor . Active Appearance Models. - IEEE Transactions on Pattern Analysis and Machine Intelligence , Vol. 23 , 2001 , No 6, pp. 484 - 498 . 43 .

43. Wang , Q. , & Yu , Y. ( 2015 ). AAM Based Facial Feature Tracking with Kinect . Cybernetics and Information Technologies , 15 ( 3 ), 127 - 139 .

44. Gunawan , A.A.S. , Face expression detection on Kinect using active appearance model and fuzzy logic . Procedia Comput. Sci . 2015 ; 59 : 268 - 274 .

45. Szwoch , M. , Pieniążek , P. , Facial emotion recognition using depth data; Proceedings of the 8th International Conference on Human System Interactions; Warsaw , Poland. 25 - 27 June 2015; pp. 271 - 277

46. Wei , W. , Jia , Q. , Chen , G. , Real-time facial expression recognition for affective computing based on Kinect; Proceedings of the IEEE 11th Conference on Industrial Electronics and Applications; Hefei, China. 5-7 June 2016; pp. 161 - 165 .

47. Sperandeo , R. , Esposito , A. , Maldonato , M. , & Dell'Orco , S. ( 2015 , May). Analyzing correlations between personality disorders and frontal functions: a pilot study . In International Workshop on Neural Networks (pp. 293 - 302 ). Springer, Cham.