=Paper=
{{Paper
|id=Vol-2524/paper12
|storemode=property
|title=Toward a technological oriented assessment in psychology: a proposal for the use of contactless devices for heart rate variability and facial emotion recognition in psychological diagnosis
|pdfUrl=https://ceur-ws.org/Vol-2524/paper12.pdf
|volume=Vol-2524
|authors=Raffaele Sperandeo,Alfonso Davide Di Sarno,Teresa Longobardi,Daniela Iennaco,Lucia Luciana Mosca,Nelson Mauro Maldonato
|dblpUrl=https://dblp.org/rec/conf/psychobit/SperandeoSLIMM19
}}
==Toward a technological oriented assessment in psychology: a proposal for the use of contactless devices for heart rate variability and facial emotion recognition in psychological diagnosis==
https://ceur-ws.org/Vol-2524/paper12.pdf
Toward a technological oriented assessment in
psychology: A proposal for the use of contactless devices
for Heart Rate Variability and facial emotion recognition
in psychological diagnosis
Raffaele Sperandeo1, Alfonso Davide Di Sarno1, Teresa Longobardi1,
Daniela Iennaco1, Lucia Luciana Mosca1, Nelson Mauro Maldonato2
1
SiPGI - Postgraduate School of Integrated Gestalt Psychotherapy, Torre Annunziata, Naples,
Italy
2
Department of Neuroscience and Reproductive Sciences and Odontostomatology of the Uni-
versity of Naples Federico II, Naples, Italy
raffaele.sperandeo@gmail.com
Abstract.Diagnosis is a complex cognitive process that takes shape within an
interpersonal relationship. It aims at the evaluation of mental and affective
processes that make the patient suffer, through their classification and identifi-
cation of the mechanisms and psychological factors that originated them. This
process can be made more effective thanks to the introduction in the diagnostic
context of technological tools, non-intrusive and relatively simple to use for the
detection of biomedical parameters. In the first section, this work highlights
some of the critical issues related to psychological diagnosis; subsequently the
methods of detecting physiological parameters, such as the Heart Rate Variabil-
ity and facial expressions related to the patient's emotional fluctuations, are de-
scribed. Finally, the concept of diagnosis will be introduced, assisted by compu-
tational methods, aimed at supporting the work of the clinical psychologist in
the complex procedure of diagnosis.
Keywords:Psychological Diagnosis, Heart Rate Variability, Emotion Recogni-
tion, Kinect v2
1 Introduction: Beyond the limits of diagnosis
In 2003 APA [1] defined psychological diagnosis as the evaluation of abnormal
behavior and mental and affective processes that are maladaptive and / or a source of
suffering through their classification in a recognized diagnostic system and the identi-
fication of the mechanisms and psychological factors that originated them and main-
tain them [2]. Diagnosis is also a cognitive process that takes place within an interper-
sonal relationship, which is its basis and influence it. This process allows the identifi-
cation of a psychopathology, if it is present, and can provide data necessary for the
structuring of an effective therapeutic plan [3, 4, 5]. In this perspective, the relation-
Copyright © 2019 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
2
ship established between the psychologist who performs the psychological evaluation
and the patient is of fundamental importance. In this paper, after discussing the limits
of current diagnostic models, a computational approach to diagnosis in psychopathol-
ogy is presented, which introduces the use of technologies to integrate mental rea-
gents with objective biomedical parameters [6]. Since the results of the Rosenham
experiment in the 1970s [7], psychiatric diagnoses have been extremely influenced by
the subjectivity of the observer. However, the current nosographies (DSM and
ICD10) have solved this problem by tightening the operative definitions of the syn-
dromes and have generated nosographies with poor naturalistic adherence [8]. The
debate is still strong and one of the possibilities presented by scholars for the defini-
tion of naturalistic nosographic criteria that reflect modern knowledge in the biomedi-
cal and neuroscientific fields, emphasize the use of objectivable biomedical parame-
ters [9].
Without claiming to identify biological markers for mental disorders, the detection
of psychophysiological signals can be clinically very useful [10].
The signals generated by the activity of the autonomic nervous system are suitable
for integrating the description of the emotional state of the patients.
One of the most significant physiological parameters is the Heart Rate Variability
which is significantly correlated to individual emotional responses [11].
2 A question of heart: the implication of the Heart Rate
Variability in cognition and emotion
The term "Heart Rate Variability" (HRV) indicates the time difference between
two sequential heart beats. It is also called R-R variability since it is given by the
measurement of the interval of two peaks "R" in the reading of the QSR complex of
an electrocardiographic trace. Physiological, cognitive and affective events of differ-
ent nature can cause HRV fluctuations. HRV is controlled by the autonomic nervous
system. As is known, the latter consists substantially in the parasympathetic system,
active when low levels of arousal are present (eg. rest, digestion) and in the sympa-
thetic nervous system, conversely active when elevations of the arousal state are
present, for example in stress conditions. The parasympathetic system decreases the
heart rate, increasing HRV, the sympathetic system increases the heart rate by de-
creasing the HRV. This type of mechanics necessarily also involves arousal fluctua-
tions related to changes in the emotional state: low activation states, and therefore a
high HRV, seems to be related to a condition of substantial well-being, whereas, in-
stead, it is shown that in different psychopathological conditions such as anxiety[12],
depression [13, 14], bipolar disorder [15, 16], phobic manifestations [17] and panic
disorder [18] there is a fall in HRV. This makes it possible to define this value as an
index of individual self-regulatory abilities [19]. Therefore, the HRV indicates the
health state of the autonomic nervous system, as a high HRV is associated with great-
er flexibility of physiological processes and the production of adaptive responses to
environmental stimuli and changes, inversely to what happens to individuals with low
HRV [20, 21].
3
2.1 The need to switch from "contact" to "contactless
The HRV can be measured by a high number of sensors of various degrees of
complexity. The golden standard in HRV measurement is the electrocardiogram
(ECG), which, although it is an effective and accurate detection, at the same time
implies the use of a complex device directly connected to the subject's skin, potential-
ly inconvenient and intrusive. In fact, a traditional ECG system requires that at least
three bioelectrodes are positioned in different parts of the body to obtain an effective
detection, significantly limiting the patient's mobility, making it inapplicable in psy-
chopathological assessments [22]. This resulted in the need to implement equipment
that could detect the heart rate in a non-intrusive manner, as an alternative to the
ECG, such as less intrusive devices such as smartwatch or Heart rate chest strap [23]
which, however, introduce a foreign element in the interaction between the patient
and the psychodiagnostic. In order to eliminate any element of disturbance in data
collection a series of methodologies have been developed; they are based on small
changes in the color of the skin of the face, invisible to the human eye but visible
through digital devices. Methods based on the photoplethysmographic(PPG) approach
[24, 25, 26] are described in the literature, they allow to identify microvascular blood
volume changes in tissues, through the micro variations of the cutaneous absorption
of light [27], which is proportional to the variation of blood flow [28]. PPG technolo-
gy has the advantages of being relatively simple as it is composed of a light source
and a photodetector, comfortable for the patient and economically sustainable [29].
The PPG approach was implemented by other authors [30] through the Eulerian Vid-
eo Magnification (EVM)[31], introduced to amplify the imperceptible variations in
skin color. The EVM amplifies the color in a video sequence and deconstructs it in
different temporal space bands by detecting the color change of the skin over time
[32]. Gambi et al [30] propose using frames of a human face obtained from the input
of a Kinect V2 as a RedGreenBlue(RGB) camera processing the area of the face
through the EVM algorithm. Through a process of extraction of ROI (Region of inter-
est), they limit the detection to the face and neck areas, so the Fast Fourier Transform
algorithm is applied to the video signal, which converts the data collected into a col-
lection of coefficients of a combination linear of sinusoids. The variations of the fre-
quency of the sinusoidal curves allow to obtain as output the HRV [33]. Tools such as
Kinect v2 was chosen because a single contactless device makes available a series of
additional data such as the analysis of the subject's movements or facial expressions,
detectable simultaneously with the images of the RGB camera. The Kinect v2, was
built in 2014 by Microsoft, and is composed of two cameras, RGB and Infra Red (IR),
allowing to obtain different information streams such as: stream of 2D color image
frames, a stream of 3D depth image frames. These features allow it to function as a
valid depth sensor. Through the Software Development Kit (SDK) [34], made availa-
ble by the manufacturer, it provides a skeleton tracker that gives a stable tracking of
the individual, providing 3D information on the position of 25 joints per person al-
lowing the detection and recognition of complex movements [35, 36]. These methods
are not without criticality, as pointed out by Wang et al.[37], which show that subtle
color changes or head movements may not be recorded during detection or due to
4
camera distortions or changes in light conditions, an avoidable eventuality through a
rigorous control of the setting in which this methodology is applied.
3 Beyond HRV: towards an integrated diagnosis
In view of the functional integration of data to psychological diagnosis, it is ex-
tremely important not only to detect the physiological change of emotion or the pa-
tience’s experience, but also how this is expressed. Humans use different signals to
express emotions, such as facial expressions, gesticulation and voice.
It is known that non-verbal aspects give the most of the information. It has been es-
timated that the expression of emotions is conveyed through facial expressions for at
least 55% of the communication, while 7% is instead attributed to the expression
through verbal language [38]. Seven basic human emotions are commonly recog-
nized: joy, surprise, anger, sadness, fear, disgust and neutral; the recognition proce-
dure of an emotional experience is extremely complex, above all because different
emotional expressions share some salient expressive characteristics and an observer
could recognize the emotion but not be able to identify the different nuances of that
experience with clarity: for example a sad smile or a fear caused by disgust[39]. The
researchers relied on different theoretical approaches to apply technological metho-
dologies to the recognition of emotions. Therefore, some studies was been inspired by
the detailed work developed by Ekman, the Facial Action Coding System
(FACS)[40], a system based on the change of facial muscles characteristic of the in-
dividual expression of human emotions. This system has coded the movement of spe-
cific facial muscles called "Action Units" (AU), which reflect the continuous changes
in facial expressions. Based on Ekman's studies, over 46 fundamental AUs, producing
the facial expressions of emotions, have been codified [41]. Kinect face tracking is
based on Active Appearance Model (AAM), one of the most popular methods for
pattern recognition applied to deformable objects. It is an algorithm for matching the
statistical model of the shape and appearance of the object to a new image, widely
used in the localization of the features of the faces [42]. Although some authors con-
sider it desirable to use different sensors for an effective expressive-emotional recog-
nition, this method was implemented through the use of depths and RGB data of the
Kinect camera [43]. Several studies have revealed the effectiveness of using Kinect in
facial expression recognition [44, 45, 46].
4 Conclusions
Psychological diagnosis is a complex interactive relational process of fundamental
importance for the preparation and management of the patient's therapeutic plan and
also an important indication of the relational modalities that the therapist can follow
during the treatment.
This complex relationship can be enriched by the introduction in the diagnostic
context of technological tools, non-intrusive, economic and relatively simple to use
for the detection of fundamental biomedical parameters. The detection of both the
5
HRV and the expression of emotions through facial expression can be essential to
obtain the most reliable and objectable psychological assessment model possible. For
a near future we mean to conceive differently the psychopathological diagnosis, bas-
ing it on new neuro-psychophysiological discoveries and introducing a diagnostic
standard based on computational methods in order to support the work of the clinical
psychologist in the complex definition of the treatment protocol [47].
References
1. APA (American PsychologicalAssociation), Parere sulla diagnosi. 2003.
2. Sperandeo, R., Esposito, A., Maldonato, M., & Dell’Orco, S. (2015, May). Analyzing cor-
relations between personality disorders and frontal functions: a pilot study. In International
Workshop on Neural Networks (pp. 293-302). Springer, Cham.
3. Cantone, D., Sperandeo, R., Maldonato, M. N., Cozzolino, P., &Perris, F. (2012). Disso-
ciative phenomena in a sample of outpatients. Rivista di psichiatria, 47(3), 246-253.
4. Maldonato, N. M., Sperandeo, R., Moretto, E., & Dell'Orco, S. (2018). A non-linear pre-
dictive model of borderline personality disorder based on multilayer perceptron. Frontiers
in psychology, 9, 447.
5. Dazzi, N., Lingiardi, V., Gazzillo, F., La diagnosi in psicologia clinica. Personalità e psi-
copatologia, Milano, Raffaello Cortina, 2014
6. Repetto, C., Serino, S., Maldonato, M., Longobardi, T., Sperandeo, R., Iennaco, D., & Ri-
va, G. (2019, April). Immersive Episodic Memory Assessment with 360° Videos: The Pro-
tocol and a Case Study. In International Symposium on Pervasive Computing Paradigms
for Mental Health (pp. 117-128). Springer, Cham.
7. Rosenhan, D. L., On being sane in insane places. Science. 1973;179:250–258
8. Sperandeo, R., Maldonato, M., Moretto, E., & Dell’Orco, S. (2019). Executive Functions
and Personality from a Systemic-Ecological Perspective. In Cognitive Infocommunica-
tions, Theory and Applications (pp. 79-90). Springer, Cham.
9. Double, D. (2002). The limits of psychiatry. Bmj, 324(7342), 900-904.
10. Krystal, J. H., & State, M. W. (2014). Psychiatric disorders: diagnosis to therapy. Cell,
157(1), 201–214. doi:10.1016/j.cell.2014.02.042
11. Williams, D., Cash, C., Rankin, C., Bernardi, A., Koenig, J., & Thayer, J. (2015). Resting
heart rate variability predicts self-reported difficulties in emotion regulation: a focus on
different facets of emotion regulation. Frontiers In Psychology, 6. doi:
10.3389/fpsyg.2015.00261
12. Gorman, J. M., & Sloan, R. P. (2000). Heart rate variability in depressive and anxiety dis-
orders. American Heart Journal, 140(4), S77–S83. doi:10.1067/mhj.2000.109981
13. Kidwell, M., &Ellenbroek, B. A. (2018). Heart and soul: heart rate variability and major
depression. Behavioural pharmacology, 29(2), 152-164.
14. Dominique, L., Musselman, D. L., Charles, B. N., &Nemeroff, C. B. (1998). The relation-
ship of depression to cardiovascular disease. Arch Gen Psychiatric, 55, 580-592.
15. Chang, H. A., Chang, C. C., Kuo, T. B., & Huang, S. Y. (2015). Distinguishing bipolar II
depression from unipolar major depressive disorder: Differences in heart rate variability.
The World Journal of Biological Psychiatry, 16(5), 351-360.
6
16. Voggt, A., Berger, M., Obermeier, M., Löw, A., Seemueller, F., Riedel, M., ... &Severus,
E. (2015). Heart rate variability and Omega-3 Index in euthymicpatients with bipolardisor-
ders. EuropeanPsychiatry, 30(2), 228-232.
17. Kawachi, I., Sparrow, D., Vokonas, P. S., & Weiss, S. T. (1995). Decreased heart rate va-
riability in men with phobic anxiety (data from the Normative Aging Study). The Ameri-
can journal of cardiology, 75(14), 882-885.
18. Klein, E., Cnaani, E., Harel, T., Braun, S., & Ben-Haim, S. A. (1995). Alteredheart rate va-
riability in panicdisorderpatients. Biologicalpsychiatry, 37(1), 18-24.
19. Segerstrom, S. C., &Nes, L. S. (2007). Heart rate variability reflects self-regulatory
strength, effort, and fatigue. Psychological science, 18(3), 275-281.
20. Meerwijk, E. L., Chelsa, C. A., Weiss, S. J. (2014). Psychological pain and reduction rest-
ing-state heart rate variability in adults with a history of depression. Psychophysiology51
247–256. 10.1111/psyp.12175
21. Haiblum-Itskovitch, S., Czamanski-Cohen, J., &Galili, G. (2018). Emotional response and
changes in heart rate variability following art-making with three different art materials.
Frontiers in psychology, 9.
22. Zhang, Z.,Photoplethysmography-based heart rate monitoring in physical activities via
joint sparse spectrum reconstruction. IEEE Trans Biomed Eng 2015;62(8): 1902–1910
23. Sartor, F., Gelissen, J., van Dinther, R., Roovers, D., Papini, G. B., & Coppola, G. (2018).
Wrist-worn optical and chest strap heart rate comparison in a heterogeneous sample of
healthy individuals and in coronary artery disease patients. BMC sports science, medicine
& rehabilitation, 10, 10. doi:10.1186/s13102-018-0098-0
24. Verkruysse, W., Svaasand, L.O., Nelson, J.S. Remote plethysmographic imaging using
ambient light. Opt. Express. 2008;16:21434–21445
25. Benezeth, Y., Li, P., Macwan, R., Nakamura, K., Gomez, R., & Yang, F. (2018, March).
Remote heart rate variability for emotional state monitoring. In 2018 IEEE EMBS Interna-
tional Conference on Biomedical & Health Informatics (BHI) (pp. 153-156). IEEE.
26. Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N., Self-Adaptive
Matrix Completion for Heart Rate Estimation From Face Videos Under Realistic Condi-
tions; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
(CVPR); Seattle, WA, USA. 27–30 June 2016
27. Saquib, N., Papon, M. T. I., Ahmad, I., & Rahman, A. (2015, January). Measurement of
heart rate using photoplethysmography. In 2015 International Conference on Networking
Systems and Security (NSysS) (pp. 1-6). IEEE.
28. Wang, C., Li, Z., Wei, X., Monitoring heart and respiratory rates at radial artery based on
PPG. Opt Int J Light Electron Opt 2013;124(4):3954–3956.
29. Sviridova, N., Sakai, K., Human photoplethysmogram: new insight into chaotic characte-
ristics. Chaos Solitons & Fractals. 2015;77:53–63
30. Gambi, E., Agostinelli, A., Belli, A., Burattini, L., Cippitelli, E., Fioretti, S., ... Spinsante,
S. (2017). Heart Rate Detection Using Microsoft Kinect: Validation and Comparison to
Wearable Devices. Sensors (Basel, Switzerland), 17(8), 1776. doi:10.3390/s17081776
31. Wu, H. Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., & Freeman, W. (2012). Eule-
rian video magnification for revealing subtle changes in the world.
32. Liu, Y. F., Vuong, C., Walker, P. C., Peterson, N. R., Inman, J. C., Filho, P. A., & Lee, S.
C. (2016). Noninvasive Free Flap Monitoring Using Eulerian Video Magnification. Case
reports in otolaryngology, 2016, 9471696. doi:10.1155/2016/9471696
33. Al-Naji, A., Gibson, K., &Chahl, J. (2017). Remote sensing of physiological signs using a
machine vision system. Journal of medical engineering & technology, 41(5), 396-405.
34. Microsoft Corporation, Kinect for windows SDK programming guide, version 1.5, 2012.
7
35. Giancola, S., Corti, A., Molteni, F., & Sala, R. (2016, November). Motion capture: an
evaluation of kinect V2 body tracking for upper limb motion analysis. In International
Conference on Wireless Mobile Communication and Healthcare (pp. 302-309). Springer,
Cham.
36. Hwang, S., Tsai, C. Y., & Koontz, A. M. (2017). Feasibility study of using a Microsoft
Kinect for virtual coaching of wheelchair transfer techniques. Biomedical Engineer-
ing/BiomedizinischeTechnik, 62(3), 307-313.
37. Wang, C., Pun, T., & Chanel, G. (2018). A Comparative Survey of Methods for Remote
Heart Rate Detection From Frontal Face Videos. Frontiers in bioengineering and biotech-
nology, 6, 33. doi:10.3389/fbioe.2018.00033
38. Mehrabian A. Communication without words. IOJT. 2008:193–200
39. Alabbasi, H. A., Moldoveanu, P., & Moldoveanu, A. (2015). Real time facial emotion rec-
ognition using kinect V2 sensor. IOSR J. Comput. Eng. Ver. II, 17(3), 2278-2661.
40. Ekman, R. (1997). What the face reveals: Basic and applied studies of spontaneous expres-
sion using the Facial Action Coding System (FACS). Oxford University Press, USA.
41. Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Pro-
ceedings of the National Academy of Sciences of the United States of America, 111(15),
E1454–E1462. doi:10.1073/pnas.1322355111
42. Coots, T. F., G. J. Edwards, C. J. Taylor. Active Appearance Models. – IEEE Transactions
on Pattern Analysis and Machine Intelligence, Vol. 23, 2001, No 6, pp. 484-498. 43.
43. Wang, Q., & Yu, Y. (2015). AAM Based Facial Feature Tracking with Kinect. Cybernet-
ics and Information Technologies, 15(3), 127-139.
44. Gunawan, A.A.S., Face expression detection on Kinect using active appearance model and
fuzzy logic. Procedia Comput. Sci. 2015;59:268–274.
45. Szwoch, M., Pieniążek, P., Facial emotion recognition using depth data; Proceedings of
the 8th International Conference on Human System Interactions; Warsaw, Poland. 25–27
June 2015; pp. 271–277
46. Wei, W., Jia, Q., Chen, G., Real-time facial expression recognition for affective computing
based on Kinect; Proceedings of the IEEE 11th Conference on Industrial Electronics and
Applications; Hefei, China. 5–7 June 2016; pp. 161–165.
47. Sperandeo, R., Esposito, A., Maldonato, M., & Dell’Orco, S. (2015, May). Analyzing cor-
relations between personality disorders and frontal functions: a pilot study. In International
Workshop on Neural Networks (pp. 293-302). Springer, Cham.