Alzheimer's disease (AD) is a form of dementia that affects memory, thinking and behavior, and whose symptoms gradually worsen over time. Moreover, AD can be accompanied by autonomic system alterations, which mainly affect the psychophysiological state of the individual as well as the thermoregulatory activity. The goal of this study is to investigate differences in autonomic activity between mild AD patients and healthy controls (HC) during the execution of cognitive assessment tests in order to support the early diagnosis of the disease. Those differences were investigated by applying artificial intelligence (AI) techniques to thermal infrared imaging (IRI) data. Indeed, IRI allows contactless monitoring of autonomic activity and its thermoregulatory expression without interfering with the psychophysiological state of the subject, thus preserving free interaction with the doctor. Multivariate IRI signal analysis based on partial least square regression and support vector regression allowed to differentiate AD patient from HC with an overall accuracy of the two methods of 80% and 85% respectively. This study demonstrated the capabilities of IRI and AI to discriminate AD patients from HC, paving the way for a potential novel tool able to assess cognitive failures from physiological indices, easily employed in clinical practice.
Alzheimer’s Disease (AD) is a progressive, degenerative disease that attacks brain's nerve cells,
resulting in loss of memory, thinking and language skills, and behavioral changes [
According to the latest National Institute on Aging and Alzheimer’s Association workgroup, initial
screening procedure is generally based on the administration of cognitive/memory test [
2020 Copyright for this paper by its authors.
diagnostic tool, it has been extensively employed for scoring dementia and mild AD [
To this end, contactless technologies that are able to detect psychophysiological states and preserve the ecological condition under which the tests are administered in the clinical routine are preferable. By contrast, classical technologies used for monitoring physiological parameters and ANS activity (e.g. heart rate variability, skin conductance level) typically require contact sensors, which cause manipulation of the subjects’ body, potentially biasing the measurement of psychophysiological/emotional states.
IRI is one of the most suitable techniques to be considered for this purpose. In fact, it is a contactless
and non-invasive technique used to measure the peripheral ANS activity through the modulation of the
cutaneous temperature [
In this paper, the autonomic modulations in AD and HC subjects during the execution of cognitive assessment tests were investigated by means of IRI and supervised learning methods. Supervised machine learning approaches are part of Artificial Intelligence (AI) algorithms, able to automatically learn functions that map an input to an output based on known input–output pairs (training dataset) [22]. They can be further grouped into regression and classification algorithms. Both algorithms aim to develop a model that can predict the value of the output variable from the input ones. The difference between the two algorithms relies on the dependent variable which is numerical for regression and categorical for classification. In this study two different supervised regression algorithm were used to estimate the MMSE score based on the autonomic modulation inferred through key features of the extracted thermal signal. The regression algorithms used were the partial least square regression (PLSR) and the support vector regression (SVR). To test the generalization performances of both models, a leave-one-subject-out cross-validation was utilized. After the cross-validation process, the estimated MMSE score from both regression approaches was used to classify AD patients from HC.
31 participants were recruited in this study. The sample population was composed of 21 healthy subjects (mean age ± standard deviation (SD): 67.3 ± 7.7 years; M/F: 9/12) and 10 mild AD patients (mean age ± SD: 72.3 ± 4.9 years; M/F: 6/4). The AD patients had a diagnosis of mild probable Alzheimer’s disease, according to the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5, APA 2003). The participants selection was performed in the geriatric ward of the university hospital of Chieti, Italy.
This study was conducted according to the Declaration of Helsinki and it was approved by the Research Ethics Board of the University of Chieti-Pescara. Before the experiment, every participant signed the informed consent, and they could withdraw from it at any time. All sessions were scheduled at the same time of day to minimize possible effects related to circadian rhythm fluctuation. 2.2.
Prior to testing, each subject was left in the experimental room for about 10 minutes in order to allow participants to achieve proper acclimatization to the room’s environmental conditions and the baseline skin temperature to stabilize [23]. Participants were seated comfortably on a chair during acclimatization and measurement periods. During the administration of the test, the participants, and the examiner (i.e. the doctor) were seated in front of each other and they had to interact. The experimental paradigm consisted of an initial period of rest in which the participants were requested to stay still for a few minutes without performing any task. Soon after, the MMSE test was administered by the doctor. The test consisted of a 30-point questionnaire that included simple questions and problems in a number of areas, such as the time and place of the test, repeating lists of words, arithmetic calculation, language use and comprehension, and basic motor skills. The administration of the test was conducted in accordance to [24] and lasted about 8 minutes. The experimental paradigm is described in figure 1.
The subject’s psychophysiological response was inferred through his/her facial cutaneous temperature changes over time, recorded by thermal camera FLIR SC660. The camera was equipped with a Focal Plane Array of 640 × 480 detectors, 0.02-s time resolution, 0.03 K temperature sensitivity/Noise Equivalent Temperature Difference, and the capability of collecting the thermal radiation in the 7.5–14μm band. The thermal camera response was blackbody-calibrated in order to remove noise-effects related to the sensor drift/shift dynamics and optical artifacts [25]. The sampling rate for thermal imaging was set at 5 frames/s. All the observations were made in a climate-controlled room according to the International Academy of Thermology (IACT) guidelines [26]. The room temperature was set at 23 ± 1 °C; relative humidity at 50–55% with no direct ventilation on the subject and no direct sunlight. The camera was placed at 60 cm from the participant and focused on the subject’s face. Other than thermal imaging, visible imaging was also recorded trough the Logitech C920 HD Pro webcam with 1920 x 1080 video resolution. 2.4.
For all participants, the time series of facial thermal images were visually inspected to ensure
adequate quality of the recordings. Variations in cutaneous temperature of facial regions of interest
were then analyzed using customized Matlab programs. The nasal tip (NT) area was selected as a region
of interest (ROI) which is known to be sensitive to autonomic activity [
Facial landmarks’ automatic recognition in the visible domain using the OpenFace library. OpenFace is an open source tool, able to detect facial landmarks, estimate head pose, recognize facial action units and estimate eye-gaze on standard RGB images [27].
ROI identification with respect to the facial landmarks.
Co-registration of the ROI in the visible video with the corresponding ROI in the thermal video. The co-registration was performed by applying a geometric transformation of the visible coordinates (calculated based on different field of view), resolution, and position.
For each subject, the tracking algorithm permitted to obtain the pattern of the NT average temperatures across all episodes of the experimental paradigm. The thermal signal was further corrected from residual motion artifacts. Motion errors were identified and replaced using a linear interpolation of the value neighboring the artifact. Figure 2b shows an example of a subject’s NT temperature time course after being visually inspected and corrected for motion artifacts. The temperature time courses during the experimental condition were normalized with respect to the resting condition by subtracting the average temperature value of the rest.
From the normalized thermal signal four representative feature were computed: 1. Average value 2. Standard deviation 3. Slope evaluated as: = ( max − )⁄ − (1) 4. Difference between the average of the signal in the first 5 s and in the last 5 s.
These features resulted to be the most used in thermal signal analysis studies and were therefore chosen in the present study.
Firstly, a machine learning approach was utilized to predict the MMSE score assigned by the doctor, relying on features extracted from thermal signals. Specifically, two type of supervised learning algorithm were tested for this specific task. Supervised learning uses labeled training data to learn the mapping function that turns input variables (X) into the output variable (Y). In other words, it solves for f in the following equation:
Y = f (X) (2)
This allows us to accurately generate outputs when given new inputs. As a type of supervised learning, regression algorithms were employed to predict the MMSE score. In details, a partial least square regression (PLSR) and a support vector regression (SVR) with linear kernel were used. Linear regression algorithms were chosen rather than the non-linear ones because of the small sample size and a possible over-fitting effect. The PLSR method is a widely used regression techniques that generalizes and combines features from principal component analysis and multiple regression. The goal of PLSR is to predict Y from X and to describe the common structure underlying the two variables. Indeed, it creates orthogonal components by using the existing correlations between explanatory variables and corresponding outputs while also keeping most of the variance of explanatory variables [28]. On the other hand, SVR is an extension of the well-known support vector machine algorithm. Unlike other regression methods that try to minimize the error between the real and predicted value, the SVR tries to fit the error inside a certain threshold, which means that it attempt to approximate the best value within a given margin [29].
Both PLSR and SVR were employed for MMSE score evaluation and then for the AD and HC classification. Both procedures were performed in Matlab environment. In detail, PLSR algorithm was implemented using number of components equal to 4, whilst the SVR procedure was performed using a linear kernel with automatic scale and standardized training data. The MMSE score was employed as the regression output. Because of the multivariate (4 regressors) approach, in-sample performance of the procedure did not reliably estimate the out-of-sample performance. Therefore, to avoid any overfitting, the procedure performance was estimated with a leave-one-subject-out cross-validation: the algorithm was trained using data from all subjects but one, and then was tested on the left-out subject, providing unbiased estimations of the regression for small datasets [30]. This procedure was iterated for all the subjects, and further statistical analyses were performed on the out-of-training-sample estimation of MMSE score. The PLSR algorithm prediction outcome was called MMSE_PLSR whilst the one from SVR was called MMSE_SVR.
Secondly, the MMSE_PLSR and MMSE_SVR were used to provide a classification between AD patients and HC. The two classes (AD and HC) were a priori assigned based on the doctor’s diagnosis. Since the two classes did not have an equal number of samples, a bootstrap procedure was implemented to test classification performance on balanced classes [31]. The performances of the classification were evaluated by means of receiver operating characteristic (ROC) analysis [32]. Figure 3 shows the flow chart related to the described machine learning approaches. employed as regression algorithms. A leave-one-subject-out cross-validation was performed to test the generalization of the regression. The results of the regressions (MMSE_PLSR resulted from PLSR algorithm and MMSE_SVR from SVR) were then individually used to perform a two-level classification of the subjects i.e. AD patient or Healthy control. Since the two classes were not balanced, a bootstrap procedure was implemented. The classification was performed using ROC analysis.
A significant correlation between the MMSE score and the predicted output from both PLSR and SVR algorithm were obtained (r = 0.480, p = 0.006 and r=0.390, p=0.032, respectively) demonstrating a good performance of the multivariate analysis.
Concerning the classification process Figure 4a reports the among iterations average ROC curves (bootstrap performed for n = 1000 iterations) obtained by the MMSE_PLSR and MMSE_SVR respectively. The average area under curve (AUC) related to the MMSE_PLSR and MMSE_SVR classification was 0.77 and 0.81 with standard deviation of 0.045 and 0.053, respectively. The distribution of the two AUC obtained after the bootstrap are reported in Figure 4b.
By choosing a specific threshold for MMSE_PLSR and MMSE_SVR respectively, the sensitivity, specificity, and accuracy values of both classification procedures were obtained as reported in the confusion matrix (Table 1). The confusion matrix shows balanced classes as obtained by bootstrap procedure. In detail, MMSE_PLSR resulted in a classification sensitivity, specificity, and accuracy of 80% whilst MMSE_SVR in a sensitivity of 90%, specificity of 80% and accuracy of 85%.
AD is a widely studied disorder with research focusing on cognitive and functional impairments, behavioral and psychological symptoms. Despite the importance of autonomic dysfunctions which are common in dementia, they have received less attention in systematic studies [33]. The aim of this study was to assess possible differences in the autonomic activity between AD and HC during the performance of MMSE. Consequently, the capability of IRI and AI to classify AD from HC subjects based on their autonomic activity was investigated. The leading hypothesis of the study was that performing cognitive tasks, such as the MMSE, designed to assess specific impairments typical of AD, could raise a different state of stress or anxiety in AD patients compared to HC. In fact, the performance in cognitive tasks could amplify the autonomic dysregulation of AD patients and thus the differences from HC.
However, in order to avoid the effect of contact-sensor instrumentation on the subjects’ performance, it would be advisable to control the autonomic and psychophysiological state of the participants in a non-invasive manner. To this end, IRI appears to be one of the most suitable tools, thanks to its contactless and non-invasive features. This technique is widely used to measure the modulation of the cutaneous temperature to infer the ANS activity.
Multivariate machine learning approaches based on PLSR and SVR were employed to estimate the MMSE score through peculiar thermal features extracted from facial ROI. The regression outcome resulted in a significant correlation between the MMSE score and the score predicted by both PLSR and SVR algorithm, named MMSE_PLSR and MMSE_SVR (r = 0.480, p = 0.006 and r=0.390, p=0.032, respectively). The results showed a moderate estimation [34] of the MMSE score through the considered thermal features. MMSE_PLSR and MMSE_SVR were then used to perform the classification between mild AD and HC subjects. The ROC analysis showed a good performance of both classifier with an average AUC of 0.77 and 0.81 for MMSE_PLSR and MMSE_SVR respectively. Specifically, the classification based on MMSE_PLSR showed an accuracy of 80% (sensitivity =80% specificity = 80%) whereas MMSE_SVR exhibited an accuracy of 85% (specificity =80% and sensitivity = 90%). It is worth noting that the cross-validation and the bootstrap procedures provided the generalization performances of the model, testing its applicability to a wide cohort of subjects. The results demonstrated the capabilities of this approach to detect the cognitive decline associated to AD. Although IRI has been already used to highlight difference in the autonomic activity of AD with respect to HC subjects, to the best of our knowledge this is the first time that AI approaches have been used for thermal signal analysis and AD classification. This approach could pave the way for novel and effective diagnostic tools designed to support early AD diagnosis. Moreover, this method could easily be employed in outpatient environment in routinely clinical practice. Further work may be performed to better characterize the underlying physiological alterations that could explain our findings. Moreover, despite the high classification performance, the purpose of further studies would be to increase the sample size in order to corroborate these results. Increasing the sample size may permit further increase of the performance by decreasing a possible in-sample overfitting effect of the classifier. However, although the study dataset can be considered rather small, the investigation was conducted employing a leave-one-out cross-validation procedure, thus intrinsically evaluating the out of sample performance. Therefore, the results obtained are indeed generalizable.
Importantly, it may be worth investigating the capability of this AI framework to discriminate the cognitive decline associated with different kind of dementia. Hence, further study should be performed including in the sample size patients affected by other degenerative cognitive pathologies. It could be also interesting to combine IRI recordings with other physiological measurements of autonomic activity (i.e. heart rate, galvanic skin response) as well as to combine it with functional neuroimaging techniques to assess the relationship between the autonomic and central nervous system activity.
In this study an innovative method to assess autonomic activity during the execution of cognitive test and classify mild AD patients was presented. The autonomic activity was investigated in a completely ecological manner, by means of IRI technique and AI algorithms. The cognitive test used was the MMSE which is commonly employed in clinical settings for cognitive impairment evaluation. By using AI approaches, it was possible to classify mild AD patients with a good level of accuracy. Thus, providing a physiological hint that might serve as an indicator of mild AD, aimed at supporting the clinical diagnosis of this dementia not only based on the cognitive score or behavioral analysis, but also from a psychophysiological perspective.
This study opens the possibility of using novel data-driven approaches for IRI signal analysis, in order to improve its diagnostic capabilities and to extend its application field.
This research was funded by PON MIUR SI-ROBOTICS, ARS01_01120.
[17] C. Filippini, E. Spadolini, D. Cardone, and A. Merla, ‘Thermal Imaging Based Affective Computing for Educational Robot’, in Multidisciplinary Digital Publishing Institute Proceedings, 2019, vol. 27, p. 27. [18] D. Cardone, P. Pinti, and A. Merla, ‘Thermal infrared imaging-based computational psychophysiology for psychometrics’, Computational and mathematical methods in medicine, vol. 2015, 2015. [19] H. Veltman and W. Vos, ‘Facial temperature as a measure of operator state’, Foundations of augmented cognition, vol. 293, 2005. [20] D. Perpetuini et al., ‘Autonomic impairment in Alzheimer’s disease is revealed by complexity analysis of functional thermal imaging signals during cognitive tasks’, Physiological measurement, vol. 40, no. 3, p. 034002, 2019. [21] D. Perpetuini, D. Cardone, R. Bucco, M. Zito, and A. Merla, ‘Assessment of the Autonomic Response in Alzheimer’s Patients During the Execution of Memory Tasks: A Functional Thermal Imaging Study’, Current Alzheimer Research, vol. 15, no. 10, pp. 951–958, Sep. 2018, doi: 10.2174/1871529X18666180503125428. [22] D. Cardone et al., ‘Driver Stress State Evaluation by Means of Thermal Imaging: A Supervised Machine Learning Approach Based on ECG Signal’, Applied Sciences, vol. 10, no. 16, Art. no. 16, Jan. 2020, doi: 10.3390/app10165673. [23] M. S. Panasiti, D. Cardone, E. F. Pavone, A. Mancini, A. Merla, and S. M. Aglioti, ‘Thermal signatures of voluntary deception in ecological conditions’, Scientific reports, vol. 6, no. 1, pp. 1– 10, 2016. [24] J. P. Dick et al., ‘Mini-mental state examination in neurological patients.’, Journal of Neurology,
Neurosurgery & Psychiatry, vol. 47, no. 5, pp. 496–499, May 1984, doi: 10.1136/jnnp.47.5.496. [25] D. Cardone and A. Merla, ‘New frontiers for applications of thermal infrared imaging devices:
Computational psychopshysiology in the neurosciences’, Sensors, vol. 17, no. 5, p. 1042, 2017. [26] ‘Thermography Guidelines. Standards and protocols.’ https://www.iactorg.org/professionals/thermog-guidelines.html (accessed Sep. 22, 2020). [27] B. Amos, B. Ludwiczuk, and M. Satyanarayanan, ‘OpenFace: A general-purpose face recognition library with mobile applications’, p. 20. [28] D. M. Pirouz, ‘An Overview of Partial Least Squares’, Social Science Research Network,
Rochester, NY, SSRN Scholarly Paper ID 1631359, Oct. 2006. doi: 10.2139/ssrn.1631359. [29] A. J. Smola and B. Schölkopf, ‘A tutorial on support vector regression’, Statistics and Computing, vol. 14, no. 3, pp. 199–222, Aug. 2004, doi: 10.1023/B:STCO.0000035301.49549.88. [30] S. Simons, D. Abasolo, and J. Escudero, ‘Classification of Alzheimer’s disease from quadratic sample entropy of electroencephalogram’, Healthc Technol Lett, vol. 2, no. 3, pp. 70–73, May 2015, doi: 10.1049/htl.2014.0106. [31] G. Dupret and M. Koda, ‘Bootstrap re-sampling for unbalanced data in supervised learning’, European Journal of Operational Research, vol. 134, no. 1, pp. 141–156, Oct. 2001, doi: 10.1016/S0377-2217(00)00244-7. [32] M. H. Zweig and G. Campbell, ‘Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine’, Clin Chem, vol. 39, no. 4, pp. 561–577, Apr. 1993, doi: 10.1093/clinchem/39.4.561. [33] E. Engelhardt and J. Laks, ‘Alzheimer disease neuropathology: understanding autonomic dysfunction’, Dement Neuropsychol, vol. 2, no. 3, pp. 183–191, 2008, doi: 10.1590/S198057642009DN20300004. [34] R. M. Dodhia, ‘A Review of Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (3rd ed.)’, Journal of Educational and Behavioral Statistics, vol. 30, no. 2, pp. 227–229, Jun. 2005, doi: 10.3102/10769986030002227.