Before the introduction of fully autonomous vehicles with all their benefits and positive impact on quality of life (e.g., increased mobility options, reduced carbon footprint, road safety), researchers propose an era of conditionally automated vehicles where the driver must take over (resume control of the automated vehicle) in critical situations. In terms of human-computer interaction (HCI) during the take-over process, the driver's physiological signals seem promising as they could be read and understood by the vehicle. In this paper, we quantify the physiological responses to take-over requests (TOR), i.e., we determine their amplitudes, delays, and durations. We measured and examined drivers' heart rate, pupil diameter, horizontal gaze dispersion, blink rate, skin conductance response, and skin temperature. Values before the TOR were compared with values after the TOR, averaged over different time intervals. In addition, the duration until the first noticeable change in each physiological response (delay) and the duration until the signals stabilized to their normal values (duration) were measured. The results showed that the relatively greatest effect of TOR was observed in skin conductance (from -62% to 142%). The fastest response (on average) to TOR was observed in pupil diameter (2.24 s ± 2.48 s), followed by skin conductance and heart rate. Manual or automatic artifact correction has not yet been performed and should be included in further analysis.
Increasing automation in vehicular technology is about to increase the overall quality of life (QoL).
With the introduction of autonomous vehicles, elderly people and children will be able to make their
daily trips without a supervisor [
Current technology is almost ready to adopt the third level of automation technology as defined by
SAE (Society of Automotive Engineers): conditionally automated driving [
The crucial problem of human-computer interaction (HCI) SAE level 3 vehicles is how to design a
take-over request (TOR) to communicate with the driver to take over the vehicle when it cannot
continue driving in autonomous mode [
In addition to vehicle-related data, such as speed, acceleration, time to collision, or lateral
displacements, physiological data seem to have potential in research, although there is still little
consensus on their potential use cases [
The longest used and most studied physiological measure is heart rate. The time differences between successive heartbeats vary (oscillate) in response to respiratory activity (breathing). This phenomenon is referred to as heart rate variability [8]. Carsten et al. showed that the average heart rate is higher during semi-automated or manual driving than during autonomous driving [11]. However, Stephenson et al. found no significant difference in the heart rate of drivers before and after a take-over [12].
Pupil diameter is one of the most common physiological measurements in research. According to Mathot, the pupil responds to three types of stimuli: it constricts in response to brightness, it constricts in response to near fixations, and it dilates in response to increased mental effort [13]. In driving environment, it could be used to measure the drivers’ cognitive load [14], [15]. However, the measurements could be difficult because the illumination changes drastically when the gaze is directed toward or away from the screen. Zhou et al. did not find any changes in pupil diameter related to situational awareness [15].
Electrodermal activity (skin conductance) consists of a slowly varying tonic activity called skin conductance level (SCL) and a rapidly varying phasic activity called skin conductance response (SCR) [16]. When a person feels stressed or experiences cognitive load, their glands begin to sweat, resulting in SCR. Therefore, SCRs in the few seconds after a stimulus (e.g., a TOR) are attributed to that stimulus [17]. Li et al. showed that counting the number of SCRs above the threshold or summing the amplitudes of SCRs (depending on the time window) are the suggested arousal metrics in automated driving systems, as they increase significantly with higher cognitive load [18].
Drivers stress can be monitored by measuring skin temperature [19]. Yamakoshi et al. showed that the peripheral skin temperature gradually dropped when driving under stressed [20]. They also suggested that the difference between body and peripheral skin temperature could be used as an indicator of drivers’ stress. Jang et al. on the other hand did not find any change in skin temperature while monitoring reaction of drivers in virtual environments [21]. 1.2.
Due to many contradictory findings presented and all the opened questions regarding the normal and expected values of physiological responses, we first need to quantify the physiological responses during a take-over to be able to reliably use the data for HCI purposes. Some papers with limited analysis of physiological responses, specific to take-over procedure, have recently been published [22], but they do not report on measured general values of physiological responses, rather just on the differences, caused by predefined circumstances. We determined the typical values and changes for physiological responses during different periods of a TO. Additionally, we determined the delay and stabilization time (effect duration) for each physiological measure. These are the pre-results of an ongoing analysis.
Summarized into single statements, our research questions were: 1. How much are different physiological signals affected by the TO? 2. How long after a TOR can a physiological response be expected and for how long? The rest of the paper is structured as follows: Section 2 presents the methodology, i.e., the data collection and processing. Section 3 presents the results of analysis, i.e., the typical values and timing of physiological signals during TO. Section 4 provides a brief discussion of the results and a conclusion.
We conducted an exploratory user study in a driving simulator where 30 participants (15 female) drove a conditionally autonomous vehicle, which issued three take-over requests during each driving session. The study was conducted in accordance with the Code of ethics of the University of Ljubljana and with the Declaration of Helsinki. An informed consent was obtained from every participant.
The participants’ task was to drive the conditionally autonomous vehicle as he or she would in a normal life. Therefore, when the vehicle had been in autonomous mode, the drivers could perform any other task by their preference. Some choose to read a magazine, some played games on a smartphone, and the others just looked around the place. However, if the vehicle had requested a take-over, the driver had to intervene and drove manually until the desired automation level became available again. 2.1.
The used NervtechTM driving simulator [23] consists of three curved TV screens and a 4-DOF motion platform with a driver’s seat, pedals, steering wheel, dashboard display, and a gearbox. The simulation software was AVSimulation’s SCANeR studio 1.7 [24].
The driving scenario featured a 13 km long city road (see Figure 1), the speed limit was 50 km/h unless otherwise specified by the traffic signs. Some surrounding traffic and pedestrians were included in the scenario to make it more realistic. A take-over request was issued by the vehicle with an auditory alert sound (4 kHz beep), and a visual alert icon on the screen, featuring a head-up display (HUD). During the drive, the vehicle issued three take-over requests: one was considered urgent as a pedestrian ran in front of the vehicle to cross the road, one was issued due to road infrastructure no longer supporting the autonomous mode, and one due to poor driving conditions (the absence of lane marks). 2.2.
Drivers’ physiological responses were measured with two devices: • Tobii Pro Glasses 2 [25], a wearable eye tracker, a. measuring pupil diameter (PD) in mm, b. measuring gaze direction, c. sampling frequency: 50 Hz. • Empatica E4 [26], a medically certified wristband, a. measuring blood volume pulse (BVP) with photoplethysmography (PPG) sensor, sampling frequency: 64 Hz, b. calculating inter-beat interval (IBI) in seconds from BVP, with automatic removal of corrupted samples due to excessive motion, c. measuring electro-dermal activity (EDA) in µS, sampling frequency: 4 Hz, d. measuring skin temperature in °C, sampling frequency: 4 Hz. 2.3.
A user session began with a short explanation of its purpose and procedure. The participants were then invited to sign an informed consent form and fill in a demographic questionnaire. Following the paperwork, measurement devices were attached (worn) and the driver was seated in the simulator.
The driving part started with a short test drive, so that the participants became familiar with the simulator and its features. They were able to try autonomous driving, manual driving and different possibilities of taking over (i.e., by steering the wheel, pressing the brake, or pressing a sophisticated button). A following measuring period lasted for about 20 minutes. The drivers were instructed to act as they would in normal life, driving a conditionally automated vehicle, i.e., take-over when requested.
After the measurement, the drivers were asked about their experience and invited to participate in the upcoming user studies. 2.4.
The processing and plotting were performed in python 3.9.12.
A heart rate signal was extracted from the IBI using equation (1) and averaged over 10 s, as proposed by the E4 device manufacturer [27] to exclude the HRV phenomenon due to breathing. It should be noted that the E4’s proprietary algorithm automatically removes corrupt samples due to excessive motion and therefore there were sometimes not enough samples to perform the analysis. We excluded the trials that contained less than three samples per 10 s time window.
= 60 / (1)
As the raw pupil diameter data was noisy and therefore unreliable, we applied a moving average filter with a window width of one second (50 samples). We assumed that all portions of the screens were equally illuminated and therefore pupil diameter was not affected by looking at different points. Additionally, we determined eye blinks by searching the eye-position data for consequently missing samples. As research shows that blinks last from 60 to 700 ms, we tagged every sample that followed a consequent miss of data greater than 60 ms, as proposed by Al-Gawwam and Benaissa [28].
Raw electrodermal activity data was decomposed into tonic skin conductance, phasic skin conductance, and sparse driver of phasic component using the methods of convex optimization – the cvxEDA algorithm by Greco et al. [29].
The raw temperature data included an exponential increase due to the driver adapting to the driving simulator environment. We therefore first applied a curve fitting algorithm from scipy library [30] to fit a 3rd level polynomial function and subtracted it from the original data. Due to sensors’ quantization noise, the data had to be low-pass-filtered (moving average over 10 seconds).
The drivers’ physiological responses preceding a TOR were absolutely and relatively compared to responses after the TOR. The time window for pre-TOR responses was 60 seconds. Post-TOR responses were calculated over many empirically determined time windows (2 s, 5 s, 10 s, 15 s, 20 s, 30 s, 60 s).
The calculated (and compared) parameters included in the analysis were: • average heart rate (HR), • average pupil diameter (PD), • horizontal gaze dispersion (HGD) – standard deviation of horizontal gaze coordinate, relative to the width of one simulator screen, • blink rate (BR) – the number of blinks per the duration of time window, • skin conductance response (SCR) – the sum of amplitudes of skin conductance responses, measured in µS/s, • average skin temperature (TEMP).
Regarding the timing of physiological responses, we determined the delay and duration of responses, following a TOR. The delay was measured from the moment of a TOR until the first noticeable change in physiological parameters, i.e., when the parameter exceeded the threshold – its pre-TOR mean ± one standard deviation [31]. Similarly, the duration of response was measured from the moment of a TOR until the stabilization of the parameter, i.e., when the parameter again reaches its overall mean ± one quarter of a standard deviation. For representation, see.
The parameters, included in the analysis were: • heart rate, • pupil diameter, • phasic skin conductance, • skin temperature.
Table 1 describes the changes in pre- vs. post-TOR responses for different time windows.
The results in Table 1 demonstrate that with respect to pre-TOR interval, the heart rate (HR) first declines for about 2 % (3–5 s after TOR) and then increases again about 20–30 s after the TOR. We speculate that the decline is an early response, while the later increase in HR is probably the delayed result of manual driving, as Carsten et al. [11] suggested. From Table 2 we can expect that the mentioned increase starts about 16 s after the TOR and neutralizes again about a minute later.
The pupil diameter seems to increase after the TOR for 2–5% on average. The first increase is detected quite soon after the TOR (2.24 s ± 2.48 s) and lasts for about 10 s. Following Mathot’s [13] suggestion, the TOR probably increased drivers’ cognitive load and therefore the pupil diameter. We observe that horizontal gaze dispersion (HGD) in the first few intervals after the TOR declines rapidly. We believe this is due to increased focus on a single point while taking over the vehicle. In the 30 s post-TOR interval, the HGD increased again, probably due to driver extensively scanning the driving environment, thus increasing situational awareness. In a normal, healthy subject, blinks occur about 17 times a min, which is once every 3.5 s [32]. Therefore, the 2 s and 5 s post-TOR intervals may be irrelevant and provide noise as no blinks could happen at the time. Regardless, it seems that blinks happen more often immediately after the TOR and get more rear after some time.
The amplitude sum of skin conductance responses (SCRs) seems to get lower immediately following the TOR for about 50% and then exceeds the pre-TOR value for more than 100% after 20–30 s. We can observe from Table 2 that this increase starts on average about 7 s after the TOR and lasts for 14 more seconds on average. The noticed SCR could reliably correspond to the TOR, as suggested by Dawson et al. [17].
The skin temperature is the most unreliable of all the measured data, as the deviations are very high, relative to the absolute values. For now, we could not reliably state whether and how much did the skin temperature increase or decrease after the TOR. This is in contrast with Yamakoshi et al. [20], who state that skin temperature declines under stress, as we believe that the TOR induces stress to the driver.
Overall, the relatively largest impact of TOR was observed with skin conductance (sum of amplitudes of SCRs), followed by HGD, BR, PD and HR. The (on average) fastest response to TOR was observed with pupil diameter, followed by SCR and HR.
It should be discussed as a limitation, that no manual artifact correction was performed on data so far. E.g., no sharp edges are naturally possible in raw electrodermal activity data and are possibly caused by electrode displacement or movement. In the future, automatic identification, and correction of artifacts, such as proposed by Taylor et al. [33], could be performed. Also, the data was currently only analyzed with respect to the take-over request (TOR). It would make sense to also include the information about the actual take-over in the analysis of timing and shape of physiological responses.
This work was supported in part by the Slovenian Research Agency within the research program ICT4QoL - Information and Communications Technologies for Quality of Life [grant number P20246]; and in part by HADRIAN (Holistic Approach for Driver Role Integration and Automation Allocation for European Mobility Needs), which has received funding from the European Union’s Horizon 2020 research and innovation program [grant number 875597].
[8] S. Laborde, E. Mosley, and J. F. Thayer, “Heart Rate Variability and Cardiac Vagal Tone in Psychophysiological Research – Recommendations for Experiment Planning, Data Analysis, and Data Reporting,” Front. Psychol., vol. 8, 2017, doi: 10.3389/fpsyg.2017.00213. [9] W. Boucsein, Electrodermal Activity. Springer Science & Business Media, 2012. [10]R. H. Spector, “The Pupils,” in Clinical Methods: The History, Physical, and Laboratory Examinations, 3rd ed., H. K. Walker, W. D. Hall, and J. W. Hurst, Eds. Boston: Butterworths, 1990. Accessed: Nov. 06, 2022. [Online]. Available: http://www.ncbi.nlm.nih.gov/books/NBK381/ [11]O. Carsten, F. C. H. Lai, Y. Barnard, A. H. Jamson, and N. Merat, “Control Task Substitution in Semiautomated Driving: Does It Matter What Aspects Are Automated?,” Hum Factors, vol. 54, no. 5, pp. 747–761, Oct. 2012, doi: 10.1177/0018720812460246. [12]A. C. Stephenson et al., “Effects of an Unexpected and Expected Event on Older Adults’ Autonomic Arousal and Eye Fixations During Autonomous Driving,” Front Psychol, vol. 11, Sep. 2020, doi: 10.3389/fpsyg.2020.571961. [13]S. Mathôt, “Pupillometry: Psychology, Physiology, and Function,” J Cogn, vol. 1, no. 1, p. 16, 2018, doi: 10.5334/joc.18. [14]T. Čegovnik, K. Stojmenova, G. Jakus, and J. Sodnik, “An analysis of the suitability of a low-cost eye tracker for assessing the cognitive load of drivers,” Applied Ergonomics, vol. 68, pp. 1–11, Apr. 2018, doi: 10.1016/j.apergo.2017.10.011. [15]F. Zhou, X. J. Yang, and J. C. F. de Winter, “Using Eye-Tracking Data to Predict Situation Awareness in Real Time During Takeover Transitions in Conditionally Automated Driving,” IEEE Transactions on Intelligent Transportation Systems, pp. 1–12, 2021, doi: 10.1109/TITS.2021.3069776. [16]M. Benedek and C. Kaernbach, “A continuous measure of phasic electrodermal activity,” Journal of Neuroscience Methods, vol. 190, no. 1, pp. 80–91, Jun. 2010, doi: 10.1016/j.jneumeth.2010.04.028. [17]M. E. Dawson, A. M. Schell, and D. L. Filion, “The electrodermal system,” in Handbook of psychophysiology, 3rd ed, New York, NY, US: Cambridge University Press, 2007, pp. 159–181. doi: 10.1017/CBO9780511546396.007. [18]P. Li, Y. Li, Y. Yao, C. Wu, B. Nie, and S. E. Li, “Sensitivity of Electrodermal Activity Features for Driver Arousal Measurement in Cognitive Load: The Application in Automated Driving Systems,” IEEE Transactions on Intelligent Transportation Systems, pp. 1–14, 2021, doi: 10.1109/TITS.2021.3135266. [19]H. Asano, T. Muto, and H. Ide, “Stress evaluation while prolonged driving operation using the facial skin temperature,” Transactions of the Society of Instrument and Control Engineers, vol. 47, no. 1, pp. 2–7, 2011. [20]T. Yamakoshi, K. Matsumura, H. Kobayashi, Y. Gotoh, and H. Hirose, “Feasibility study on assessment of driver’s stress from differential skin temperature measurement under simulated monotonous driving,” Transactions of Japanese Society for Medical and Biological Engineering, vol. 48, no. 2, pp. 163–174, 2010. [21]D. P. Jang, I. Y. Kim, S. W. Nam, B. K. Wiederhold, M. D. Wiederhold, and S. I. Kim, “Analysis of Physiological Response to Two Virtual Environments: Driving and Flying Simulation,” CyberPsychology & Behavior, vol. 5, no. 1, pp. 11–18, Feb. 2002, doi: 10.1089/109493102753685845. [22]N. Du, X. J. Yang, and F. Zhou, “Psychophysiological responses to takeover requests in conditionally automated driving,” Accident Analysis & Prevention, vol. 148, p. 105804, Dec. 2020, doi: 10.1016/j.aap.2020.105804. [23]“Simulation Technologies,” Nervtech. https://www.nervtech.com (accessed Nov. 03, 2022). [24]“SCANeR studio,” AVSimulation. https://www.avsimulation.com/scaner-studio/ (accessed Nov.
03, 2022). [25]“Tobii Pro Glasses 2 wearable eye tracker,” Jun. 25, 2015. https://www.tobiipro.com/productlisting/tobii-pro-glasses-2/ (accessed Feb. 06, 2020). [26]“E4 wristband | Real-time physiological signals | Wearable PPG, EDA, Temperature, Motion sensors,” Empatica. https://www.empatica.com/research/e4 (accessed Aug. 12, 2020). [27]“Recommended tools for signal processing and data analysis,” Empatica Support. https://support.empatica.com/hc/en-us/articles/202872739-Recommended-tools-for-signalprocessing-and-data-analysis (accessed Nov. 06, 2022). [28]S. Al-Gawwam and M. Benaissa, “Eye Blink Detection Using Facial Features Tracker,” in Proceedings of the International Conference on Bioinformatics Research and Applications 2017, New York, NY, USA, Dec. 2017, pp. 27–30. doi: 10.1145/3175587.3175588. [29]A. Greco, G. Valenza, A. Lanata, E. P. Scilingo, and L. Citi, “cvxEDA: A Convex Optimization Approach to Electrodermal Activity Processing,” IEEE Transactions on Biomedical Engineering, vol. 63, no. 4, pp. 797–804, Apr. 2016, doi: 10.1109/TBME.2015.2474131. [30]“scipy.optimize.curve_fit — SciPy v1.9.3 Manual.” https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html (accessed Nov. 07, 2022). [31]D. A. Smith, “A Quantitative Method for the Detection of Edges in Noisy Time-Series,”
Philosophical Transactions: Biological Sciences, vol. 353, no. 1378, pp. 1969–1981, 1998. [32]“How does blinking affect eye tracking. Tobii Connect.” https://connect.tobii.com (accessed Nov.
07, 2022). [33]S. Taylor, N. Jaques, W. Chen, S. Fedor, A. Sano, and R. Picard, “Automatic identification of artifacts in electrodermal activity data,” Annu Int Conf IEEE Eng Med Biol Soc, vol. 2015, pp. 1934–1937, 2015, doi: 10.1109/EMBC.2015.7318762.