Due to the progressive shift of responsibility from the driver to the vehicle itself in automated vehicle technologies, driver-centered innovations represent a key point for its advance. The socalled Driver Monitoring Systems (DMS) are therefore increasingly gaining importance in this context. One of the main aims of DMS is to estimate the driver's arousal levels in order to infer their cognitive state and capabilities. Even though the scientific literature is riddled with useful psychophysiological indices to estimate arousal levels [1], nowadays, arousal estimation is based on broad, mostly blink/gaze-related, indices. The reason is that actual implementation of reliable sensors in a feasible system able to collect, analyze, and interpret measurements in real-life conditions is still an open challenge. One of the alternatives to signal different cognitive states is facial skin temperature [2][3]. Infrared sensors that monitor heat loss have been shown useful to track facial skin temperature that indicate arousal modulations while driving [2][3]. Such intensive, laborious work to extract and analyze temperature changes in some facial landmarks is not reasonable in real-life applications [2]. Here, we present the preliminary results obtained with a new software able to track, in real-time, drivers' facial-skin temperature changes.
Due to the progressive shift of responsibility from the driver to the vehicle itself in automated vehicle
technologies, driver-centered innovations represent a key point for its advance. The so-called Driver
Monitoring Systems (DMS) are therefore increasingly gaining importance in this context. One of the
main aims of DMS is to estimate the driver’s arousal levels in order to infer their cognitive state and
capabilities. Even though the scientific literature is riddled with useful psychophysiological indices to
estimate arousal levels [
We used a sensorized driving simulator (Nervtech™ solution, see Figure 1) running a SCANeR studio software (AVSimulation, v.DT2.5). Participants’ facial skin temperature was constantly monitored with a thermographic camera (FLIR A325sc, with a resolution of 320 × 240, a NETD < 50mK and an accuracy of ±2°C or ±2% of reading) synchronized with a color camera (infrared color camera, Intel® Realsense). 2.2.
To extract participants’ facial-skin temperature at specific locations, we developed an algorithm (Figure 2) able to identify two facial landmarks (Points of Interest [POIs]), the tip of the nose and the forehead, as well as the background in a thermographic image. The solution was based on a dual camera setup (i.e., color camera and thermographic camera), with a spatial correspondence between the two.
ℎ
= → ℎ = [0 1 0 ( ) ( ) 0 0 = [ 0
→ ℎ = 0 1 ] 0 1 − 0 0 1 0 1 2.3.
(1) (2) (3) (4) (5) multiplication of three matrices (2). The first describes a translation with coordinates [ , ] (3), the second describes a rotation around the center of the screen with angle (4) and the third describes a scaling with parameters [ , ] (5). Once the positions of the landmarks in the thermographic image space were found, the POIs temperature values were read in the image. Finally, we multiplied the result by the skin emissivity (0.98) to obtain the skin temperature.
The described system first needed to be calibrated to determine the parameters of the transformation matrix → ℎ
: , , , and . Filippini and colleagues used a similar set-up and performed
the calibration using a custom checkerboard [
To test our algorithm, we designed a 2 (traffic vs. low-traffic Traffic density) × 2 (automated [ADL4] vs. manual [MD] Driving modality) within-participants experiment. Thirty-five expert drivers (mean age = 41.61 years, standard deviation = 6.26 years) drove along two virtual scenarios (∼ 20 minutes [min] each) with varying traffic density. In both scenarios, the participants performed 10 min in MD and 10 min in ADL4. The order of the traffic density and driving mode was randomly balanced across drivers. During ADL4, they were instructed to supervise the system. We expect the arousal level of the drivers to be modulated by these conditions, as the manual driving mode and the high traffic condition are more demanding than the autonomous one and the low traffic condition respectively.
In order to validate the proposed algorithm, we selected randomly one of the two 20-min recordings (high or low traffic) for each driver. Then, we extracted one pair of color and thermographic images each 20 seconds. We obtained 65 images per driver and 2,340 in total. We then developed an annotation software to manually extract the temperature on these images. For each image, we pointed at two landmarks: the driver’s forehead and the driver’s nose tip. Four trained annotators performed the same procedure on the 2,340 images. Then, we computed on each image the mean of the four annotated positions of the nose tip and the forehead to establish the reference location of the nose tip and the forehead. Finally, we extracted the temperature at these locations to define the reference temperature. 2.6.
After removing the drivers on who the algorithm performed worse (see section 3.1.2), we used the algorithm described in Section 2.2 to extract the face temperature of the remaining drivers (n = 28). In order to obtain more measurement points, the algorithm at this stage ran at a higher frequency compared to the validation phase: one each 2 sec instead of one each 20 sec. We were therefore able to remove extreme values (lower than 25°C and higher than 37°C) as well as the outliers by applying a moving median thresholding procedure. We finally took the mean of the remaining points on each of the four segments: High traffic – Manual driving, High traffic – Automated driving, Low traffic – Manual driving, Low traffic – Automated driving. This gave us four data points per driver that we later used in our statistical analysis.
We first analyzed the algorithm performance in terms of position error in the thermographic image, measured in pixels. We computed the position error between the algorithm’s output and the mean values provided by the four annotators (see 2.5). We also compared the position error of the mean position error of each annotator with respect to the overall mean value. Then, we analyzed the consequences of the algorithm position error in terms of temperature error, measured in degrees Celsius (°C).
In Figure 6, we present the errors’ distributions of the algorithm and the annotators. As a reference, in our setup the nose tip measures approximately 10 x 10 pixels. When pointing at the nose, the algorithm performed worse than the annotators with respect to the mean of the annotators. The two main causes for high mismatches were landmarks estimation errors of MediaPipe and spatial correspondence errors due to head movements (head turning or bending). Surprisingly, the algorithm outperformed slightly the annotators on the forehead with respect to the mean of the annotators. Our interpretation is that for a human, it could be hard to define a precise location on a large area with no points of reference such as the forehead.
Figure 7 shows the errors of the final temperature values computed by the algorithm. On the forehead, the temperature gradient was low, so the temperature error resulting from the position error was small. On the nose, however, the temperature gradient was higher, so the temperature error was also much higher compared to the forehead. Interestingly, the temperatures computed at the positions annotated by one annotator are consistently smaller than the temperatures computed at the mean of the annotated positions. This is because the face temperature exhibits a local peak on the nose and one individual annotator is further from this peak than the mean position of the four annotators.
Looking at Figure 8, we see that the mean absolute error of the nose temperature highly depends on the driver (it goes up to 1.6 °C for some drivers). For the statistical study, we excluded the 6 participants with an absolute error higher than 0.8 °C.
The present work describes the first results obtained with an algorithm for tracking a driver’s facial
skin temperature during driving interactions. The algorithm consistently and effectively tracked
participants’ facial-skin temperature without interfering with their driving tasks. We have analyzed the
position and temperature errors and for some drivers, tracking the nose tip temperature remains a
challenge. Future systems should improve both the initial landmarks detection and the landmark
transfer. The later could be achieved by measuring the distance between the cameras and the driver’s
face like previous studies [
This study was funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No. 875597 - HADRIAN (Holistic Approach for Driver Role Integration and Automation Allocation for European Mobility Needs) project. This document reflects only the authors' view, the European Climate, Infrastructure and Environment Executive Agency (CINEA) is not responsible for any use that may be made of the information it contains. We thank Leila Maboudi (Polytechnic University of Turin, Italy) for her comments and assistance in language edition.