=Paper=
{{Paper
|id=Vol-1812/JARCA16-paper-1
|storemode=property
|title=Estimating the Stress for Drivers and Passengers Using Deep Learning
|pdfUrl=https://ceur-ws.org/Vol-1812/JARCA16-paper-1.pdf
|volume=Vol-1812
|authors=Víctor Corcoba Magaña,Mario Muñoz Organero,Jesús Arias Fisteus,Luis Sánchez Fernández
}}
==Estimating the Stress for Drivers and Passengers Using Deep Learning==
https://ceur-ws.org/Vol-1812/JARCA16-paper-1.pdf
Estimating the Stress for Drivers and Passengers Using Deep Learning
Víctor Corcoba Magaña, Mario Muñoz Organero, Jesús Arias Fisteus, and Luis Sánchez Fernández
Dpto. de Ingeniería Telemática
Universidad Carlos III
vcorcoba@it.uc3m.es
others (15%).
There are many proposals on measuring and
quantifying the driver’s cognitive load and stress
Abstract levels. In [Gao et al., 2014] the authors propose a
The number of vehicles in circulation has method for detecting stress based on facial
become a problem both for safety and for the expressions.They employed a near-infrared (NIR)
citizens health. Public transport is a solution to camera to capture the near frontal view of the driver’s
reduce its impact on the environment. One of face. Tracking face is made using a supervised descent
the keys to encourage users to use it is to method (SDM) [Xiong and Torre, 2013].
improve comfort. On the other hand, In [Eilebrecht, 2012], the research analyzed the
numerous studies highlight that drivers are suitability of the heart rate variability (HRV) to
more likely to suffer physical and measure the driving workload. The results conclude
psychological illnesses due to the sedentary that the HRV could be used as a good workload
nature of this work and workload. In this indicator, although it is also affected by many other
paper, we propose a model to predict the stress factors that may have an influence on it. To solve this
level on drivers and passengers. The solution problem, the author suggested using other parameters
is based on deep learning algorithms. The such as: skin conductivity, gripping pressure on the
proposal employs the Heart Rate Variability steering wheel, respiration or blood pressure. In the
(HRV) and telemetry from the vehicle in order paper [Rodrigues et al., 2015], the authors also
to anticipate the incoming stress. It has been observed that variability is not enough to detect the
validated in a real environment on distinct stress. They also mentioned the meaning of previous
routes. The results show that it predicts the driver experience. Novice drivers are more likely to get
stress by 86 % on drivers and 92% on stressed driving in difficult situations.
passengers. This algorithm could be used to Currently, the vehicle has become an essential
develop driving assistants that recommend element, both for the freight transport and the city
actions to smooth driving, reducing the residents. In Japan [Elbanhawi et al ,2015] statistics
workload and the passenger stress. show that each person travels 7500 Km on average per
year. On the other hand, many of these trips are made
by bus or rail. In the United States [Innamaa and
1. Introduction Penttinen, 2014], commuters travel more than 20
Cognitive errors appear in highly cognitive demanding billion miles by bus and 9 billion miles by commuter
situations in which the cognitive load as perceived by rail. In the literature there are many methods for
the driver is high and the actions taken by the driver to evaluating the comfort of the passenger. The author in
handle those situations are in many occasions not [da Silva, 2002] proposes various measures based on
appropriate. The data presented in [U.S. Department of vibrations, noise, temperature and air quality for this.
Transportation, 2008] categorizes the major risk He also mentioned that actions such as accelerating,
factors responsible for traffic accidents as follows braking, or engaging the clutch impact on the level of
(according to their impact): human factors (92%), comfort in the driver. The road geometry is another
vehicle factors (2.6%), road/environmental factors factor to consider.
(2.6%), and others (2.8%). Among these, drivers’ [Hoberock, 1977] studies the relationship between
human factors consist of cognitive errors (40.6%), the passenger comfort and the longitudinal motion. He
judgment errors (34.1%), execution errors (10.3%), and estimated nominal acceleration values in the range of
Copyright © by the paper’s authors. Copying permitted only for private
0.11 g to 0.15 g and a maximum jerk region of 0.30
and academic purposes. g/sec in the longitudinal direction to maintain a good
In: Zoe Falomir, Juan A. Ortega (eds.): Proceedings of JARCA 2016,
confort level. For safe emergency response, maximum
Almería, Spain, June 2016, published at http://ceur-ws.org acceleration of 0.52 g is estimated to prevent
dislodgment. The author also noted that these results 2.2 Input variables
are affected by passenger type. In addition, he
In this work, the input variables can be classified into
concluded that there are few studies analyzing the over
two groups: variables related to the stress level and
short distances. In this scenario, the start-stop variables associated with the vehicle telemetry.
movements are very frequents, disturbing the
passengers.
Measurements of stress level
The authors in [Wang et al., 2001] propose a a
Heart Rate signals are employed as an indicator for
driving assistant that estimates the discomfort in real
the Autonomic Nervous System (ANS) neuropathy for
time. This solution is based on the analysis of the
normal, fatigued and drowsy states because the ANS is
accelerations. The objective of this study is that the influenced by the sympathetic nervous system and
driver change his driving style in order to reduce the
parasympathetic nervous systems. This indicator is not
passenger ride discomfort and provide passenger-
intrusive. A high heart rate correlates with the driver
friendly maneuvers.
experiencing stress.
The authors in [Innamaa and Penttinen, 2014] made
Among the different variables previously described
a study about the impact of Green-driving applications
as presented in the existing literature we have used the
on fuel consumption, speeding and passenger confort. Heart Rate Variability (HRV) since it has been
The results demostrated that the driving assistant
assessed as one having a higher correlation with stress
improved the decelerations and the driver’s service
levels together with Skin Conductivity (SC).
attitude in peak traffic significantly. This was also
One major limitation of the HRV signal in order to
experienced as more pleasant by passengers. All these
estimate the level of stress and cognitive load is that
works is based on questionnaires whose responses are
there are other factors such as the physical exercise that
subjective. The consequence is a large variation among also impact the measured values. As described in the
the different results.
previous section, the experiment has been designed to
In conclusion, there are many proposals to detect the
minimize the impact that factors outside the study have
driver stress and measure the comfort of the
on the measurements. In this way, only data from
passengers. However, there are no solutions to predict
drivers driving alone to work and back home in similar
a worsening for the tension on drivers and passengers
situations each day (same hour, same traffic conditions,
in real scenarios. In this paper, we propopse a model with moderated previous walking to get into the car
that allows us to estimate if the driver or passenger will
and a relaxation period of 30 seconds before driving,
experience an increase in the stress level.
with the mobile phone muted, with the radio switched
The aim is that assistive tools to prevent the stress or
off and without using any navigation system) have
avoid it could be developed in the future based on our
been taken. The same methodology was applied to
proposed algorithm. This algorithm could be used to
passengers.
build a driving assistant that recommends an optimal In addition, we analyze the driving behavior. The
vehicle speed in order to maximize confort while
combination of these two groups of variables allows us
driving. This recommended speed will minimize the
to build a model to predict the stress on drivers and
driving workload and the speed fluctuations. The
passengers accurately.
reduction in the number and intensity of the
We can consider Heart Rate Variability (HRV) from
accelerations will improve the comfort of the drivers
two different domains: Time and Frequency Time
and passengers. domain analysis of HRV implicates quantifying the
mean or standard deviation of RR intervals. Frequency
domain analysis means calculating the power of the
Proposal for the estimation of stress level respiratory-dependent high frequency and low
on drivers and passengers frequency components of HRV. In our case, we are
going to use measures on the time domain. There are
many HRV that can be defined on this domain such as:
2.1 Objective
mean RR interval (mRR), mean heart rate (mHR),
Our goal is to predict the stress, both for drivers and standard deviation of RR interval (SDRR) or standard
passengers, based on the current measurements of deviation of heart rate (SDHR).
stress level and vehicle telemetry. We have chosen the following variables based on
real tests:
• Average HeartRate (b.p.m): This variables has
a high value when the driver or passenger
experiences high levels of stress.
• Average RR (ms): It measures the time
between beat-beat (consecutive heartbeats). Its
value decreases when there is an event that
causes stress on the driver. On the contrary, a
high value means that the driver is relaxed.
• Standard deviation of RR intervals (ms): the
variation between beat and beat (inter-beat
period) increases when the driving workload
is high.
• RR50: It is the number of pairs of successive
RRs that differ by more than 50 ms. A high
number allows us to detect stress situations. Fig. 1. Schema of the driving assistant.
• Average acceleration (positive and negative):
Frequently, these actions cause stress. For In order to minimize the random errors introduced
example, if the car in front braked suddenly, by the sensors in each sample, values are averaged
the driver from behind will have to react each 5 seconds. The measured values in the last 5
quickly. This situation increases the tension, seconds of driving are then used to estimate the
both for drivers and for passengers and upcoming stress level for the next 5 seconds of driving.
demands attention. Therefore, a 10 second window is selected for the
computations and validation.
Driving behavior
Figure 1 captures the driving speed profile and the
result of the difference between the current RR and the 2.3 Method
previous RR. We can observe that the driver’s In order to validate results we consider two cases.
perceived tension increases between seconds 2 and 4,
when the driver is accelerating. In addition, the stress The first scenario is to predict the drivers stress. The
level also worsens in the interval between 15 and 23 second case is to estimate the incoming stress for the
seconds. During this period, the driver was accelerating passengers.
and braking frequently. In the first case, 6 different users using 2 different
cars in 2 different regions have been selected. The
• Positive Kinetic Energy (PKE): This variable regions were Sheffield in the UK and Madrid in Spain.
measures the aggressiveness of driving. Its
The vehicle models were an Opel Zafira Tourer and a
value depends on the intensity and frequency
of the accelerations. If it is high means that the Citroen Xsara Picasso. In total, we have obtained 100
driver accelerated sharply and frequently. This test drives with around 2000 minutes of driving. Each
driving style has a negative impact on the test drive comprised both urban and inter-urban (rural
stress level, both for drivers and passengers. and highway) sections.
In the case of the driver it implies that he or One major limitation of the HRV signal in order to
she has to make faster decisions in order to
estimate the level of stress and cognitive load is that
avoid accidents.
there are other factors such as the physical exercise that
• The intensity of turning: We detected during
testing that the tension increased in the also impact the measured values. The experiment has
majority of the drivers when there were curves been designed to minimize the impact that factors
on the road. The degree of impact depends on outside the study have on the measurements. In this
the road angle (intensity of turning required). way, only data from drivers driving alone to work and
back home in similar situations each day (same hour,
same traffic conditions, with moderated previous
walking to get into the car and a relaxation period of at
least 30 seconds before driving, with the mobile phone The intensity of turning is estimated using the
muted, with the radio switched off and without using following formula:
any navigation system) have been taken.
*+ ∙ *+-.
In the second case, 6 different bus drivers and 6 !"# = cos−1 ; 1# > 3ℎ
*+ *+-.
passengers have participated. The regions were near
Sheffield in the UK. In total, we have obtained 30 test
drives with around 1200 minutes of driving. Each test Where the numerator represents the dot product
drive comprised both urban and inter-urban sections. between the average direction vectors in the last 5
As in the first case, the experiments were conducted seconds and the average direction vectors in the next 5
under similar conditions in order to get accurate results. seconds and the denominator captures the norm of such
A Polar H7 band was used to record the HRV signal. averaged vectors. The direction vectors are calculated
The band was paired with a Nexus 6 Android Mobile from the GPS coordinates. The average over a period
device running an application implemented for the of 5 seconds is used to minimize the impact of random
experiment which recorded the HRV together with errors in the GPS signal. In order to eliminate the errors
GPS data and telemetry data such as the driving speed. introduced at low speeds, a threshold in the speed is
In order to predict the stress level, we only use a used. This threshold has been empirically evaluated
heart rate strap because this device is not intrusive and and a value of 1 m/s has been found to perform well
the cost is low. These features allow us to make tests and therefore selected for the experiment.
with a broad population. The aim is to maximize the
acceptability. The user only has to buy a heart rate 3 Validation of the proposal
strap that costs about 50$. There are many solutions We capture the results for drivers and passengers in
[Mohan et al., 2016] which demonstrate that only with two different sub-sections.
this variable, you can get a good stress estimate.
The current acceleration of the vehicle is calculated 3.1 Prediction of driver stress
based on the measured speed as follows: This section captures the results of training the
algorithms leaving out one particular driver and using
%" - %"'( the trained algorithms with the data coming from the
!" = other drivers (5 drivers) to predict the stress levels. To
)" - )"'( validate results, we have used 5 different algorithms to
In which vi represents the speed at the sample capture different families of machine learning
techniques: Support Vector Machines (SVM), Multi-
number i, ai the estimated acceleration at that sample
Layer Perceptron (MLP), Naïve Bayes, C4.5 and Deep
and the derivative of the speed is estimated by dividing Learning. The confusion matrixes are captured in
the increment in speed by the time elapsed between the tables 1 to 5.
consecutive samples i-1 and i. We can see that the Deep Learning algorithm has
The positive kinetic energy (PKE) is estimated over obtained the best result both in terms of accuracy and
a period of time as follows: positive hits. This algorithm is able to predict stress
situations by 86%. On the other hand, other algorithms
predict high driving workload by 52%. In this case, the
&' - &')* + algorithm which presents the best best accuracy is c4.5.
!"# = ; Deep Learning is a method that introduces a new
,
way to train multilayer networks. This technique
Where vi is the speed (meters/seconds) and d is the trip allows us to discover the complex relationships
distance (meters). between variables.
TABLE I
SVM FOR PREDICTING STRESS (DRIVER) TABLE 6
Actual/Predicted Yes No SVM FOR PREDICTING STRESS (PASSENGER)
Yes 0.70 0.30 Actual/Predicted Yes No
No 0.35 0.65 Yes 0.64 0.36
No 0.28 0,72
TABLE 2
MLP FOR PREDICTION STRESS (DRIVER) TABLE 7
Actual/Predicted Yes No MLP FOR PREDICTION STRESS (PASSENGER)
Yes 0.43 0.57 Actual/Predicted Yes No
No 0.26 0.74 Yes 0.70 0.30
No 0.20 0.80
TABLE 3
NAÏVE BAYES FOR PREDICTION STRESS (DRIVER) TABLE 8
Actual/Predicted Yes No NAÏVE BAYES FOR PREDICTION STRESS (PASSENGER)
Yes 0.35 0.65 Actual/Predicted Yes No
No 0.16 0.84 Yes 0.85 0.15
No 0.35 0.65
TABLE 4 TABLE 9
C4.5 FOR PREDICTION STRESS (DRIVER) C4.5 FOR PREDICTION STRESS (PASSENGER)
Actual/Predicted Yes No Actual/Predicted Yes No
Yes 0.59 0.41 Yes 0.75 0.25
No 0.32 0.68 No 0.23 0.77
TABLE 5 TABLE 10
DEEP LEARNING FOR PREDICTION STRESS (DRIVER) DEEP LEARNING FOR PREDICTION STRESS (PASSENGER)
Actual/Predicted Yes No Actual/Predicted Yes No
Yes 0.86 0,14 Yes 0.92 0,08
No 0.17 0.83 No 0.03 0.97
3.2 Prediction of passenger stress
This section captures the results of training the 3 Conclusions
algorithms leaving out one particular passenger and The results demostrate the suitability of deep
using the trained algorithms with the data coming from learning algorithms to predict the perceived stress both
the other passengers (5 drivers) to predict the stress for drivers and passengers. We have also observed that
levels. The confusion matrixes are captured in tables 6 the prediction of stressful situations is easier for the
to 10. passengers. Drivers are subjected to more difficult
In this case we see that the prediction of the situations. Therefore, it is more complex to anticipate
passenger stress is more accurate than the first scenario upcomming reactions for drivers than for passengers.
taking into account all the algorithms. We have found In addition, both types of users do not start from the
during experiments that passengers are always more same stress level. Driving is a difficult task because
relaxed than the driver. Stress only increases when drivers have to do multiple actions at the same time.
there is a high stress event such as high deceleration, This causes that the drivers stress levels are higher than
high acceleration or jerk. In this case, Deep Learning the passengers stress levels for the same road
algorithm is able to predict stress situations by 92%. conditions.
The remaining algorithms only obtained a hit rate of As future work, we want to add more information in
74% on average. order to increase accuracy in the prediction of the
driver stress. The previous activity level, sleeping time,
working time, the upcoming traffic signs, and the Communication and Signal Processing (ICCSP),
driving style from the nearby driver are variables that Melmaruvathur, 2016, pp. 1141-1144. doi:
can be useful to improve the solution. On the other 10.1109/ICCSP.2016.7754331
hand, the results of this paper could be employed to [Rodrigues et al., 2015] J. G. P. Rodrigues, M.
develop driving assistants that avoid or reduce the Kaiseler, A. Aguiar, J. P. Silva Cunha and J. Barros,
driving workload. "A Mobile Sensing Approach to Stress Detection
and Memory Activation for Public Bus Drivers," in
Acknowledgments IEEE Transactions on Intelligent Transportation
The research leading to these results has received Systems, vol. 16, no. 6, pp. 3294-3303, Dec.
funding from the “HERMES-SMART DRIVER” 2015.doi: 10.1109/TITS.2015.2445314.
project TIN2013-46801-C4-2-R funded by the Spanish [U.S. Department of Transportation, 2008] U.S.
MINECO, from the grant PRX15/00036 from the Department of Transportation “National motor
Ministerio de Educación Cultura y Deporte and from a vehicle crash causation survey,” Washington, DC,
sabbatical leave by the Carlos III of Madrid University. USA, Tech. Rep. DOT HS 811 059, July 2008.
[Wang et al., 2001] F. Wang, N. Ma and H. Inooka, "A
driver assistant system for improvement of
References passenger ride comfort through modification of
[da Silva, 2002] M. C. G. da Silva, “Measurements of driving behaviour," Advanced Driver Assistance
comfort in vehicles,” Meas. Sci. Technol., vol. 13, Systems, 2001. ADAS. International Conference on
no. 6, p. R41, 2002. (IEE Conf. Publ. No. 483), Birmingham, 2001, pp.
38-42. doi: 10.1049/cp:20010494
[Eilebrecht, 2012] B. Eilebrecht, "The relevance of
HRV parameters for driver workload detection in [Xiong and Torre, 2013] X. Xiong and F. De la Torre,
real world driving," 2012 Computing in Cardiology, “Supervised Descent Method and Its Applications
Krakow, 2012, pp. 409-412. to Face Alignment,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR),
[Eilebrecht, 2012] B. Eilebrecht, "The relevance of
2013, pp. 532–539.
HRV parameters for driver workload detection in
real world driving," 2012 Computing in Cardiology,
Krakow, 2012, pp. 409-412.
[Gao et al., 2014] H. Gao, A. Yüce and J. P. Thiran,
"Detecting emotional stress from facial expressions
for driving safety," 2014 IEEE International
Conference on Image Processing (ICIP), Paris,
2014, pp. 5961-5965. doi:
10.1109/ICIP.2014.7026203.
[Hoberock, 1977] Hoberock, L. L. (1977). A survey of
longitudinal acceleration comfort studies in ground
transportation vehicles. Journal of Dynamic
Systems, Measurement, and Control, 99(2), 76-84.
[Innamaa and Penttinen, 2014] S. Innamaa and M.
Penttinen, "Impacts of a green-driving application
in city buses on fuel consumption, speeding and
passenger comfort," in IET Intelligent Transport
Systems, vol. 8, no. 5, pp. 435-444, August 2014.
doi: 10.1049/iet-its.2013.0039
[Mohan et al., 2016] P. M. Mohan, V. Nagarajan and S.
R. Das, "Stress measurement from wearable
photoplethysmographic sensor using heart rate
variability data," 2016 International Conference on