In near future scenarios, self-driving vehicles will circulate in urban environments, and their behaviour should be adapted with respect to diferent types of pedestrians. In particular, vehicles should be able to provide efective feedback, especially when dealing with the most vulnerable people, such as older adults and impaired subjects. Within this perspective, this paper illustrates the experimental settings and protocol to study pedestrian and autonomous vehicle interaction, especially focusing on the safeness felt by each subject in diferent crossing conditions. To this end, besides traditional self assessment questionnaires and video recordings, movement and physiological data are collected as indicators of stress. From the analysis of this multimodal data, diferent classes of pedestrians could be defined, that will guide the definition of proper vehicle behaviour depending on their level of confidence and safety feeling. A preliminary data collection have been performed and is here described in a controlled urban-like crossing environment. Subjects of various ages were considered, as well as diferent dynamic behaviours of a properly prepared vehicle, running in both human-controlled and self-driving modes.
The vehicle-pedestrian interaction while crossing a road is a crucial aspect in the feeling of safe
walking, and subjective emotions must be taken into account, considering diferent degrees of
pedestrians vulnerability (age, gender, disabilities). In future scenarios, self-driving vehicles will
circulate in urban environments, and will need to adapt to the feelings of pedestrians, being
able to provide them efective feedback, properly tuned for the most vulnerable ones, such
as older adults and impaired people [
To this end, we here propose an experimental setting performed in a controlled urban-like crossing environment. The main research questions that guided the definition of the experimental protocol can be summarized as follows: • Q1: the safeness felt by the pedestrian while crossing a street decreases with respect to the perceived level of autonomy of the vehicle. • Q2: the safeness felt by the pedestrian while crossing a street increases as the safety gap increases (i.e., the distance between the pedestrian and where the vehicle ends its sharp deceleration); • Q3: the safeness felt by the pedestrian while crossing a street decreases with increasing age of the pedestrian.
Video cameras and proper self assessment questionnaires are adopted to profile the subjects and assess their confidence with respect to self-driving vehicles. We rely on physiological responses to assess the subjective feeling of safe crossing. In our investigation we consider PhotoPlethysmoGraphy (PPG) that measures the blood volume registered just under the skin, which can be used to calculate the heart rate of the subject, and Galvanic Skin Response (GSR), that measures the skin sweat, as they are both efective to detect emotional arousal. Arousal is a physiological and psychological state that can be related to sensory alertness, mobility, and readiness to respond, activated as a defensive reaction to preserve safety. Moreover, motion data, measuring the muscle activity with Electromyography (EMG), are also collected, in an integrated approach to study pedestrian walkability. Relying on diferent signal sources that register both physiological and dynamic walking responses will provide accurate results for afective state recognition tasks. We have been encouraged to perform this research, by having obtained positive results in a previous experiment on the pedestrian interaction with traditional vehicles, whose aim was to collect movement and physiological data as reliable indicators of stress, during safe walking and road crossing [13]. The interaction between pedestrians and autonomous vehicles has been already considered in few literature works in which the perceived stress of pedestrians during crossing is detected using GSR signals [14, 15]. These analyses, however, use data collected in scenarios developed in immersive virtual reality setup, thus not representative of real crossing situations.
For the experiments, two distinct groups of subjects are taken into consideration. The first group, with an age between 18 and 35 years and the second group of over 65 years old. The inclusion criteria are: i) age in one of the classes mentioned; ii) absence of major medical disorders (neurological disorders, epilepsy, severe cognitive disorders); iii) no presence of pharmacotherapy that could interfere with the measured data (psychotropic drugs, anti-depressants); iv) no significant visual impairment (all with normal visual acuity or corrected to normal); v) no significant hearing impairment; and vi) autonomous mobility without the need for supports. The suitability of the participants is verified through a self-report questionnaire on personal medical history. Before participating in the study, each individual is informed by the investigator about the characteristics of the research, both verbally and through an information document. The participant signs an informed consent. Participation in the trial takes place following the voluntary participation of the subjects. The experiment is carried out in a private parking lot of the Università di Milano - Bicocca, during the weekend or at other non-working hours in a controlled urban crossing environment, as depicted in Figure 1. Three video cameras are used in order to record diferent points of view of the interaction between the subject and the vehicle. Video camera 1, on the top left of the scene depicted in Figure 1 records the subject approaching the zebra crossing to study her/his appraisal behaviour. Video camera 2 and video camera 3 (on the bottom right of the same Figure) record the crossing behaviour and the facial expressions of the subjects. A van is positioned a few meters before the crossing, in order to partially occlude the path of the vehicle, so that the subject can visually perceive the vehicle only when she/he is nearly arrived at the crossing. The participants wear wearable sensors that measure heart beat through Photopletysmography (PPG), Galvanic Skin response (GSR) and muscle activity using noninvasive Electromyography (EMG). The PPG and GSR sensors are placed on the fingers of the dominant hand, while the EMG sensors are placed on one leg. The sensors used are noninvasive and completely painless. The sensors used to collect physiological data are Shimmer3 GSR+ and Shimmer3 EMG/ECG [16]. Both these sensors interface with a software named ConsensysPRO, made by Shimmer as well, used to setup our trials, superimposing markers to raw data, and to partially pre-process collected data. To answer research questions 2 and 3 (related to the safeness feeling with respect to the perceived vehicle autonomy and with respect to varying safety gaps), the following conditions have been considered: • crossing the zebra with no vehicle interaction (C); • crossing the zebra interacting with the vehicle both in manually-driven (M) and self-driven (S) conditions; • self-driven conditions are implemented by putting a person on the passenger seat that is not paying attention to the road; • crossing the zebra with two diferent safety gaps ( DS distance short, and DL distance long), adopted by the vehicle in both conditions (M or S).
The experimental protocol also includes self-assessment questionnaires, for evaluating the self-esteem (SE) levels of the participants, filling the Rosenberg questionnaire (see [ 17]), the personality traits, filling a short version of the BIG 5 form, [18], and the level of safety felt after each crossing (SC). Before starting the experiment a questionnaire about the subject’s confidence with respect to both self and manually-driven vehicles is administered (IC). Moreover, at the end of the experiment, the subject has to answer if she/he had noticed the type of guide for every cross (manual or autonomous) (ATT ).
The whole protocol is described as follows: • Questionnaires filling: SE, BIG 5, and IC. • Baseline: 2 minutes session to acquire the reference physiological signals, where the subject has to stay straight up and still, to record her/his physiological responses in absence of any tasks. • Experiment Core: 6 repetitions of C, 2 repetitions of M DL, 2 repetitions of M DS, 2 repetitions of S DL, 2 repetitions of S DS, 6 repetition of a 60 seconds baseline recording, also intended to bring the subject back to a neutral state before the next task. Each task is followed with the crossing questionnaire filling ( SC). Within the experiment core, the order of the tasks is randomly selected for each subject, in order to avoid possible biases introduced by the experimental setting.
• Questionnaire filling: ATT.
The experiment lasts about 40 minutes, 20 of which are dedicated to self-assessment of the questionnaires, long enough to collect usable data and short enough to prevent subjects from getting used to the task at hand.
For the experiment, we used the prototype of autonomous vehicle shown in Figure 2. It is equipped with two front-mounted single-plane LiDARs, to sense the environment and detect obstacles (including pedestrians), and one front camera, used by the operator at the passenger seat. The hardware and software architecture of the vehicle base was developed to ensure safe and reliable experiments. A custom relay board was developed for switching between manual control and self-driving mode. A DC motor was mounted on the steering wheel to control the self-driving car. In addition, the throttle is controlled by a digital potentiometer via the vehicle’s electronic control unit. An absolute encoder was mounted under the steering mechanism to sense the steering angle, and two encoders were mounted on the rear wheels to measure the vehicle’s speed and enable closed-loop control. The digital potentiometer and steering wheel are controlled by an STM32 microcontroller. The linear and angular velocity commands are sent from the main computer to a Raspberry Pi, which relays the messages to the microcontroller.
The safety of the experiment is also ensured by several watchdogs. On the Raspberry Pi, a software watchdog was implemented to send a stop command to the STM32 if no valid speed command was received within a certain time window. The same safety measure was implemented on the STM32 using a hardware watchdog that is updated every time a valid control command is received. If no valid command is received in time, the hardware watchdog runs down and stops both the main motor and the steering wheel. As an additional safety measure, several emergency stop buttons have been installed on the vehicle: if something unexpected happens, the operator can immediately interrupt the autonomous control loop and slow down the vehicle. The entire vehicle infrastructure is shown in Figure 3.
Left wheel w/ rotary encoder
Right wheel w/ rotary encoder
Absolute encoder Steering wheel motor H bridge Main motor
Electronic Control Unit
STM32 mcu
Ethernet Switch
Photon Focus front camera Custom relay board
Raspberry Pi Emergency Stop
Main Computer
LMS-111 SICK LMS-111 SICK
LiDAR LiDAR
To ensure the repeatability of the experiments, we opted for map-based localization and navigation. The grid-based map is generated ofline using pre-recorded data with GMapping [19]. For the localization of the vehicle, we use a scan matching-based localization algorithm [20], which estimates the current pose of the vehicle by comparing the map with laser scan data acquired with the LiDAR.
The various components of the software architecture use the ROS framework for communication [21]. The typical approach to robot navigation using ROS involves what is called a global planner that generates the best path to a destination. However, to ensure that the vehicle always follows the same trajectory, we developed a custom planner that always sends the same straight path to the lower layers, along with the velocity profile we described earlier.
As a safety measure, we developed a simple obstacle detection component that stops the vehicle when an obstacle is detected within a range of 1 . This distance may seem very short, but the vehicle approaches the crosswalk at walking speed (0.6 /) after full braking. Therefore, the stopping distance is also very short.
(a) (b)
The experiment here described is still ongoing. Data so far collected comes from a population of
young adults, 10 male and 4 female, with average age of 23 (standard deviation = 4.1), collected
from Bachelor, Master and PhD students, and from 5 older adults with mean age of 65.6 (standard
deviation = 1.34), including 3 female and 2 male subjects, for a total of 19 subjects.
Most of the data has not yet been analyzed, so here we report few of the preliminary analysis
and our proposal of future works. Among all the questionnaires administered, we report here
the results of the IC questionnaire, acquired before the execution of the crossings. With this
questionnaire we want to investigate the initial subject’s confidence with respect to both self
and manually-driven vehicles. The two questions on a 5 point Likert scale are:
• Thinking of traditional vehicles, when crossing a road without trafic lights, how much
trust do you have in drivers? 1: no trust; 2: limited trust; 3: trust; 4: high trust; 5 total
trust;
• Thinking about self-driving vehicles, if you were to cross a street without trafic lights,
how much confidence would you have in the vehicle? 1: no trust; 2: limited trust; 3: trust;
4: high trust; 5 total trust
The histograms of Figure 4 summarize the results collected from the questionnaire according
to the population considered (young or older adults) as well as the type of driven condition
(manually or self). The following comments could be drawn starting from the values collected:
• The results obtained applying the non-parametric Wilcoxon Rank Sum Test showed
that there are no significant diferences in the subject’s confidence level when self and
manually-driven vehicles are taken into account (p-value = 0.86, alpha = 0.05). In both
cases, the average of the collected values is in the range [
These preliminary observations are quite interesting but, at the same time, require a higher number of subjects responses.
In future studies, the raw physiological signals collected during the experiment will be preprocessed and normalized to reduce noise and moderate subjects dependencies. From each of the four normalized signals, a proper set of features will be evaluated as characteristics useful to describe them. From PPG signals, handcrafted features will be considered including temporal features and frequency domain features. Concerning the GSR, statistical and peak related features extracted from phasic component will be examined, as well as the regression coeficient feature obtained from the tonic part of the signal. Finally, two features will be computed from EMG signals: the Root Mean Square [22] and the walking frequency, known as Stride Frequency, evaluated in terms of number of steps per second [23]. To verify the research hypotheses, statistical analyzes will be carried out to compare feature distributions of diferent crossing conditions, comparing the two groups of populations. In particular, the analyses will focus on identifying if there are significant diferences in subjects’ physiological signals during crossing in presence or absence of vehicles. Furthermore, statistical test could also be used to evaluate variations in people’s behaviour and physiological parameters according to the perceived level of autonomy of the vehicle and on diferent safety gaps, collected by the self assessment questionnaires. Finally, tests will be performed to verify if there are statistically significant diferences between the two populations considered during the same crossing conditions. Furthermore the analysis of the videos recorded by the three cameras that acquire diferent points of view of the interaction between the vehicle and the participant could provide behaviour understanding especially in the appraisal phase, and during the crossing, also in terms of emotion recognized from the facial expressions.
The experiment here described aims at defining perceivable classes of pedestrians with respect to their level of confidence and safety feeling, while crossing a street in presence of a self-driven vehicle. A proper vehicle dynamic behaviour can thus be defined depending on the perceived class of pedestrians, to increase his/her sense of safety. The adaptation of the safety gap to the class of pedestrian is technically possible, as it requires the sensing suite of the vehicle to be able to classify the pedestrians nearby the crossing, and then the corresponding dynamic behaviour is executed. This paper reports the details of the experimental protocol, which makes a useful tool for the repeatability of the experiments. Some preliminary analysis is also reported.
This research is partially supported by the FONDAZIONE CARIPLO “LONGEVICITY-Social Inclusion for the Elderly through Walkability” (Ref. 2017-0938). We also thanks the many students that helped develop the vehicle in the previous years: P. Colombo, D. Gerosa, J. Maltagliati, M. Pugno, I. Rizzi, and others. tonomous vehicle-pedestrian interaction, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–12. [13] F. Gasparini, M. Giltri, S. Bandini, Experimental approach to study pedestrian dynamics towards afective agents modeling., in: ATT@ ECAI, 2020, pp. 31–38. [14] M. Mudassar, A. Kalatian, B. Farooq, Analysis of pedestrian stress level using gsr sensor in virtual immersive reality, arXiv preprint arXiv:2111.11492 (2021). [15] K. Kamal, B. Farooq, M. Mudassar, A. Kalatian, Ordered-logit pedestrian stress model for trafic flow with automated vehicles, arXiv preprint arXiv:2204.11367 (2022). [16] A. Burns, E. P. Doheny, B. R. Greene, T. Foran, D. Leahy, K. O’Donovan, M. J. McGrath, Shimmer™: an extensible platform for physiological signal capture, in: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, 2010, pp. 3759–3762. [17] M. Rosenberg, The association between self-esteem and anxiety., Journal of Psychiatric
Research (1962). [18] C. Chiorri, F. Bracco, T. Piccinno, C. Modaferi, V. Battini, Psychometric properties of a revised version of the ten item personality inventory., European Journal of Psychological Assessment 31 (2015). [19] G. Grisetti, C. Stachniss, W. Burgard, Improved techniques for grid mapping with raoblackwellized particle filters, IEEE transactions on Robotics 23 (2007) 34–46. [20] E. Pedrosa, A. Pereira, N. Lau, Eficient localization based on scan matching with a continuous likelihood field, in: 2017 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), 2017, pp. 61–66. [21] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y. Ng, Ros: an open-source robot operating system, in: ICRA workshop on open source software, volume 3, Kobe, Japan, 2009. [22] A. Phinyomark, C. Limsakul, P. Phukpattaranont, A novel feature extraction for robust emg pattern recognition, arXiv preprint arXiv:0912.3973 (2009). [23] F. Gasparini, A. Grossi, K. Nishinari, S. Bandini, Age-related walkability assessment: A preliminary study based on the emg, in: International Conference of the Italian Association for Artificial Intelligence, Springer, 2020, pp. 423–438.