Pedestrian and autonomous vehicle interaction: towards affective crossing Domenico Giorgio Sorrenti1 , Francesca Gasparini1 , Fabio D’Elia1 , Andrea Ballini1 , Simone Fontana2 , Federica Di Lauro1 , Alessandra Grossi1 , Stefano Dessena1 and Stefania Bandini1,3 1 Dept. Informatica, Sistemistica e Comunicazione, Università di Milano - Bicocca, Italy 2 School of Law, Università di Milano - Bicocca, Italy 3 RCAST Research Center for Advanced Science & Technology, University of Tokyo, Tokyo, Japan Abstract In near future scenarios, self-driving vehicles will circulate in urban environments, and their behaviour should be adapted with respect to different types of pedestrians. In particular, vehicles should be able to provide effective feedback, especially when dealing with the most vulnerable people, such as older adults and impaired subjects. Within this perspective, this paper illustrates the experimental settings and protocol to study pedestrian and autonomous vehicle interaction, especially focusing on the safeness felt by each subject in different crossing conditions. To this end, besides traditional self assessment questionnaires and video recordings, movement and physiological data are collected as indicators of stress. From the analysis of this multimodal data, different classes of pedestrians could be defined, that will guide the definition of proper vehicle behaviour depending on their level of confidence and safety feeling. A preliminary data collection have been performed and is here described in a controlled urban-like crossing environment. Subjects of various ages were considered, as well as different dynamic behaviours of a properly prepared vehicle, running in both human-controlled and self-driving modes. Keywords vehicle pedestrian interaction, autonomous vehicle, physiological data, electromyography, photoplethys- mography, galvanic skin response 1. INTRODUCTION The vehicle-pedestrian interaction while crossing a road is a crucial aspect in the feeling of safe walking, and subjective emotions must be taken into account, considering different degrees of pedestrians vulnerability (age, gender, disabilities). In future scenarios, self-driving vehicles will circulate in urban environments, and will need to adapt to the feelings of pedestrians, being able to provide them effective feedback, properly tuned for the most vulnerable ones, such The 2nd workshop on social robots for personalized, continuous and adaptive assistance (ALTRUIST), December 16, 2022, Florence, Italy $ domenico.sorrenti@unimib.it (D. G. Sorrenti); francesca.gasparini@unimib.it (F. Gasparini); f.delia2@campus.unimib.it (F. D’Elia); a.ballini@campus.unimib.it (A. Ballini); simone.fontana@unimib.it (S. Fontana); f.dilauro2@campus.unimib.it (F. D. Lauro); alessandra.grossi@unimib.it (A. Grossi); s.dessena@campus.unimib.it (S. Dessena); stefania.bandini@unimib.it (S. Bandini)  0000-0002-6279-6660 (F. Gasparini); 0000-0001-7823-8973 (S. Fontana); 0000-0003-1308-8497 (A. Grossi); 0000-0002-7056-0543 (S. Bandini) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) as older adults and impaired people [1]. As introduced by Franzoni et al. in [2], the new era of information society has produced a significant revolution related to the creation of strong interactions between humans and machines. In particular, there is no field related to robotics and Artificial Intelligence that is not, directly or indirectly, related to the implementation of emo- tional values. Emotion recognition represents a fruitful research direction for the assessment of safe walking feeling, introducing quantitative evaluation tools for the measurement of affective walkability [3]. Several studies have examined the impact of autonomous vehicles’ behaviour on passengers [4, 5, 6, 7]. However, the effects on crossing pedestrians have been much less studied. We believe that studying the impact of autonomous vehicles on the emotional state of crossing pedestrians is fundamental to building better and more livable cities. In addition, we believe that communication between the vehicle and pedestrians is also fundamental to increasing the feeling of safety. In conventional vehicles, eye contact between pedestrians and drivers is one of the most important cues to convey a sense of safety to a crossing pedestrian. On the other hand, the introduction of autonomous vehicles brings new challenges in this area, as eye contact is obviously not possible and therefore other ways of communication have to be found [8]. Examples include LED strips used to signal attention to pedestrians [8, 9, 10]. Similar techniques have also been tested with non-autonomous vehicles to improve pedestrian awareness in poor lightning conditions [11]. In addition, Mahadevan et al. also experimented the use of led strips in conjunction with a display showing smiling faces and auditory cues to communicate pedestrian detection [12]. The acoustic cues could consist of human-like voices reproduced by the autonomous vehicle, as well as sounds played back by the pedestrian phone. Within this perspective, the aim of this study, is to investigate if it is possible to classify the pedestrians with respect to their level of confidence and safety feeling crossing the street, in particular in the presence of a self-driving vehicle. Different vehicle behaviours depending on the level of confidence and safety feeling of different classes of pedestrians, could then be properly defined. We believe that, in addition to auditory and visual cues, the behaviour of the vehicle, i.e., the way it decelerates, may play an important role in signaling pedestrian detection. For this reason, we also investigate whether this other form of communication, i.e., change in dynamics, can be established between the subject and the vehicle and whether it is effective in increasing the subject’s sense of safety. To this end, we here propose an experimental setting performed in a controlled urban-like cross- ing environment. The main research questions that guided the definition of the experimental protocol can be summarized as follows: • Q1: the safeness felt by the pedestrian while crossing a street decreases with respect to the perceived level of autonomy of the vehicle. • Q2: the safeness felt by the pedestrian while crossing a street increases as the safety gap increases (i.e., the distance between the pedestrian and where the vehicle ends its sharp deceleration); • Q3: the safeness felt by the pedestrian while crossing a street decreases with increasing age of the pedestrian. Video cameras and proper self assessment questionnaires are adopted to profile the subjects and assess their confidence with respect to self-driving vehicles. We rely on physiological responses to assess the subjective feeling of safe crossing. In our investigation we consider PhotoPlethysmoGraphy (PPG) that measures the blood volume registered just under the skin, which can be used to calculate the heart rate of the subject, and Galvanic Skin Response (GSR), that measures the skin sweat, as they are both effective to detect emotional arousal. Arousal is a physiological and psychological state that can be related to sensory alertness, mobility, and readiness to respond, activated as a defensive reaction to preserve safety. Moreover, motion data, measuring the muscle activity with Electromyography (EMG), are also collected, in an integrated approach to study pedestrian walkability. Relying on different signal sources that register both physiological and dynamic walking responses will provide accurate results for affective state recognition tasks. We have been encouraged to perform this research, by having obtained positive results in a previous experiment on the pedestrian interaction with traditional vehicles, whose aim was to collect movement and physiological data as reliable indicators of stress, during safe walking and road crossing [13]. The interaction between pedestrians and autonomous vehicles has been already considered in few literature works in which the perceived stress of pedestrians during crossing is detected using GSR signals [14, 15]. These analyses, however, use data collected in scenarios developed in immersive virtual reality setup, thus not representative of real crossing situations. 2. Experimental Design For the experiments, two distinct groups of subjects are taken into consideration. The first group, with an age between 18 and 35 years and the second group of over 65 years old. The inclusion criteria are: i) age in one of the classes mentioned; ii) absence of major medical disorders (neurological disorders, epilepsy, severe cognitive disorders); iii) no presence of phar- macotherapy that could interfere with the measured data (psychotropic drugs, anti-depressants); iv) no significant visual impairment (all with normal visual acuity or corrected to normal); v) no significant hearing impairment; and vi) autonomous mobility without the need for supports. The suitability of the participants is verified through a self-report questionnaire on personal medical history. Before participating in the study, each individual is informed by the investigator about the characteristics of the research, both verbally and through an information document. The participant signs an informed consent. Participation in the trial takes place following the voluntary participation of the subjects. The experiment is carried out in a private parking lot of the Università di Milano - Bicocca, during the weekend or at other non-working hours in a controlled urban crossing environment, as depicted in Figure 1. Three video cameras are used in order to record different points of view of the interaction between the subject and the vehicle. Video camera 1, on the top left of the scene depicted in Figure 1 records the subject approaching the zebra crossing to study her/his appraisal behaviour. Video camera 2 and video camera 3 (on the bottom right of the same Figure) record the crossing behaviour and the facial expressions of the subjects. A van is positioned a few meters before the crossing, in order to partially occlude the path of the vehicle, so that the subject can visually perceive the vehicle only when she/he is nearly arrived at the crossing. The participants wear wearable sensors that measure heart beat through Photopletysmography (PPG), Galvanic Skin response (GSR) and muscle activity using noninvasive Electromyography (EMG). The PPG and GSR sensors are placed on the fingers of the dominant hand, while the EMG sensors are placed on one leg. The sensors used are noninvasive Figure 1: The experimental setting in a private parking lot of the Università di Milano - Bicocca. and completely painless. The sensors used to collect physiological data are Shimmer3 GSR+ and Shimmer3 EMG/ECG [16]. Both these sensors interface with a software named ConsensysPRO, made by Shimmer as well, used to setup our trials, superimposing markers to raw data, and to partially pre-process collected data. To answer research questions 2 and 3 (related to the safeness feeling with respect to the perceived vehicle autonomy and with respect to varying safety gaps), the following conditions have been considered: • crossing the zebra with no vehicle interaction (C); • crossing the zebra interacting with the vehicle both in manually-driven (M) and self-driven (S) conditions; • self-driven conditions are implemented by putting a person on the passenger seat that is not paying attention to the road; • crossing the zebra with two different safety gaps (DS distance short, and DL distance long), adopted by the vehicle in both conditions (M or S). The experimental protocol also includes self-assessment questionnaires, for evaluating the self-esteem (SE) levels of the participants, filling the Rosenberg questionnaire (see [17]), the personality traits, filling a short version of the BIG 5 form, [18], and the level of safety felt after each crossing (SC). Before starting the experiment a questionnaire about the subject’s confidence with respect to both self and manually-driven vehicles is administered (IC). Moreover, at the end of the experiment, the subject has to answer if she/he had noticed the type of guide for every cross (manual or autonomous) (ATT ). The whole protocol is described as follows: • Questionnaires filling: SE, BIG 5, and IC. • Baseline: 2 minutes session to acquire the reference physiological signals, where the subject has to stay straight up and still, to record her/his physiological responses in absence of any tasks. • Experiment Core: 6 repetitions of C, 2 repetitions of M DL, 2 repetitions of M DS, 2 repetitions of S DL, 2 repetitions of S DS, 6 repetition of a 60 seconds baseline recording, also intended to bring the subject back to a neutral state before the next task. Each task is followed with the crossing questionnaire filling (SC). Within the experiment core, the order of the tasks is randomly selected for each subject, in order to avoid possible biases introduced by the experimental setting. • Questionnaire filling: ATT. The experiment lasts about 40 minutes, 20 of which are dedicated to self-assessment of the questionnaires, long enough to collect usable data and short enough to prevent subjects from getting used to the task at hand. 3. The self-driving vehicle For the experiment, we used the prototype of autonomous vehicle shown in Figure 2. It is equipped with two front-mounted single-plane LiDARs, to sense the environment and detect obstacles (including pedestrians), and one front camera, used by the operator at the passenger seat. The hardware and software architecture of the vehicle base was developed to ensure safe and reliable experiments. A custom relay board was developed for switching between manual control and self-driving mode. A DC motor was mounted on the steering wheel to control the self-driving car. In addition, the throttle is controlled by a digital potentiometer via the vehicle’s electronic control unit. An absolute encoder was mounted under the steering mechanism to sense the steering angle, and two encoders were mounted on the rear wheels to measure the vehicle’s speed and enable closed-loop control. The digital potentiometer and steering wheel are controlled by an STM32 microcontroller. The linear and angular velocity commands are sent from the main computer to a Raspberry Pi, which relays the messages to the microcontroller. Figure 2: The vehicle approaching the crosswalk. In this picture, the vehicle is in autonomous mode, with an operator on the passenger seat, wearing a hat occluding eye-contact with the pedestrian. Nobody is sitting at the steering wheel. The safety of the experiment is also ensured by several watchdogs. On the Raspberry Pi, a software watchdog was implemented to send a stop command to the STM32 if no valid speed command was received within a certain time window. The same safety measure was implemented on the STM32 using a hardware watchdog that is updated every time a valid control command is received. If no valid command is received in time, the hardware watchdog runs down and stops both the main motor and the steering wheel. As an additional safety measure, several emergency stop buttons have been installed on the vehicle: if something unexpected happens, the operator can immediately interrupt the autonomous control loop and slow down the vehicle. The entire vehicle infrastructure is shown in Figure 3. STM32 mcu Absolute encoder Photon Focus Ethernet front camera Switch Steering wheel motor H bridge Custom Left wheel relay board w/ rotary encoder Raspberry Pi LMS-111 SICK LMS-111 SICK Main motor LiDAR LiDAR Electronic Control Unit Right wheel Emergency Stop Main Computer w/ rotary encoder Figure 3: Vehicle infrastructure including the control system and sensors To ensure the repeatability of the experiments, we opted for map-based localization and navigation. The grid-based map is generated offline using pre-recorded data with GMapping [19]. For the localization of the vehicle, we use a scan matching-based localization algorithm [20], which estimates the current pose of the vehicle by comparing the map with laser scan data acquired with the LiDAR. The various components of the software architecture use the ROS framework for commu- nication [21]. The typical approach to robot navigation using ROS involves what is called a global planner that generates the best path to a destination. However, to ensure that the vehicle always follows the same trajectory, we developed a custom planner that always sends the same straight path to the lower layers, along with the velocity profile we described earlier. As a safety measure, we developed a simple obstacle detection component that stops the vehicle when an obstacle is detected within a range of 1 𝑚. This distance may seem very short, but the vehicle approaches the crosswalk at walking speed (0.6 𝑚/𝑠) after full braking. Therefore, the stopping distance is also very short. (a) (b) Figure 4: Histograms of subject’s confidence with respect to both self and manually-driven vehicles collected using the IC questionnaire. The left histogram (a) refers to young adults, (b) refers to older adults. 4. Preliminary and future analysis The experiment here described is still ongoing. Data so far collected comes from a population of young adults, 10 male and 4 female, with average age of 23 (standard deviation = 4.1), collected from Bachelor, Master and PhD students, and from 5 older adults with mean age of 65.6 (standard deviation = 1.34), including 3 female and 2 male subjects, for a total of 19 subjects. Most of the data has not yet been analyzed, so here we report few of the preliminary analysis and our proposal of future works. Among all the questionnaires administered, we report here the results of the IC questionnaire, acquired before the execution of the crossings. With this questionnaire we want to investigate the initial subject’s confidence with respect to both self and manually-driven vehicles. The two questions on a 5 point Likert scale are: • Thinking of traditional vehicles, when crossing a road without traffic lights, how much trust do you have in drivers? 1: no trust; 2: limited trust; 3: trust; 4: high trust; 5 total trust; • Thinking about self-driving vehicles, if you were to cross a street without traffic lights, how much confidence would you have in the vehicle? 1: no trust; 2: limited trust; 3: trust; 4: high trust; 5 total trust The histograms of Figure 4 summarize the results collected from the questionnaire according to the population considered (young or older adults) as well as the type of driven condition (manually or self). The following comments could be drawn starting from the values collected: • The results obtained applying the non-parametric Wilcoxon Rank Sum Test showed that there are no significant differences in the subject’s confidence level when self and manually-driven vehicles are taken into account (p-value = 0.86, alpha = 0.05). In both cases, the average of the collected values is in the range [2, 3], proving a medium-low confidence of the pedestrians towards any type of vehicle (self or manually driven). • For each participant, the confidence level reported in case of crossing with self and manually-driven vehicles are usually similar and near to the central value (3) of the confidence range. Only two subjects reported opposite answers in the two questions: a young adult that indicated a high confidence in presence of traditional vehicle (4) and limited confidence in case of self-driving vehicle (2), and an older person (the oldest one, 68 years old) that, instead, reported an higher trust in self-driving vehicles (3) then in traditional ones (1). • In general, young adults seems having a higher confidence with vehicle (overall average value = 2.57) with respect to older people (overall average value = 2.2). These preliminary observations are quite interesting but, at the same time, require a higher number of subjects responses. In future studies, the raw physiological signals collected during the experiment will be pre- processed and normalized to reduce noise and moderate subjects dependencies. From each of the four normalized signals, a proper set of features will be evaluated as characteristics useful to describe them. From PPG signals, handcrafted features will be considered including temporal features and frequency domain features. Concerning the GSR, statistical and peak related features extracted from phasic component will be examined, as well as the regression coefficient feature obtained from the tonic part of the signal. Finally, two features will be computed from EMG signals: the Root Mean Square [22] and the walking frequency, known as Stride Frequency, evaluated in terms of number of steps per second [23]. To verify the research hypotheses, statistical analyzes will be carried out to compare feature distributions of different crossing conditions, comparing the two groups of populations. In particular, the analyses will focus on identifying if there are significant differences in subjects’ physiological signals during crossing in presence or absence of vehicles. Furthermore, statistical test could also be used to evaluate variations in people’s behaviour and physiological parameters according to the perceived level of autonomy of the vehicle and on different safety gaps, collected by the self assessment questionnaires. Finally, tests will be performed to verify if there are statistically significant differences between the two populations considered during the same crossing conditions. Furthermore the analysis of the videos recorded by the three cameras that acquire different points of view of the interaction between the vehicle and the participant could provide behaviour understanding especially in the appraisal phase, and during the crossing, also in terms of emotion recognized from the facial expressions. 5. Conclusions The experiment here described aims at defining perceivable classes of pedestrians with respect to their level of confidence and safety feeling, while crossing a street in presence of a self-driven vehicle. A proper vehicle dynamic behaviour can thus be defined depending on the perceived class of pedestrians, to increase his/her sense of safety. The adaptation of the safety gap to the class of pedestrian is technically possible, as it requires the sensing suite of the vehicle to be able to classify the pedestrians nearby the crossing, and then the corresponding dynamic behaviour is executed. This paper reports the details of the experimental protocol, which makes a useful tool for the repeatability of the experiments. Some preliminary analysis is also reported. Acknowledgments This research is partially supported by the FONDAZIONE CARIPLO “LONGEVICITY-Social Inclusion for the Elderly through Walkability” (Ref. 2017-0938). We also thanks the many students that helped develop the vehicle in the previous years: P. Colombo, D. Gerosa, J. Maltagliati, M. Pugno, I. Rizzi, and others. References [1] A. Rasouli, J. K. Tsotsos, Autonomous vehicles that interact with pedestrians: A survey of theory and practice, IEEE Transactions on Intelligent Transportation Systems 21 (2019) 900–918. [2] V. Franzoni, A. Milani, D. Nardi, et al., Emotional machines: The next revolution, in: Web Intelligence, volume 17, IOS Press, 2019, pp. 1–7. [3] S. Bandini, F. Gasparini, Towards affective walkability for healthy ageing in the future of the cities., in: AI* AAL@ AI* IA, 2019, pp. 1–9. [4] N. Dillen, M. Ilievski, E. Law, L. E. Nacke, K. Czarnecki, O. Schneider, Keep calm and ride along: Passenger comfort and anxiety as physiological responses to autonomous driving styles, in: Proceedings of the 2020 CHI conference on human factors in computing systems, 2020, pp. 1–13. [5] M. Elbanhawi, M. Simic, R. Jazar, In the passenger seat: investigating ride comfort measures in autonomous cars, IEEE Intelligent transportation systems magazine 7 (2015) 4–17. [6] N. Mohajer, S. Nahavandi, H. Abdi, Z. Najdovski, Enhancing passenger comfort in au- tonomous vehicles through vehicle handling analysis and optimization, IEEE Intelligent Transportation Systems Magazine 13 (2020) 156–173. [7] P. Launonen, A. O. Salonen, H. Liimatainen, Icy roads and urban environments. passenger experiences in autonomous vehicles in finland, Transportation research part F: traffic psychology and behaviour 80 (2021) 34–48. [8] V. M. Lundgren, A. Habibovic, J. Andersson, T. Lagström, M. Nilsson, A. Sirkka, J. Fagerlönn, R. Fredriksson, C. Edgren, S. Krupenia, et al., Will there be new communication needs when introducing automated vehicles to the urban context?, in: Advances in human aspects of transportation, 2017, pp. 485–497. [9] E. Florentine, M. A. Ang, S. D. Pendleton, H. Andersen, M. H. Ang, Pedestrian notification methods in autonomous vehicles for multi-class mobility-on-demand service, in: Pro- ceedings of the Fourth International Conference on Human Agent Interaction, 2016, p. 387–392. [10] K. Wang, G. Li, J. Chen, Y. Long, T. Chen, L. Chen, Q. Xia, The adaptability and challenges of autonomous vehicles to pedestrians in urban china, Accident Analysis & Prevention 145 (2020). [11] V. Balasubramanian, R. Bhardwaj, Pedestrians’ perception and response towards vehicles during road-crossing at nighttime, Accident Analysis & Prevention 110 (2018) 128–135. [12] K. Mahadevan, S. Somanath, E. Sharlin, Communicating awareness and intent in au- tonomous vehicle-pedestrian interaction, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–12. [13] F. Gasparini, M. Giltri, S. Bandini, Experimental approach to study pedestrian dynamics towards affective agents modeling., in: ATT@ ECAI, 2020, pp. 31–38. [14] M. Mudassar, A. Kalatian, B. Farooq, Analysis of pedestrian stress level using gsr sensor in virtual immersive reality, arXiv preprint arXiv:2111.11492 (2021). [15] K. Kamal, B. Farooq, M. Mudassar, A. Kalatian, Ordered-logit pedestrian stress model for traffic flow with automated vehicles, arXiv preprint arXiv:2204.11367 (2022). [16] A. Burns, E. P. Doheny, B. R. Greene, T. Foran, D. Leahy, K. O’Donovan, M. J. McGrath, Shimmer™: an extensible platform for physiological signal capture, in: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, 2010, pp. 3759–3762. [17] M. Rosenberg, The association between self-esteem and anxiety., Journal of Psychiatric Research (1962). [18] C. Chiorri, F. Bracco, T. Piccinno, C. Modafferi, V. Battini, Psychometric properties of a revised version of the ten item personality inventory., European Journal of Psychological Assessment 31 (2015). [19] G. Grisetti, C. Stachniss, W. Burgard, Improved techniques for grid mapping with rao- blackwellized particle filters, IEEE transactions on Robotics 23 (2007) 34–46. [20] E. Pedrosa, A. Pereira, N. Lau, Efficient localization based on scan matching with a continuous likelihood field, in: 2017 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), 2017, pp. 61–66. [21] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y. Ng, Ros: an open-source robot operating system, in: ICRA workshop on open source software, volume 3, Kobe, Japan, 2009. [22] A. Phinyomark, C. Limsakul, P. Phukpattaranont, A novel feature extraction for robust emg pattern recognition, arXiv preprint arXiv:0912.3973 (2009). [23] F. Gasparini, A. Grossi, K. Nishinari, S. Bandini, Age-related walkability assessment: A preliminary study based on the emg, in: International Conference of the Italian Association for Artificial Intelligence, Springer, 2020, pp. 423–438.