The modeling of a new generation of agent-based simulation systems supporting pedestrian and crowd management taking into account affective states represents a new research frontier. As in the case of any study of pedestrian dynamics, adding an affective component implies the rigorous design of experimental protocols and data acquisition sets. The integration of multi-modal signal sources considering both data coming from physical activity and uncontrolled reactions related to affective responses provides new perspectives to study pedestrian dynamics and pedestrian interaction with traditional vehicles as well as with autonomic and autonomous transportation systems. The designed in-vivo experimental protocol devoted to the collection of movement and physiological data as reliable stress indicators during walking and road crossing, and the related analysis will be illustrated.
According to the World Health Organization (WHO) [
In order to protect especially these frail categories, it is important to properly study road safety in order to analyze the pedestrianvehicle interaction: profiling the attitude that people engage in while being a pedestrian is important to include a more realistic behaviour for the agents in simulation models.
The development of models for intelligent agents that are able
to incorporate, use and express affects into reasoning and
interaction processes or supporting decision making activities is becoming
a novel research area in the field of agent-based simulation [
Numerous studies have been developed during recent years in
order to investigate different aspects of the pedestrian behaviour,
focusing on non-signalized pedestrian crossings [
As it is highlighted in [
In the case of agent-based and crowd and pedestrian dynamics
simulations, the modeling of a new generation of systems,
supporting crowd management that takes into account affective states,
represents a new research frontier, involving also many human
disciplines [
New approaches of Artificial Intelligence that rely on
affective computing are becoming crucial to design new generations of
computer-based systems supporting the creation of services for the
future cities [
Nowadays, the significant improvement of sensor technology and
the progressive lowering of sensors costs allow their adoption in
many new experimental scenarios, measuring inertial data and
physiological signals during, for example, daily life activities [
The integration of multi-modal signal sources considering both data coming from physical activity and uncontrolled reactions related to affective responses provides new perspectives to study pedestrian dynamics and pedestrian interaction with traditional vehicles as well as with autonomic and autonomous transportation systems.
In order to incorporate affective parameters in the development of
agent-based models, a formal design of experimental protocols and
sets is crucial both for assessing the validity of the model, and
facing data and approaches coming from the related scientific
community [
Within this framework, in this paper we illustrate the in-vivo experimental protocol designed to perform the collection of movement and physiological data during walking and road crossing, the performed experimental sessions and some first analyses on the collected data, done in order to detect meaningful patterns referring to the level of stress of subjects during walking or road crossing.
The structure of the paper is the following. In Section 2 the in-vivo experiment carried out in an uncontrolled outdoor environment is described. Signal processing on physiological data, in particular to remove noise and normalize the responses of the subjects, is described in section 3. This step, together with proper feature extraction, is required to analyze the data in an intra and inter subjects comparison with the aim of finding characteristic patterns corresponding to different affective states. In Section 4 the subjective responses to the self assessment questionnaires as well as statistical inferences from physiological data are presented. Finally in the Conclusion final remarks and future developments are drawn. 2
In order to focus on the pedestrians’ perception of safe road crossing and walking, an experiment in an uncontrolled urban scenario has been carried out. To this end, a two way road in correspondence to a crossroad, without traffic lights, has been considered. In figure 1 this experimental environment is depicted. The zebra crossing where the experiment was conducted is highlighted with a red rectangle.
This crossing is considered moderately dangerous for the pedestrians for the following reasons:
There are no traffic lights to control the traffic flow for both cars and pedestrians.
There are parking lots surrounding the crosswalk, thus limiting the view of the pedestrians.
A lot of different vehicles travel along this road, ranging from bicycles to cars to trucks and buses.
Thus, wanting to test the subjects conditions while traversing a stressing crosswalk, the one located at this crossroads presented some difficulties that could effectively elicit a stressful affective state. The only indication the subjects were given was to try and cross the road when cars were approaching the intersection, in order to make the experience more realistic. 2.1
The subjects involved in this study were chosen from a same social and age group. A total of 14 participants were engaged, 7 males and 7 females, aged between 20 and 26 years (mean = 24.42, standard deviation = 1,65), and they were all students enrolled in one of the scientific faculties at the University of Milano-Bicocca. Because of their attendance on campus, those students were familiar to the chosen intersection, especially since most of them usually crossed the street in that same location in order to reach the Department of Informatics, Systems and Communication.
The experimental procedure had been explained in all of its parts to the participants, in order to let them know what their tasks consisted of. All of the subjects in this experiments were volunteers who provided informed consent. This study has been approved by the Ethical Committee of the University of Milano-Bicocca. 2.2
For this experimentation, the chosen sensors aimed at recording the
physiological responses of the participants, focusing in particular
on three different signals: the Galvanic Skin Response (GSR), also
known as Skin Conductance (SC), which is connected to
sweating and perspiration on the skin and is a reliable stress indicator;
the Plethysmography (PPG), that measures the blood volume
registered just under the skin, which can be used to obtain the heart
rate of the subject; the Electromyography (EMG), measured as
surface electromyography, which measures the muscle activity of the
person. In order to properly record these three signals, two
different sensors have been adopted, from the Irish company Shimmer
(www.shimmersensing.com). These low-cost wearable sensors were
already utilized in different experiments concerning physiological
signals analysis and affective state recognition with encouraging
results [
Human affective states are influenced not only by the
environmental stimuli, but also by several subjective characteristics. In
particular personality traits strongly condition the affective responses [
10. At times I think I am no good at all.
Custom questionnaire about the crossing task: this questionnaire was used to collect subjective perception about the crossing task, such as the stress level of the participant, his/her confidence in drivers, disturbing elements etc. The participant needs to classify every item of this survey as NULL, LOW or HIGH. The items of this questionnaire are the following: 1. Stress level during the crossing. 2. Confidence level towards the cars during the crossing. 3. Interference level brought by other means of transportation during the crossing.
5. Confidence level in the crossing without traffic control or traffic lights. 6. Confidence level in the crossing with disturbing elements (parked cars, partially blocked view...) 2.4 The experimental protocol consists of different parts that include the questionnaire filling, the crossing task and some baseline recordings which could have helped in the data analysis at the end of the experiment. Furthermore, the whole experiment has been video recorded.
After reaching the chosen crossroads, the participants were instructed on the following procedure:
Experiment Core: repeated 4 times – Walking on sidewalk (non-stressing task), as depicted in figure 3. – 30 seconds baseline recording, where the subject had to stay straight up and still to record his/her physiological response in absence of tasks. – Crossing the road and coming back at the start point (stressing task), as depicted in figure 4. 4 – 30 seconds baseline, same as before, also intended to bring the subject back to a neutral state before the next crossing. – Crossing questionnaire filling.
During this in-vivo data acquisition, a problematic emerged: because of the very low temperatures registered during the trial of three 4 In order to better understand the participant’s behaviour, this task was also filmed with a full HD camera. Every participant has consented the recording of their crossings. of the participants, the GSR+ sensor had some difficulties recording the GSR and PPG signals, thus rendering those three recordings unusable for our analysis. This likely happened because the GSR+ sensor has an optimal temperature range between 20 °-28 °C in order to function properly, while the registered temperatures during those days were around 8 °-10 °C. The whole experiment has been video recorded. 3
Before passing on to evaluate the stressful state deriving from crossing the street, a preliminary analysis on the recorded physiological responses is required. The raw signals obtained during the experimentation needed to be pre-processed and cleaned, and proper features needed to be extracted from the signals before performing the analysis, since the original recordings may contain noise and artifacts that can throw off the results.
The recorded signals were sampled with a frequency of 128Hz for the GSR and the PPG, and of 512Hz for the EMG. For the GSR and the PPG filtering step, we used a zero-phase filter in order to properly remove the noise and the possible high-frequency artifacts that we could expect, while for the EMG we decided to use a zerolag Butterworth bandpass filter with a cut-off frequency of 20Hz. In figures 5 and 6 two examples of unfiltered and filtered signals for both GSR and EMG are reported.
The filtered signals were normalized with a z-score function in order to have all of the signals to confront in a same reference range, and then were split into different segments following the markers directions. This way, for every participant, we obtained a total of 22 segments:
After this step we then proceeded to display all of the signals overlapping with the markers we activated during the experiment in order to properly highlight the different tasks, obtaining for everyone of the remaining 11 participants a graphic similar to the one displayed in figure 7.
With such a visualization, every task of the experiment can be easily distinguished. The different event markers were created ad hoc beforehand and were differentiated using different heights, and these are the experimental phases corresponding to those different sizes:
Y=2: Questionnaire period (Q) Y=4: Walking period (W) Y=6: Baseline period (B) Y=8: Crossing period (C) 3.1
We calculated a total of 13 features from the acquired data. Table 1 shows all of these features and for what signals we computed them in order to perform the following analysis 5.
In this section the subjective responses to the self assessment questionnaires as well as statistical inferences from physiological data are presented.
The only thing that needs to be addressed is that, in order to correctly compute the features for the GSR, we firstly had to separate the two different components of this signal: the Skin Conductance Level (SCL) and the Skin Conductance Response (SCR). The SCL comprises all of the low frequencies of the GSR signal, thus giving the general trend of the signal, while the SCR includes all of the high frequencies and shows clearly all of the peaks that can be categorized as ”natural peaks” or ”elicitation peaks” (that are more relevant in our analysis since they highlight the person’s response to external events and elicitations). In order to do this, we derived the SCL by using a low-pass filter at 0.05Hz, obtaining the tonic part of the GSR, and the SCR was derived using a high-pass filter with the same frequency, thus generating the phasic part.
All of the GSR features were calculated from the phasic part of the various GSR signals whit the exception of the Regression Coefficient, which was obtained from the tonic part since it contained the necessary information about the signal slope. 5 IBI is the Inter-Beat Interval feature, while RMSSD is the Root Mean Square of the Successive Differences feature 4.1
Looking at the results obtained from the Rosenberg’s Self-Esteem Scale, what we understood about our subject sampling was that all of the participants had a very good self-perception, tending to approach in a serene way new tasks given them. This was to be expected since, as we said before, the subjects we took into consideration were young students in good health. This analysis also confirm the homogeneity of the population considered in the experiment reducing the variables to be considered. One of the first analysis performed on the obtained features was a Kruskal-Wallis test, in order to understand if the physiological feature distributions coming from different tasks recorded during our experimentation (baseline, walking and crossing) were statistically different, thus corroborating the hypothesis that the physiological response of a subject can differentiate between different states of being of the person.
The Kruskal-Wallis test provides a null hypothesis, for which two distribution provided as input are similar enough to be considered as coming from the same initial distribution. If the returned result of the test, the p-value, is lower than a certain significance level (that for us was fixed as = 0:05), the null hypothesis is rejected and the two input distributions are deemed as statistically different and thus diversifiable. Needless to say, our goal was to obtain low p-values in order to confirm that physiological features differed while in different states.
The first test we performed was about comparing the features distributions of the Walking tasks with the ones from the Crossing tasks, and table 2 shows the obtained results.
The green values highlighted in the table are the ones that were lower than the significance level we put. In this case, we can see how almost half of the performed tests comparing feature distributions from different activities were found to be genuinely diverse, and this kind of results corroborate our hypothesis.
The same, if not better, response is also achieved from the comparison of Crossing and Baseline and of Walking and Baseline, whose Kruskal-Wallis test results are reported in tables 3 and 4.
Another thing that emerges from the analysis of the above mentioned tables is that the PPG and the EMG signals does not seem to be really correlated to an affective state (stress or non-stress) but to be more connected to movement in general: comparing table 2 with tables 3 and 4 it is clear that the distributions coming from walking and crossing tasks for these two signals, the PPG in particular, seem more similar (thus not passing the KW test) than in the other two cases.
After this signal analysis, we decided also to perform a sample checking analysis in order to better understand what impression of the crossing task the participants had. Therefore, the first thing we did was gather all of the custom questionnaire answers for all of the subjects and all of their crossing tasks, thus obtaining a total of 56 answers to every question we created. From this set we then computed the percentages of NULL, LOW and HIGH answers given by the participants, obtaining the graphic that can be seen in figure 10.
As we can see, the majority of the crossings delivered low to null stress to the subjects, and only a few high stress levels were reported through the custom questionnaires after the task. This data is not unexpected since, as we previously highlighted, the subject sample for this experiment was narrowed down to healthy and young students who are also accustomed to crossing this particular intersection while walking through the university campus.
Figure 11, on the other hand, shows the correlation matrix obtained by checking the relations between the answers, a test performed using Pearson correlation index. From left to right, and from low to high, we have these categories: Stress Level, Confidence (Vehicles), Interference (Other Vehicles), Interference (Other Pedestrians), Confidence (Crossing without Controls), Confidence (Crossing with Disturbances). We can see how the highest Pearson correlation coefficient (0.4574) is between Confidence (Crossing without Controls) and Confidence (Crossing with Disturbances): this can mean that many participants were less confident in crossing the street for both these factors. The lowest Pearson correlation coefficient (0.4089), on the other hand, is between Stress Level and Confidence (Crossing without Controls).
Even if from the self assessment questionnaires emerges that the subjects involved in the experiment were not particularly stressed by the crossing tasks, the physiological data clearly shows different patterns with respect to the different activities as well as differences in the feature distributions that are statistically significant. These considerations are important hints towards the adoptions of physiological signals as indicators of uncontrolled affective reactions of subjects in the pedestrians-vehicles interaction. 5
The paper illustrated an in-vivo experiment to evaluate the
pedestrian-vehicle interactions from an affective point of view.
Collected physiological data has shown to be reliable indicators of
variations of affective states during walking and road crossing. The
results of this research will drive the design of agent-based models for
pedestrian dynamics simulation, taking in account the representation
of affective states, namely, stress during road crossing. Moreover,
parallel experiments conducted in in-vitro environments (as
illustrated in[
It will also be important to explore in follow-up experiments how
Answer percentages for every evaluation category obtained for the custom questionnaires about the crossing experience of our subjects. affective states and quantitative measures can be correlated. This kind of connection between a pedestrian’s mood and his reaction times, speed and direction may bring great value to agent simulators, being especially useful to help calibrating them for different types of pedestrians. Moreover as the whole experiment has been video recorded, the analysis of these video will be helpful to further analysed pedestrian behaviour, and related physiological responses in order to integrate our findings in pedestrian dynamic modelling.
This analysis will be object of our future works.
This research is partially supported by the FONDAZIONE CARIPLO “LONGEVICITY-Social Inclusion for a Elderly through Walkability” (Ref. 2017-0938) and by the Japan Society for the Promotion of Science (Ref. L19513). We want to give our thanks to Maria Elena Manisera and Gianluca Toffanin, for their supporting work during the experimentation and data analysis.