Evaluating a custom-made agent-based driving simulator Andreas Gregoriades1 Maria Pampaka2 1 Cyprus University of Technology, Limassol, Cyprus 2 The University of Manchester, Manchester, UK andreas.gregoriades@cut.ac.cy, maria.pampaka@manchester.ac.uk Abstract non-performance errors that accounts for about 7% of the crashes. These categories however, are highly interrelated. We present the design and evaluation of a custom- For instance, overloading will affect decision and recognition made driving simulator, which was conducted that could lead to a crash. through an experiment with users. Objective and self-reported measures of driving behaviour are Driving style is directly linked to accidents. Many studies used to validate the simulator. Objective data in- analyse driving style, and particularly aggressive driving, clude situation awareness and workload measures, since this is highly related to crashes. Evaluating the effect of quantified with SAGAT and physiological esti- different driving styles on accidents can be performed in mates, while self-reported data focused on driving different ways, one of which is through driving simulation. behaviour perceptions from a standardised driving Alternatively surveys such as the Manchester Driving Style style questionnaire. To evaluate the simulator, we questionnaire [Reason et al., 1990] can be used. Simulation firstly check that the synthetic environment does not can be described as a method of reproducing a situation simi- overload the participants and enable them to have a lar to reality. To test driving style, it is necessary to design an sufficient level of situation awareness. Secondly, a environment with identical stimuli to a real situation. In the correlation analysis is conducted between observed field of driving, simulations are used to generate situations and self-reported driving style to examine the level that produce the same response to participants as real-life of their covariance and similarity. Results showed driving, without having the risks of injury. Driving simula- that participants exhibited a similar driving behav- tors have been developed ranging from small-scale such as iour as that reported with self-reports. This indicates the NADS miniSim [FHA, 2013] to large scale models such that the simulator provides realistic driving condi- as the Daimler-Benz [Kading, 1995]. The advantages of us- tions that encourage participants to behave in a real- ing simulators as a research tool include the design of exper- istic way. iments that are easily replicated and the dynamic collection of relevant drivers’ variables for safety analysis. Driving simulators are designed for specific purposes and hence re- 1. Introduction quire validation in order to produce correct results. However Driver related factors, according to the literature [Evans, due to the inherent complexity of such systems the prediction 1991; NHTSA, 2015a] constitute the main cause of accident of travellers and drivers’ behaviour is becoming increasingly in three out of five crashes, while they contribute to the oc- harder. For these reasons, agent-based simulation, which currence of 95% of all accidents. The National Highway adopts an individual-centered approach, is one of the most Traffic Safety Administration -NHTSA [NHTSA, 2015a], relevant paradigms to design and implement such applica- classified driver-related accident causes into recognition er- tions [Mastio et al., 2018]. rors, decision errors, performance errors, and non- performance errors [Reason et al., 1990]. The most frequent Human driver behaviour modelling has been the subject of of these errors (more than 40%) are recognition errors which many studies. The car-following model [Reuschel, 1950] has include driver’s inattention, internal and external distrac- been used to describe driver behaviour at the micro level and tions, and inadequate surveillance. Decision errors, such as is based on control theory (predictive control, optimal con- driving too fast under certain conditions, too fast for given trol, etc.). The model expresses how vehicles follow one curves, false assumption of others’ actions, illegal manoeu- another on a roadway, the minimum space and time gap be- vres and misjudgement of gap between other vehicles or oth- tween them and the behaviour of the driver with regards to ers’ speed, account for more than 30% of accidents keeping a “safe distance” from the leading vehicle, driving at [NHTSA, 2015b]. Performance related errors such as over- a desired speed, or choosing acceleration pattern to maintain loading, poor directional control, etc., account for 11% of the a comfortable range from the vehicle in front. The aim is to crashes. Sleep is the most common critical reason among mimic different driving styles, such as: Aggressive driving, that has high accelerations and deceleration patterns with iour describes driving habits that define the way a driver almost no anticipation, eco driving style, where sufficient chooses to drive [Lajunen et al., 2011]. The Driver Behav- anticipation is evident to avoid unnecessary acceleration and iour Questionnaire (DBQ) [Reason et al., 1990] is one of the braking, and normal driving, an intermediate driving style most widely used instruments for measuring driving style. between the above two styles. Alternative driver behaviour According to Reason et al. [1990] driving errors and viola- modelling methods include neural networks, and fuzzy logic. tions are two different groups of behaviour, which is overall The latter models human perceptions by fuzzy sets and fuzzy categorised into violations, errors, slips and lapses. Viola- mathematics combined with knowledge-based logic. These tions are deliberate deviations from practices believed neces- methods are often used to mimic processes that have compli- sary to maintain safe operation of a potentially hazardous cated mathematical models. Fuzzy logic has gained attention system, while errors are defined as the failure of planned in modelling tactical driver behaviour [Khaisongkram et al., actions to achieve intended outcomes. The research instru- 2010], however, it requires a large number of experimental ment DBQ developed by Reason et al. [1990] considers this data. classification. Slips and lapses refer to attention and memory failures such as: attempt to drive away from the traffic lights The use of driving simulation in driver behaviour analysis is in wrong gear, forgetting where you park the car etc. Viola- considered essential due to the difficulty in eliminating con- tions are more serious and include close following vehicle founding effects on control measures in field experiments. ahead (tailgating), speeding, risky overtaking etc. Errors re- However, unrealistic simulation conditions may affect the fer to behaviours such as failing to notice pedestrians- driving behaviour of participants in experiments which could crossing, missing Give Way signs etc. A further classifica- influence the validity of any study. Commercial driving sim- tion [Lawton et al., 1997] divides violations into aggressive ulators, on the other hand do not provide the required level of violations, and ordinary violations, which are deliberate de- customisability necessary for researchers to design experi- viations without aggressive behaviour. ments. Therefore, most experiments are designed using cus- tom made simulators. The main limitation of driving simula- An important skill that affects driver safety is anticipation of tion studies is that by removing the risk of harm to partici- events. Experienced drivers can predict the traffic situation, pants, their driving behaviour may be altered. Therefore, the hence are ready when a hazardous event occurs. This ability conclusions made could be inaccurate. This paper contributes is referred to as driver’s situation awareness. Gaining situa- in resolving this problem by addressing the following re- tion awareness involves perception and pattern recognition, search questions: (1) Does self-report driving behaviour of attention and comprehension, and decision-making [Ensley, participants differ from the observed objective driving be- 2012]. Hence, drivers identify, process, and comprehend the haviour in the simulator? The assumption here is that drivers critical information cues from the environment to predict should demonstrate similar driving style as their self-report how future events could unfold. Drivers’ decision-making in the driving behaviour questionnaire. (2) Does the simula- process is not only based on the current environmental state, tor enable the drivers to have sufficient situation awareness? but also extrapolates the current situation to future projec- The assumption is that a realistic driving simulator should tions. Well aware drivers analyse the current state of their enable drivers to have a minimum situation awareness (SA), environment using multiple information sources, then predict which is the capability of understanding what is going on the next states. Situation awareness is an important feature in around them and make decisions accordingly. (3) Is aggres- driver safety. In normal condition an average driver has a sive driving associated with situation awareness? (4) Does minimum level of situation awareness, which is required in the simulation environment overload the drivers? This exam- order to navigate the vehicle. This driver property can be ines if the workload of participants is within the accepted used as an indicator of the quality and realism of a driving levels. An indication of overloading in a normal driving sce- simulator, assuming that an unrealistic simulator will not nario could indicate a problem with the realism of the simu- enable drivers to maintain minimum situation awareness. In lator. (5) Do imprudent drivers consume more cognitive re- this study, self-reports of driver style gives an indication of sources than prudent drivers? capability to maintain sufficient situation awareness, along The paper is organised as follows. The next two sections with driver behaviour. Therefore, a driver that reports in describe the literature relating to driver behaviour, SA, work- DBQ that he/she is making a few errors and lapses is ex- load and driver simulation. The next section describes the pected to demonstrate sufficient situation awareness in the process of designing a custom made driving simulator, fol- simulator. lowed by a section that addresses its validation process. The paper concludes with the main results of this study. Amongst the various methods for assessing drivers’ situation awareness, this study employs the Situation Awareness 2. Driving behaviour, Situation Awareness and Global Assessment Technique SAGAT [Endsley, 2004; workload Endsley & Jones, 2012], which is a dynamic query technique Driving performance is associated with driving skills that are that questions participants’ recent memory of the situation manifested on driver behaviour which, in turn, affects driv- by freezing the simulation and hiding all information. ing style. Driving skills include information processing and motor skills, which improve with experience. Driving behav- Measuring driver workload is of great significance for im- driver behaviour research, prototype intelligent transporta- proving the understanding of driver behaviours and support- tion systems validation and training. ing the development of driver assistance systems. Workload expresses the demands placed on the driver from secondary There are different categories of driving simulations. Micro- tasks that could potentially interfere with the primary driving simulation is widely considered as a method to study drivers’ task. Workload is defined as the competition in driver re- behaviors, as in the example of parking choice simulators sources (perceptual, cognitive, or physical) between the driv- PARKIT and PARKAGENT [Bonsall and Palmer, 2004]. ing task and a concurrent secondary task, occurring over that Among micro-simulation programs, multi agent-based mod- task’s duration. Driving tasks for instance, require physical elling simulation environments, such as NetLogo [Sklar, and cognitive resources that are dynamically varied under 2007] and Archisim [Doniec et al., 2008], allow researchers different driving conditions. to investigate the connection between micro-level behaviors of individuals, and macro-level patterns coming from their There are three main methods to measure cognitive work- interactions. Intelligent agents in multi agent systems per- load: subjective, performance-based, and physiological. Sub- form three functions: they perceive the dynamic conditions jective knowledge acquisition techniques such as surveys, from their environment, they perform actions, and reason to interviews, and observations are commonly used to assess interpret perceptions, solve problems, draw inferences, and cognition workload during tasks [Lehto et al., 1992]. Per- determine best course of actions. formance based measures are usually classified as either primary task or secondary task performance. Depending on Traffic models are also classified into microscopic and mac- the type of secondary task performed, objective measures of roscopic models. The latter analyse traffic flow as a whole, workload include lane departures, and lateral deviations. while the former focus on specific actions of the driver and Additionally, performance based assessments include task the physical laws of motion. Thus, in the case of macroscop- time, reaction time, accuracy, and error rate. Physiological ic models, overall shockwaves are analysed but do not con- measures encompass audiology, cardiovascular, urodynamic, sider each car individually. Macroscopic models are not suit- gastrointestinal, respiratory, neurophysiology, and ophthal- able for driving behavior modeling since they do not exam- mic physiology [Rusnock, 2018]. Using physiology is advan- ine individual vehicle behaviours. Microscopic approaches tageous, as assessment occurs continuously in real-time. are more suitable for driving behaviour analysis and are Physiological quantitative data of a subject’s state can be based on the models of: car-following [Brackstone and linked to complex constructs such as mental workload, fa- McDonald, 1999], intelligent agents [Hidas, 2002], fuzzy tigue, situation awareness, health, and emotion [Endsley, logic [McDonald et al., 1997], and cell transmission [Da- 1996; Kelly, 2003]. By assessing a user’s physiological state, ganzo, 1993] for simulation of traffic. Car-following theory a designer will receive feedback that cannot be expressed is an effective method to study the interaction between vehi- verbally or written by the user. cles in a microscopic simulator. The method used in this work is based on a microscopic model utilising the agent- In this study drivers’ workload was measured by electroen- based approach, with each vehicle represented by a software cephalography (EEG) and lateral deviation. The algorithm agent having autonomy to behave based on some predeter- implemented in the NeuroSky EEG device measures the at- mined rules that define basic driving styles. tentional resources consumed while the participant per- formed a task. Data from the EEG was monitored on a simu- The aim of this work is to provide a simulation environment lation time-step basis and automatically mapped to road sec- that is fully customisable. This is necessary to eliminate the tions. The optimum level of driver performance is achieved effects of confounding variables from a driver behaviour with a medium level of workload [Gregoriades et al., 2006], experiment due to unfamiliarity with the infrastructure. which implies an EEG reading around 50%. Hence, as part Hence, it was necessary to model the road network in the of the simulator validation, it is hypothesised that a normal simulator prior to the analysis. driving scenario should not overload the participants. Over- loading users in a simple scenario could indicate unrealistic 4 Designing the driving simulator driving conditions that require participants to devote extra Much effort has been put in implementing driving simulators cognitive resources to process unfamiliar task-related situa- in the last years [Biurrun-Quel et al., 2017; Rossetti et al., tions (unexpected acceleration, steering etc). 2013; Almeida et al., 2013; Gonçalves et al., 2012, 2013; Alves et al., 2013]. These methods and tools allow the repre- 3 Driving simulation sentation of complex, realistic traffic situations for evaluat- By definition, driving simulators are complex systems of ing specific traffic situations or testing new technological software and hardware which simulate real life environ- applications and their influence on the driver. The simulator ments, behaviours and physical systems. Driving simulators was implemented using UNITY game engine which apprais- are used in a variety of applications, from training new driv- es rapid application development through a component-based ers in a safe environment to testing new car technologies. software engineering approach. The driving environment They are often developed as part of traffic modelling and was designed using generic models that make up driving conditions and road infrastructure. The modelling of the road network was achieved by extracting a section of the Nicosia Interactivity between the user and the simulator was realised road network from OpenStreetMap to generate a 3D model in Unity through C# scripting languages. Finally, the simula- of the cropped area in UNITY. The selection of the road tor was designed with the capability to record in log files the network was based on identified accident black spots [Gre- driving behaviour of users in real-time. Specifically, for each goriades, 2013] on the road network: roads suffering from simulation time-step the simulator records drivers’ headway, high accident rates. The assumption is that drivers consume lateral deviations, speed, acceleration and deceleration. Thus, more cognitive resources at these locations hence they are it enables the analysis of the data collected on a section-by more susceptible to accidents. The selection of the car mod- section basis. A screenshot of the simulator’s user interface els was based on car types and brands currently used in Cy- from the driver’s perspective is depicted in Figure 1 along prus, in order to enhance the realism factor of the simulated with the road network under study divided into 63 sections. environment. Traffic conditions were specified though the use of autonomous agent-based vehicles that are able to nav- The main components of the simulator are: i) the Unity game igate independently in the network based on pre-set driving engine that controls the physical and environmental aspects behaviours. The vehicle behaviours were based on a prelimi- of the simulation; ii) the host vehicle controller that enables nary analysis of traffic routing in the modelled traffic net- the navigation of the host vehicle using the pedals and steer- work. The accident time statistics of the modelled section of ing wheel; iii) the data-logger that records the driving behav- the road network were used to pinpoint the most critical time iour of participants in experiments, along with additional on the selected black spot and accordingly replicate the traf- data relating to the traffic conditions; iv) the Multi screen fic conditions in the simulator. controller, that is responsible for the synchronization of the 4 screens in the cave automatic virtual environment (CAVE) facility; v) the autonomous multi agent vehicle controller, that controls the vehicle-agents in the simulation. This com- ponent is responsible for recreating different traffic condi- tions depending on the scenarios that need to be modelled. Each autonomous vehicle agent dynamically decides its route, avoids obstacles in its way and alters its speed depend- ing on the traffic. vi) The final component, is the road infra- structure manager component is the facility used for the de- velopment of the road network and the surrounding envi- ronment. 4.1 Autonomous vehicle agents In order to mimic a real driving experience, it is essential to model all external environmental factors such as surrounding vehicles dynamics, traffic lights and so on. The behaviour of vehicles around a car is modelled based on the car-following model and using the agent-based paradigm. The goal here is for participants to experience the same feelings as if they were in the real vehicle in naturalistic settings. This is achieved by embedding each agent with a driving behaviour model with the following features: path finding, speed selec- tion, obstacle avoidance, and acceleration and deceleration models. Each vehicle agent interacts with other vehicle agents and with infrastructure and traffic control agents as shown in Figure 3. The exchanged messages enable each agent to achieve its goals which are to avoid colliding with other vehicles or the infrastructure, maintain a normal speed, abide to the traffic regulations (drive on left lane etc.). The route followed by each agent is dynamically defined, howev- er collectively all agents device routes that mimic realistic Figure 1: Screenshot of the virtual road design (bottom) and traffic conditions. the first-person-view from the driver’s seat (top) Infrastructure to follow in a similar way as Navmesh method [He et al., AGENT Ii Msg Traffic 2016]. Agents choose their path at runtime, hence deciding Vehicle Msg control AGENT Ai the path based on what is currently happening around them. For this study vehicle agents had no specific destination. AGENT Ai Msg Msg Msg Their role is to move autonomously, in a non-predefined Vehicle AGENT Ai Msg Vehicle AGENT Ai path, in the road network under study, to mimic the traffic condition at the specific network. The driving behaviour model used was the car-following and the traffic volumes for the particular part of the road network was specified based Anatomy of agent Sensors Sensors Sensors on results from a previous study using the VISTA macro- scopic simulation model [Gregoriades et al., 2013]. Speed Brake controller Steer controller Target finder controller For the path finding model to be operational, the road net- work was modelled using waypoints (Figure 3). This enables Agent’s Interface Bottleneck controller GOALS Controller vehicle agents to know their location on the network, the number of lanes at each point and the flow direction at each lane. Waypoints represent all the possible paths a car can Agent’s sensors take on the network. Waypoints are connected in a way that Figure 2: Multi agent system architecture each road lane has a predefined direction. Two types of way- points were used: simple and connector waypoints. Simple waypoints are used by the vehicle agents as targets to follow on a path, which resembles the road lane they are currently on (Figure 4). Connector waypoints are used as simple way- points, with the added functionality to connect two different road sections. For example, in an intersection, you exit the road section you are currently on to enter a different section on your path. In this case, these two sections are connected with connector waypoints. Vehicle agents follow waypoints, to create a path to follow. As mentioned before, the path is not specified at the begin- ning of an agent’s life. Instead they start by defining the first two target waypoints and each time a target is reached, the agent changes its target to the next waypoint it had already Figure 3: Vehicle Path finding component selected before, while choosing a new “next” target. The selection of a target waypoint is done randomly, based on where the target waypoint is connected to, along the direc- tion of the car. All possible connections from each waypoint are stored in an array, and are dynamically accessed by the vehicle agent at each time-step of the simulation. To implement this functionality, all waypoints were pre- specified on the network model in the form of invisible event-based UNITY objects (Figure 4). Waypoints’ objects act as placeholders of infrastructural information that auto- nomic cars utilise. Vehicle agents access this information when targeting a waypoint, or when they are in the process of selecting a new target waypoint. In order to mimic the driving conditions of the road network under study, the au- tonomous agents’ controller assesses the number of vehicles Figure 4: Waypoints on the infrastructure, for vehicle path that are on the road at each time-step of the simulation and finding accordingly increase or reduce the traffic volume so as to replicate the expected traffic conditions. Path finding refers to the process of finding the path to fol- low in order to reach a destination or an objective. For in- Vehicle steering and acceleration is performed after the vehi- stance, an agent might be seeking the shortest path to a desti- cle has selected its next target. As soon as a car has a new nation, or the path with the smaller number of obstacles or target to reach, it starts calculating the steering angle and traffic. Path finding agents analyse all available paths, and acceleration required to effectively reach the target waypoint. based on their objectives and restrictions, decide which one The steering angle is adjusted dynamically depending on the driving environment would enable participants to have ade- position of the vehicle, its desired destination, and speed. quate level of SA and workload. During the experiment par- The steering functionality also addresses issues with regards ticipants were informed to drive in their normal driving style to obstacles or bottlenecks. In case of obstacles, the steering in a pre-specified path in the road network. During the exper- to be applied is calculated based on the direction the car iment the simulator was collecting data regarding their needs to follow in order to avoid the obstacle. Acceleration speed, acceleration, deceleration, EEG, headway, lateral and speed are calculated based on distance to the preceding movements and breaking patterns. Upon completion of the vehicle. Lane change behaviour is stochastic. experiment participants completed the post-test questionnaire about their driving experience in the simulator. Post- 5. Validating the Simulator experiment questionnaire addressed the following dimen- To be confident that the driving simulator correctly mimics sions: realism of the simulator’s general features, user inter- reality, two validation studies were conducted: a preliminary face, ease of learning, capabilities, usefulness, ease of use, validation and a more extensive human factors validation. how the simulator supports their situation awareness. Each For the former, a number of professional taxi drivers were dimension was assessed on a 1-7 point response scale with 1 asked to drive in a modelled road section in the VR settings being negative ratings and 7 positive (figure 5). Results show using the virtual host vehicle. Experts tested vehicle’s steer- percentage of positive scores (scores of 5 and above). ing sensitivity, acceleration and deceleration, and evaluated the realism factor of the virtual environment. Initially, sever- al problems were identified with regards to vehicle steering, acceleration and deceleration behaviours. In addition, the early versions of the driving simulator suffered from low refresh rate that led to motion sickness. In order to overcome these issues, several modifications were performed to the simulation scripts until a satisfactory vehicle behaviour was achieved. The revised version of the simulator was revalidat- ed by 5 taxi drivers who all agreed that its behaviour was realistic. The main simulator validation study aimed to identify the Figure 5. Percentages of positive responses above 4, in each suitability of the developed synthetic environment for human of the measured dimensions factors analysis. Therefore, for this purpose an experiment was conducted with participants in a hypothetical driving Participants’ post-test response shown as percentage of posi- scenario of a replica road section of Nicosia, with the same tive responses (above 4) in Figure 5, reveal that overall the infrastructure, traffic control and similar traffic conditions. simulator was perceived as satisfactory in mimicking a real- The simulation was performed in the VR cave with physical istic driving situation. Moreover, the level of realism was steering wheel and petals. The research was conducted in adequate (71%). However, in one case the participant suf- three stages: before, during and after the experiment. Before fered of a minor incident of motion sickness. the experiment, participants completed the Manchester Driv- ing Style questionnaire [Reason et al., 1990] and after the During the objective SA assessment, the simulator was driving experience questionnaire. stopped at different points and participants were asked a number of questions relevant to the driving situation to the Seventeen participants from the local population, with a val- freezing point. Questionnaire responses from this process id driver’s licence and either 20/20 vision or wearing correc- were assessed on a 0-100 score and analysed by comparing tive glasses or lenses were involved in all stages of the ex- the actual situation with what the participants reported in periment. Given that driving skill is a significant factor in the their results for the 3 freezing points. Answers from these visual search strategies of drivers, and subsequently SA [Un- questions were analysed and an average collated score for all derwood, 2007], the subjects selected had at least seven questions designated the level of SA. Results showed that all years’ driving experience and were under 55 years old. Prior participants maintained an adequate level of SA with an av- to the experiment, participants were screened for colour erage score of (69.6%) in 3 freezing points. This was slightly blindness. They were introduced to the various simulator less than the subjective rating of participants as shown in controls, made adjustments to the seat and were given a five- Figure 5 which was about 72%. However both indicate a minutes training session in a road section other than the sec- satisfactory level of SA. An additional evaluation of SA was tion used in the experiment. The average age of participants conducted using objective data from lateral deviations as was 37.1 years and the gender distribution was 55% female recorded by the simulator for all 63 road sections. These to 45% male. were analysed to identify points of reduced SA due to sharp lateral movements. This is phenotype behaviour related to The main variables of interest in this study were workload both overloading and low SA. From the diagram in Figure 6 and Situation awareness (SA), hypothesising that a realistic it is evident that the deviations are relatively smooth which indicates an acceptable level of SA. This, in turn, shows that acceleration was positively correlated with “errors”, “lapses” participants were actively engaged with the driving task. and negatively correlated with “SA”. This means that ag- Moreover, smooth deviations also indicate a relatively easy gressive driving reduces drivers’ SA while it increases errors task undertaken by participants. The three points with high and lapses. Essentially, our initial assumptions regarding deviations (sections 23, 47 & 58) represent the points with self-report driver behaviour and observed driver behaviour the pre-set obstacles. were met. Specifically, self-reported aggressive behaviour was found to be positively related to increased speed and acceleration patterns in the simulator, hence indicating that Average leteral deviations per Section the simulation environment provides a realistic setting that 0.1 enables participants to drive in the same manner as they do in their everyday life. This, as a result, is a promising indica- 0.05 tor towards the validity of the designed simulator Serious Errors Lapses Acceleration Aggressive Tailgate Situation 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 (SR) (SR) (SR) (O) acceleration (SR) awareness -0.05 (O) (O) Speed (O) -0.267 -.288* -.296* 0.396** -.325* -0.268 0.212 -0.1 Aggression 0.512** 0.401** .518** -0.204 .277* -0.189 -0.151 (SR) serious(SR) .622** .507** -0.245 0.275 .333* -.302* EEG per Section errors(SR) .857** -0.241 .389** 0.125 -0.155 70 lapses(SR) -0.133 .402** -0.012 -0.197 60 Aggressive 0.068 -.416** acceleration 50 (O) 40 30 Table 1. Pearson correlations (and significance level) among 20 observed (O) and self-reported (SR) behaviours (N=50 or 10 51, *p<0.05, **p<0.01) 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 Workload analysis To answer the fourth research question in relation to drivers’ Figure 6: Workload (bottom) and Lateral deviations (top) of workload, both EEG readings and lateral deviations per road all participants per road section section (Figure 6 &7) were utilised. The former is a physio- logical objective measure and the latter a phenotype objec- Driver Style Analysis tive measure. Given that the participants were driving in a To answer the first research question it was necessary to in- normal driving scenario with easy traffic conditions, the as- vestigate the extent of the association between participants’ sumption here was that there would be no overloading of self-reported and observed driving style. The assumption is participants. If that occurred then it could indicate a problem that, if self-reported and observed driving behaviours are with the simulator’s level of realism. The hypothesis is that similar then the simulator provides the means for participants drivers under optimum driving condition (no hazards and to behave in a realistic manner and hence is considered as low traffic flow) should not experience overloading. If this valid. occurs then it could designate that the simulator requires the drivers to utilise extra cognitive resources to figure out how For the self-report stage, participants were asked prior to the to drive optimally in the synthetic environment. It is evident experiment, to fill in the Manchester Driving Style question- from these results that on average all participants experience naire [Reason et al., 1990]. This aimed to elicit the driving an optimum level of workload. This was between 45 to 65 in style of participants, along with demographic information. terms of EEG readings (Figure 6). Similar results are depict- Their observed driving style data were collected by the simu- ed in the 3D analysis of the frequency distribution of EEG lator for each time-step of the simulator and assigned to rele- ratings (Figure 7) per road section. This shows that the ma- vant road sections. jority of participants experience optimum level of workload in all road sections. The EEG ratings are slightly high at the Collected data underwent pre-processing and subsequently first road sections but still within the acceptable range of analysed in SPSS to investigate the magnitude and signifi- optimality. The second measure of workload utilised here is cance of the link between observed (simulator) and self- lateral deviations. Results of Figure 6 show that there was no report (questionnaires) behaviours. Results in Table 1 indi- significant deviations by participants and hence indicating cate that aggression variable is correlated positively (and that the level of workload was optimal throughout the exper- significantly) with the variables “serious violation”, “errors”, iment. “lapses” and “aggressive acceleration”. Observed aggressive Limitations of this work concentrate on the simulator’s level Aggressive Serious Errors Lapses Tailgating (SR) of immersion factors and the issue of motion sickness known (SR) (SR) (SR) (SR) in VR settings. Simulated settings do not currently offer the EEG .225 .477** .181 .320* .148 resolution of the real world, and so these may affect driving behaviour and human factors analyses. Table 2. Pearson correlations (and significance level) among observed EEG (O) and self-reported (SR) behaviours (N=50 References or 51, *p<0.05, **p<0.01) [Alves, J. Gonçalves, et al 2013], "Forward collision warning Optimum workload systems using heads-up displays: Testing usability of two new metaphors," IEEE Intelligent Vehicles Symposium (IV), Gold Coast, QLD, 2013, pp. 1-6. [Davenne D el al, 2012].,. “Reliability of simulator driving tool for evaluation of sleepiness, fatigue and driving per- formance”, Accident Analysis and Prevention, 45, pp.677- 682. [Doniec A., R. Mandiau, et al,2008] “A behavioural multi- agent model for road traffic simulation,” Eng. Appl. Artif. Road s Intell.,vol. 21, no. 8, pp. 1443–1454, 2008. Sections [Eksler V. et al, 2008] “Regional analysis of road mortality in Europe”, Public Health, 122, pp.826-837. Endlsey M. R. (2004). Designing for situation awareness: an approach to user-centered design, CRC press [Endsley MR, Jones DG (2012) Designing for situation Figure 7: 3D EEG frequency distribution per road sections awareness: an approach to human-centered design, 2nd (vertical axis) and EEG level (horizontal axis), showing the edn. Taylor and Francis, London concentration of frequency peaks around the optimal level of [Gonçalves et al., 2014] "Testing Advanced Driver Assis- workload tance Systems with a serious-game-based human factors analysis suite," IEEE Intelligent Vehicles Symposium To answer the fifth research question, an analysis was con- Proceedings, Dearborn, MI, 2014, pp. 13-18. ducted to examine the link between drivers’ style and work- [Gonçalves, R. J. F. et al 2012] "IC-DEEP: A serious games load. Correlation results showed that drivers who are charac- based application to assess the ergonomics of in-vehicle terised as inattentive (i.e. commit high level of lapses) in information systems," 2012 15th International IEEE Con- their self-report driving style experiences high readings of ference on Intelligent Transportation Systems, Anchorage, EEG (Table 2). This confirms the assumption that careless AK, 2012, pp. 1809-1814. and inattentive drivers (imprudent) consume more cognitive [Gregoriades. A, et al, 2007] “Workload prediction for im- resources to engage with the driving scenarios. proved design and reliability of complex systems,” Reliab. Eng. Syst. Saf., 39, n.4, pp.530–549. 6. Conclusions [Gregoriades, A., et al, 2013] Black spots identification The paper describes the design and validation of a custom through a Bayesian Networks quantification of accident made driving simulator for driver behaviour analysis. The risk index. Transportation Research Part C 28, 28-43 developed driving simulator is agent-based with the infra- [Gregoriades A, et al, 2010] “Human-Centred Safety Analy- structure being developed using a component based ap- sis of Prospective Road Designs”, IEEE Transactions on proach. This allows the analyst to easily customize the road Systems, Man and Cybernetics, Part A, Vol 40, 2, pp 236- infrastructure for what-if scenario analyses and the design of 250. experimental settings for a variety of scenarios. [He Z, M. et al, 2016] "Research and application of path- finding algorithm based on unity 3D," 2016 IEEE/ACIS Results from the analysis of the data collected during the 15th International Conference on Computer and Infor- experiment, revealed that the simulator satisfies the mini- mation Science (ICIS), Okayama, 2016, pp. 1-4. mum requirements for vehicle control since participants [Reason J, et al, 1990], Errors and violations on the road: a maintain satisfactory level of SA and workload. Additional- real distinction? Ergonomics 33:1315–1332 ly, results indicate that what the users experienced during [NHTSA, 2015a], The National Highway Traffic Safety their interaction with the simulator and what they actually Administration. denoted as their opinion in the post-test questionnaire point [NHTSA, 2015b], Critical Reasons for Crashes Investigated to the same conclusion. Finally, self-reported driver style of in the National Motor Vehicle Crash Causation Survey, participants was correlated with observed behaviour during The traffic safety facts, DOT HS 812 115 the use of the simulator, pointing to the conclusion that the [Kading, W., 1995]. The Advanced Daimler-Benz driving artificial settings did not alter their driving style, hence it is simulator. PC-8 Technical Paper No 9530012. Society of realistic and considered as valid. Automotive Engineers of Japan, Inc [Plochl, M., et al, 2007] Driver models in automobile dy- namics application, Vehicle Syst. Dynamics, 45, 699-741 [Reuschel, 1950], Vehicle Movements in a Platoon with Uni- form Acceleration or Deceleration of the Lead Vehicle, Zeitschrift des Oesterreichischen Ingenieur-und Archi- tekten-Vereines, No.95, 59-62 and 73-77 [Khaisongkram, W.,et al, 2009]. Driver behavior model- ing and parameter identification during car-following situation on urban road, in Proc. of 27thAnnual Conf. of the Robotics Society of Japan. [Lajunen, T., et al, 2011]. Self-report instruments and meth- ods. In B. E. Porter (Ed.), Handbook of Traffic Psycholo- gy (pp. 43-59). London: Elsevier. [Mastio M., et al, 2018], "Distributed Agent-Based Traffic Simulations," in IEEE Intelligent Transportation Systems Magazine, vol. 10, no. 1, pp. 145-156, Spring 2018. [Lawton, R., et al, 1997] "The Role of Affect in Predicting Social Behaviors: The Case of Road Traffic Violations.", Blackwell Publishing Ltd, [Bonsall, P., et al, 2004] Modelling drivers car parking be- haviour using data from a travel choice simulator. Trans- portation Research Part C, 12, pp. 321–347. [Sklar E, et al, 2007] NetLogo, a multi-agent simulation en- vironment. Artif. Life. 13(3) [McDonald, M., et al, 1997] Development of a Fuzzy Logic Based Microscopic Motorway Simulation Mode. Proceed- ing of the ITSC97 Conference, Boston, U.S.A. [Wu, J., et al, 2000] Fuzzy Sets and Systems for a Motorway Microscopic Simulation Model. Fuzzy Sets and Systems, special issue on fuzzy sets in traffic and transport systems Vol. 116 (Issue 1) [Hidas, P., 2002] Vehicle Interactions by Intelligent Agents, University of New South Wales. [Daganzo C.F. 1993]. The cell-transmission model. Part I: A simple dynamic representation of highway traffic. Califor- nia PATH Research report, UCB-ITS-PRR-93-7, Inst. Trans. Studies, University California, Berkeley [Evans, L. 1991]. Traffic Safety and the Driver. Van Nos- trand Relnhold, New York . [Rusnock C, et al 2018] Workload profiles: A continuous measure of mental workload, International Journal of In- dustrial Ergonomics, Volume 63, 2018, [FHA, 2013] Making Driving Simulators More Useful for Behavioral Research— Simulator Characteristics Compar- ison and Model-Based Transformation, Technical Report, USDT 2013