A developed by authors application for real-time visualization of augmented and virtual reality is considered, which enables to simulate the traffic accidents on a mobile platform with the Android operating system. The application is developed on the basis of the Unity 3D and is able to visualize various types of road accidents and also link them to a terrain, different conditions of the urban environment, and map of the accident statistics. The system has an archive of videos for the accident 3D-scenes. The application interface and implementation features are considered. The subsystems of virtual and augmented reality are described in detail with examples. This application may be used to train novice drivers, learn the rules of the road, and inform experienced drivers about new road junctions. In the system environment, the distribution of the number of traffic violations depending on the driving experience for three groups (no experience, less than 3 years, more than 3 years) was investigated. The study of the usefulness of the system on the Likert scale on 80 cadets of driving schools is presented.
According to statistics, annually more than 1 million people over the world die in road traffic accidents and about 50 million people are injured. Only in 2019, 164358 road accidents were registered in the Russian Federation, as a result of which 16981 people died and 210877 were injured. In this case, the number of accidents committed due to traffic violations, has 146688, which killed 14420 people. Thus, it is clear that more than three quarters of all accidents are associated with violations of traffic rules. * Publication financially supported by RFBR grant №18-08-01484
Based on the statistics provided by the portal stat.gibdd.ru, it can be concluded that
it is necessary to develop and adopt a set of measures to improve the skills and
knowledge of traffic rules to ensure road safety. At the same time, studies conducted
in different countries, for example, by the authors F. Wegman, H.-Y. Berg, I.
Cameron, C. Thompson, S. Siegrist, W. Weijermars in four jurisdictions - Western
Australia, the Netherlands, Sweden and Switzerland showed that the human factor is
especially important - a correct assessment of the traffic situation [
The solution of the problems considered is not limited to the development of the
urban road network. Various intelligent transport systems are offered. For example, in
the works of Shaurya Agarwal, Pushkin Kachroo Emma Regentova [
Interesting research authors A. Paz, N. Veeramisti, R. Khaddar, H. de la
FuenteMella and L. Modorcea [
To address this problem, i.e., to mitigate driving problems caused by excessive
information, it is proposed to work on making it easier to view navigation instructions
without taking your eyes off the road» [
Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 3 not know how to "scan" the environment quickly enough cannot cope with the additional flow of information. Therefore, during the research it was revealed that for the novice drivers and cadets of driving schools the most acute problem is the problem of correctly recognizing the road situation, in which there can be almost no time for making a decision. It is necessary to act as quickly as possible and, at the same time, take into account all aspects of the current situation, such as the signal of a traffic light, marking, signs, the position of cars and their speed. In addition, when you get into an extreme situation, the student may become confused, which can lead to even more deplorable results.
At the same time, studies conducted in the work D. Sobel, K. Jędrasiak, K. Daniec,
J. Wrona, P. Jurgaś and A.M. Nawrat [
In this study, the solution to this problem can be the proposed application for a mobile platform intended, primarily for novice drivers and cadets driving schools.
Two ways of ensuring road safety are considered: 1. Preview an unfamiliar path section using the augmented reality. Pointing the camera smartphone on a map of the city, for example, on Yandex map, the driver can see a realistic model of a specific section of the road infrastructure on the smartphone screen with all signs and markings.
2. By immersing in virtual reality, a driver can see in a stereo mode a living situation of an accident that occurred on a particular section of a realistically modeled part of a particular city. View this situation from different points of view, like the victim, the culprit or the casual witness and understand all the reasons for this incident. And also for the cadets of driving schools, a system of dynamic tests is offered, performed on really existing sites with increased statistics of road accidents in specific cities. Subsequent paragraphs, however, are indented. 3
The most dangerous parts of the road network are considered. Since the application is
aimed at obtaining traffic information, the following risk factors are studied as the
contextual information [
Ki = OP/N, where OP is the number of traffic accidents with this factor at a given location; N - average for all similar events in the city. All factors are modeling parameters. The most dangerous sections of the city roads, corresponding to each factor Ki (1), are identified, and dangerous road situations are modeled on these sections. Recorded video showing: 1) erroneous behavior of the driver and 2) how to properly act in a similar situation. A video bank has been prepared, which stores videos demonstrating traffic rules. 4
The application interface contains two main modes of operation: Dynamic testing and View scenes (Figure 1).
When the first mode is selected, the marker tracking and snapping mechanism is enabled. In the process of pointing the camera of the smartphone to the Yandex map, this mechanism is allowed to see all possible scenes available for testing. To load a specific scene, the user needs to put his hand on the virtual button “Loading a scene” while holding the smartphone so that the model remains visible on the screen. When you select View scenes, the application displays a list of all available scenes on the smartphone screen. The user can examine the selected scene in detail from various points of view and from different faces of road users.
The application has two ways to pass the test: normal (control and navigation only using a smartphone) and using virtual reality glasses. 5
To implement the application on the Android platform, the integrated development environment of Unity 3D is used. For the traffic-accident areas appropriate scenes are designed to simulate typical accidents associated with this location. Models of scene’s participants are pre-created in the Autodesk Maya and 3DsMax tool environment as the low-polygon objects having trajectories parameterized relatively to the road, roadside objects and other traffic participants. These scenes are then embedded in the Unity3D environment (Figure 2).
Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 5
The augmented reality functionality was realized using Vuforia SDK (www.vuforia.com), the worlds widely used as AR platform for Unity 3D. Blocks of our application are united via the C# programming language. The augmented reality realized in the application via the Vuforia Object Scanner. Vuforia plug-in launches a mechanism for searching for tags, created in advance for the city map, and reproduces as the augmented reality a tooltip about the degree of danger or the correspondent accident 3D scenes for this part of the road.
The scenes for the most dangerous areas can be also integrated onto the map via special labels for learning such places. The Vuforia plug-in recognizes the image of one of the label on the map received from the smartphone's camera, based on information about the position of the label (marker) in space. For 3D reconstruction of the scene the calculation of the reference points with simultaneous tracking is performed, thereby the objects of three-dimensional scene will be correctly oriented in the space. The application projects the corresponding model onto the label, and the scene of the accident case is loaded on the screen. The scene is also played in the third person view mode (Figure 3). 6
Block VR (virtual reality) is designed to view the model in virtual reality mode without using a label. From the list of suggested scenes, the user selects the model of interest. The screen of the smartphone or tablet is loaded with a scene, with the ability to view it from different angles and from different road users with full or partial immersion.
For a full immersion, a virtual reality helmet is required. A virtual camera layer is added to the scene. The choice is made in favor of the product Fibrum SDK, because it has all the necessary settings, and it does not overload the system with additional plug-ins. Using the Fibrum SDK object, two cameras are installed, for each eye, respectively, the distance between the cameras, the distortion of the image and the required cylindrical distortion. To view the virtual reality with full immersion requires a screen with a diagonal of 4 to 6 inches, as it is necessary to have in the phone sensors accelerometer and gyro.
To manage the view mode the standard internal plug-in Unity 3D - Characters Controller is used. When you download a scene on a smartphone in the virtual reality mode, you can view the situation from various road participants, whether the culprit, the victim, or the free travel in the scene. When the user is in the free travel mode, he
Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 7 can look around, explore the terrain and the road infrastructure surrounding it, or navigate around the stage using the standard control joystick.
Buttons for selecting the view mode are at the top of the screen. Back button - to exit the scene, to the main menu.
In the virtual reality scene view mode using the virtual reality glasses (Figure 4), to switch to viewing a scene from another participant in traffic, the user must "look" at the button until the load bar appears. After loading it, within a few seconds, the program will redirect the user to the selected menu item. To select a view of the scene from another participant of the road, the user needs to "look" at the corresponding button on the screen. For a more complete immersion, objects inside the scene cast a shadow from static and dynamic light sources. 7
During the studding in driving schools trainees are offered static illustrations in a
book with tests that give the student time to think, against the dynamic situation on
the road, in which time is very short for making a decision. Here, as an alternative, the
system of dynamic tests, performed on really existing sites with increased statistics of
road accidents on the example of a particular city, is proposed. The most dangerous
parts of the road-transport interchange are taken as a basis [
As a result of the investigation of these areas and road accidents, a number of test tasks were compiled with situations in which accidents most often occur, as well as a number of complex road sections with which drivers have problems. On each site, several variants of tasks are considered, based on the real location of traffic signs, marking and possible obstacles (for example, other cars that violate the rules, so the driver may not notice the traffic sign).
In the opposite of standard test (Figure 5), during the passage of the dynamic test, the view from the driver's seat is displayed on the monitor screen just before the learner (Figure 6). The test person can turn his head around, as it does in real life.
The procedure for passing the test is as follows. When the user's machine approaches the testing stage, for example, the intersection, the candidate is offered a choice of several action options.
If the tested person chooses the correct action, then the score is recorded, and the machine continues to move to a predetermined position. If an incorrect action is selected, penalty points is recorded. In some cases, if an incorrect choice can lead to an accident, the scene of the accident is reproduced.
During the one test, 20 tasks total are used, according to the results of which the final score is announced and tips are given on the incorrectly passed stages if any. The level of complexity of the task and the number of earned points is determined based on the complexity of the road segment via the integral factor Ki (like the level of difficulty in a game) and determines the level of the driver's qualification. At each stage, from 1 to 5 possible outcomes are realized, where each outcome is a next task that is randomly loaded into the system, depending on the choice made earlier.
There are two modes for the tests: a mono visualization mode, and a full immersion mode using a virtual reality helmet, or its light version - VR glasses.
The application has several Check Point, the points of the beginning of the simulation of road traffic. After loading the character at this point, he is given the task to arrive at a Finish point. The user starts the movement. On the way to the Finish point, he encounters various obstacles and random events. The following factors are taken into account: the road surface, visibility, the time of movement of the character, the number of oncoming vehicles, and the discipline of the driver. If a character is disciplined, then he moves from random points to other given points according to the rules of the road with different speed limits - from slow to normal.
Characters "offenders" - clearly show what kind of behavior on the roads can cause an emergency or create interference on the road. They constitute 5% of all NPS (NonPlayer Character - a character controlled by a non-player) involved in road traffic and at this stage are programmed for three main violations: speed limit (61-90 km/h), going beyond the stop line at intersections, rebuilding from strip to strip "game of checkers".
If you violate the rules of the road, penalties are charged on the table as in the exam when you receive the rights to drive a car. The description of the error with decoding appears on the screen. At the end of the trip, the user can view all the violations and refresh their knowledge of the rules of the road.
With the accumulation of five penalty points, the program stops the driving process, displays a list of errors, offers to re-read the rules of the road and pass the site.
These events are generated randomly so that the user gets to a familiar site every time, but with different conditions. This factor keeps the user in constant attention, which in turn forms the skill of quick response in unforeseen situations.
Examples of violations recorded by the program:
Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 9
Collision with another vehicle, obstacle, or exit to the sidewalk. When colliding with third-party objects (Figure 7), the simulation stops, five points are added to the error register, and a corresponding message is displayed on the screen with an error. For simplicity, the calculation of collisions for each object has its own invisible contour. For example, for cars, it is a rectangle.
Movement on a given lane. The program tracks the position and dimensions (spirit) of the car. If the user crosses the marking line or continues to move along it for five seconds without performing the appropriate actions, such as rebuilding, overtaking, returning to its own lane, then the user is charged with an error in the amount of one point. And the remark “Violated the rules for the location of the vehicle on the carriageway” is added to the registry of errors. When a violation is repeated, the script repeats.
The application is modeled tram tracks. When traveling on the opposite direction, the transparent plate (oncoming traffic plane) intersects the vehicle’s contour, the program stops the simulation, charges five points and displays the critical error “Took off the oncoming traffic (except when allowed) or on the tram tracks of the opposite direction”. By the same logic, the recognition of a double solid marking line is arranged.
Traffic at the prohibitory traffic light sign. During the red signal of the traffic light at the level of the stop line is an invisible plate, which has its own contour (Figure 8). When crossing the contours of the car and the transparent wall, the “violation” algorithm is launched. 5 points are added to the register, the error “I passed the prohibitory signal of the traffic light” or “Crossing the stop line” is displayed on the screen. Switching the colors of traffic signals is a sequential change in the properties of object materials (color, emission) in accordance with the operation of real traffic lights in the studied area of the city.
The task of changing the color of traffic lights must be performed periodically, it is
suggested to use the Unity coroutine for implementing traffic lights in order to receive
updates regularly, but not every single [
When prohibiting road signs are installed on the road, the application establishes an invisible plate with an outline. When this contour is intersected, the simulation stops, an error of 5 points is recorded in the register, an error is displayed on the screen: “Did not fulfill the requirements of priority signs, prohibiting and prescriptive signs, road markings, as well as special prescription signs”.
To control the digital model of the car requires a controller that communicates with the program. Currently, two control systems are popular:
− buttons placed on a smartphone or VR glasses such as a magnet switch (its disadvantage is incompatibility with smartphones that are not equipped with a magnetometer) and a capacitive button (it works with most types of smartphones since it does not require any special sensor);
− gamepad / controller (joystick), interacting with a smartphone using Bluetooth technology. The controller is chosen because it is universal and gives more opportunities to the user and the application in its development. It does not restrict the user and gives a sense of freedom, as his hands are closer to the real position on the steering wheel, and not on the helmet of virtual reality.
When the user is in the application menu, the joystick allows you to select the desired section of the system, and you can confirm the selection with the appropriate selection button. This allows you not to remove the virtual reality helmet from your head and not use the smartphone screen. At the time when the user is immersed in virtual reality, the digital car is controlled by an external joystick.
Three groups of drivers with different driving experiences were formed. Each group consisted of twenty people. The main characteristics of dynamic tests have been verified by users of these three groups. The results are shown in Table 1. No Driving experience Total responses for:* 1 2 3 4 5 1 Without experience 5 7 2 6 0 2 Up to 3 years 3 5 3 5 4 3 More than 3 years 1 4 3 3 9 * Traffic offenses: 1 → “Journey on forbidding signal of a traffic light”; 2 → “Driving on the wrong side of the road, unless there is an obstruction”; 3 → “Not giving priority to pedestrians in pedestrians’ crossings”; 4 → “Noncompliance of vehicles drivers with the traffic signs and directions”; 5 → “There are no traffic offenses” It can be seen from the table that the greatest number of mistakes are made by inexperienced drivers who have no experience in driving a car.
Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 11 1 2 3 4 5 6 7 8 No The mobile software system was also offered to eighty students from four driving schools. At the end of the training cycle, they filled out a questionnaire to express your opinion on 8 statements about the educational process using an AR/VR-based elearning tool. Table 2 presents the received survey responses on the Likert scale. Research has shown that the use of a dynamic testing system makes driving learning easier and more enjoyable. 8
The application of virtual and augmented reality technologies for driver training is investigated. The mode of dynamic training and testing in the mode of virtual and augmented reality with various degrees of immersion is offered. To build scenarios for the creation of accident-dangerous cases, statistics, and situational descriptions of accidents were used.
The application for a mobile platform was realized in the Unity 3D integrated development environment and intended to use in practical classes in driving schools. The reason to research is no possibility in driving schools to pass all difficult sites of roads and to disassemble all non-standard situations with the cadet.
This research carried out confirms that virtual worlds and simulators improve
learning outcomes [
It can also be applied when new complex traffic junctions are put into operation. For example, in Nizhny Novgorod, when the metro bridge was put into operation, many drivers did not use it due to difficulty, in their opinion, entrances the bridge both from one side of the river and on the other. Timely information of drivers via virtual reality will avoid many dangerous situations and, accordingly, will contribute to improving road safety. The pilot project of this training simulator works on smartphones with mid-price segment processors.