=Paper=
{{Paper
|id=Vol-2744/paper69
|storemode=property
|title=Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality
|pdfUrl=https://ceur-ws.org/Vol-2744/paper69.pdf
|volume=Vol-2744
|authors=Tatiana Tomchinskaya,Marina Shaposhnikova,Nikolay Dudakov
}}
==Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality==
Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality* Tatiana Tomchinskaya1 [0000-0002-4693-7478], Marina Shaposhnikova1 [0000-0002-8443-1005], Nikolay Dudakov1 [0000-0002-7844-6593] 1Nizhny Novgorod state technical university n.a. R.Е. Alexeev, Russia tomchinskaya@mail.ru,mar1900@yandex.ru, dudakovnu@gmail.com Abstract. A developed by authors application for real-time visualization of augmented and virtual reality is considered, which enables to simulate the traf- fic accidents on a mobile platform with the Android operating system. The ap- plication is developed on the basis of the Unity 3D and is able to visualize vari- ous types of road accidents and also link them to a terrain, different conditions of the urban environment, and map of the accident statistics. The system has an archive of videos for the accident 3D-scenes. The application interface and im- plementation features are considered. The subsystems of virtual and augmented reality are described in detail with examples. This application may be used to train novice drivers, learn the rules of the road, and inform experienced drivers about new road junctions. In the system environment, the distribution of the number of traffic violations depending on the driving experience for three groups (no experience, less than 3 years, more than 3 years) was investigated. The study of the usefulness of the system on the Likert scale on 80 cadets of driving schools is presented. Keywords: Road Traffic Accidents, Transport, Dynamic Tests, 3D-simulator, Mobile Platform, Traffic Rules, E-learning. 1 Introduction According to statistics, annually more than 1 million people over the world die in road traffic accidents and about 50 million people are injured. Only in 2019, 164358 road accidents were registered in the Russian Federation, as a result of which 16981 people died and 210877 were injured. In this case, the number of accidents committed due to traffic violations, has 146688, which killed 14420 people. Thus, it is clear that more than three quarters of all accidents are associated with violations of traffic rules. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons Li- cense Attribution 4.0 International (CC BY 4.0). * Publication financially supported by RFBR grant №18-08-01484 2 T. Tomchinskaya, M. Shaposhnikova, N. Dudakov Based on the statistics provided by the portal stat.gibdd.ru, it can be concluded that it is necessary to develop and adopt a set of measures to improve the skills and knowledge of traffic rules to ensure road safety. At the same time, studies conducted in different countries, for example, by the authors F. Wegman, H.-Y. Berg, I. Camer- on, C. Thompson, S. Siegrist, W. Weijermars in four jurisdictions - Western Austral- ia, the Netherlands, Sweden and Switzerland showed that the human factor is espe- cially important - a correct assessment of the traffic situation [1]. Noted that though “the past decades, road safety in highly-motorized countries has made significant progress”, however, the four cases (Western Australia, the Netherlands, Sweden and Switzerland) show that the fully evidence-driven road-safety management is too com- plicated at the moment. The authors of Nobuyuki Uchida, Maki Kawakoshi, Takashi Tagawa and Tsutogo Mochid especially noted incidents at intersections [2]. 2 Related works The solution of the problems considered is not limited to the development of the ur- ban road network. Various intelligent transport systems are offered. For example, in the works of Shaurya Agarwal, Pushkin Kachroo Emma Regentova [3], S. Agarwal, P. Kachroo, S. Contreras [4], an automatic system for the detection of road accidents is proposed which analyzes data and effectively predicts incidents with a high degree of confidence. To determine the factors associated with secondary accident likeli- hood, conditions such as traffic speed, duration of the primary accident, hourly vol- ume, rainfall intensity, and number of vehicles involved in the primary accident. Interesting research authors A. Paz, N. Veeramisti, R. Khaddar, H. de la Fuente- Mella and L. Modorcea [5], offering an architecture for interactive motion modeling. The proposed architecture combines several types of modeling, including motion- based modeling of driving, simulation of pedestrians, modeling of motorcycles and bicycles, and traffic modeling. Hanshin Expressway Company Limited considers the risk of an accident as one of the important concepts of its safe driving assistance and traffic management for the next generation. The company analyzes various driving environmental data that it has as a network operator and records of road traffic acci- dents in order to be able to predict road traffic incidents for given driving conditions. It is proposed to use information about the risks of accidents for a comfortable and safe driving, in particular, one of the "last trends" in the construction of transport sys- tems is offered. It is the technology of displaying objects in 3D format with the help of augmented reality on the windshield of the car. Applications of augmented reality are mainly intended for navigation, which helps drivers better navigate the road (https://newsroom.konicaminolta.eu). «Safe automobile driving requires drivers to process large amounts of dynamic information under time pressure. However, drivers can attend to only a small percentage of visual stimuli at once. To address this problem, i.e., to mitigate driving problems caused by excessive in- formation, it is proposed to work on making it easier to view navigation instructions without taking your eyes off the road» [6]. Such an application can be more useful for experienced drivers. Partially covers the windshield and the novice driver who does Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 3 not know how to "scan" the environment quickly enough cannot cope with the addi- tional flow of information. Therefore, during the research it was revealed that for the novice drivers and cadets of driving schools the most acute problem is the problem of correctly recognizing the road situation, in which there can be almost no time for making a decision. It is necessary to act as quickly as possible and, at the same time, take into account all aspects of the current situation, such as the signal of a traffic light, marking, signs, the position of cars and their speed. In addition, when you get into an extreme situation, the student may become confused, which can lead to even more deplorable results. At the same time, studies conducted in the work D. Sobel, K. Jędrasiak, K. Daniec, J. Wrona, P. Jurgaś and A.M. Nawrat [7], in particular, related to evaluating the effec- tiveness of AR to improve educational experience among abstract or technical courses showed excellent results. “Importance of the study is associated with research efforts performed to evaluate effectiveness of AR on improving learning experiences among abstract or technical courses. It has been tried to find out if usage of AR can give a rise to better learning experiences for especially difficult courses, and the intersection of both virtual and real environment can form a better way for learning rather than using other e-learning approaches. According to the findings obtained via evaluation works, the software system has shown effective and successful performance on im- proving learning experiences, and been accepted as better and more enjoyable way according to other e-learning approaches.” In this study, the solution to this problem can be the proposed application for a mobile platform intended, primarily for novice drivers and cadets driving schools. Two ways of ensuring road safety are considered: 1. Preview an unfamiliar path section using the augmented reality. Pointing the camera smartphone on a map of the city, for example, on Yandex map, the driver can see a realistic model of a specific section of the road infrastructure on the smartphone screen with all signs and markings. 2. By immersing in virtual reality, a driver can see in a stereo mode a living situa- tion of an accident that occurred on a particular section of a realistically modeled part of a particular city. View this situation from different points of view, like the victim, the culprit or the casual witness and understand all the reasons for this incident. And also for the cadets of driving schools, a system of dynamic tests is offered, performed on really existing sites with increased statistics of road accidents in specific cities. Subsequent paragraphs, however, are indented. 3 Proposed mobile application The most dangerous parts of the road network are considered. Since the application is aimed at obtaining traffic information, the following risk factors are studied as the contextual information [8]: speed, lateral interval, priority, distance; weather condi- tions: ice conditions, black ice, snow-covered road, snowfall, rain, fog; road surface. Each factor may be described by corresponding coefficient as below: Ki = OP/N, (1) 4 T. Tomchinskaya, M. Shaposhnikova, N. Dudakov where OP is the number of traffic accidents with this factor at a given location; N - average for all similar events in the city. All factors are modeling parameters. The most dangerous sections of the city roads, corresponding to each factor Ki (1), are identified, and dangerous road situations are modeled on these sections. Recorded video showing: 1) erroneous behavior of the driver and 2) how to properly act in a similar situation. A video bank has been prepared, which stores videos demonstrating traffic rules. 4 Interface The application interface contains two main modes of operation: Dynamic testing and View scenes (Figure 1). Fig. 1. Application main screen. When the first mode is selected, the marker tracking and snapping mechanism is ena- bled. In the process of pointing the camera of the smartphone to the Yandex map, this mechanism is allowed to see all possible scenes available for testing. To load a specif- ic scene, the user needs to put his hand on the virtual button “Loading a scene” while holding the smartphone so that the model remains visible on the screen. When you select View scenes, the application displays a list of all available scenes on the smartphone screen. The user can examine the selected scene in detail from various points of view and from different faces of road users. The application has two ways to pass the test: normal (control and navigation only using a smartphone) and using virtual reality glasses. 5 Augmented Reality Unit To implement the application on the Android platform, the integrated development environment of Unity 3D is used. For the traffic-accident areas appropriate scenes are designed to simulate typical accidents associated with this location. Models of scene’s participants are pre-created in the Autodesk Maya and 3DsMax tool environment as the low-polygon objects having trajectories parameterized relatively to the road, road- side objects and other traffic participants. These scenes are then embedded in the Unity3D environment (Figure 2). Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 5 Fig. 2. Images of low-polygonal three-dimensional scenes (bottom), which are examples of Object for recognizing via the Vuforia Object Scanner one of the city square from the positions marked by tags on the map (top). The augmented reality functionality was realized using Vuforia SDK (www.vuforia.com), the worlds widely used as AR platform for Unity 3D. Blocks of our application are united via the C# programming language. The augmented reality realized in the application via the Vuforia Object Scanner. Vuforia plug-in launches a mechanism for searching for tags, created in advance for the city map, and reproduces as the augmented reality a tooltip about the degree of danger or the correspondent accident 3D scenes for this part of the road. The scenes for the most dangerous areas can be also integrated onto the map via special labels for learning such places. The Vuforia plug-in recognizes the image of one of the label on the map received from the smartphone's camera, based on infor- mation about the position of the label (marker) in space. Fig. 3. Loading a Scene on a Smartphone by a Map fragment. 6 T. Tomchinskaya, M. Shaposhnikova, N. Dudakov For 3D reconstruction of the scene the calculation of the reference points with simul- taneous tracking is performed, thereby the objects of three-dimensional scene will be correctly oriented in the space. The application projects the corresponding model onto the label, and the scene of the accident case is loaded on the screen. The scene is also played in the third person view mode (Figure 3). 6 Virtual Reality Unit Block VR (virtual reality) is designed to view the model in virtual reality mode with- out using a label. From the list of suggested scenes, the user selects the model of in- terest. The screen of the smartphone or tablet is loaded with a scene, with the ability to view it from different angles and from different road users with full or partial im- mersion. For a full immersion, a virtual reality helmet is required. A virtual camera layer is added to the scene. The choice is made in favor of the product Fibrum SDK, because it has all the necessary settings, and it does not overload the system with additional plug-ins. Using the Fibrum SDK object, two cameras are installed, for each eye, re- spectively, the distance between the cameras, the distortion of the image and the re- quired cylindrical distortion. To view the virtual reality with full immersion requires a screen with a diagonal of 4 to 6 inches, as it is necessary to have in the phone sensors accelerometer and gyro. Fig. 4. Two Cameras for Stereo Imaging. To manage the view mode the standard internal plug-in Unity 3D - Characters Con- troller is used. When you download a scene on a smartphone in the virtual reality mode, you can view the situation from various road participants, whether the culprit, the victim, or the free travel in the scene. When the user is in the free travel mode, he Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 7 can look around, explore the terrain and the road infrastructure surrounding it, or nav- igate around the stage using the standard control joystick. Buttons for selecting the view mode are at the top of the screen. Back button - to exit the scene, to the main menu. In the virtual reality scene view mode using the virtual reality glasses (Figure 4), to switch to viewing a scene from another participant in traffic, the user must "look" at the button until the load bar appears. After loading it, within a few seconds, the pro- gram will redirect the user to the selected menu item. To select a view of the scene from another participant of the road, the user needs to "look" at the corresponding button on the screen. For a more complete immersion, objects inside the scene cast a shadow from static and dynamic light sources. 7 Dynamic Testing Unit During the studding in driving schools trainees are offered static illustrations in a book with tests that give the student time to think, against the dynamic situation on the road, in which time is very short for making a decision. Here, as an alternative, the system of dynamic tests, performed on really existing sites with increased statistics of road accidents on the example of a particular city, is proposed. The most dangerous parts of the road-transport interchange are taken as a basis [9]. As a result of the investigation of these areas and road accidents, a number of test tasks were compiled with situations in which accidents most often occur, as well as a number of complex road sections with which drivers have problems. On each site, several variants of tasks are considered, based on the real location of traffic signs, marking and possible obstacles (for example, other cars that violate the rules, so the driver may not notice the traffic sign). Fig. 5. Example of the Usual Test Strategy. Fig. 6. Example of the Dynamic Test Sys- tem. In the opposite of standard test (Figure 5), during the passage of the dynamic test, the view from the driver's seat is displayed on the monitor screen just before the learner (Figure 6). The test person can turn his head around, as it does in real life. 8 T. Tomchinskaya, M. Shaposhnikova, N. Dudakov The procedure for passing the test is as follows. When the user's machine ap- proaches the testing stage, for example, the intersection, the candidate is offered a choice of several action options. If the tested person chooses the correct action, then the score is recorded, and the machine continues to move to a predetermined position. If an incorrect action is se- lected, penalty points is recorded. In some cases, if an incorrect choice can lead to an accident, the scene of the accident is reproduced. During the one test, 20 tasks total are used, according to the results of which the fi- nal score is announced and tips are given on the incorrectly passed stages if any. The level of complexity of the task and the number of earned points is determined based on the complexity of the road segment via the integral factor Ki (like the level of dif- ficulty in a game) and determines the level of the driver's qualification. At each stage, from 1 to 5 possible outcomes are realized, where each outcome is a next task that is randomly loaded into the system, depending on the choice made earlier. There are two modes for the tests: a mono visualization mode, and a full immer- sion mode using a virtual reality helmet, or its light version - VR glasses. The application has several Check Point, the points of the beginning of the simula- tion of road traffic. After loading the character at this point, he is given the task to arrive at a Finish point. The user starts the movement. On the way to the Finish point, he encounters various obstacles and random events. The following factors are taken into account: the road surface, visibility, the time of movement of the character, the number of oncoming vehicles, and the discipline of the driver. If a character is disci- plined, then he moves from random points to other given points according to the rules of the road with different speed limits - from slow to normal. Characters "offenders" - clearly show what kind of behavior on the roads can cause an emergency or create interference on the road. They constitute 5% of all NPS (Non- Player Character - a character controlled by a non-player) involved in road traffic and at this stage are programmed for three main violations: speed limit (61-90 km/h), going beyond the stop line at intersections, rebuilding from strip to strip "game of checkers". If you violate the rules of the road, penalties are charged on the table as in the ex- am when you receive the rights to drive a car. The description of the error with decod- ing appears on the screen. At the end of the trip, the user can view all the violations and refresh their knowledge of the rules of the road. With the accumulation of five penalty points, the program stops the driving pro- cess, displays a list of errors, offers to re-read the rules of the road and pass the site. These events are generated randomly so that the user gets to a familiar site every time, but with different conditions. This factor keeps the user in constant attention, which in turn forms the skill of quick response in unforeseen situations. Examples of violations recorded by the program: Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 9 Fig. 7. Simulated accident. Collision with another vehicle, obstacle, or exit to the sidewalk. When colliding with third-party objects (Figure 7), the simulation stops, five points are added to the error register, and a corresponding message is displayed on the screen with an error. For simplicity, the calculation of collisions for each object has its own invisible contour. For example, for cars, it is a rectangle. Movement on a given lane. The program tracks the position and dimensions (spirit) of the car. If the user crosses the marking line or continues to move along it for five seconds without performing the appropriate actions, such as rebuilding, overtaking, returning to its own lane, then the user is charged with an error in the amount of one point. And the remark “Violated the rules for the location of the vehicle on the car- riageway” is added to the registry of errors. When a violation is repeated, the script repeats. The application is modeled tram tracks. When traveling on the opposite direction, the transparent plate (oncoming traffic plane) intersects the vehicle’s contour, the program stops the simulation, charges five points and displays the critical error “Took off the oncoming traffic (except when allowed) or on the tram tracks of the opposite direction”. By the same logic, the recognition of a double solid marking line is ar- ranged. Traffic at the prohibitory traffic light sign. During the red signal of the traffic light at the level of the stop line is an invisible plate, which has its own contour (Figure 8). When crossing the contours of the car and the transparent wall, the “violation” algo- rithm is launched. 5 points are added to the register, the error “I passed the prohibitory signal of the traffic light” or “Crossing the stop line” is displayed on the screen. Fig. 8 . Sign of an invisible plate connected with the stop line. 10 T. Tomchinskaya, M. Shaposhnikova, N. Dudakov Switching the colors of traffic signals is a sequential change in the properties of object materials (color, emission) in accordance with the operation of real traffic lights in the studied area of the city. The task of changing the color of traffic lights must be performed periodically, it is suggested to use the Unity coroutine for implementing traffic lights in order to receive updates regularly, but not every single [10]. When prohibiting road signs are installed on the road, the application establishes an invisible plate with an outline. When this contour is intersected, the simulation stops, an error of 5 points is recorded in the register, an error is displayed on the screen: “Did not fulfill the requirements of priority signs, prohibiting and prescriptive signs, road markings, as well as special prescription signs”. To control the digital model of the car requires a controller that communicates with the program. Currently, two control systems are popular: − buttons placed on a smartphone or VR glasses such as a magnet switch (its dis- advantage is incompatibility with smartphones that are not equipped with a magne- tometer) and a capacitive button (it works with most types of smartphones since it does not require any special sensor); − gamepad / controller (joystick), interacting with a smartphone using Bluetooth technology. The controller is chosen because it is universal and gives more opportuni- ties to the user and the application in its development. It does not restrict the user and gives a sense of freedom, as his hands are closer to the real position on the steering wheel, and not on the helmet of virtual reality. When the user is in the application menu, the joystick allows you to select the de- sired section of the system, and you can confirm the selection with the appropriate selection button. This allows you not to remove the virtual reality helmet from your head and not use the smartphone screen. At the time when the user is immersed in virtual reality, the digital car is controlled by an external joystick. Three groups of drivers with different driving experiences were formed. Each group consisted of twenty people. The main characteristics of dynamic tests have been verified by users of these three groups. The results are shown in Table 1. Table 1. The distribution of the traffic offenses number depending on the driving experience. No Driving experience Total responses for:* 1 2 3 4 5 1 Without experience 5 7 2 6 0 2 Up to 3 years 3 5 3 5 4 3 More than 3 years 1 4 3 3 9 * Traffic offenses: 1 → “Journey on forbidding signal of a traffic light”; 2 → “Driving on the wrong side of the road, unless there is an obstruction”; 3 → “Not giving priority to pe- destrians in pedestrians’ crossings”; 4 → “Noncompliance of vehicles drivers with the traf- fic signs and directions”; 5 → “There are no traffic offenses” It can be seen from the table that the greatest number of mistakes are made by inexpe- rienced drivers who have no experience in driving a car. Training Beginners and Experienced Drivers using mobile-based Virtual and Augmented Reality 11 Table 2. Survey statements and the received responses. No Statement Total responses for:* 1 2 3 4 5 “I enjoyed the educational process performed with the support 1 0 2 3 29 46 of this application.” 2 “It was difficult to use the tool-application.” 31 46 2 1 0 “With this application, it was more effective to learn the rules 3 0 0 4 67 9 of the road.” “I don’t want to take part again in such an educational pro- 4 5 70 3 2 0 cess.” “Materials (animations, videos…etc.) viewed on the tool – 5 0 0 5 31 44 application were attractive.” “My achievement level has been improved, thanks to this tool- 6 0 1 4 43 32 application.” 7 “This tool-application has enabled me to learn faster.” 0 2 3 33 42 “After viewing videos from the archive of videos of accidents 8 0 1 4 39 36 it becomes more successful to complete tasks ” * Likert Scale: 1→ “strongly disagree”; 2→ “disagree”; 3 → “no opinion”; 4→ “agree”; 5→ “strongly agree” The mobile software system was also offered to eighty students from four driving schools. At the end of the training cycle, they filled out a questionnaire to express your opinion on 8 statements about the educational process using an AR/VR-based e- learning tool. Table 2 presents the received survey responses on the Likert scale. Re- search has shown that the use of a dynamic testing system makes driving learning easier and more enjoyable. 8 Conclusion The application of virtual and augmented reality technologies for driver training is investigated. The mode of dynamic training and testing in the mode of virtual and augmented reality with various degrees of immersion is offered. To build scenarios for the creation of accident-dangerous cases, statistics, and situational descriptions of accidents were used. The application for a mobile platform was realized in the Unity 3D integrated de- velopment environment and intended to use in practical classes in driving schools. The reason to research is no possibility in driving schools to pass all difficult sites of roads and to disassemble all non-standard situations with the cadet. This research carried out confirms that virtual worlds and simulators improve learning outcomes [11],[12], but here on the example the driver training. The applica- tion is possible to significantly improve the training level of driving school cadets via more complete acquaint them with the complex situations that arise in specific places of the road infrastructure. It can also be applied when new complex traffic junctions are put into operation. For example, in Nizhny Novgorod, when the metro bridge was put into operation, many drivers did not use it due to difficulty, in their opinion, entrances the bridge both from one side of the river and on the other. Timely information of drivers via 12 T. Tomchinskaya, M. Shaposhnikova, N. Dudakov virtual reality will avoid many dangerous situations and, accordingly, will contribute to improving road safety. The pilot project of this training simulator works on smartphones with mid-price segment processors. References 1. Wegman, F., Berg, H.-Y., Cameron, I., Thompson, C., Siegrist, S., Weijermars, W.: Evi- dencebased and data-driven road safety management. IATSS Res. 39 (1), 19–25 (2015). 2. Uchida, N., Kawakoshi, M., Tagawa, T., Mochida T.: An investigation of factors contrib- uting to major crash types in Japan based on naturalistic driving data. IATSS Res. 34 (1), 22-30 (2010). 3. Agarwal, S., Kachroo, P., Regentova, E.: A hybrid model using logistic regression and wavelet transformation to detect traffic incidents. IATSS Research 40 (1), 56–63 (2016). 4. Agarwal, S., Kachroo, P., Contreras, S.: A dynamic network modeling-based approach for traffic observability problem. IEEE Trans. Intell. Transp. Syst 17(4), 1168-1178 (2016). 5. Paz, A., Veeramisti, N., Khaddar, R., de la Fuente-Mella, H., Modorcea L.: Traffic and Driving Simulator Based on Architecture of Interactive Motion. The Scientific World Journal 2015 (2015). 6. Abdi, L., Abdallah, F.B., Meddeb, A.: In-Vehicle Augmented Reality Traffic Information System: A New Type of Communication Between Driver and Vehicle. Procedia Computer Science 73, 242–249 (2015). 7. Sobel, D., Jędrasiak, K., Daniec, K., Wrona J., Jurgaś P., Nawrat, A. M.: Camera Calibra- tion for Tracked Vehicles Augmented Reality Applications. In: Innovative Control Sys- tems for Tracked Vehicle Platforms, pp. 147–162. Springer (2014). 8. Pundt, H.: Integration of context-information to support spatial decision systems. IADIS International Journal on Computer Science and Information Systems 12(2), 50-64 (2017). 9. Eliseev, M., Tomchinskaya, T., Galkina, E., Dudakov, N., Musarsky, R.: Virtual Reality and Navigation subsystems of the Interactive System for Road Safety Improving. IOP Conference Series: Materials Science and Engineering, 386 (2018). 10. Unity User Manual (2018.3), https://docs.unity3d.com/ru/2018.4/Manual, last accessed 2020/07/30. 11. Gallastegui, S.: Between immersion and emersion: orientating digital games towards virtu- al and physical spaces. IADIS International Journal on Computer Science and Information Systems 11(1), 49-62 (2016). 12. Dafoulas, G., Saleeb, N., Loomes, M.: What learners want from educational spaces? A framework for assessing impact of architectural decisions in virtual worlds. IADIS Interna- tional Journal on WWW/Internet 14(2), 72-90 (2017).