Benefits and Challenges of Smart Highways for the User Gesa Wiegand fortiss GmbH LMU Munich Munich, Germany wiegand@fortiss.org ABSTRACT This information can be used in different ways to benefit the dri- In order to improve the user experience on the highway, a smart ver. Many applications are not immediately apparent to the driver highway sensor system can be connected to the car and other driving on the highway. As an indirect beneficiary e.g. the highly devices. It can provide the driver with the fastest and safest journey automated vehicle of the driver uses the infrastructure information on the highway by improved route planning. This paper discusses to improve the driving performance. By transmitting information the potential benefits and challenges of the use of a connected from the infrastructure to the vehicle the sensor data of the vehi- highway sensor system. The provided information is based on a cle can be improved. The highly automated vehicle itself can then focus group of experts (N=9) in the scope of the project Providentia. send information to the infrastructure which adds to the data of Within this project, a sensor system is built on the highway to track infrastructure. A direct benefit would occur in case of the driver traffic objects. Different user groups that potentially benefit of a informing him/herself of the route by infrastructure sensors. A smart highway system are identified: Drivers of autonomous and first study was conducted to assess the improvement of the driving semi-autonomous vehicles, highway operators and teleoperators performance by informing the driver about the highway situation of autonomous cars. The main benefit for users is the redundant in a dangerous scenario [16]. In Figure 3 the driver of the car gets sensor system that provides a far-reaching view. This additional this visualization of the highway in an augmented representation information about the highway situation enables improved route within the cockpit. In both cases, the infrastructure data extends planning for connected vehicles. the knowledge of the highway situation. If a person is sitting in an autonomous car, this knowledge can be used to validate the sensor CCS CONCEPTS information of the autonomous car. In a manually steered car, the information can be used to warn the driver of traffic situations • Computing methodologies → Modeling and simulation; • Human- on the highway. One traffic situation is a fast braking car further centered computing → Visualization systems and tools; • Hard- ahead. If this car brakes right in front of you the vehicle sensors ware → Emerging technologies. would be able to detect the reduction of the velocity. In case of a braking car three cars ahead of the driver, the scenario would just KEYWORDS be detectable by connected vehicles or infrastructure sensors. Then Smart infrastructure; Intelligent highway; Sensor system the sensors need to transmit this information in order to shorten ACM Reference Format: the reaction and thus braking time of the driver. In this scenario, the Gesa Wiegand. 2019. Benefits and Challenges of Smart Highways for the traffic management can prevent a braking cascade by informing all User. In Joint Proceedings of the ACM IUI 2019 Workshops, Los Angeles, USA, vehicles and drivers of this scenario. If there is bad weather on the March 20, 2019. ACM, New York, NY, USA, 4 pages. highway the driver can be warned of an accident that is not yet well visible. These examples show the direct benefits of this system for 1 INTRODUCTION the driver. Nonetheless, you need an intelligent interface to warn Smart Infrastructure connects traffic lights with cars, for example, the driver and adjust the route or driving behavior of the vehicle to provide users with infrastructure information such as a red traffic accordingly. Within the project Providentia a smart voice assistant light. On the highway, sensors can collect information about the is implemented so people can already inform themselves about the vehicles or transmit information about the highway to the user. test bed. With that you can, for example, ask, how many vehicles One example is a smart highway system that collects information are on the route and which lane is best in order to get fastest to the about road usage to power on or power off lighting on the highway destination. This smart infrastructure holds challenges like the con- [14]. Within the project Providentia [7] a sensor system is built to struction and the costs thereof or the architecture of those systems. observe the traffic on the highway. One measurement point con- Nevertheless, there are several benefits that might not be directly sists of four radars and four cameras to cover far and near range in apparent to the user of smart infrastructure. both directions (see Figure 1 and Figure 2). Within Providentia a distance of approximately 2.2 kilometers of a highway is covered. 2 FOCUS GROUP As a result, the vehicles’ velocities and positions can be tracked. In the following the benefits and possible use cases of a sensor system on the highway, as identified in a focus groups (N = 9), is IUI Workshops’19, March 20, 2019, Los Angeles, USA presented. The focus group consists of sensor, infrastructure, and Copyright © 2019 for the individual papers by the papers’ authors. Copying permitted data fusion experts. A first brainstorming session was conducted by for private and academic purposes. This volume is published and copyrighted by its editors. all participants in order to identify user groups and use cases of the infrastructure system. This brainstorming session was realized by IUI Workshops’19, March 20, 2019, Los Angeles, USA Gesa Wiegand a more robust system that covers all weather conditions is used than the sensor systems of single vehicles [7]. • Early Warning In case of a highway sensor system that is distributed along several kilometers of the highway the driver can be warned early on of possible accidents or dangerous situations along Figure 1: Sensor setup on the highway. the route [16]. The infrastructure system has the advantage of having an overall overview of accurate information about the traffic on the highway. Today’s advanced driver assis- asking the participants of use cases and user groups that potentially tance systems (ADAS) detect objects and traffic situations benefit from a highway sensor system. In a second session, the in close proximity and warn the driver of braking situations participants were divided into three groups. Each group was told in close proximity of the ego vehicle. Providentia though is to brainstorm use cases for different user groups. One group, for able to detect braking scenarios that lead to brake cascades. example, thought of use cases for teleoperated drivers, one about A brake cascade originates from a fast braking car and can the driver and co-driver of a manually driven, highly automated an lead to traffic jams and accidents of following vehicles. An autonomous vehicle (SAE Level 2-5 [8]) and one about operators of early warning of fast braking cars can, therefore, result in the highway. smoother traffic flow and a faster and safer journey. The driver of an autonomous car benefits from smart infrastructure 3 BENEFITS OF SMART INFRASTRUCTURE rather indirectly as he/she does not distinguish between the sources FOR THE USER that the vehicle needs to drive. The sensor system on the highway The benefits that arise of the use of the smart infrastructure can is a redundant sensor system that can be used by the algorithm be provided to different user groups of the system. Those are: Dri- of the autonomous car to validate its own sensor information or vers of manual, semi-autonomous and autonomous cars (SAE Level extend the sensor information of the car. Autonomous trucks that 2-5 [8]), the operator of the highway and a teleoperator of an au- are connected to other trucks via sensors can drive in a platoon on tonomous car. the highway. Infrastructure sensors provide a way to add sensor information and control values to those platoons. By optimized traf- 3.1 Driver of manual, semi-autonomous and fic flow, those platoons can get information about the route ahead autonomous cars to improve their route planning. Current projects that research Manually driven vehicles do not have much sensor information but vehicle platooning have the goal to maintain a fixed gap between can benefit from infrastructure information by brought-in devices. vehicles or perform evasive maneuvers such as emergency braking In case of the driver using his/her phone for navigational infor- [1]. mation, infrastructure information can be provided to the driver. Warnings of dangerous traffic situations or route information can 3.2 Operator of the highway be transmitted. The driver of a semi-autonomous car can be warned The operator of the highway needs to maintain the highway, know of different traffic scenarios that can be classified by the information about damages and accidents in order to redirect the traffic flow. provided by the sensor system. Following incomplete list includes The additional information collected by smart infrastructure can some examples of traffic scenarios: be transmitted to the operator in order to improve traffic planning. • Standing Vehicle Emergency vehicles can be supervised from far in order to find In case of a vehicle that breaks down on the highway, the the best and fastest way to an accident. Currently, the traffic on other cars on the highway can be warned that a vehicle the highway is directed manually by changing speed limitations stands on the highway. or by indicating that the emergency lane can be used by vehicles. • Ghost driver Traffic flow management could be enabled by smart infrastructure In case of a driver that mistakenly enters the highway in the by predicting the vehicles trajectories and behavior. By providing wrong way, the sensor system can identify this vehicle and the operator with an accurate traffic density on the road, the average warn other vehicles on the highway of the ghost driver. speed of the vehicles and predicted maneuvers part of the traffic • Bad weather flow control could be automated. The attention of the driver can be During bad weather, the orientation on the highway might directed to critical situations on the highway and as a consequence be difficult for the driver. In this case, the surrounding traffic make the reaction time faster. and the distance between vehicles can be provided to the user. By combining sensor data from the vehicles on the road 3.3 Teleoperator of an autonomous car and sensor data from Providentia, the accuracy of detected Autonomous cars will likely face limits in operability in certain objects could be increased. The cameras that use a deep situations in which their sensors break down or there is not enough learning based object detection approach and the radars that environment information to securely operate the car. In such situa- provide object detection fuse the data in a data fusion unit. tions, a teleoperator might be able to steer the car to its destination By combining the strengths of the sensors (cameras are good [15]. By providing the teleoperator information about the surround- classification sensors, radars determine velocities and angles) ings of the car he/she is able to steer the car from the distance for Benefits and Challenges of Smart Highways for the User IUI Workshops’19, March 20, 2019, Los Angeles, USA could communicate to the driver that the fog will probably lift itself before the vehicle arrives at the scene. It is imaginable that the user interface needs to inform the driver of out-of-view scenarios on the highway in case of more infrastructure information. 4.2 Operator of the highway Operators of highways observe the highway and control it in a limited way. The velocity of the traffic can sometimes be regulated and the emergency lane can be blocked or authorized for traffic 1 . The design of control rooms consists of several working desks that have screens to show control applications or the videos of the highway [9] [11]. The challenges of large screen applications are the loss of orientation on large screens. Looking for the mouse cur- sor on large displays creates high physical demand. One possibility of improving control operators input techniques is eye tracking as suggested by Lischke et al. [11]. Some control rooms, such as Figure 2: Picture of radars on the highway tracking the traf- nuclear power industries, have a strong focus on safety and per- fic. formance of operators [13]. Therefore the control rooms need to be designed in a way that operators have no spatial constraint and have fast interaction possibilities. In the automotive domain espe- example by joystick [12]. Even if the sensors of the vehicle itself cially tunnels need to be observed and in case of an emergency need do not work, the surroundings of the car are monitored by infras- to be closed for incoming traffic right away. Therefore the operator tructure sensors and therefore do not face the same limitations as needs the information of dangerous scenarios on the highway right the car sensors. Through this redundant interface, the lost informa- away. Then emergency vehicles can be informed and traffic can be tion of the car can be replaced. The visualization of the highway controlled. prevents situations in which the occupants of the car are not able to continue their journey. 4.3 Teleoperator of an autonomous car Teleoperation is needed in several contexts, e.g. drone control, robot 4 INTERFACE DESIGN control in space operations or medicine. In automotive vehicles, the In an interface that provides the user with information about the design of teleoperation interfaces is under research [4]. Teleoper- infrastructure, more information can be communicated. Therefore ated driving requires a network that has a high uplink data rate and the existing design spaces for driver-based automotive user inter- a much lower downlink data rate according to Boban et al. [3]. To faces [10] and windshield applications [6] need to be extended to get an experience similar to that of a regular driver of the car, sev- consider design dimensions covering use cases of infrastructure eral sensors (two or more cameras and other sensor information) information. need to transmit their information to the teleoperator interface. Berggren et al. test a teleoperated bus on a test bed [2]. To transmit 4.1 Driver of a semi-autonomous and the environment information and steering relevant information of autonomous car the bus to the teleoperator, a driving simulator interface is used. The operator sees a camera image of the cockpit of the bus on a Even though design spaces for windshield applications and driver- screen in front of him/her. The input control can be manipulated based automotive user interfaces exist [10][6], there is not yet a by a steering wheel that is handled by the remote operator. The design space for a highly automated vehicles. To put an emphasis bus speeds up to 20 km/h. In another study by Georg et al. [5] on infrastructure interaction, the design dimensions need to be teleoperated driving with head-mounted displays is compared with adjusted according to the input of the infrastructure information. teleoperated driving with conventional computer screens. Even As in theory the information of the whole length of the highway though they did not find significant differences between the two can be detected, scenario information also needs to be accompanied output modalities, the participants thought the top down view of by distance information. If the scenario is close by, the notification the vehicle surroundings was helpful. With Providentia this top- modality needs to differ from the modality of notifications of sce- down view and all other visualization angles could be realized in narios in a far distance. Otherwise, the occupant of the vehicle can teleoperated driving. Situation awareness would then increase if not estimate if immediate action needs to be taken or if the scenario the whole environment would be visualized. might change over time. Predicting whether a scenario will change in time for the vehicle to arrive at that predicted point on the high- 5 CHALLENGES way could help the driver make an informed decision about the route to take. In case of an accident, the algorithm could calculate, Even though the benefits of the system promise great potential that it might take some time to clear the highway. Therefore the in the future of autonomous driving there are some challenges prediction could be very certain that the driver will lose time on the 1 http://www.stmb.bayern.de/vum/strasse/verkehrsmanagement/verkehrssteuerung/index.php, route. In case of fog in the morning in the far distance the prediction Accessed: 2019-02-13 IUI Workshops’19, March 20, 2019, Los Angeles, USA Gesa Wiegand REFERENCES Figure 3: Representation of the surroundings of the highway [1] Carl Bergenhem, Steven Shladover, Erik Coelingh, Christoffer Englund, and within the cockpit [16]. Sadayuki Tsugawa. 2012. Overview of platooning systems. In Proceedings of the 19th ITS World Congress, Oct 22-26, Vienna, Austria (2012). [2] Viktor Berggren, Aneta Vulgarakis, Athanasios Karapantelakis, Elena Fersman, Keven Wang, Leonid Mokrushin, Nicolas Schrammar, and Rafia Inam. 2019. 5G teleoperated vehicles for future public transport. https://www.ericsson.com/en/ that need to be addressed. In order to use the system as redundant blog/2017/6/5g-teleoperated-vehicles-for-future-public-transport. Accessed: sensor system for a teleoperator, the system needs to be spread to 2019-02-14. the whole length of the considered highway. Teleoperated driving [3] Mate Boban, Apostolos Kousaridas, Konstantinos Manolakis, Josef Eichinger, and Wen Xu. 2018. Connected roads of the future: Use cases, requirements, and is preferably latency free, therefore a minimal delay between the design considerations for vehicle-to-everything communications. ieee vehicular steering decision of the teleoperator and the movement of the car technology magazine 13, 3 (2018), 110–123. [4] Martijn Bout, Anna Pernestål Brenden, Maria Klingegård, Azra Habibovic, and would be desirable [5]. The user might not have transparency of Marc-Philipp Böckle. 2017. A Head-Mounted Display to Support Teleoperations the used system that is in control of his/her vehicle. In order to of Shared Automated Vehicles. In Proceedings of the 9th International Conference regain control of the vehicle this information might be necessary on Automotive User Interfaces and Interactive Vehicular Applications Adjunct. ACM, 62–66. though. Therefore, a clear and transparent communication of the [5] Jean-Michael Georg, Johannes Feiler, Frank Diermeyer, and Markus Lienkamp. used systems is necessary to gain trust of the user. In order to pre- 2018. Teleoperated Driving, a Key Technology for Automated Driving? Com- vent others to harmfully interact with the system, safety is also an parison of Actual Test Drives with a Head Mounted Display and Conventional Monitors. In 2018 21st International Conference on Intelligent Transportation Sys- important topic that needs to be addressed. If wrongful information tems (ITSC). IEEE, 3403–3408. gets distributed the algorithm of the car might make a decision that [6] Renate Haeuslschmid, Bastian Pfleging, and Florian Alt. 2016. A design space to support the development of windshield applications for the car. In Proceedings of might result in an accident or undesired behavior. Therefore, the the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5076– accuracy of the sensors and the classified and identified vehicles on 5091. the road must be high enough in order to make accurate steering [7] Gereon Hinz, Josef Eichinger, Martin Büchel, and Alois Knoll. 2017. Proactive Video-based Use of Telecommunications Technologies in Innovative Motorway decisions. Scenarios. 5th GI/ITG KuVS Fachgespräch Inter-Vehicle Communication (2017), 41. [8] SAE International. 2016. Taxonomy and definitions for terms related to driving 6 CONCLUSION automation systems for on-road motor vehicles. Smart infrastructure provides great potential for users of the road. [9] Toni Ivergard and Brian Hunt. 2008. Handbook of control room design and er- gonomics: a perspective for the future. CRC Press. Facing new technology such as autonomous cars, connected cars, [10] Dagmar Kern and Albrecht Schmidt. 2009. Design space for driver-based au- brought-in devices or platooning the users of the highway will tomotive user interfaces. In Proceedings of the 1st International Conference on have different levels of autonomy. Therefore a perfectly controlled Automotive User Interfaces and Interactive Vehicular Applications. ACM, 3–10. [11] Lars Lischke, Sven Mayer, Andreas Preikschat, Markus Schweizer, Ba Vu, Paweł W traffic flow is not possible. By providing all users with information WoŹniak, and Niels Henze. 2018. Understanding Large Display Environments: independent of their vehicles’ abilities, a shared knowledge base is Contextual Inquiry in a Control Room. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW134. created. This knowledge base can be retrieved by the system itself [12] Chang Liu, Hong Yu Zheng, and Chang Fu Zong. 2013. Study on Control Strategy but also by the driver on the highway. A brought-in device such of Steer-by-Wire Vehicle with Joystick. In Advanced Materials Research, Vol. 694. as a phone or tablet, for example, can retrieve information like the Trans Tech Publ, 2061–2064. [13] Michael N Louka, Morten A Gustavsen, and Svein Tore Edvardsen. 2006. Using accurate traffic situation on the highway. This provides the user of virtual reality to support multi-participant human-centered design processes the infrastructure system with more traffic information in order to for control room design. In Proceedings of 5th international topical meeting on improve the safety of the drivers on the road. nuclear plant instrumentation, controls, and human machine interface technology (NPIC&HMIT 2006) at the American Nuclear Society 2006 meeting, November. American Nuclear Society, Albuquerque. 7 ACKNOWLEDGEMENT [14] Ahmad M Mustafa, Omar M Abubakr, Ahmed H Derbala, Essam Ahmed, and Bassem Mokhtar. 2017. Towards a Smart Highway Lighting System Based on This research has been funded by the Federal Ministry of Transport Road Occupancy: Model Design and Simulation. In Applications for Future Internet. and Digital Infrastructure in the project Providentia. I am partic- Springer, 22–31. [15] Tito Tang, Frederic Chucholowski, and Markus Lienkamp. 2014. Teleoperated ularly grateful for the assistance given by the Providentia Team, driving basics and system design. ATZ worldwide 116, 2 (2014), 16–19. namely Vincent Aravantinos, Maida Bakovic, Markus Bonk, Mar- [16] Gesa Wiegand, Christian Mai, Yuanting Liu, and Heinrich Hußmann. 2018. Early Take-Over Preparation in Stereoscopic 3D. In Adjunct Proceedings of the 10th tin Büchel, Dhiraj Gulati, Gereon Hinz, Annkathrin Krämmer, Juri International Conference on Automotive User Interfaces and Interactive Vehicular Kuhn, Venkat Lakshmi, Daniel Malovetz, Philipp Quentin, Maxim- Applications. ACM, 142–146. ilian Schnettler and Christoph Schöller and their contribution to this paper.