Outdoors Mobile Augmented Reality for Coastal Erosion Visualization Based on Geographical Data Minas Katsiokalis Lemonia Ragia Katerina Mania mkatsiokalis@isc.tuc.gr lemonia.ragia@gmail.com amania@isc.tuc.gr Technical University of Crete Athena Research Innovation Center Technical University of Crete in Information Communication Figure 1: The real-world coast (left), the real-world coast with seamless 3D sea segment (right). ABSTRACT KEYWORDS This paper presents a Mobile Augmented Reality (MAR) system Augmented Reality, Mobile AR, Coastal Erosion, Outdoors AR, Mo- for coastal erosion visualization based on geographical data. The bile Application, Landscape Visualization system is demonstrated at the beach of Georgioupoli in Chania, Reference Format: Crete Grece, in challenging, sunny, outdoors conditions. The main Minas Katsiokalis, Lemonia Ragia, and Katerina Mania. 2020. Outdoors focus of this work is the 3D on-site visualization of the future Mobile Augmented Reality for Coastal Erosion Visualization Based on Geo- state of the beach when the shoreline will inevitably progresses graphical Data. In Cross-Reality (XR) Interaction, ACM ISS 2020 (International in-land based on the impact of severe coastal erosion, taking place Workshop on XR Interaction 2020). across the Mediterranean sea but also worldwide. We feature two future scenarios in three locations of the beach. A 3D sea segment is matched to the user’s actual position. The visualization as seen CROSS-REALITY INTERACTION through a smartphone’s screen presents an unprecedented seamless Reality and Virtuality are the two edges of a continuum, the one view of the 3D sea segment joined with the real-world edge of the describes the physical-real world and the other a non physical- sea, achieving accurate registration of the 3D segment with the virtual environment. Cross-Reality is the bridging of those two real-world. Position tracking is performed by utilizing the phone’s worlds and stands between the two edges. In this presented work, GPS and the computer vision capabilities of the presented AR frame- we introduce a cross-reality interface where the users can inter- work. A location-aware experience ensures that 3D rendering is act with a virtual environment through their smartphones while space-aware and timely according the user’s position at the coast. they reside on the real location of the environment and witness By combining AR technologies with geo-spatial data we aim to the scenery to change. The virtual world combines with the real motivate public awareness and action in relation to critical envi- one to provide a cross-reality experience where the user is able to ronmental phenomena such as coastal erosion. view the future of a specic coast area using Augmented Reality technology. The virtual content that presented alongside with the CCS CONCEPTS real scenery depicts the future state of the coast and manages to • Computing methodologies  Mixed / augmented reality ; • visualize the coastline changes on top of the physical world. The Applied computing  Interactive learning environments; Envi- virtual environment enhances visually the real world in real time, ronmental sciences. while the user can interact on both of them. Achieving to provide Copyright © 2020 for this paper by its authors. Use permied under Creative virtual content without extinguish the real-world factor, is essential Commons License Aribution 4.0 International (CC BY 4.0). on cross-reality interaction. Cross-Reality (XR) Interaction, ACM ISS 2020, November 8 2020, Lisbon, Portugal International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal Katsiokalis, et al. 1 INTRODUCTION Mobile Augmented Reality (MAR) is an open research area due to the emergence and widespread uptake of smart-phones that pro- vide powerful platforms for supporting Augmented Reality on a mobile device. Littering behavior is a global issue a�ecting most countries, regardless of their development status. Despite the wider applications of MAR in areas such as cultural heritage [8], [6], [2] and shopping [4], MAR systems of environmental context are still rare because of technical challenges when AR is occurring out- doors and the need for reliable environmental and geo-located data. Standard 3D simulation has been employed in the past to high- light environmental issues such as the impact of tsunami waves [5]. However, MAR technology on-location could raise environmental awareness and provoke environmental action compared to meth- Figure 2: MAR system architecture. ods such as radios, maps and handheld displays [3], [1]. Landscape visualization can be particularly e�ective when communicating future changes to community groups and policymakers [10]. The 2 IMPLEMENTATION visualization of potential environmental changes is a powerful tool for public understanding. MAR can o�er the ability to experience The navigation scene was developed using Mapbox SDK for Unity3D, future changes of the environment as if happening now, with the suitable for building systems from real-world map data, enabling potential to provoke shock and even disbelief. interaction with Mapbox web services APIs (Maps, Geocoding and The phenomenon this paper focuses on, is the erosion of the Directions APIs) via a C#-based API. Having access to a device’s coastal zone and the tremendous changes of the coastline. Across GPS, an area map is loaded based on the user’s geo-location. Points- the Mediterranean sea, coastal erosion will increase provoking of-interest (POIs) are added guiding the user to where the AR visu- disastrous outcomes for the regions. The coastline is the physical alization takes place. The Directions API provides directions from line where the land meets the sea. Nowadays, coastline extraction the user’s geo-location to the POI’s geo-location while a script and tracking of its changes has become of high importance because calculates the distance between them at every second. of global warming and rapid growth of population. Our goal was Planar faces in the real world (the ground, walls etc.) were recog- to visualize the shoreline at a speci�c spot in Crete, Greece, in nized based on plane detection. The user’s position and orientation its near future state, on the basis of mathematical models that in physical space were tracked by the smartphone’s motion track- showcase the beach retreat prediction, e.g. the tendency of the ing. Then, the virtual content appeared on top of the recognized beach to erode, without any human corrective measures, enhancing physical world. In order to enable Unity3D’s AR Foundation’s func- public awareness. tionality, an ’AR Session’ component controlled the life-cycle and We propose a MAR system for the visualization of coastal ero- con�guration options for the AR session and an ’AR Session Origin’ sion on-site (Fig. 1), putting forward successful location-aware component represented the device. The user viewed and interacted recognition outdoors, based on geo-referenced spatial location data, with the 3D scene using the GameObject that contains this compo- addressing MAR challenges such as occlusions, large variations in nent. Rotation or movements of this object represents rotation and lighting, the impossibility of modifying the environment, as well movement of the user in the scene. The ’AR Camera’ GameObject as unpredictable weather conditions and pollution. represents what the user sees rendered through the camera. The proposed work is based an on-site Mobile Augmented Reality Attached to the ’AR Session Origin’ object, there were an ’AR (MAR) system for the 3D on-site visualization of the future shoreline Plane Manager’ and an ’AR Point Cloud Manager’ component. The when it will inevitably progresses in-land , o�ering a seamless view ’AR Plane Manager’ selects the data about the scanned planar sur- of the 3D sea segment joined with the real-world edge of the sea faces adding each plane that has been detected into a list, creating (Fig. 1). The system is designed to operate in challenging outdoors, a GameObject for it. The ’AR Point Cloud Manager’ collects data sunny conditions, using a consumer’s handheld smartphone or about feature points scanned by the device and a point cloud is cre- tablet supporting both Android and iOS devices. The MAR system ated for depth recognition and tracking to the app. The interaction presented consists of two main phases (Fig. 2). The �rst phase guides with the real-world was achieved by ray-casting the tracked planes. the user’s navigation on the beach. Once at one of the set Points A ray is sent from the center of the ’AR Camera’. If a tracked plane of Interest (PoIs), the second phase allows the user to experience is hit by it, then we are able to interact with this speci�c plane. Here, the visualization, after a brief process of calibration as shown in this method tracks a plane surface (mostly the ground) and shows steps 1-4 in Fig. 3). Over three PoIs, the MAR system visualizes two an indicator at the point where the ray hits the plane, updated in possible future scenarios of the coastline for sea level elevation by every frame following the device’s pose. If there is no tracked plane, 0.5 meters and 1 meter, where the coastline is estimated to penetrate the indicator disappears. While the indicator is active, the ’Place 3.6 and 7.7 meters inland, respectively (Fig. 4). Here’ button is activated and the user places the virtual sea content at the pointed direction, aligned with the real shoreline. The virtual content is instantiated at the orientation of the indicator (steps 2,3 Fig 3) and the virtual shoreline is placed where the indicator points Mobile Augmented Reality for Coastal Erosion Visualization International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal Figure 4: Shoreline retreat scenarios, SLR 0.5m/1m. A terrain was created based on the shape of the beach at the speci�c location [9]. Using the terrain, interaction with the water was possible corresponding to the shoreline. The aerial image used is a real sub-scale map exported with GIS software showing the beach retreat (Fig. 4). The model that accurately extracts the beach retreat prediction at the speci�c shoreline, with characteristics of low slope and sediment of sand is (1) representing the low mean of the beach retreat prediction [7]. ( = 0.05U 2 + 8.12U 0.46 (1) U: Sea Level Rise (SLR) in meters Figure 3: Process of placing the 3D sea extending the real- The image (Fig. 4) is geo-referenced to the Hellenic Geodetic world sea coastline (as seen on smartphone’s camera . Reference System 1987 (HGRS87) and is showcasing the retreat of the beach in two scenarios (SLR = 0.5 and SLR = 1). The image was imported in Unity as 1:1 scale so 1 unit in the engine corresponds to 1 meter in the real world. For each possible scenario, an elevation to. The user relocates the virtual content if the match with the real layer was created as well as one corresponding to the current state world is o� by pressing the setting button (step 3 of Fig. 3). An of the shoreline. A realistic looking water shader was created and intuitive user interface (UI) helps navigation (see Fig. 3). During attached to a planar surface acquiring water properties such as step 4 of Fig. 3, three UI signs are enabled by pressing the ’info re�ection, transparency, waviness, foam e�ects on collision etc. button’ at the top right of the screen. A ray starts from the touched In order to animate the rising of the water, the lerping method position of the sign on screen. If the ray hits a sign, a window pops gradually moves an object from one position to another during a up providing information about coastal erosion (Fig. 5). time window at a given speed. International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal Katsiokalis, et al. coastal erosion, bridging the gap between reality and virtuality of widely available XR technologies. REFERENCES [1] Majed Abdullah Alrowaily and Manolya Kavakli. 2018. Mobile Augmented Reality for Environmental Awareness: A Technology Acceptance Study. In Proceedings of the 2018 10th International Conference on Computer and Automation Engineering (Brisbane, Australia) (ICCAE 2018). Association for Computing Machinery, New York, NY, USA, 36–43. https://doi.org/10.1145/3192975.3193002 [2] Silvia Blanco-Pons, Berta Carrión-Ruiz, Michelle Duong, Joshua Chartrand, Stephen Fai, and José Lerma. 2019. Augmented Reality Markerless Multi-Image Outdoor Tracking System for the Historical Buildings on Parliament Hill. Sus- tainability 11 (08 2019), 4268. https://doi.org/10.3390/su11164268 [3] Te-Lien Chou and Lih-Juan ChanLin. 2012. Augmented Reality Smartphone Environment Orientation Application: A Case Study of the Fu-Jen University Mobile Campus Touring System. Procedia - Social and Behavioral Sciences 46 (2012), 410 – 416. https://doi.org/10.1016/j.sbspro.2012.05.132 4th WORLD CONFERENCE ON EDUCATIONAL SCIENCES (WCES-2012) 02-05 February 2012 Barcelona, Spain. [4] Scott G. Dacko. 2017. Enabling smart retail settings via mobile augmented reality shopping apps. Technological Forecasting and Social Change 124 (2017), 243 – 256. https://doi.org/10.1016/j.techfore.2016.09.032 [5] Alexandros Giannakidis, Giannis Giakoumidakis, and Katerina Mania. 2014. 3D photorealistic scienti�c visualization of tsunami waves and sea level rise. In 2014 IEEE International Conference on Imaging Systems and Techniques (IST) Proceedings. IEEE, 167–172. [6] A. Haugstvedt and J. Krogstie. 2012. Mobile augmented reality for cultural heritage: A technology acceptance study. In 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 247–255. [7] I.N. Monioudi, A. Karditsa, A. Chatzipavlis, G. Alexandrakis, O.P. Andreadis, A.F. Velegrakis, S.E. Poulos, G. Ghionis, S. Petrakis, D. Sifnioti, T. Hasiotis, M. Lipakis, N. Kampanis, T. Karambas, and E. Marinos. 2014. Assessment of vulnerability of the eastern Cretan beaches (Greece) to sea level rise. Regional Environmental Change (2014). https://doi.org/10.1007/s10113-014-0730-9 cited By 0; Article in Figure 5: Information signs in AR. Press. [8] Chris Panou, Lemonia Ragia, Despoina Dimelli, and Katerina Mania. 2018. An architecture for mobile outdoors augmented reality for cultural heritage. ISPRS International Journal of Geo-Information 7, 12 (2018), 463. 3 EVALUATION AND CONCLUSIONS [9] Lemonia Ragia and Pavlos Krassakis. 2019. Monitoring the changes of the coastal areas using remote sensing data and geographic information systems. In Seventh We received feedback about the functionality of the application International Conference on Remote Sensing and Geoinformation of the Environment and its usefulness, involving users at the beach, using the think (RSCy2019), Vol. 11174. International Society for Optics and Photonics, SPIE, 289 – 297. https://doi.org/10.1117/12.2533659 aloud usability evaluation methodology. Users involved were either [10] Stephen Sheppard. 2005. Landscape visualisation and climate change: The poten- non-experts with AR technology or experts in AR. The non-expert tial for in�uencing perceptions and behaviour. Environmental Science Policy 8 users were fascinated while the experts focused on the function- (12 2005), 637–654. https://doi.org/10.1016/j.envsci.2005.08.002 ality. Certain users mentioned that they would prefer an AR head mounted experience while others had no problem with the use of the smartphone. Initially, users found it challenging to accurately place the virtual content depending on the location, mostly due to the physiology of the area. After training, they easily used the app and navigated around. We received enthusiastic feedback con- cerning the photorealistic 3D water and its seamless integration with the real-world shoreline. The signs and the UI in general were simple and intuitive, communicating the impact of coastal erosion. Certain users preferred the UI to be assigned more visible colours. The application was hard to use during bad weather (clouds, wind etc.). When the sea was wavy, it was hard to accurately anchor the 3D content as it was drifting in the scene. Tracking and registration in AR are far from solved. Future work could automate shoreline detection, exempting user from the calibration process. Addition of more scenarios and locations. Lighting of the AR digital content can be improved for a stronger feeling of depth and better photo- realism. Concluding, we showcased the design of a mobile Augmented Reality application aimed for consumer-grade mobile phones with the ultimate goal of increasing the environmental awareness of the public audience. By employing AR, we enhance user awareness of