Mixed Reality for Archeology and Cultural Heritage Paolo Fogliaroni Vienna University of Technology, Austria paolo.fogliaroni@geo.tuwien.ac.at https://orcid.org/0000-0002-0578-8904 Abstract Archeological and cultural heritage data may also consist of a 3D geospatial component, which for certain scenarios, is better consumed by means of immersive technologies such as Augmented, Mixed, and Virtual Reality. This short paper provides an introduction to the different types of virtual environments and discusses possible application scenarios within the archeological and cultural heritage domain. Also, the limits of current technologies are presented and the challenge of integrating Augmented and Mixed Reality technologies into the geospatial domain is discussed. Finally, a new set of outdoor MR application scenarios is envisioned. 2012 ACM Subject Classification Human computer interaction (HCI), Interaction paradigms, Mixed / augmented reality Keywords and phrases Augmented Reality, Mixed Reality, Spatial HCI Digital Object Identifier 10.4230/LIPIcs.COARCH.2018. Category Invited paper 1 Introduction Mixed reality (MR) is a Human-Computer Interaction (HCI) environment blending together the real world and virtual (i.e., digital) elements. Milgram and Kishino [11] describe it as an HCI environment placed anywhere in the so-called virtuality continuum (see Figure 1), that goes from the completely real to the completely virtual environment. According to this taxonomy, Augmented Reality (AR) and Augmented Virtuality (AV) are special cases of MR, with the major difference among them lying in the ratio of real to virtual contents. More specifically, AR superimposes virtual content on the real surroundings of the user while AV brings some real objects into a virtual world. As of today, the terms AR and MR found their way into the general public terminology, while AV remained circumscribed to a more technical and specialized domain. Mixed Reality Real Augmented Augmented Virtual Environment Reality Virtuality Environment Figure 1 The virtuality continuum stretches from completely real environments to totally virtual ones. Image adapted from [11]. © Paolo Fogliaroni; 2nd Workshop on Computing Techniques for Spatio-Temporal Data in Archaeology and Cultural Heritage. Editors: Pierre Hallot, Sara Migliorini, Alberto Belussi, and Roland Billen 13 MR for Archeology and Cultural Heritage (a) Physical Reality (b) Augmented Reality (c) Mixed Reality (d) Virtual Reality Figure 2 A representation of the Colosseum (Rome, Italy) in four different nuances of reality. Physical reality (a) can be directly experienced through sensory-motor abilities or via conventional video captures. In Augmented Reality (AR) environments (b) real-world features are enriched through the overlay of digital information. In Mixed Reality (MR) environments (c) virtual objects interact with the real ones. Both AR and MR experiences can be enjoyed through either conventional or head-mounted see-through displays. Virtual Reality (VR) environments (d) present a completely digital world that can be experienced through either traditional or head-mounted displays. According to Azuma [3] AR environments should expose the following characteristics: combine real and virtual: virtual and real objects are blended together into a unique experience; interactive in real time: the user is to be able to interact with the augmentation either via conventional controllers or more advanced types of interactions (e.g., gaze-based [4, 7] or hand-based [10] interactions); registered in 3D: the virtual and real objects have to be precisely aligned in real-time in order to provide a seamless experience. Arguably, AR is a special case of MR where the blending of virtual objects into the real world is mainly about the enrichment of the real elements by means of an overlay of digital content. Conversely, in MR environments the goal is to create a seamless integration of real and virtual objects, with the latter (at least) visually behaving as real items—e.g., they occlude and are occluded by other real and virtual elements. In MR, the virtual objects can or cannot obey physical rules, allowing for creating a variety of different experiences ranging from environments where the reality is enriched with virtual objects in a much natural way, to fantastic environments where the virtual objects escape one or more physical law—e.g., floating virtual objects that are not constrained to gravity. Figure 2 provides an example that illustrates how the same subject (the Colosseum) could be represented in different nuances of reality (physical, augmented, mixed, and virtual reality). 14 P. Fogliaroni This paper provides an introduction to the main concepts underlying current AR/MR technology and discusses its application in the archeology and cultural heritage domain. Also, the limitations of current technology are highlighted and it is argued that AR/MR technology would benefit from an integration with the Geographic Information Science (GIS) domain. Finally, under the assumption that this integration is realized, novel applications for the visualization and interaction with archeological and cultural heritage data are envisioned. In the remainder of this work we mainly focus on Mixed Reality (that, as discussed above, is a generalization of AR), sometimes referring to it as Holographic Experience. 2 Virtual Experiences in the Archeology and Cultural Heritage The application of Virtual Reality (VR) and Mixed Reality (MR) to the archeological and cultural heritage domain is not new and has been already investigated in the past. For example, Abbott [1] applies VR for the visualization and analysis of Stonehenge; Gaitatzes et al. [5] implement a virtual journey through a digital reconstruction of the city of Miletus (among other ancient cities); Ledermann and Schmalstieg [9] utilize a volumetric display to visualize the Heidentor ruin (located in Carnuntum, Austria) and a superimposition of the missing part of the construction. Similar projects and research endeavors have been also focused on the application of Augmented and Mixed Reality technologies. For example, the ARCHEOGUIDE project [12] led to the implementation of a system providing on-site personalized guide and an AR visualization of the digital reconstruction of ancient ruins; Kretschmer et al. [8] also developed a mobile augmented reality system for the historic city of Heidelberg that provides story-telling as the user moves within the historical site; Hall et al. [6] investigate the effect of MR technology on the social and learning performance during the visit of historical sites. 3 Behind the Holographic Illusion To obtain realistic and seamless MR experiences, holograms (i.e., the virtual objects blended into the real world) should consistently retain their position and orientation in space. That is, as the user moves and rotates his viewpoint in space, the relative position, orientation, and size of the holograms have to be updated to offer the illusion that they are real objects. (a) (b) Figure 3 Perception of real objects. An observer is firstly located at an initial position (a), obtaining a given perspective on the observed scene. At later point in time, the observer is located and oriented differently (b), obtaining a different perspective on the scene. 15 C OA R C H MR for Archeology and Cultural Heritage To obtain a realistic holographic illusion, one has to artificially reproduce the mechanics undergoing the perception of real objects. When we sense reality through our eyes what is actually happening is that light reflects on object surfaces. The light waves reflected towards an observer are captured by his eyes and processed by his brain into a meaningful image. Such a process happens naturally and continuously. That is, as the observer moves and rotates in space, at each point in time the scene is perceived from a different location and orientation, producing a different perspective. This process is graphically illustrated in Figure 3. Note how the perceived relative position, orientation, and size of the two observed objects changes as the observer moves in space from the position depicted in Figure 3a to the one in Figure 3b. Now, assume we are in a MR environment where the tree is a real object while the colored cube is an hologram anchored at a specific location in space, with a given orientation and scale. In order to obtain a realistic holographic illusion the position, orientation, and size of the virtual cube relative to the MR user have to be computed at any point in time in order to display it in the visual field of the user in the same way a real object would be perceived. This process is referred to as posing the hologram. There exist two main techniques to pose holograms. They are called marker-based and location-based. In the former, a graphical marker is chosen beforehand and the hologram is anchored to it. Every time the same marker is visible to the cameras of the MR device, the hologram is projected on top of it. The relative position and orientation necessary to project the hologram are derived from the distortion of the marker with respect to its original shape. This technique does not require to know the pose of the user in space, nor that of the hologram. On the down side, however, this approach requires a setup of the scene where the marker has to be defined and the hologram has to be located on top of it. (a) (b) Figure 4 Holographic projection. To realistically project holograms in the MR user view field, the relative position and orientation of the holograms have to be updated at any point in time. This can be easily achieved if we know the absolute pose of both the hologram and the user. Conversely, the location-based posing approach requires to know the pose of both the user and the hologram in order to compute the relative pose of the hologram via vector difference, as depicted in Figure 4. This technique does not require special preparation but demands more sophisticated MR hardware that is capable of tracking the position of the user in space—a process that is typically done through simultaneous localization and mapping (SLAM) techniques [2]. 16 P. Fogliaroni 4 Limitations and Challenges of Current MR Technology A number of toolkits—e.g., Google ARCore1 , Apple ARKit2 , and Microsoft MixedReality- Toolkit3 —have been released recently that allow for setting up an holographic experience quite smoothly. The majority of such toolkits provides support for both marker-based and location-based holographic pose. However, the location-based functionalities are only conceived for indoor usage. One may naively think that the same techniques can be straightforwardly applied for outdoor space. However, indoor and outdoor spaces expose very different characteristics that do not really allow for a direct application of indoor techniques for outdoor—or vice versa. Spatial representation itself is completely different for indoor and outdoor spaces, with the representation of the former typically being based on local (Cartesian) coordinate reference systems while the representation of the latter relies on geographic coordinate reference systems. This means that the realization of outdoor holographic experiences is still not properly supported and custom (and typically cumbersome) workarounds have to be developed in order to let the MR system to work properly in geographic space. Indeed, this requires, for example, some sort of bidirectional mapping from geographic space to local space in order to leverage the MR functionalities supported by the toolkit at hand. Another important feature that typically has to be addressed in a custom manner concerns the registration of the user position and orientation, that has to be extremely accurate in order to offer a smooth and consistent holographic experience. Outdoor localization is typically done through GNSS/GPS signals which, however, might be distorted, poor, or completely absent in particular areas (e.g., areas with a high concentration of high buildings and narrow streets, or in the proximity of large glass buildings). Even more problematic is global heading. Electronic compasses are often unreliable as they are highly sensible to distortions of the Earth electromagnetic field. Other approaches aim at deriving heading information by fusing registrations from different sensors such as inertial measurement units (IMUs), accelerometers, and gyroscopes. However, measurements based on these sensors are subject to drifting over time. These and other challenges that have to be tackled to realize outdoor MR applications have been investigated in the field of Geographic Information Science (GIS) for decades. So, arguably, a core integration of MR and GIS technologies would allow for enlarging the range of holographic experiences (as described in the next section for the domain of archeology and cultural heritage), ultimately allowing for a greater dissemination and enjoyment of holographic experiences. 5 Envisioning Outdoor MR Scenarios Under the assumption that the integration of MR and GIS technologies is set in place, we now envision some possible scenarios in the archeological and cultural heritage domain. 1 https://developers.google.com/ar/ 2 https://developer.apple.com/arkit/ 3 https://github.com/Microsoft/MixedRealityToolkit 17 C OA R C H MR for Archeology and Cultural Heritage 5.1 Excavation Support for Archeological Sites Typically, an archeological site is accurately mapped prior to start the excavation phase. The mapping is done using GNSS/GPS surveying equipment to delimit an excavation grid on the site. Also, other instruments such as ground penetrating radar (GPR) are used to detect underground artifacts and to generate a stratigraphy of the area to be dug. This information can also be embedded in a geographic reference system. After this preliminary phases, the excavation process starts during which the archeologists have to continuously refer the underground mappings in order to avoid damaging precious artifacts. This makes the excavation process a delicate, laborious, and long-lasting effort. Arguably, an outdoor MR application capable of loading and showing in real time the 3D pose of the underground features would potentially result in a much quicker, more secure, and simpler excavation process. 5.2 Holographic Historical Reconstruction There is a countless number of historical and cultural heritage buildings and artifacts that are partly destroyed. For example, this can be the consequence of a natural disaster such as an earthquake or simply the effect of time. The digital reconstruction of historical sites and buildings is a technic that is already largely employed today. Digital reconstructions support scientists in understanding better the usage and the historical implications of a given site and are powerful means to offer a scenic historical experience to non-academics. As of today, historical reconstructions are largely used in laboratories or museums. The realization of an outdoor MR application to visualize such reconstructions, however, would allow to enjoy the experience directly on-site. Thanks to the spatial contextualization this may actually bring to greater insights that would be hard to discover from within a remote laboratory. 5.3 Historical Time Lapses Digital reconstructions and historical simulations can actually be converted into animated holographic experiences or time-lapses. For example, one may use historical information to simulate landscape modification over years and centuries. An outdoor MR application capable of showing terrain modification time-lapses of the visible surroundings would be a great support tool. For example, this might offer a better understanding of an archeological site, or help to elaborate a geophysical explanation of an historical event that took place in the same area when this was actually looking completely different than it looks today. 6 Conclusions This short paper provided an overview of the concepts of Mixed reality and its application to the archeology and cultural heritage domain. The mechanics underlying this technology is presented in an easy-to-understand manner. The main characteristics of today MR technology are discussed as well as one of its more outstanding limitation: the lack of integration with GIS. The result is that the application of MR technology to outdoor scenarios is cumbersome and requires custom adaptations. Finally, under the assumption that the integration of MR and GIS domains is in place, three exemplary application scenarios for the archeological and cultural heritage domain are envisioned. Within the scope of the proposed scenarios, it is highlighted how MR environments may support analytical efforts, archeological excavations, and understanding of historical processes, respectively. 18 REFERENCES References 1 Marcus Abbott. Visualising stonehenge: A virtual archaeology. In David Arnold, Jaime Kaminski, Franco Niccolucci, and Andre Stork, editors, VAST: International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage - Short and Project Papers. The Eurographics Association, 2012. 2 Josep Aulinas, Yvan R Petillot, Joaquim Salvi, and Xavier Lladó. The slam problem: a survey. CCIA, 184(1):363–371, 2008. 3 Ronald T Azuma. A survey of augmented reality. Presence: Teleoperators & Virtual Environments, 6(4):355–385, 1997. 4 Andrew T Duchowski. Gaze-based interaction: A 30 year retrospective. Computers & Graphics, 73:59–69, 2018. 5 Athanasios Gaitatzes, Dimitrios Christopoulos, and Maria Roussou. Reviving the past: cultural heritage meets virtual reality. In Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage, pages 103–110. ACM, 2001. 6 Tony Hall, Luigina Ciolfi, Liam Bannon, Mike Fraser, Steve Benford, John Bowers, Chris Greenhalgh, Sten-Olof Hellström, Shahram Izadi, Holger Schnädelbach, et al. The visitor as virtual archaeologist: explorations in mixed reality technology to enhance educational and social interaction in the museum. In Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage, pages 91–96. ACM, 2001. 7 Peter Kiefer, Ioannis Giannopoulos, and Martin Raubal. Using eye movements to recognize activities on cartographic maps. In Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pages 488–491. ACM, 2013. 8 Ursula Kretschmer, Volker Coors, Ulrike Spierling, Dieter Grasbon, Kerstin Schneider, Isabel Rojas, and Rainer Malaka. Meeting the spirit of history. In Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage, pages 141–152. ACM, 2001. 9 Florian Ledermann and Dieter Schmalstieg. Presenting an archaeological site in the virtual showcase. In Proceedings of the 4th International conference on Virtual Reality, Archaeology and Intelligent Cultural Heritage, pages 123–130. Eurographics Association, 2003. 10 Cristina Manresa, Javier Varona, Ramon Mas, and Francisco J Perales. Hand tracking and gesture recognition for human-computer interaction. ELCVIA Electronic Letters on Computer Vision and Image Analysis, 5(3):96–104, 2005. 11 Paul Milgram and Fumio Kishino. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12):1321–1329, 1994. 12 Vassilios Vlahakis, John Karigiannis, Manolis Tsotros, Michael Gounaris, Luis Almeida, Didier Stricker, Tim Gleue, Ioannis T Christou, Renzo Carlucci, and Nikos Ioannidis. Archeoguide: first results of an augmented reality, mobile computing system in cultural heritage sites. Virtual Reality, Archeology, and Cultural Heritage, 9, 2001. 19 C OA R C H