=Paper=
{{Paper
|id=Vol-2230/paper_02
|storemode=property
|title=Mixed Reality for Archeology and Cultural Heritage
|pdfUrl=https://ceur-ws.org/Vol-2230/paper_02.pdf
|volume=Vol-2230
|authors=Paolo Fogliaroni
}}
==Mixed Reality for Archeology and Cultural Heritage==
https://ceur-ws.org/Vol-2230/paper_02.pdf
Mixed Reality for Archeology and Cultural
Heritage
Paolo Fogliaroni
Vienna University of Technology, Austria
paolo.fogliaroni@geo.tuwien.ac.at
https://orcid.org/0000-0002-0578-8904
Abstract
Archeological and cultural heritage data may also consist of a 3D geospatial component, which
for certain scenarios, is better consumed by means of immersive technologies such as Augmented,
Mixed, and Virtual Reality. This short paper provides an introduction to the different types of
virtual environments and discusses possible application scenarios within the archeological and
cultural heritage domain. Also, the limits of current technologies are presented and the challenge
of integrating Augmented and Mixed Reality technologies into the geospatial domain is discussed.
Finally, a new set of outdoor MR application scenarios is envisioned.
2012 ACM Subject Classification Human computer interaction (HCI), Interaction paradigms,
Mixed / augmented reality
Keywords and phrases Augmented Reality, Mixed Reality, Spatial HCI
Digital Object Identifier 10.4230/LIPIcs.COARCH.2018.
Category Invited paper
1 Introduction
Mixed reality (MR) is a Human-Computer Interaction (HCI) environment blending together
the real world and virtual (i.e., digital) elements. Milgram and Kishino [11] describe it as
an HCI environment placed anywhere in the so-called virtuality continuum (see Figure 1),
that goes from the completely real to the completely virtual environment. According to this
taxonomy, Augmented Reality (AR) and Augmented Virtuality (AV) are special cases of MR,
with the major difference among them lying in the ratio of real to virtual contents. More
specifically, AR superimposes virtual content on the real surroundings of the user while AV
brings some real objects into a virtual world. As of today, the terms AR and MR found
their way into the general public terminology, while AV remained circumscribed to a more
technical and specialized domain.
Mixed Reality
Real Augmented Augmented Virtual
Environment Reality Virtuality Environment
Figure 1 The virtuality continuum stretches from completely real environments to totally virtual
ones. Image adapted from [11].
© Paolo Fogliaroni;
2nd Workshop on Computing Techniques for Spatio-Temporal Data in Archaeology and Cultural Heritage.
Editors: Pierre Hallot, Sara Migliorini, Alberto Belussi, and Roland Billen
13
MR for Archeology and Cultural Heritage
(a) Physical Reality (b) Augmented Reality
(c) Mixed Reality (d) Virtual Reality
Figure 2 A representation of the Colosseum (Rome, Italy) in four different nuances of reality.
Physical reality (a) can be directly experienced through sensory-motor abilities or via conventional
video captures. In Augmented Reality (AR) environments (b) real-world features are enriched
through the overlay of digital information. In Mixed Reality (MR) environments (c) virtual objects
interact with the real ones. Both AR and MR experiences can be enjoyed through either conventional
or head-mounted see-through displays. Virtual Reality (VR) environments (d) present a completely
digital world that can be experienced through either traditional or head-mounted displays.
According to Azuma [3] AR environments should expose the following characteristics:
combine real and virtual: virtual and real objects are blended together into a unique
experience;
interactive in real time: the user is to be able to interact with the augmentation either
via conventional controllers or more advanced types of interactions (e.g., gaze-based [4, 7]
or hand-based [10] interactions);
registered in 3D: the virtual and real objects have to be precisely aligned in real-time in
order to provide a seamless experience.
Arguably, AR is a special case of MR where the blending of virtual objects into the real
world is mainly about the enrichment of the real elements by means of an overlay of digital
content. Conversely, in MR environments the goal is to create a seamless integration of real
and virtual objects, with the latter (at least) visually behaving as real items—e.g., they
occlude and are occluded by other real and virtual elements. In MR, the virtual objects can
or cannot obey physical rules, allowing for creating a variety of different experiences ranging
from environments where the reality is enriched with virtual objects in a much natural way,
to fantastic environments where the virtual objects escape one or more physical law—e.g.,
floating virtual objects that are not constrained to gravity. Figure 2 provides an example that
illustrates how the same subject (the Colosseum) could be represented in different nuances
of reality (physical, augmented, mixed, and virtual reality).
14
P. Fogliaroni
This paper provides an introduction to the main concepts underlying current AR/MR
technology and discusses its application in the archeology and cultural heritage domain.
Also, the limitations of current technology are highlighted and it is argued that AR/MR
technology would benefit from an integration with the Geographic Information Science (GIS)
domain. Finally, under the assumption that this integration is realized, novel applications for
the visualization and interaction with archeological and cultural heritage data are envisioned.
In the remainder of this work we mainly focus on Mixed Reality (that, as discussed above, is
a generalization of AR), sometimes referring to it as Holographic Experience.
2 Virtual Experiences in the Archeology and Cultural Heritage
The application of Virtual Reality (VR) and Mixed Reality (MR) to the archeological and
cultural heritage domain is not new and has been already investigated in the past. For
example, Abbott [1] applies VR for the visualization and analysis of Stonehenge; Gaitatzes
et al. [5] implement a virtual journey through a digital reconstruction of the city of Miletus
(among other ancient cities); Ledermann and Schmalstieg [9] utilize a volumetric display to
visualize the Heidentor ruin (located in Carnuntum, Austria) and a superimposition of the
missing part of the construction.
Similar projects and research endeavors have been also focused on the application of
Augmented and Mixed Reality technologies. For example, the ARCHEOGUIDE project
[12] led to the implementation of a system providing on-site personalized guide and an
AR visualization of the digital reconstruction of ancient ruins; Kretschmer et al. [8] also
developed a mobile augmented reality system for the historic city of Heidelberg that provides
story-telling as the user moves within the historical site; Hall et al. [6] investigate the effect
of MR technology on the social and learning performance during the visit of historical sites.
3 Behind the Holographic Illusion
To obtain realistic and seamless MR experiences, holograms (i.e., the virtual objects blended
into the real world) should consistently retain their position and orientation in space. That
is, as the user moves and rotates his viewpoint in space, the relative position, orientation,
and size of the holograms have to be updated to offer the illusion that they are real objects.
(a) (b)
Figure 3 Perception of real objects. An observer is firstly located at an initial position (a),
obtaining a given perspective on the observed scene. At later point in time, the observer is located
and oriented differently (b), obtaining a different perspective on the scene.
15
C OA R C H
MR for Archeology and Cultural Heritage
To obtain a realistic holographic illusion, one has to artificially reproduce the mechanics
undergoing the perception of real objects. When we sense reality through our eyes what
is actually happening is that light reflects on object surfaces. The light waves reflected
towards an observer are captured by his eyes and processed by his brain into a meaningful
image. Such a process happens naturally and continuously. That is, as the observer moves
and rotates in space, at each point in time the scene is perceived from a different location
and orientation, producing a different perspective. This process is graphically illustrated in
Figure 3. Note how the perceived relative position, orientation, and size of the two observed
objects changes as the observer moves in space from the position depicted in Figure 3a to
the one in Figure 3b.
Now, assume we are in a MR environment where the tree is a real object while the colored
cube is an hologram anchored at a specific location in space, with a given orientation and
scale. In order to obtain a realistic holographic illusion the position, orientation, and size of
the virtual cube relative to the MR user have to be computed at any point in time in order
to display it in the visual field of the user in the same way a real object would be perceived.
This process is referred to as posing the hologram.
There exist two main techniques to pose holograms. They are called marker-based and
location-based. In the former, a graphical marker is chosen beforehand and the hologram
is anchored to it. Every time the same marker is visible to the cameras of the MR device,
the hologram is projected on top of it. The relative position and orientation necessary to
project the hologram are derived from the distortion of the marker with respect to its original
shape. This technique does not require to know the pose of the user in space, nor that of the
hologram. On the down side, however, this approach requires a setup of the scene where the
marker has to be defined and the hologram has to be located on top of it.
(a) (b)
Figure 4 Holographic projection. To realistically project holograms in the MR user view field,
the relative position and orientation of the holograms have to be updated at any point in time. This
can be easily achieved if we know the absolute pose of both the hologram and the user.
Conversely, the location-based posing approach requires to know the pose of both the
user and the hologram in order to compute the relative pose of the hologram via vector
difference, as depicted in Figure 4. This technique does not require special preparation but
demands more sophisticated MR hardware that is capable of tracking the position of the user
in space—a process that is typically done through simultaneous localization and mapping
(SLAM) techniques [2].
16
P. Fogliaroni
4 Limitations and Challenges of Current MR Technology
A number of toolkits—e.g., Google ARCore1 , Apple ARKit2 , and Microsoft MixedReality-
Toolkit3 —have been released recently that allow for setting up an holographic experience
quite smoothly. The majority of such toolkits provides support for both marker-based
and location-based holographic pose. However, the location-based functionalities are only
conceived for indoor usage.
One may naively think that the same techniques can be straightforwardly applied for
outdoor space. However, indoor and outdoor spaces expose very different characteristics that
do not really allow for a direct application of indoor techniques for outdoor—or vice versa.
Spatial representation itself is completely different for indoor and outdoor spaces, with the
representation of the former typically being based on local (Cartesian) coordinate reference
systems while the representation of the latter relies on geographic coordinate reference
systems.
This means that the realization of outdoor holographic experiences is still not properly
supported and custom (and typically cumbersome) workarounds have to be developed in
order to let the MR system to work properly in geographic space. Indeed, this requires, for
example, some sort of bidirectional mapping from geographic space to local space in order to
leverage the MR functionalities supported by the toolkit at hand.
Another important feature that typically has to be addressed in a custom manner concerns
the registration of the user position and orientation, that has to be extremely accurate in
order to offer a smooth and consistent holographic experience. Outdoor localization is
typically done through GNSS/GPS signals which, however, might be distorted, poor, or
completely absent in particular areas (e.g., areas with a high concentration of high buildings
and narrow streets, or in the proximity of large glass buildings).
Even more problematic is global heading. Electronic compasses are often unreliable as
they are highly sensible to distortions of the Earth electromagnetic field. Other approaches
aim at deriving heading information by fusing registrations from different sensors such as
inertial measurement units (IMUs), accelerometers, and gyroscopes. However, measurements
based on these sensors are subject to drifting over time.
These and other challenges that have to be tackled to realize outdoor MR applications
have been investigated in the field of Geographic Information Science (GIS) for decades. So,
arguably, a core integration of MR and GIS technologies would allow for enlarging the range
of holographic experiences (as described in the next section for the domain of archeology
and cultural heritage), ultimately allowing for a greater dissemination and enjoyment of
holographic experiences.
5 Envisioning Outdoor MR Scenarios
Under the assumption that the integration of MR and GIS technologies is set in place, we
now envision some possible scenarios in the archeological and cultural heritage domain.
1
https://developers.google.com/ar/
2
https://developer.apple.com/arkit/
3
https://github.com/Microsoft/MixedRealityToolkit
17
C OA R C H
MR for Archeology and Cultural Heritage
5.1 Excavation Support for Archeological Sites
Typically, an archeological site is accurately mapped prior to start the excavation phase.
The mapping is done using GNSS/GPS surveying equipment to delimit an excavation grid
on the site. Also, other instruments such as ground penetrating radar (GPR) are used to
detect underground artifacts and to generate a stratigraphy of the area to be dug. This
information can also be embedded in a geographic reference system. After this preliminary
phases, the excavation process starts during which the archeologists have to continuously
refer the underground mappings in order to avoid damaging precious artifacts. This makes
the excavation process a delicate, laborious, and long-lasting effort. Arguably, an outdoor
MR application capable of loading and showing in real time the 3D pose of the underground
features would potentially result in a much quicker, more secure, and simpler excavation
process.
5.2 Holographic Historical Reconstruction
There is a countless number of historical and cultural heritage buildings and artifacts that
are partly destroyed. For example, this can be the consequence of a natural disaster such as
an earthquake or simply the effect of time. The digital reconstruction of historical sites and
buildings is a technic that is already largely employed today. Digital reconstructions support
scientists in understanding better the usage and the historical implications of a given site
and are powerful means to offer a scenic historical experience to non-academics. As of today,
historical reconstructions are largely used in laboratories or museums. The realization of an
outdoor MR application to visualize such reconstructions, however, would allow to enjoy the
experience directly on-site. Thanks to the spatial contextualization this may actually bring
to greater insights that would be hard to discover from within a remote laboratory.
5.3 Historical Time Lapses
Digital reconstructions and historical simulations can actually be converted into animated
holographic experiences or time-lapses. For example, one may use historical information
to simulate landscape modification over years and centuries. An outdoor MR application
capable of showing terrain modification time-lapses of the visible surroundings would be a
great support tool. For example, this might offer a better understanding of an archeological
site, or help to elaborate a geophysical explanation of an historical event that took place in
the same area when this was actually looking completely different than it looks today.
6 Conclusions
This short paper provided an overview of the concepts of Mixed reality and its application
to the archeology and cultural heritage domain. The mechanics underlying this technology is
presented in an easy-to-understand manner. The main characteristics of today MR technology
are discussed as well as one of its more outstanding limitation: the lack of integration with
GIS. The result is that the application of MR technology to outdoor scenarios is cumbersome
and requires custom adaptations. Finally, under the assumption that the integration of MR
and GIS domains is in place, three exemplary application scenarios for the archeological and
cultural heritage domain are envisioned. Within the scope of the proposed scenarios, it is
highlighted how MR environments may support analytical efforts, archeological excavations,
and understanding of historical processes, respectively.
18
REFERENCES
References
1 Marcus Abbott. Visualising stonehenge: A virtual archaeology. In David Arnold, Jaime
Kaminski, Franco Niccolucci, and Andre Stork, editors, VAST: International Symposium
on Virtual Reality, Archaeology and Intelligent Cultural Heritage - Short and Project
Papers. The Eurographics Association, 2012.
2 Josep Aulinas, Yvan R Petillot, Joaquim Salvi, and Xavier Lladó. The slam problem: a
survey. CCIA, 184(1):363–371, 2008.
3 Ronald T Azuma. A survey of augmented reality. Presence: Teleoperators & Virtual
Environments, 6(4):355–385, 1997.
4 Andrew T Duchowski. Gaze-based interaction: A 30 year retrospective. Computers &
Graphics, 73:59–69, 2018.
5 Athanasios Gaitatzes, Dimitrios Christopoulos, and Maria Roussou. Reviving the past:
cultural heritage meets virtual reality. In Proceedings of the 2001 conference on Virtual
reality, archeology, and cultural heritage, pages 103–110. ACM, 2001.
6 Tony Hall, Luigina Ciolfi, Liam Bannon, Mike Fraser, Steve Benford, John Bowers, Chris
Greenhalgh, Sten-Olof Hellström, Shahram Izadi, Holger Schnädelbach, et al. The visitor
as virtual archaeologist: explorations in mixed reality technology to enhance educational
and social interaction in the museum. In Proceedings of the 2001 conference on Virtual
reality, archeology, and cultural heritage, pages 91–96. ACM, 2001.
7 Peter Kiefer, Ioannis Giannopoulos, and Martin Raubal. Using eye movements to
recognize activities on cartographic maps. In Proceedings of the 21st ACM SIGSPATIAL
International Conference on Advances in Geographic Information Systems, pages 488–491.
ACM, 2013.
8 Ursula Kretschmer, Volker Coors, Ulrike Spierling, Dieter Grasbon, Kerstin Schneider,
Isabel Rojas, and Rainer Malaka. Meeting the spirit of history. In Proceedings of the 2001
conference on Virtual reality, archeology, and cultural heritage, pages 141–152. ACM,
2001.
9 Florian Ledermann and Dieter Schmalstieg. Presenting an archaeological site in the
virtual showcase. In Proceedings of the 4th International conference on Virtual Reality,
Archaeology and Intelligent Cultural Heritage, pages 123–130. Eurographics Association,
2003.
10 Cristina Manresa, Javier Varona, Ramon Mas, and Francisco J Perales. Hand tracking
and gesture recognition for human-computer interaction. ELCVIA Electronic Letters on
Computer Vision and Image Analysis, 5(3):96–104, 2005.
11 Paul Milgram and Fumio Kishino. A taxonomy of mixed reality visual displays. IEICE
TRANSACTIONS on Information and Systems, 77(12):1321–1329, 1994.
12 Vassilios Vlahakis, John Karigiannis, Manolis Tsotros, Michael Gounaris, Luis Almeida,
Didier Stricker, Tim Gleue, Ioannis T Christou, Renzo Carlucci, and Nikos Ioannidis.
Archeoguide: first results of an augmented reality, mobile computing system in cultural
heritage sites. Virtual Reality, Archeology, and Cultural Heritage, 9, 2001.
19
C OA R C H