=Paper=
{{Paper
|id=Vol-2618/paper1
|storemode=property
|title=iFAR: mobileAR for Cultural Heritage
|pdfUrl=https://ceur-ws.org/Vol-2618/paper1.pdf
|volume=Vol-2618
|authors=Gerhard Sprung,Agnesa Haxha
}}
==iFAR: mobileAR for Cultural Heritage==
https://ceur-ws.org/Vol-2618/paper1.pdf
iFAR: mobileAR for Cultural Heritage
Gerhard Sprung*
Agnesa Haxha**
FH JOANNEUM
They should be able to look at non-existent objects, buildings and
ABSTRACT other environments from different angles and be able to interact
with or even change them.
Commercially available smart phones are already capable of using But there are several questions that have to be tackled before
markerless inside-out tracking, particularly “Simultaneous implementing such a system in cultural heritage sites or museums.
Localization and Mapping” (SLAM). With these possibilities, One field of research is the accuracy and reliability of „inside-out“
developers of Augmented Reality (AR)-applications can realize tracking methods (visual-inertial odometry) as provided by mobile
ideas for applications that, until recently, were restricted to high- phones equipped with ARKit or ARToolkit in situations where
end devices such as the Microsoft Hololens. Smartphones can now many moving objects like cars, other visitors or trees are
be used to provide a view into a virtual alternative reality, for interfering. Additionally, it has to be researched, if it is possible to
example to show buildings that no longer or don’t yet exist. display the other participants on the AR-Screen of the mobile
device convincingly and accurately as avatars and to adapt the
In this paper we describe the idea and realization of a prototype user’s movement and gestures to the avatar.
using this concept as well as collaborative social interaction in AR
as a work in progress. Using this prototype, we intend to research 2 RELATED WORK
the acceptance and advantages/disadvantages of such solutions and
try to find out whether social interaction in AR can be meaningful 2.1 Definition of AR and VR
and expedient to convey facts and emotions to visitors of cultural Milgram and Kishino defined the “virtuality continuum” as the
heritage sites. span between reality and Virtual Environments [1]. Everything
between these two poles is called Mixed Reality. In this definition
Keywords: mobileAR, cultural heritage, social media, tracking,
Augmented Reality enhances and extends reality without replacing
avatar, serious game.
it completely. As the term was defined differently by several
Index Terms: H.5.1 [Information Interfaces and Presentation]: companies its usage frequently leads to confusion. Foni et al. [2]
Multimedia Information Systemes—Artificial, augmented, and therefore introduced a 4D coordinate set to categorize the different
virtual realities I.4.8 [Image Processing and Computer Vision]: approaches. While there is no generally accepted term for this type
Scene Analysis—Tracking; I.4.9 [Image Processing and Computer of technology, we will use Mobile Augmented Reality (Mobile AR)
Vision]: Applications in the following to describe a partial overlay of the real image with
computer-generated, preferably 3-dimensional objects, depending
CCS Concepts: Human-centered computing → Mixed / on location and position using a smart phone or similar mobile
augmented reality; device.
1 INTRODUCTION 2.2 VR and AR for presenting Cultural Heritage Objects
Often it is hard or even impossible to imagine what a historic site Cultural Computing uses computer technologies to represent and
looked like in the past. Such an experience would be immensely extend culture and social sciences [3][4], especially to present
helpful in conveying the importance of historical events or helping Cultural Heritage (CH) assets [5] and scenes [6][7][8]. Several EU
one to develop an opinion concerning urban planning decisions. In projects like CHESS and EMOTIVE deal with the possibilities as
order to achieve such experiences, usually interactive PC- regards technics and content [9][10][11].
Applications or Virtual reality (VR) experiences are developed. But
both have severe limitations like the inability of exploring such a In recent years, museums and cultural heritage sites have tried to
situation on site and/or together with other participants. use AR and VR to convey their content and strengthen the
With our approach we will develop an application for mobile emotional bond between visitors and exhibits [12][13]. Bozelli et
devices like smartphones or tablets to look through an "AR- al. [14] describe a framework for interactive experiences including
Window" into the past or into a possible future. Interested VR and AR. Real world applications and research projects show
individuals will be able to use it while walking around on a large solutions for digitizing and visualizing of artefacts, environments
site and be able to interact with non-player characters as well as and architecture in VR [5][15]. In addition, solutions for location
avatars of other participants. based AR-Applications (GeoAR) with GPS and sensors including
occlusion of computer generated content by real objects are
provided [16]. Because of the lack of possibilities for exact tracking
of the devices, many solutions are limited to small objects, or
solutions are sought to replace or improve the tracking of the
* gerhard.sprung@fh-joanneum.at mobile devices [17].
** agnesa.haxha@edu.fh-joanneum.at
The effectiveness and acceptance of AR and VR is researched and
is already partly approved for some specific applications [18] like
Copyright © 2020 for this paper by its authors. Use permitted under AR in CH [14] or urban heritage tourism [19]. Smith et al. [20]
Creative Commons License Attribution 4.0 International (CC BY 4.0). describe the importance of proximity of the provided information
for mobile guides to historic places. Efstratios and George [21] buildings had the largest expansion in the 19th century. Large parts
introduce a concept for virtual portals to use mobile devices to look of the castle were demolished in 1853 and only parts were
into alternative realities. preserved (Figure 1: Situation before 1853 and Figure 2: present
situation). While the wing constructed by Archduke Charles in
2.3 State of the art of AR technology 1570, a gothic hall, a late gothic chapel, and a double spiral
staircase going back to 1499 has remained largely intact, parts like
The creation of 3D models or avatars with the help of the early renaissance portal leading to the inner courtyard, a work
photogrammetry or 3D-scanning devices is already used widely in of Domenico dell'Aglio, have been abandoned.
the field of archaeology, CH and museums. [5][14][22]
Mobile location-based applications need to know exactly where the
mobile device is at a given moment. Among others, this can be
achieved using GPS, NFC, recognition of markers or triangulation
of Bluetooth signals. Especially in outdoor situations, GPS is often
used but has a rather low accuracy [23], [24][25]
For a few years now, the flagship models of smartphones have been
able to obtain their location in real time with their camera or, in
some cases, the combination of camera and depth information.
They use this information to calculate the changes that must be
made in the displayed content. This technology is called SLAM
[26], which stands for Simultaneous Location and Mapping and
was previously only available for expensive AR devices such as the
Microsoft Hololens and Daqri Smart Glasses. The combination of
the measurement data from the sensors a.k.a. odometry (compass,
rotation and acceleration sensor) and the results of the calculations
from the camera image enables the device to calculate the position
with reasonable accuracy over a longer period of time [27].
2.4 Social AR
Cooperation, exchange of ideas and social contacts are omnipresent
in mobile apps. Furthermore, in VR and AR social interaction is Figure 1: Postcard of the entrance to the Castle before 18533
expected to bring added value to the users’ experience. Social
applications are already widespread for VR devices (VRChat1).
Latoschik et al. [28], Roth et al. [29] and Sprung et al. [30] are
concerned with the creation of avatars and the appropriate
representation in VR. Interaction with a non-player character
(NPC) [31] and playful elements (Neanderthal App2) have been
combined with location-based AR-applications. Lewi et al. [32] is
investigating the potential of social AR, collaborative tasks in AR
are implemented and researched by [33][34]. Poretsky et al. [35]
research possible challenges concerning collaborative interaction in
AR.
The use of serious game elements is currently mostly limited to 2D
representations, 360° images and menus or text. Bozelli et al. [14]
uses serious games, storytelling, exploration, choices for the
VR/AR framework ArkaeVision project and Papagiannakis et al.
[36] describe the implementation of gamification und storytelling Figure 2: Present Situation
in AR applications.
3 THE APPLICATION
3.2 The Application
3.1 The Situation With the help of mobile devices capable of using SLAM, it has
The Graz Castle was chosen as the content of the prototype because become possible to offer a “window” into a virtual or past reality
it is historically interesting and due to the partial destruction in the with tablets and mobile phones. With this method, visitors can be
19th century it seems to be a perfect example of how historical offered a look into the past.
buildings can be made visible again.
The castle of Graz (the ensemble with the cathedral, the mausoleum
and the Jesuit college is also known as the city crown) was
recognized as a World Heritage Site by UNESCO in 1999 [37].
Parts of the castle go back to the 14th century, the group of
3 BY-NC-SA 4.0: Austrian Centre for Digital Humanities, Karl-
1 https://www.vrchat.com/ Franzens-Universität Graz
2 https://www.neanderthal.de/de/app.html
We designed a mobile AR-application for Android and iOS
smartphones that can show the demolished parts of the castle by
looking through the device like a window. It is possible to walk
around the whole site and look at the buildings from all sides. By
using personal devices, we try to avoid the obstacles that teachers
and museums usually face, like the prohibitive acquisition cost of
devices, maintenance, insurance and much more. Compared to VR-
devices we can eliminate the need for cables, high-end computers
and tracking devices. In addition, we create the possibility of
interacting with other participants, walking around and exploring
everything from different angles.
We use Unity as developing platform, ARFoundation as platform
independent library to implement AR-functionality and Photon
engine4 to implement multiuser tracking and real-time interaction
in AR.
The application provides a menu to choose the time and a
username. Then the users join a so-called Room and can move the
smartphone and look “through” the screen into another time. There
they can see abandoned buildings in place occluding the image of
existing objects. (Figure 3: Screenshot). The players can interact
with objects in the scene, e.g. view, push or collect them by
focusing or touching. In this view they also can see the avatars of
other users who are currently in the same Room. The other players
are represented by avatars located exactly where the real users are.
The other participants are integrated into the AR storytelling as
historically dressed avatars. One can interact with others and is Figure 3: Screenshot looking at the demolished grand staircase.
forced to work together with them to perform certain tasks and
unlock achievements. It is possible to chat with these persons via a
chat-function, either privately or in a common chatroom which is
REFERENCES
visible to all users in the Room.
Non-player characters can appear and talk to the players [1] P. Milgram and F. Kishimo, “A taxonomy of mixed reality,”
(headphones are recommended) and tell them about the historical IEICE Trans. Inf. Syst., vol. 77, no. 12, pp. 1321–1329, 1994.
events and assign tasks to them which have to be solved, either
[2] A. E. Foni, G. Papagiannakis, and N. Magnenat-Thalmann, “A
alone or in collaboration with another person in the Room. It is
possible to combine these tasks with other locations where similar Taxonomy of Visualization Strategies for Cultural Heritage
applications are provided. Applications,” J. Comput. Cult. Herit., vol. 3, no. 1, 2010.
[3] M. Haydar, D. Roussel, M. Maidi, S. Otmane, and M. Mallem,
“Virtual and augmented reality for cultural computing and
4 DISCUSSION
heritage: A case study of virtual exploration of underwater
The long-term goal of our research activities is to assess the archaeological sites (preprint),” Virtual Real., vol. 15, pp. 311–
usability of mobile AR utilizing SLAM with user owned devices
(Bring Your Own Device BYOD). We will especially look at the 327, 2011.
impact for sensitizing visitors to historical content and the [4] M. K. Bekele, R. Pierdicca, E. Frontoni, E. S. Malinverni, and J.
importance of dealing responsibly with historical buildings and Gain, “A Survey of Augmented, Virtual, and Mixed Reality for
ensembles. The social interaction and the acceptance of the Cultural Heritage,” J. Comput. Cult. Herit., vol. 11, no. 2, 2018.
activities have to be tested along with the technical restrictions and [5] S. Gonizzi Barsanti, G. Caruso, L. Micoli, M. Covarrubias, and
accuracy of tracking. In addition to the visualization of the
historical buildings at a predefined historical date, we introduce and G. Guidi, “3D Visualization of Cultural Heritage Artefacts with
test activities to improve the emotional connection with history and Virtual Reality devices,” ISPRS - Int. Arch. Photogramm. Remote
the understanding of the processes. Sens. Spat. Inf. Sci., vol. XL-5/W7, pp. 165–172, 2015.
[6] V. Vlahakis et al., “Archeoguide: First Results of an Augmented
Based on the results, solutions for simple design of such Reality, Mobile Computing System in Cultural Heritage Sites,” in
applications, conclusions about acceptance and sustainability and
other areas of application such as urban planning and tourism shall Proceedings of the 2001 Conference on Virtual Reality,
be described. Hopefully we can contribute to a future where AR- Archeology, and Cultural Heritage, 2001, pp. 131–140.
applications can help students understand the importance of [7] G. Casella and M. Coelho, “Augmented Heritage: Situating
historical events as well as enable urban planners to better present Augmented Reality Mobile Apps in Cultural Heritage
and discuss their visions to those involved, on site and in full scale. Communication,” in Proceedings of the 2013 International
Conference on Information Systems and Design of
Communication, 2013, pp. 138–140.
[8] D. Petrelli, “Making virtual reconstructions part of the visit: An
exploratory study,” Digit. Appl. Archaeol. Cult. Herit., 2019.
4 https://www.photonengine.com/
[9] L. Pujol, M. Roussou, S. Poulou, O. Balet, M. Vayanou, and Y. [25] C. Bauer and Christine, “On the (In-)Accuracy of GPS Measures
Ioannidis, “Personalizing interactive digital storytelling in of Smartphones,” in Proceedings of International Conference on
archaeological museums: the CHESS project,” Archaeol. Digit. Advances in Mobile Computing & Multimedia - MoMM ’13,
Era. Pap. from 40th Annu. Conf. Comput. Appl. Quant. Methods 2013, pp. 335–341.
Archaeol., p. to appear, 2013. [26] J. Leonard and H. Durrant-Whyte, “Mobile robot localization by
[10] R. G. Boboc, M. Duguleana, G. D. Voinea, C. C. Postelnicu, D. tracking geometric beacons,” Robot. Autom. IEEE Trans., vol. 7,
M. Popovici, and M. Carrozzino, “Mobile augmented reality for pp. 376–382, 1991.
cultural heritage: Following the footsteps of Ovid among different [27] Y. Chen et al., “The Accuracy Comparison of Three
locations in Europe,” Sustain., vol. 11, no. 4, 2019. Simultaneous Localization and Mapping (SLAM)-Based Indoor
[11] A. Katifori et al., “The EMOTIVE Project - Emotive Virtual Mapping Technologies,” Sensors (Basel)., vol. 18, no. 10, p.
Cultural Experiences through Personalized Storytelling,” in 3228, Sep. 2018.
CIRA@EuroMed, 2018. [28] M. Latoschik, D. Roth, D. Gall, J. Achenbach, T. Waltemate, and
[12] S. Perry, “The Enchantment of the Archaeological Record,” Eur. M. Botsch, “The Effect of Avatar Realism in Immersive Social
J. Archaeol., vol. 22, no. 3, pp. 354–371, 2019. Virtual Realities,” in Proceedings of the 23rd ACM Symposium
[13] J. Häkkilä et al., “Visiting a virtual graveyard: designing virtual on Virtual Reality Software and Technology, 2017.
reality cultural heritage experiences,” in MUM 2019, 2019. [29] D. Roth et al., “Technologies for Social Augmentations in User-
[14] G. Bozzelli et al., “An integrated VR/AR framework for user- Embodied Virtual Reality,” in 25th ACM Symposium on Virtual
centric interactive experience of cultural heritage: The Reality Software and Technology, 2019.
ArkaeVision project,” Digit. Appl. Archaeol. Cult. Herit., 2019. [30] G. Sprung, A. Egger, A. Nischelwitzer, R. Strohmaier, and S.
[15] A. Gaitatzes, D. Christopoulos, and M. Roussou, “Reviving the Schadenbauer, “ViReSt - Storytelling with Volumetric Videos,”
Past: Cultural Heritage Meets Virtual Reality,” in Proceedings of in Proceedings of the 11th Forum Media Technology and 4th All
the 2001 Conference on Virtual Reality, Archeology, and Cultural Around Audio Symposium, 2018, p. 74.
Heritage, 2001, pp. 103–110. [31] P. Breuss-Schneeweis, “‘The Speaking Celt’: Augmented Reality
[16] J.-E. Kilimann, D. Heitkamp, and P. Lensing, “An Augmented Avatars Guide through a Museum -- Case Study,” in Proceedings
Reality Application for Mobile Visualization of GIS-Referenced of the 2016 ACM International Joint Conference on Pervasive
Landscape Planning Projects,” in The 17th International and Ubiquitous Computing: Adjunct, 2016, pp. 1484–1491.
Conference on Virtual-Reality Continuum and Its Applications in [32] H. Lewi et al., “Shared digital experiences supporting
Industry, 2019. collaborative meaning-making at heritage sites,” in The
[17] J. B. Madsen and C. B. Madsen, “Handheld Visual Routledge International Handbook of New Digital Practices in
Representation of a Castle Chapel Ruin,” J. Comput. Cult. Herit., Galleries, Libraries, Archives, Museums and Heritage Sites,
vol. 9, no. 1, Nov. 2015. Routledge, 2019, pp. 143–156.
[18] C. Flavián, S. Ibáñez-Sánchez, and C. Orús, “The impact of [33] J. Laviole, L. Thevin, J. Albouys-Perrois, and A. Brock, “Nectar:
virtual, augmented and mixed reality technologies on the Multi-User Spatial Augmented Reality for Everyone: Three Live
customer experience,” J. Bus. Res., 2019. Demonstrations of Educative Applications,” in Proceedings of
[19] M. C. Tom Dieck and T. H. Jung, “Value of augmented reality at the Virtual Reality International Conference - Laval Virtual,
cultural heritage sites: A stakeholder approach,” J. Destin. Mark. 2018.
Manag., vol. 6, no. 2, pp. 110–117, 2017. [34] A. Jing, C. Xiang, S. Kim, M. Billinghurst, and A. Quigley,
[20] W. Smith, H. Lewi, D. Constantinidis, and H. Stitt, “Directed “SnapChart: An Augmented Reality Analytics Toolkit to
looking and proximal content: two concepts for designing mobile Enhance Interactivity in a Collaborative Environment,” in The
guides to historic urban places,” Proc. 26th Aust. Comput. 17th International Conference on Virtual-Reality Continuum and
Interact. Conf. Des. Futur. Futur. Des., pp. 400–403, 2014. Its Applications in Industry, 2019.
[21] G. Efstratios and P. George, “Case Study For Augmented Reality [35] L. Poretski, J. Lanir, and O. Arazy, “Normative tensions in shared
( AR ) Application Of Portals For The Preservation Of Cultural augmented reality,” Proc. ACM Human-Computer Interact., vol.
Heritage,” Heraklion, Greece. 2, no. CSCW, Nov. 2018.
[22] C. Bolognesi and D. Aiello, “FORGOTTEN ARCHITECTURE: [36] G. Papagiannakis et al., “Mixed Reality, Gamified Presence, and
SMART TOOLS FOR CULTURAL TOURISM IN THE Storytelling for Virtual Museums,” in Encyclopedia of Computer
CLOISTER OF THE PRIOR (SANTA MARIA DELLE Graphics and Games, Springer International Publishing, 2019,
GRAZIE, MILAN),” ISPRS - Int. Arch. Photogramm. Remote pp. 1–13.
Sens. Spat. Inf. Sci., vol. XLII-2/W11, pp. 247–253, 2019. [37] “Convention concerning the protection of the World Cultural and
[23] P. A. Zandbergen and S. J. Barbeau, “Positional accuracy of Natural Heritage,” Marrakesh, 2000.
assisted GPS data from high-sensitivity GPS-enabled mobile
phones,” J. Navig., vol. 64, no. 3, pp. 381–399, 2011.
[24] P. A. Zandbergen, “Accuracy of iPhone Locations: A
Comparison of Assisted GPS, WiFi and Cellular Positioning,”
Trans. GIS, vol. 13(s1), pp. 5–26, 2009.