Commercially available smart phones are already capable of using markerless inside-out tracking, particularly “Simultaneous Localization and Mapping” (SLAM). With these possibilities, developers of Augmented Reality (AR)-applications can realize ideas for applications that, until recently, were restricted to highend devices such as the Microsoft Hololens. Smartphones can now be used to provide a view into a virtual alternative reality, for example to show buildings that no longer or don't yet exist. In this paper we describe the idea and realization of a prototype using this concept as well as collaborative social interaction in AR as a work in progress. Using this prototype, we intend to research the acceptance and advantages/disadvantages of such solutions and try to find out whether social interaction in AR can be meaningful and expedient to convey facts and emotions to visitors of cultural heritage sites.
Often it is hard or even impossible to imagine what a historic site looked like in the past. Such an experience would be immensely helpful in conveying the importance of historical events or helping one to develop an opinion concerning urban planning decisions. In order to achieve such experiences, usually interactive PCApplications or Virtual reality (VR) experiences are developed. But both have severe limitations like the inability of exploring such a situation on site and/or together with other participants. With our approach we will develop an application for mobile devices like smartphones or tablets to look through an "ARWindow" into the past or into a possible future. Interested individuals will be able to use it while walking around on a large site and be able to interact with non-player characters as well as avatars of other participants. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). They should be able to look at non-existent objects, buildings and other environments from different angles and be able to interact with or even change them.
But there are several questions that have to be tackled before implementing such a system in cultural heritage sites or museums. One field of research is the accuracy and reliability of „inside-out“ tracking methods (visual-inertial odometry) as provided by mobile phones equipped with ARKit or ARToolkit in situations where many moving objects like cars, other visitors or trees are interfering. Additionally, it has to be researched, if it is possible to display the other participants on the AR-Screen of the mobile device convincingly and accurately as avatars and to adapt the user’s movement and gestures to the avatar. 2 2.1 2.2
Cultural Computing uses computer technologies to represent and extend culture and social sciences [3][4], especially to present Cultural Heritage (CH) assets [5] and scenes [6][7][8]. Several EU projects like CHESS and EMOTIVE deal with the possibilities as regards technics and content [9][10][11].
In recent years, museums and cultural heritage sites have tried to use AR and VR to convey their content and strengthen the emotional bond between visitors and exhibits [12][13]. Bozelli et al. [14] describe a framework for interactive experiences including VR and AR. Real world applications and research projects show solutions for digitizing and visualizing of artefacts, environments and architecture in VR [5][15]. In addition, solutions for location based AR-Applications (GeoAR) with GPS and sensors including occlusion of computer generated content by real objects are provided [16]. Because of the lack of possibilities for exact tracking of the devices, many solutions are limited to small objects, or solutions are sought to replace or improve the tracking of the mobile devices [17].
The effectiveness and acceptance of AR and VR is researched and is already partly approved for some specific applications [18] like AR in CH [14] or urban heritage tourism [19]. Smith et al. [20] describe the importance of proximity of the provided information for mobile guides to historic places. Efstratios and George [21] introduce a concept for virtual portals to use mobile devices to look into alternative realities. 2.3
The creation of 3D models or avatars with the help of photogrammetry or 3D-scanning devices is already used widely in the field of archaeology, CH and museums. [5][14][22] Mobile location-based applications need to know exactly where the mobile device is at a given moment. Among others, this can be achieved using GPS, NFC, recognition of markers or triangulation of Bluetooth signals. Especially in outdoor situations, GPS is often used but has a rather low accuracy [23], [24][25] For a few years now, the flagship models of smartphones have been able to obtain their location in real time with their camera or, in some cases, the combination of camera and depth information. They use this information to calculate the changes that must be made in the displayed content. This technology is called SLAM [26], which stands for Simultaneous Location and Mapping and was previously only available for expensive AR devices such as the Microsoft Hololens and Daqri Smart Glasses. The combination of the measurement data from the sensors a.k.a. odometry (compass, rotation and acceleration sensor) and the results of the calculations from the camera image enables the device to calculate the position with reasonable accuracy over a longer period of time [27]. 2.4
Cooperation, exchange of ideas and social contacts are omnipresent in mobile apps. Furthermore, in VR and AR social interaction is expected to bring added value to the users’ experience. Social applications are already widespread for VR devices (VRChat1). Latoschik et al. [28], Roth et al. [29] and Sprung et al. [30] are concerned with the creation of avatars and the appropriate representation in VR. Interaction with a non-player character (NPC) [31] and playful elements (Neanderthal App2) have been combined with location-based AR-applications. Lewi et al. [32] is investigating the potential of social AR, collaborative tasks in AR are implemented and researched by [33][34]. Poretsky et al. [35] research possible challenges concerning collaborative interaction in AR.
The use of serious game elements is currently mostly limited to 2D representations, 360° images and menus or text. Bozelli et al. [14] uses serious games, storytelling, exploration, choices for the VR/AR framework ArkaeVision project and Papagiannakis et al. [36] describe the implementation of gamification und storytelling in AR applications. 3 3.1
The Graz Castle was chosen as the content of the prototype because it is historically interesting and due to the partial destruction in the 19th century it seems to be a perfect example of how historical buildings can be made visible again.
The castle of Graz (the ensemble with the cathedral, the mausoleum and the Jesuit college is also known as the city crown) was recognized as a World Heritage Site by UNESCO in 1999 [37]. Parts of the castle go back to the 14th century, the group of 1 https://www.vrchat.com/ 2 https://www.neanderthal.de/de/app.html buildings had the largest expansion in the 19th century. Large parts of the castle were demolished in 1853 and only parts were preserved (Figure 1: Situation before 1853 and Figure 2: present situation). While the wing constructed by Archduke Charles in 1570, a gothic hall, a late gothic chapel, and a double spiral staircase going back to 1499 has remained largely intact, parts like the early renaissance portal leading to the inner courtyard, a work of Domenico dell'Aglio, have been abandoned. With the help of mobile devices capable of using SLAM, it has become possible to offer a “window” into a virtual or past reality with tablets and mobile phones. With this method, visitors can be offered a look into the past.
3 BY-NC-SA 4.0: Austrian Centre for Digital Humanities, KarlFranzens-Universität Graz We designed a mobile AR-application for Android and iOS smartphones that can show the demolished parts of the castle by looking through the device like a window. It is possible to walk around the whole site and look at the buildings from all sides. By using personal devices, we try to avoid the obstacles that teachers and museums usually face, like the prohibitive acquisition cost of devices, maintenance, insurance and much more. Compared to VRdevices we can eliminate the need for cables, high-end computers and tracking devices. In addition, we create the possibility of interacting with other participants, walking around and exploring everything from different angles.
We use Unity as developing platform, ARFoundation as platform independent library to implement AR-functionality and Photon engine4 to implement multiuser tracking and real-time interaction in AR.
The application provides a menu to choose the time and a username. Then the users join a so-called Room and can move the smartphone and look “through” the screen into another time. There they can see abandoned buildings in place occluding the image of existing objects. (Figure 3: Screenshot). The players can interact with objects in the scene, e.g. view, push or collect them by focusing or touching. In this view they also can see the avatars of other users who are currently in the same Room. The other players are represented by avatars located exactly where the real users are. The other participants are integrated into the AR storytelling as historically dressed avatars. One can interact with others and is forced to work together with them to perform certain tasks and unlock achievements. It is possible to chat with these persons via a chat-function, either privately or in a common chatroom which is visible to all users in the Room.
Non-player characters can appear and talk to the players (headphones are recommended) and tell them about the historical events and assign tasks to them which have to be solved, either alone or in collaboration with another person in the Room. It is possible to combine these tasks with other locations where similar applications are provided. 4
The long-term goal of our research activities is to assess the usability of mobile AR utilizing SLAM with user owned devices (Bring Your Own Device BYOD). We will especially look at the impact for sensitizing visitors to historical content and the importance of dealing responsibly with historical buildings and ensembles. The social interaction and the acceptance of the activities have to be tested along with the technical restrictions and accuracy of tracking. In addition to the visualization of the historical buildings at a predefined historical date, we introduce and test activities to improve the emotional connection with history and the understanding of the processes.
Based on the results, solutions for simple design of such applications, conclusions about acceptance and sustainability and other areas of application such as urban planning and tourism shall be described. Hopefully we can contribute to a future where ARapplications can help students understand the importance of historical events as well as enable urban planners to better present and discuss their visions to those involved, on site and in full scale. 4 https://www.photonengine.com/ Taxonomy of Visualization Strategies for Cultural Heritage 327, 2011. Gain, “A Survey of Augmented, Virtual, and Mixed Reality for Augmented
Reality
Mobile
Apps in
Cultural Heritage
Communication,” in Proceedin
Information
Systems and
Desi
D. Petrelli, “Making virtual reconstructions part of the visit: An
exploratory study,” Digit. Appl. Arc
L. Pujol, M. Roussou, S. Poulou, O. Balet, M. Vayanou, and Y. Ioannidis, “Personalizing interactive digital storytelling in archaeological museums: the CHESS project,” Archaeol. Digit. Era. Pap. from 40th Annu. Conf. Comput. Appl. Quant. Methods Archaeol., p. to appear, 2013.
R. G. Boboc, M. Duguleana, G. D. Voinea, C. C. Postelnicu, D. M. Popovici, and M. Carrozzino, “Mobile augmented reality for cultural heritage: Following the footsteps of Ovid among different locations in Europe,” Sustain., vol. 11, no. 4, 2019.
A. Katifori et al., “The EMOTIVE Project - Emotive Virtual Cultural Experiences through Personalized Storytelling,” in CIRA@EuroMed, 2018.
S. Perry, “The Enchantment of the Archaeological Record,” Eur. J. Archaeol., vol. 22, no. 3, pp. 354–371, 2019.
J. Häkkilä et al., “Visiting a virtual graveyard: designing virtual
reality cultural
G. Bozzelli et al., “An integrated VR/AR framework for
usercentric interactive experience of cultural heritage: The
ArkaeVision project,” Digit. Appl. Arc
J.-E. Kilimann, D. Heitkamp, and P. Lensing, “An Augmented
Reality Application for Mobile Visualization of GIS-Referenced
Landscape Planning Projects,” in The 17th International
Conference on Virtual-Reality Continuum and Its Applications in
In
J. B. Madsen and C. B. Madsen, “Handheld Visual Representation of a Castle Chapel Ruin,” J. Comput. Cult. Herit., vol. 9, no. 1, Nov. 2015.
C. Flavián, S. Ibáñez-Sánchez, and C. Orús, “The impact of virtual, augmented and mixed reality technologies on the customer experience,” J. Bus. Res., 2019.
M. C. Tom Dieck and T. H. Jung, “Value of augmented reality at cultural heritage sites: A stakeholder approach,” J. Destin. Mark. Manag., vol. 6, no. 2, pp. 110–117, 2017.
W. Smith, H. Lewi, D. Constantinidis, and H. Stitt, “Directed looking and proximal content: two concepts for designing mobile guides to historic urban places,” Proc. 26th Aust. Comput. Interact. Conf. Des. Futur. Futur. Des., pp. 400–403, 2014.
G. Efstratios and P. George, “Case Study For Augmented Reality ( AR ) Application Of Portals For The Preservation Of Cultural Heritage,” Heraklion, Greece.
C. Bolognesi and D. Aiello, “FORGOTTEN ARCHITECTURE: SMART TOOLS FOR CULTURAL TOURISM IN THE CLOISTER OF THE PRIOR (SANTA MARIA DELLE GRAZIE, MILAN),” ISPRS - Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci., vol. XLII-2/W11, pp. 247–253, 2019.
P. A. Zandbergen and S. J. Barbeau, “Positional accuracy of assisted GPS data from high-sensitivity GPS-enabled mobile phones,” J. Navig., vol. 64, no. 3, pp. 381–399, 2011.
P. A. Zandbergen, “Accuracy of iPhone Locations: A Comparison of Assisted GPS, WiFi and Cellular Positioning,” Trans. GIS, vol. 13(s1), pp. 5–26, 2009.