Digital Augmented Reality App for Historic Architecture⋆

Abstract

This paper aims to create an augmented reality application to reconstruct damaged historical buildings. We have developed a mobile Augmented Reality (AR) application for Android devices that offers a new way to experience historic architecture. The app uses QR code scanning and GPS localization to display the reconstructed monument at the original location of the building. We have designed the app to cater to three types of users: non-logged-in users, logged-in users, and administrators. Users can explore and rate historical sites, providing valuable feedback to our development team. We have rigorously tested the app and engaged with users, and we believe it will be an effective tool for historical education and a model for future AR applications in different contexts. Our goal is to make the app user-friendly and easy to navigate, even for individuals who are not familiar with smartphones. Additionally, we plan to introduce a feature that locks the user's phone while the app is in use, allowing museums to offer phone rentals for visitors who do not have smartphones.

Keywords

• eol>Augmented Reality
• virtual reality
• mobile application
• 3D reconstruction
• cultural heritage
• QR code 1

1. Introduction

Regarding technology, both virtual Reality (VR) and Augmented Reality (AR) have their merits. VR creates a digital version of a real-world setting, while AR overlays virtual objects onto the real world [ 1 ]. After careful consideration, we have determined that AR is the superior choice for our application. Its more straightforward implementation, lower system costs, and minimal need for additional hardware make it a more practical solution. Furthermore, the potential discomfort and balance issues that some users, particularly the elderly, may experience with VR [ 2 ] could pose safety concerns for our application. By choosing AR, we can ensure a more inclusive and user-friendly experience. Unlike virtual reality, augmented reality enriches the real world with virtual objects, which creates a wholly digital environment around the user. While older AR systems require bulky hardware and software, the evolution of mobile technology has made it possible to experience augmented reality on the go. This shift has significantly increased the accessibility of AR, making it a viable and user-friendly option for our application. In this thesis, we present the design of a mobile AR application for Android devices. The app will allow users to view 3D reconstructions of historically significant buildings that no longer exist. Users can see the reconstructed monument on their mobile phone screens by scanning a QR code using the back camera. Additionally, we will implement GPS to display the reconstructed monument at the original location of the building. This feature will provide a more immersive experience for users. Unlike other attempts that require additional hardware [ 3 ], our advantage is that users already have all the necessary hardware in their pockets, making the project more cost-effective. 3rd Workshop on Artificial Intelligence for Cultural Heritage (AI4CH 2024, https://ai4ch.di.unito.it/), co-located with the 23nd International Conference of the Italian Association for Artificial Intelligence (AIxIA 2024). 26-28 November 2024, Bolzano, Italy ∗ Corresponding author - Natalia Kryvinska. † These authors contributed equally.

A few applications serve similar purposes [ 4 ], but none allow the user to enter the projected building. Our application will make this possible. This means our monument models must be more detailed and have proper interiors. Another challenge is making GPS locations highly accurate because people will move small distances inside the reconstruction [ 5 ]. We also want users to be able to rate the places they visit and optionally add them to their favorites list. This will serve as feedback for us, letting us know what people are most interested in and also letting other users know what is worth seeing.

2. Model building

The application requires input from a QR code scanned by a phone camera and GPS localization [ 6 ]. Once the code is scanned, our application will display the corresponding model and sound of the exhibition on mobile phones (Android/iOS). Depending on customer needs and resources, the museum hosts or other servers will provide the application's server-side.

System requirements o For the client-side user will need an Android 4.0+ or an ios system phone o For the server-side user will need to provide the HTTP server with models and sounds o Application will be available from Google Play and App Store Application requirements

o The application will need a phone with GPS and a camera The class diagram illustrates the program's class layers and their relationships. These class layers were created to adhere to Spring design principles. There are three primary layers: model, repository, and controller, as well as a view layer. All repositories and controllers follow the same generic class pattern. We have designed the application to accommodate three types of users: non-logged-in users, loggedin users, and administrators. Each role has access to different actions. For example, an administrator can check user accounts, objects, and personal accounts.

3. Model testing practical part

3.1. Test cases

Test case 1. Reconstruction of the textile spinning machine requirements: smartphone with working GPS module newest version of the application installed scenario: The user opens the application and picks the „see reconstruction” button in the interface. The phone's back camera will become active, and the user will be asked to scan the QR code on the podium near the exhibition. After that, guidelines will appear to help the user finish reconstructing the object. The application will use the GPS module to guide the user to the proper location and where to point their device. After a brief moment, the application will present an image on the phone.

result: the 3D image of the reconstructed textile spinning machine appears on the screen. Test case 2. Reconstruction of the watchtower requirements: smartphone with working GPS module newest version of the application installed scenario: The user opens the application and picks the „see reconstruction” button in the interface. The phone's back camera will become active, and the user will be asked to scan the QR code on the podium near the exhibition. After that, guidelines will appear to help the user finish reconstructing the object. The application will use the GPS module to guide the user to the proper location and where to point their device. After a brief moment, the application will present an image on the phone. From this moment, the user can walk inside the watchtower. result: 3D image of reconstructed watchtower, both interior and exterior. appears on the screen. Test case 3. Reconstruct the old manor house requirements: ‒ ‒ smartphone with working GPS module newest version of the application installed scenario: The user should go to the nearest manor house. The user opens the application and picks the “see reconstruction” button in the interface. Users will be asked to scan the QR code on the podium near the manor house. The user shall move the camera to where he was told in the application {reconstructed object appears on the screen, the animations with the walking environment are displayed, and sound is played}.

Test case 4. Changing reconstruction objects requirements: ‒ ‒ smartphone with working GPS module newest version of the application installed scenario: The user opens the application and picks the “see reconstruction” button in the interface. Users will be asked to scan the QR code on the podium near the exhibition. The user shall move the camera to the place where he was told in the application {reconstructed object appears on the screen}. The user shall go to the next exhibition and click the "see reconstruction" button again; as in previous steps, the user should move the camera to the place where the application displays {Other reconstructed objects appear on the screen}.

Test case 5. Changing localization while observing objects. requirements: ‒ ‒ ‒ smartphone with working GPS module newest version of the application installed rooted phone with fake GPS application {android only} scenario: The user opens the application and picks the “see reconstruction” button in the interface. Users will be asked to scan the QR code on the podium near the exhibition. Users should use fake GPS applications to change localization to other places a minimum of 50 meters. {User should be told about localization change and render process should be aborted, the main screen with "reconstruct" button should be displayed}.

3.2. Application visualization for tests

The development of our mobile Augmented Reality (AR) application [ 7, 8 ] offers a transformative approach to experiencing historical architecture that has long been lost. By leveraging widely accessible mobile technology, users can engage with 3D reconstructions of significant buildings through a seamless interface that utilizes QR code scanning and GPS localization. The advantages of AR over Virtual Reality – such as ease of implementation, reduced costs, and minimal hardware requirements – position our application as a user-friendly alternative, particularly for those facing VR technology challenges. Our application enhances the educational value of museum exhibits and fosters user interaction by allowing individuals to explore and rate historical sites, thus contributing valuable feedback to our development team. The architectural fidelity of the models, coupled with immersive audio experiences, aims to provide a rich and engaging environment that invites users to delve deeper into their cultural heritage. Through rigorous testing and user engagement, we anticipate this application will be an effective tool for historical education and a model for future AR applications in various contexts. This project will significantly enhance how individuals experience and connect with their historical surroundings.

4. Conclusions

The modern-day reliance on 2D media has made these exhibits less engaging, especially for young people, which could lead to a loss of our cultural heritage. To address this issue, we propose bringing reconstructed exhibits to life in 3D with added sound for a more immersive experience. Our research shows that the application should be user-friendly and easy to navigate, even for individuals unfamiliar with smartphones. We plan to automate the application as much as possible by utilizing GPS modules and QR codes. The GPS will help us identify the user's location, while QR codes will ensure the accurate display of relevant information. This approach will make the application simple, consistent, and easy to use. By allowing visitors to move around exhibits while experiencing the sights and sounds of the past, we aim to help them remember and connect with history on a deeper level as we provide more sensory stimuli. Our main challenge lies in creating new 3D models that are readily available. However, we hope that with the full approval of museums, we can integrate the cost of making these models into their exhibit budgets. Looking ahead, we plan to introduce a feature that locks the user's phone while the application is in use, allowing museums to offer phone rentals for visitors without smartphones. Additionally, we will explore alternative input sources for our localization mechanism, such as using Wi-Fi instead of QR code scanning, to streamline the application further.

Acknowledgements

Funded by the EU NextGenerationEU through the Recovery and Resilience Plan for Slovakia under the project No. 09I03-03-V01-000121

References

[1] J.-T. Hwang i T.-C. Chu, 3D Building Reconstruction By Multiview Images And The Integrated Application With Augmented Reality , Int. Arch. Photogramm. Remote Sens . Spat. Inf. Sci., t. XLI-B1, s. 1235 - 1241 , paź. 2016 , doi: 10.5194/ isprs-archives- XLI-B1- 1235- 2016 .

[2] C. Regan , An investigation into nausea and other side-effects of head-coupled immersive virtual reality, Virtual Real ., t. 1 , nr 1, s. 17 - 31 , cze. 1995 , doi: 10.1007/BF02009710.

[3] Y. Liu i Y. Wang , AR-View : An augmented reality device for digital reconstruction of Yuangmingyuan , International Symposium on Mixed and Augmented Reality - Arts , Media and Humanities, Orlando, USA, paź. 2009 , 3- 7 . doi: 10 .1109/ISMAR-AMH. 2009 . 5336752 .

[4] C. Panou , L. Ragia , D. Dimelli, i K. Mania , „ Outdoors Mobile Augmented Reality Application Visualizing 3D Reconstructed Historical Monuments”:, w Proceedings of the 4th International Conference on Geographical Information Systems Theory, Applications and Management, Funchal, Madeira, Portugal, 2018 , s. 59 - 67 . doi: 10 .5220/0006701800590067.

[5] B. Jach , Ł. Dłużniewski, Ł. Radosz, K. Likus , AR reconstruction of historical buildings . Seminar Work , 2021 .

[6] M. Hincapié , C. Díaz , M. I. Zapata-Cárdenas , H. Rios , D. Valencia , & D. Güemes-Castorena, Augmented reality mobile apps for cultural heritage reactivation . Computers & Electrical Engineering , 93 , 2021 , 107281 . https://doi.org/10.3390/ijgi7120463

[7] Hincapié , M. , Díaz , C. , Zapata-Cárdenas , M. I. , Rios , H. D. J. T. , Valencia , D. , & GüemesCastorena , D. Augmented reality mobile apps for cultural heritage reactivation . Computers & Electrical Engineering , 93 , 2021 , 107281 .

[8] Angelopoulou , A. , et al. Mobile augmented reality for cultural heritage . In Mobile Wireless Middleware, Operating Systems, and Applications: 4th International ICST Conference , Mobilware 2011 , London, UK, 2012 , pp. 15 - 22 .