An Intelligent Augmented Reality Mobile Application for Heritage Conservation Education Lap-Kei Lee1, Kwok Tai Chui1, Chung-Man Chiu1, Pui-Yu Lo1, Sai-Wing Tsoi1 and Nga-In Wu2 1 School of Science and Technology, Hong Kong Metropolitan University, Ho Man Tin, Kowloon, Hong Kong SAR, China 2 College of Professional and Continuing Education, Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China Abstract It has become a challenge to protect the built heritage and conserve the local traditional and cultural values of communities for future generations. Heritage conservation education is important as it increases the local communities’ awareness and appreciation of the value of the heritage and thus helps to achieve sustainable conservation. This paper presents the design of an intelligent mobile application for heritage conservation education. The mobile application has a collection of historical photos of local heritage in Hong Kong and provides a heritage map with markers for users to tour on heritages. Users can learn histories of heritages in a fun way by using its Augmented Reality function to compare the current and the past and by using its intelligent recognition function to retrieve historic photos relevant to the current camera view. A survey of 45 Hong Kong residents showed that the mobile app can serve as a fun way to learn history in Hong Kong and is a useful tool for heritage conservation education. Keywords 1 Heritage conservation education, mobile application, augmented reality, image recognition 1. Introduction The balancing between heritage conservation and urban development is hard. With the rapid growth of cities, governments open up lands for the essential needs and build supporting facilities. It has become a challenge to protect the built heritage and conserve the local traditional and cultural values of communities for future generations [1]. To achieve sustainable conservation, it is important to increase the local communities’ awareness and appreciation of the value of the heritage such that the society and communities would make wise decisions about protecting and preserving resources that define their culture and society [2-4]. Heritage conservation in Hong Kong. Hong Kong was a British Colony with very limited land and other natural resources. Since the change of its sovereignty from the British to China in 1997, heritage conservation has become one of the most popular issues due to economic, social, political, and environmental concerns [5]. Several historical constructions were demolished and led to a number of heritage preservation movements. Most of the participants were not the elders, but the younger and the better-educated persons; they know how to use the Internet, public forums, political parties, and mass media to express their views and mobilize the public so as to achieve their goals of heritage conservation [5]. This shows the importance of educating the communities about heritage conservation. International Conference on Smart Systems and Advanced Computing (Syscom-2021), December 25–26, 2021 EMAIL: lklee@hkmu.edu.hk (A. 1); jktchui@hkmu.edu.hk (A. 2); g1186390@study.hkmu.edu.hk (A. 3); g1186281@study.hkmu.edu.hk (A. 4); g1186666@study.hkmu.edu.hk (A. 5); ngain.wu@cpce-polyu.edu.hk (A. 6) ORCID: 0000-0001-9619-6041 (A. 1); 0000-0001-7992-9901 (A. 2) ©️ 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) Our contribution. This paper presents the design of an intelligent mobile application for heritage conservation education in Hong Kong. The mobile app has a collection of historical photos of local heritage in Hong Kong and provides a map with markers pointing out the locations of heritages. Due to urban renewal, historical buildings and old streets were demolished and face-lifted. The location markers are supplemented with the mobile’s GPS information such that when a user visits a location, nearby heritage can be easily identified. Then, users can use Augmented Reality to augment the camera view with historical photos, which provides an interesting way for exploring a heritage. Machine learning is also used to identify shops and objects, e.g., public transports and restaurants, such that relevant historical photos can be shown to help the users understand the difference between the current and old living styles. A survey of 45 Hong Kong residents showed that the mobile app can serve as a fun way to learn history in Hong Kong and is a useful tool for heritage conservation education. Organization. Section 2 reviews existing and related work on heritage conservation education and using augmented reality in education. Section 3 presents the design details of our mobile application. Section 4 gives the preliminary evaluation result. Section 5 discusses the preliminary evaluation result and concludes the paper. 2. Related work Augmented Reality in Education. Augmented Reality (AR) [6] is a technology that enables computer-generated virtual objects to augment the physical world, e.g., the camera view of a mobile device. One of the important applications of AR is education (see the surveys by Wu et al. [7] and Pochtoviuk et al. [8]). AR allows the simulations of various learning environments, e.g., injection and wound dressing for nurse training [9] [10], virtual block builders for learning games [11], and simulating solar system and street views for English vocabulary learning [12, 13]. AR can help engage, stimulate, and motivate students to learn [14]. AR can also promote active learning and enhance the learning experience [15]. Heritage conservation education in Hong Kong. The present solution of heritage conservation education in Hong Kong relies on the books and photo albums on history, museum exhibitions such as the “Hong Kong Story” permanent exhibition at the Hong Kong Museum of History [16], and guided tours on heritage such as the guide tours by Kaifong Tour [17]. The Hong Kong government has developed a mobile application “iM Guide” [18] to offer museum visitors a deeper understanding of the exhibits through text and multimedia content with indoor navigation (Figure 1). Figure 1: User interfaces of iM Guide Figure 2: User interfaces of Snappy Yet the non-interactive nature of books and photo albums make them not interesting and attractive enough for heritage conservation education; the museum exhibitions and guide tours also suffer from a lot of constraints including time and social distancing under the COVID-19 pandemic. Snappy [19] is a mobile application that collects old Hong Kong street view photos through Crowdsourcing (Figure 2). Everyone can view, edit, and upload photos. Yet due to crowdsourcing, some contents in the mobile application are incorrect and inconsistent. AR experiential learning in heritage tours. Experiential learning is the approach of learning from experience or learning by doing, and it has been shown as one of the best learning approaches for learners [20]. Experiential learning has been applied in various contexts, e.g., Artificial Intelligence education [21]. In heritage tourism, it is often that visitors require considerable information and knowledge to appreciate the cultural heritage fully [22] and AR has been proposed as a good solution to supplement the information and knowledge required [23]. A recent study by Han et al. [24] further showed that the AR experience can positively affect the satisfaction and the experiential value of visitors in heritage tours. Thus, we believe that AR can act as an effective and attractive tool for heritage conservation education in heritage tours. 3. Intelligent AR mobile application on heritage conservation education This section presents the detailed design of our intelligent mobile application for heritage conservation education in Hong Kong. There is a server for the storage of heritage information, including its text description, photos, and its location coordinates. The mobile application works as a client to retrieve the information for display, and it contains four components: a heritage map, heritage information pages, AR camera, and AI recognition. 3.1. Heritage map Figure 3: Heritage map: default (left), current location (middle), marker information (right) A heritage map is shown when the mobile application starts (Figure 3 left). The geographical coordinates of all the stored heritage locations are retrieved from the server database and displayed as pin markers at the locations on the Google Map. The upper right corner of the map shows a GPS button; when a user clicks on it, the map will jump to the user’s current location (Figure 3 middle). When a user clicks on a pin marker (Figure 3 right), the name of the heritage is shown. A second click on it will lead the user to its heritage information page. 3.2. Heritage information page Figure 4: Heritage information page: default (left), slider (middle), map (right) The heritage information page shows text descriptions and photos of a heritage (see Figure 4 left). It has also a slider, showing the latest and old photos of a heritage; users can slide left and right to compare the current and the past (Figure 4 middle). There are two buttons in the top navigation bar: one button is for voice guidance such that the users can listen to the speech of the text description when touring the heritage, saving the troubles of reading the text description on the mobile devices; the other button is for opening a Google map to guide navigation to the heritage (Figure 4 right). 3.3. AR camera Figure 5: AR camera: default (left), normal (middle) & AR views (right) Figure 6: Manual AR view On the heritage map, when the camera button is chosen, a selection menu for the AR camera and the other component AI recognition is shown (Figure 5 left). If AR camera is selected, the normal camera view is shown (Figure 5 middle). If the user clicks on the screen, the mobile application uses the GPS location of the mobile device to determine the nearby heritage, and the users can select a heritage and its historic photos at different times such that the photo can augment suitably the normal view (Figure 5 right). The AR camera has also a manual mode for users to augment the camera view using photos of old lifestyles, e.g., children on street (Figure 6). The AR camera provides a fun way to recur the past on the existing camera view. 3.4. AI recognition Figure 7: AI recognition: recognition result (left), relevant historic photos (right) The AI recognition function can be started from the selection menu previously introduced (Figure 5 left). To understand the culture of the tradition, a user can capture images of objects in some categories using the camera of the mobile device, e.g., restaurants and public transport, such that relevant historic photos in the corresponding category will be shown for the users to learn and compare the current and past. This function uses the cloud services [25] provided by the machine learning [26] platform IBM Watson, and more precisely, it uses the IBM Watson Image Recognition service to classify a camera image captured by the user into predefined categories. 4. Preliminary evaluation Participants and setting. We invited 45 Hong Kong residents to participate in a preliminary evaluation. Due to the COVID-19 pandemic, a screen recording video demonstrating the functionality of our mobile application and a questionnaire were provided to the participants. Result. The survey used a 5-point Likert scale (1: disagree, 2: partially disagree, 3: neutral, 4: partially agree, 5: agree). Table 1 shows the results of the survey. Table 1 Survey results on our mobile application (where the percentages are rounded to 1 decimal place) Item 1 2 3 4 5 (1) The AR camera brings the old life-style back 0.0% 6.7% 17.8% 68.9% 6.7% back to the real life. (2) Our mobile application can motivate learners 0.0% 4.4% 22.2% 57.8% 15.6% to learn the cultural values of local heritage. (3) It is fun to use our mobile application. 0.0% 0.0% 20.0% 60.0% 20.0% (4) Our mobile application is an effective way to 0.0% 2.2% 22.2% 62.2% 13.3% absorb new knowledge on heritage. 5. Discussion and conclusion In the preliminary evaluation, Table 1 shows that the majority of the participants agreed that the AR camera is an effective tool to recur the old lifestyle to reality. The majority of the participants also agreed that our mobile application provides a fun and motivating way for learners to learn the cultural values of local heritage. The majority of them also can gain new knowledge about the heritage through this application. Thus, our mobile application can serve as a fun way to learn history in Hong Kong and is a useful tool for heritage conservation education. This paper presented the design of a mobile application for heritage conservation education in Hong Kong. We hope that it can provide insights for other researchers and practitioners on heritage conservation on how the mobile application and augmented reality can be used to increase the local communities’ awareness and appreciation of the value of the heritage so as to achieve sustainable conservation. 6. References [1] M. E. Embaby, Heritage conservation and architectural education: "An educational methodology for design studios", HBRC Journal 10.3 (2014) 339-350. [2] N. Azman, S. A. Halim, I. Komoo, Integrated public education for heritage conservation: A case for Langkawi Global Geopark, in: G. Ainsworth, S. Garnett (Eds.), Dalam RIMBA: Sustainable Forest Livelihoods in Malaysia and Australia, 2009, pp. 151-156. [3] N. Azman, S. A. Halim, O. P. Liu, S. Saidin, I. Komoo, Public education in heritage conservation for geopark community, Procedia-Social and Behavioral Sciences 7 (2010) 504-511. [4] Ahmed, E., Chatzimisios, P., Gupta, B. B., Jararweh, Y., & Song, H. (2018). Recent advances in fog and mobile edge computing. Transactions on Emerging Telecommunications Technologies, 29(4), e3307. [5] T. L. D. Lu, Heritage conservation in post‐colonial Hong Kong, International Journal of Heritage Studies 15.2-3 (2009) 258-272. [6] M. Kaur, B. Gupta (2021) Metaverse Technology and the Current Market, Insights2Techinfo, pp.1 [7] H. K. Wu, S. W. Y. Lee, H. Y. Chang, J. C. Liang, Current status, opportunities and challenges of augmented reality in education, Computers & Education 62 (2013) 41-49. [8] S. Pochtoviuk, T, Vakaliuk, A. Pikilnyak, Possibilities of application of augmented reality in different branches of education, SSRN (2020). URL: https://ssrn.com/abstract=3719845. doi:10.2139/ssrn.3719845. [9] S. C. Ng, L. K. Lee, A. K. F. Lui, K. F. Wong, W. Y. Chan, H. H. Tam, Using immersive reality in training nursing students, International Journal of Innovation and Learning 27.3 (2020) 324- 343. [10] Mavneet Kaur, B. Gupta(2021) Metaverse Technologies and their Applications, Insights2Techinfo, pp.1 [11] L. K. Lee, C. P. Chau, K. N. Tsoi, N. L. Yang, N. I. Wu, A mobile game for learning English vocabulary with augmented reality block builder, in: S. Cheung, J. Jiao, L. K. Lee, X. Zhang, K. Li, Z. Zhan (Eds), Technology in Education: Pedagogical Innovations. ICTE 2019, volume 1048 of Communications in Computer and Information Science, Springer, Singapore, 2019, pp. 116- 128, doi: 10.1007/978-981-13-9895-7_11. [12] L. K. Lee, C. H. Chau, C. H. Chau, and C. T. Ng, Using augmented reality to teach kindergarten students English vocabulary, in Proceedings of 2017 International Symposium on Educational Technology, ISET 2017, IEEE, 2017, pp. 53-57. doi:10.1109/ISET.2017.20. [13] L. K. Lee, C. H. Chau, C. H. Chau, C. T. Ng, J. H. Hu, C. Y. Wong, L. C. Yu, N. I. Wu, Improving the experience of teaching and learning kindergarten-level English vocabulary using augmented reality, International Journal of Innovation and Learning 25.2 (2019) 110-125. [14] L. Kerawalla, R. Luckin, S. Seljeflot, A. Woolard, "Making it real": Exploring the potential of augmented reality for teaching primary school science, Virtual Reality 10. 3 (2006) 163-174. [15] B. M. Garrett, J. Anthony, C. Jackson, Using mobile augmented reality to enhance health professional practice education, Current Issues in Emerging eLearning 4.1 (2018) 10. [16] Hong Kong Museum of History, Audio guide for recreating a classic: The best features of the Hong Kong story, 2021. URL: http://hk.history.museum/en_US/web/mh/exhibition/current.html. [17] Kai Fong Tour, Homepage, 2021. URL: https://www.kaifongtour.com. [18] SenMedia Limited, iM Guide: Museum multimedia information guide system, 2021. URL: https://www.senmedia.com.hk/index.php/portfolio/im-guide-museum-multimedia-information- guide-system. [19] Snappy, Homepage, 2021. URL: https://snappy.collaction.hk. [20] D. A. Kolb. Experiential learning: Experience as the source of learning and development. FT Press, 2014. [21] S. F. Li, K. K. Ng, L. K. Lee, A study on the application of AI experiential learning in the architecture and design courses of a Taiwan university, in: R. Li, S. K. S. Cheung, C. Iwasaki, L. F. Kwok, M. Kageto (Eds), Blended Learning: Re-thinking and Re-defining the Learning Process. ICBL 2021, volume 128390 of Lecture Notes in Computer Science, Springer, Cham, 2021, pp. 103-115. doi:10.1007/978-3-030-80504-3_9. [22] D. L. Kerstetter, J. J. Confer, A. R. Graefe, An exploration of the specialization concept within the context of heritage tourism, Journal of Travel Research 39.3 (2001) 267-274. [23] M. C. tom Dieck, T. Jung, A theoretical model of mobile augmented reality acceptance in urban heritage tourism, Current Issues in Tourism 21.2 (2018) 154-174. [24] S. Han, J. H. Yoon, J. Kwon, Impact of experiential value of augmented reality: The context of heritage tourism, Sustainability 13.8 (2021) 4147. doi:10.3390/su13084147. [25] Wang, Z., Yang, Y., Liu, Y., Liu, X., Gupta, B. B., & Ma, J. (2020). Cloud-based federated boosting for mobile crowdsensing. arXiv preprint arXiv:2005.05304. [26] Gupta, B. B., Yamaguchi, S., & Agrawal, D. P. (2018). Advances in security and privacy of multimedia big data in mobile and cloud computing. Multimedia Tools and Applications, 77(7), 9203-9208.