Digitization of scientific and cultural heritage assets (scanning, reconstruction and modeling) makes it possible to supply an additional information layer of extended reality (augmented or virtual) for the preservation, research and active use of the most important national and sociocultural patterns in the era of the post-industrial information society. This paper analyzes the recommendations regarding the information and technical structure and implementation of specific functionality of a mobile application with augmented reality layer on the example of Ivan Puluj's digital museum. With the use of computer 3D technologies, the integration of a panoramic museum tour, the virtual space of Puluj's laboratory, and marker-based augmented reality objects was carried out within a single software and information complex dedicated to the scientific, technical, social and political activities of Ivan Puluj.
During the last decade, digitalization has been set forth to preserve cultural heritage [1]. Researchers [2] admit that high-quality digitization of cultural heritage is a rather laborious task. It requires thorough planning, significant financial investments, development of special technical and software tools. At the same time, high-quality tools and means of digitalization have great potential for further implementation [3]. The creation of digital copies and models of documents and artifacts together with digital capture and reconstruction techniques can be used for extended (virtual and augmented) reality purposes [4]. The development of new information technologies in this field will allow to enhance the exposure of cultural heritage assets to society. In some cases, it is possible to extend the life of artifacts whose physical originals have been lost. The authors [5] note that digitization technologies are widely used to support, improve and supplement traditional methods of preservation of cultural heritage elements within the framework of the CIPA Heritage Documentation initiative. At the same time, the reconstruction of objects of scientific and cultural heritage in the digital XR format extends the possibilities for the innovative technologies use in museum and exhibition activities [6].
Augmented reality (AR) technology expands the perception of the world. It allows one to enrich the environment with the computer-modeled layer which contains additional visual images and sounds. Users interact with means of presenting information through tactile perception.
Unlike virtual reality, which mostly requires the purchase of expensive immersion headsets, AR makes effective use of mobile devices. This makes it a more cost-effective option for users, developers, marketers, and museum and archive personnel.
Almost arbitrary object, any museum artifact can be digitized and reproduced using AR applications. The authors [7] emphasize that the use of virtual representation of museum exhibits has a number of advantages. In particular, it may use dynamic interactive elements and makes accessible those aspects of the artifact that may be hidden or inaccessible in the real exhibition. A common problem for museums is limiting visitors' access to artifacts due to their fragility, uniqueness, or lack of physical display space. Researchers [8] suggest using augmented reality to create new, natural interaction based opportunities for users. In particular, motion, gestures and speech allow a user to operate the digital content created for physically inaccessible exhibition samples. Digital content is text, audio accompaniment, images, animations, video files or 3D models. For effective formation of "cultural twin" [9] of appropriate completeness and complexity using heritage digitization technologies, developers must clearly understand how users will interact with interfaces of the information systems. When developing AR-applications, an important stage is the selection of the optimal combination of means by which relevant information is presented to users. At the same time, for proper perception, the content should correspond to their interests and requests.
Augmented reality improves orienting in the museum environment and enhances the effect of immersion into a particular historical location, historical period and cultural phenomenon. The user perceives information better with more natural interface. AR can also provide additional contextual information and a more realistic interaction with digital content and text annotations placed on top of images of the real-world objects. Mobile applications with elements of augmented reality have significant potential for the proper harvesting of user metadata. The collected information can be used for operative control of the application, expand the variety and functions of AR services. This will make it possible to enforce the educational capabilities of software by supplementing real-world objects with improved virtual data. In particular, AR-simulations with models and effects of interactive interaction allow intuitive, clear and comprehensive display of abstract information entities. At the same time, the users feel more immersed in the subject area. The work [10] demonstrated the achievement of improved learning outcomes and mastery of the material by a test group of users. Therefore, it is advisable to more actively introduce information technology applications for the reactivation of cultural heritage.
The goal of this article is to design an interactive mobile application with elements of augmented reality for presenting the scientific and cultural heritage of Ivan Puluj. At the same time, it is necessary to ensure the efficient reuse of the digital content. We aim to design and practically implement a convenient mobile application that can be used independently or under the guidance of museum guides to learn about the life and work of Ivan Puluj. A separate aspect of this work is the proper reproduction of the historical context of the time based on archival photos from the Electronic Fund of Ivan Puluj. The use of audio content makes it possible to improve the perception of detailed 3D models of inventions and scanned sculptural art objects.
This article analyzes the features of implementing augmented reality technology when creating a mobile application for a museum of scientific and cultural heritage. Marques and Costello [11] emphasize that modern means of content development with elements of augmented reality have become more accessible and technologically advanced. Currently, use of the AR applications are limited to museum exhibition. They can be used more actively, in particular to supplement existing exhibitions. One of the reasons for this is the limitation of development skills in the field of museum-related augmented reality. Therefore, researchers suggest more active introduction of the innovative AR elements into exhibition activities. This will contribute to the formation of positive impressions of the exhibitions in users, the creation of a memorable emotional background from casual educational activities, the improvement of communication links with the elements of the exposition and the improvement of navigation in the exhibition halls.
Authors of [12] recommend forming a clear awareness of how the use of AR elements will enrich the experience of museum visitors with new knowledge, feelings and impressions, when developing specialized museum applications. It is advisable to design display of AR elements on the basis of a mobile device perception as a kind of portal into a virtual interactive scientific and cultural space. Active movements of the mobile device in all planes should cause changes of the spatial position of the modeled entities and markers. It is necessary [13]: to ensure a comfortable reception of the information content by the user; to create intuitive instructions for working with the application; to minimize the inflow of extra information.
Since a visitor can interact with the exposition individually or as a participant of a group tour, it is necessary to provide different scenarios and levels of using the information content with the mobile AR application. To ensure the safety of users interacting with augmented reality, special conditions should be provided. In particular, the possibility of limiting the space due to active movements of the limbs and warnings about the possible running into another visitor are to be ensured. The interface of the museum application [12] should have a minimalist design and is to be as much intuitive as possible. It is advisable to use icons to indicate functional elements. The maximum of the working area of the mobile device display should be used to display AR content with high resolution and image refresh rate. The flickering will cause an overload of the visual organs and, as a result, dizziness, so this defect is to be fixed first of all. The vast majority of mobile users know how to tap, drag, and swipe across objects. Therefore, it is advisable to use these interaction patterns in the user interface of the museum AR application as well. The work [14] recommends the use of audio accompaniment, sound effects of interaction with the virtual environment, and background music. After all, all this contributes to the effect of immersion.
Mobile devices can successfully complement a visit to a museum with additional content adapted, inter alia, for the educational purposes of the youngsters [15]. At the same time, it is possible to increase the level of interaction between visitors and exhibits. Smartphones should help visitors to explore the exhibition, but should not disperse attention on the interaction with the application itself. The information system used by the mobile application should accumulate information about museum exhibits and form connections between these entities and the virtual guide. Developing a mobile application for a small museum is a non-trivial task, as it can be considered as a single-use interface that users will interact with for a short period of time. Within the framework of the cultural and educational paradigm, the museum application for pupils and students should be intuitive and visually attractive. It is important that it does not cause emotional and physical fatigue during use. Therefore, audiovisual content should be played for several minutes. Animated AR elements should be used to attract attention or clearly instruct users [10].
At the previous stage of this research [16], a description of the reconstruction works regarding the virtualization of the scientific and cultural heritage of the famous Ukrainian scientist Ivan Puluj was presented (see Figure 1). A similar technique was used in [17] when analyzing the possibilities of using augmented reality in the prototyping of cyber-physical systems to improve cyber security education. In particular, a promotional web portal and the development of a general concept of digitization of archival materials were done, including using the capabilities of the Electronic Fund of Ivan Puluj [18] in the institutional repository of the TNTU. This work describes the key stages of creating a mobile application with AR elements. The use case diagram is presented in Figure 2. The designed mobile application will be an integral element of the software complex of the digital museum of Ivan Puluj.
The photogrammetric method was used to create 3D models of the memorial sculptures of Ivan Puluj. It is based on the processing of several hundreds of photos taken using digital camera with wideangle lens in high resolution, with partial overlap of image areas taken at different angles of view. Separately, several circular passes at different heights relative to the scanning object and additional pictures of important details were performed. On the basis of a set of photographs using specialized algorithms, a dense point cloud, a depth map, a structured mesh of an object are created with the ability to generate post-processing of the model (for example, texturing) and export in the required format (see Figure 3).
In the vast majority of cases, the Reality Capture photogrammetry software is used. This software can process photo materials much faster than its competitors and operate with a large number of images on a standard configuration computer. This creates opportunities for checking data right at the shooting location. The flexible toolkit allows you to adjust the quality of the generated model, perform smoothing and simplifying, as well as reduce the file size and increase the performance of the applications into which the model is imported. However, there is a significant limitation – the production of a highpolygonal mesh is possible only on a computer with a modern Nvidia discrete graphics card that supports the software-hardware architecture of parallel computing CUDA. Also, Reality Capture has a specific pricing and payment policies for created models. However, the export of the result is organized according to the principle of PPI (Pay-Per-Input) – technology with payment for input, when the type and size of uploaded photos are evaluated. After the license purchase, the input data may be reused at no additional charge. The user can process all his data for free, get information about the evaluation of the value of the 3D model. This allows optimizing financial costs for creating completed models.
During field work, it was established that at least 300-400 photographs are needed to create a high-quality 3D model of the monument. Since direct sunlight creates an overexposed color gamut, reflections appear, surface detail decreases and this directly reduces the quality of textures. Therefore, uniform and diffused lighting is optimal.
The 3D content of the engineering and scientific heritage of Ivan Puluj includes vacuum tubes, radiometers of original design, equipment for experiments with cathode rays and X-rays, accessories for physical educational demonstrations. Autodesk 3ds Max and Autodesk Maya graphic editors were used for graphics design purposes when creating a mobile AR application. In particular, their capabilities for high-quality modeling of three-dimensional objects, animation and visualization are well-known. The level of development of modern tools for creating virtual reality (VR) makes it possible to use these information technologies to reproduce historical places that have undergone a significant transformation or even ceased to exist.
Scholars of the scientific heritage of Ivan Puluj single out in the activity of Ivan Puluj a fruitful period from 1884 till 1918, which were obtained, in particular, in the physics laboratory at the Prague Polytechnic (for a review, see [19]). It was here that Puluj discovered the ionization property of X-rays, previously unknown, during their passage through rarefied gases, investigated their place of origin and spatial distribution. In addition, he managed to give a correct scientific explanation of the nature of Xrays and their mechanism of occurrence, and the cathode tubes of their own design were of superior quality. Archive photos of Ivan Puluj's laboratory, the detailed description of scientific and demonstration equipment allows for their credible 3D reconstruction (see Figure 4).
In the virtual space of Ivan Puluj's laboratory, the interior and devices are reproduced on the basis of historical photographs. Digital models of the Crooks radiometer and instruments designed by I. Puluj namely the radiometer with a biplane cathode, the cathode lamp, the phosphorescent lamp with a cupshaped cathode, the incandescent lamp, the device for measuring the mechanical equivalent of heat, phosphorescent lamps with mica plates, and the device for determining the acceleration of free fall can be viewed from different distances and at different angles in the virtual interior of the laboratory (see Figure 5).
To solve the tasks of AR support of the exhibition of the Ivan Puluj digital museum [https://pulujmuseum.tntu.edu.ua/], it is currently optimal choice to use the Unity software ecosystem. The flexibility and effectiveness of this approach is confirmed by the experience of other research groups [8,9]. Unity is a multi-platform environment for developing augmented reality spaces, virtual visualizations, training simulators and games. It contains a fully integrated professional game engine that can provide rendering functions, environment physics simulation and implementation of scene control scripting mechanisms. Currently, Unity has a convenient, flexible and functional project porting mechanism for common OSs, game consoles, VR and AR gadgets. Unity actively supports DirectX, OpenGL and WebGL frameworks. Unity software provides opportunities for rapid prototyping and evaluation of interaction processes with virtual or augmented reality objects. AR technology allows for real-time synchronization between the entities of the virtual and real worlds. A significant advantage of the Unity environment for modeling AR elements is the availability and relative simplicity of settings when creating a mobile application using the Vuforia plugin from PTC. Vuforia includes AR-adapted modifications of computer vision algorithms, real-time marker tracking for objects with flat surfaces and multiple sides or images, the use of multi-targeting. In Vuforia, there are built-in functions for recognizing ready-made 3D models, scanned objects with rich surface texture and constant shape. Integrated image tracking technology allows you to effectively place and orient virtual 3D objects, models or multimedia content in space, associating them with real objects during viewing through mobile devices [20]. It is worth noting that the virtual object is positioned on top of the real marker image so that the observer's sight generates the illusion as if the virtual content is an integral part of the real world.
In Figure 6 the interface of interactive mobile application (requires Android 9 or newer) with elements of augmented reality for the digital museum of Ivan Puluj is shown. The UI is based on intuitive icons accompanied by textual explanations. For the convenience of the user, there is an option to switch the language of information content display. The educational part of the application, which instructs the visitor on the use of the AR functionality, is important to be read. Detailed information about the project of the Digital Museum of Ivan Puluj, the possibility of viewing full-size detailed 3D models of the scientist's inventions is provided elsewhere.
In this project the virtual panoramic tour has been adapted for mobile devices (see Figure 7). At the same time, the possibility of a guided video tour or independent research of the virtual exposition has been created. At the same time, each stand is accompanied by audiovisual support, and the user can easily adjust the accompaniment and natural tactile navigation through the museum according to his preferences. To demonstrate 3D models of the most remarkable inventions of Ivan Puluj, a set of twelve cards was created, which have embedded markers of augmented reality (see Figure 8). For this, archival drawings and photos of exhibit devices developed by a famous physicist and electrical engineer were selected. When viewing the models (see Figure 9), it is possible to examine the design of the devices in detail with audio accompaniment.
The use of augmented reality in museum mobile applications expands the possibilities of visitors' general perception of information exhibitions, allows to involve the user in active interaction with virtual elements. At the same time, an unforgettable experience of immersion in the amazing world of scientific discoveries and cultural specificity of a specific historical era is formed. The use of 3D modeling and photogrammetry makes the visual component more realistic and contributes to the formation of a sustainable interest in augmented reality and the enrichment of the scientific worldview of visitors, especially young people.
The created mobile application with elements of augmented reality can act as a hub. It allows not only acquainting visitors with the assets of the museum exhibition, but also offers them the directions of transitions-portals for a comprehensive own study of the scientific and cultural heritage of one of the most famous Ukrainian scientists, physicist, electrical engineer and public figure Ivan Puluj. The proposed format of the digital museum with elements of augmented reality has a significant potential for integration of atypical and innovative means for the preservation of the national scientific and cultural heritage, the tourist industry and scientific and educational activities.
The authors of the article and the Cyber-Physical Systems Laboratory are grateful to Ternopil Ivan Puluj National Technical University for financial support of the project.
[4] S.Z. Salleh, A.R. Bushroa, Bibliometric and content analysis on publications in digitization
technology implementation in cultural heritage for recent five years (2016–2021), Digital
Applications in Archaeology and Cultural Heritage, 25 (2022) e0025 (
Niebling, A. Seidel-Grzesińska (Eds.), Digital Research and Education in Architectural Heritage, Communications in Computer and Information Science, 817 (2018) 215-245. Springer Cham. doi: https://doi.org/10.1007/978-3-319-76992-9_13. [6] T. Caciora, G.V. Herman, A. Ilies et al, The Use of Virtual Reality to Promote Sustainable Tourism: A Case Study of Wooden Churches Historical Monuments from Romania, Remote Sensing 13 (2021) 1758. doi: https://doi.org/10.3390/rs13091758. [7] J. Malpas, Cultural heritage in the age of new media, in: New Heritage. New Media and Cultural
Heritage; Y.E. Kalay, T. Kvan, J. Affleck, (Eds.), Routledge: London, UK, 2008; 13–26.
[8] P. Kyriakou, S. Hermon, Can I touch this? Using Natural Interaction in a Museum Augmented
Reality System, Digital Applications in Archaeology and Cultural Heritage, 12 (2019) e00088.
doi: https://doi.org/10.1016/j.daach.2018.e00088.
[9] R.G. Boboc, M. Duguleană, G.-D. Voinea et al, Mobile Augmented Reality for Cultural Heritage:
Following the Footsteps of Ovid among Different Locations in Europe, Sustainability, 11 (2019)
1167 (
Intelligence. ACM Comput. Surv., (2022) 1-35. doi: https://doi.org/10.1145/3557999
[14] N. Babich. UX Design Principles for Augmented Reality, 2020. URL:
https://xd.adobe.com/ideas/principles/emerging-technology/ux-design-principles-for-augmentedreality/
[15] V. Cesario, D. Petrelli, and V. Nisi, Teenage Visitor Experience: Classification of Behavioral
Dynamics in Museums, in: Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems. Association for Computing Machinery, NY, USA, paper 207 (2020) 1–13.
doi: https://doi.org/10.1145/3313831.3376334.
[16] O. Kramar, Yu. Skorenkyy, O. Rokitskyi, and T. Kramar, Application of virtual and augmented
reality technologies for creation of a digital museum of scientific and cultural heritage of Ivan
Puluj, 1st International Workshop on Information Technologies: Theoretical and Applied
Problems, ITTAP 2021, 3039 (2021), 285-293.
[17] Yu. Skorenkyy, R. Kozak, N. Zagorodna et al, Use of augmented reality-enabled prototyping of
cyber-physical systems for improving cyber-security education, J. Phys.: Conf. Ser. 1840 (2021)
012026. doi: https://doi.org/10.1088/1742-6596/1840/1/012026
[18] Electronic Fund of Ivan Puluj. URL: http://elartu.tntu.edu.ua/handle/123456789/329
[19] R. Gajda, R. Plyatsko, Ivan Puluj: life and works, Lviv, NTSh, 2019 [in Ukrainian].
[20] I. Pedersen, N. Gale, P. Mirza-Babaei, and S. Reid, More than meets the eye: the benefits of
augmented reality and holographic displays for digital cultural heritage, J. Comput. Cult. Herit. 10
(2017) 11 (