=Paper=
{{Paper
|id=Vol-3309/short11
|storemode=property
|title=Peculiarities of Augmented Reality Usage in a Mobile Application: the Case of the Ivan Puluj Digital Museum
|pdfUrl=https://ceur-ws.org/Vol-3309/short11.pdf
|volume=Vol-3309
|authors=Taras Kramar,Oleksii Duda,Oleksandr Kramar,Oleksandr Rokitskyi,Volodymyr Pasichnyk
|dblpUrl=https://dblp.org/rec/conf/ittap/KramarDKRP22
}}
==Peculiarities of Augmented Reality Usage in a Mobile Application: the Case of the Ivan Puluj Digital Museum==
https://ceur-ws.org/Vol-3309/short11.pdf
Peculiarities of Augmented Reality Usage in a Mobile
Application: the Case of the Ivan Puluj Digital Museum
Taras Kramara, Oleksii Dudaa, Oleksandr Kramara, Oleksandr Rokitskyia and Volodymyr
Pasichnykb
a
Ternopil Ivan Puluj National Technical University, 56 Ruska St, Ternopil, UA46001, Ukraine
b
Lviv Polytechnic National University, 12 Bandera St, Lviv, UA79000, Ukraine
Abstract
Digitization of scientific and cultural heritage assets (scanning, reconstruction and modeling)
makes it possible to supply an additional information layer of extended reality (augmented or
virtual) for the preservation, research and active use of the most important national and socio-
cultural patterns in the era of the post-industrial information society. This paper analyzes the
recommendations regarding the information and technical structure and implementation of
specific functionality of a mobile application with augmented reality layer on the example of
Ivan Puluj's digital museum. With the use of computer 3D technologies, the integration of a
panoramic museum tour, the virtual space of Puluj's laboratory, and marker-based augmented
reality objects was carried out within a single software and information complex dedicated to
the scientific, technical, social and political activities of Ivan Puluj.
Keywords
Augmented reality, mobile app, cultural heritage, digital museum
1. Introduction
During the last decade, digitalization has been set forth to preserve cultural heritage [1].
Researchers [2] admit that high-quality digitization of cultural heritage is a rather laborious task. It
requires thorough planning, significant financial investments, development of special technical and
software tools. At the same time, high-quality tools and means of digitalization have great potential for
further implementation [3]. The creation of digital copies and models of documents and artifacts
together with digital capture and reconstruction techniques can be used for extended (virtual and
augmented) reality purposes [4]. The development of new information technologies in this field will
allow to enhance the exposure of cultural heritage assets to society. In some cases, it is possible to
extend the life of artifacts whose physical originals have been lost. The authors [5] note that digitization
technologies are widely used to support, improve and supplement traditional methods of preservation
of cultural heritage elements within the framework of the CIPA Heritage Documentation initiative. At
the same time, the reconstruction of objects of scientific and cultural heritage in the digital XR format
extends the possibilities for the innovative technologies use in museum and exhibition activities [6].
Augmented reality (AR) technology expands the perception of the world. It allows one to enrich
the environment with the computer-modeled layer which contains additional visual images and sounds.
Users interact with means of presenting information through tactile perception.
ITTAP’2022: 2nd International Workshop on Information Technologies: Theoretical and Applied Problems, November 22–24, 2022,
Ternopil, Ukraine
EMAIL: kramartar18@gmail.com (A.1); oleksij.duda@gmail.com (A.2); kramaroitntu@gmail.com (A.3); rokitskyi.o@gmail.com (A.4);
vpasichnyk@gmail.com (B.1)
ORCID: 0000-0001-8060-0169 (A.1); 0000-0003-2007-1271(A.2); 0000-0003-0805-3732 (A.3); 0000-0002-1621-3225 (A.4); 0000-0002-
5231-6395 (B.1)
©️ 2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
Unlike virtual reality, which mostly requires the purchase of expensive immersion headsets, AR
makes effective use of mobile devices. This makes it a more cost-effective option for users, developers,
marketers, and museum and archive personnel.
Almost arbitrary object, any museum artifact can be digitized and reproduced using AR
applications. The authors [7] emphasize that the use of virtual representation of museum exhibits has a
number of advantages. In particular, it may use dynamic interactive elements and makes accessible
those aspects of the artifact that may be hidden or inaccessible in the real exhibition. A common problem
for museums is limiting visitors' access to artifacts due to their fragility, uniqueness, or lack of physical
display space. Researchers [8] suggest using augmented reality to create new, natural interaction based
opportunities for users. In particular, motion, gestures and speech allow a user to operate the digital
content created for physically inaccessible exhibition samples. Digital content is text, audio
accompaniment, images, animations, video files or 3D models. For effective formation of "cultural
twin" [9] of appropriate completeness and complexity using heritage digitization technologies,
developers must clearly understand how users will interact with interfaces of the information systems.
When developing AR-applications, an important stage is the selection of the optimal combination of
means by which relevant information is presented to users. At the same time, for proper perception, the
content should correspond to their interests and requests.
Augmented reality improves orienting in the museum environment and enhances the effect of
immersion into a particular historical location, historical period and cultural phenomenon. The user
perceives information better with more natural interface. AR can also provide additional contextual
information and a more realistic interaction with digital content and text annotations placed on top of
images of the real-world objects. Mobile applications with elements of augmented reality have
significant potential for the proper harvesting of user metadata. The collected information can be used
for operative control of the application, expand the variety and functions of AR services. This will make
it possible to enforce the educational capabilities of software by supplementing real-world objects with
improved virtual data. In particular, AR-simulations with models and effects of interactive interaction
allow intuitive, clear and comprehensive display of abstract information entities. At the same time, the
users feel more immersed in the subject area. The work [10] demonstrated the achievement of improved
learning outcomes and mastery of the material by a test group of users. Therefore, it is advisable to
more actively introduce information technology applications for the reactivation of cultural heritage.
The goal of this article is to design an interactive mobile application with elements of augmented
reality for presenting the scientific and cultural heritage of Ivan Puluj. At the same time, it is necessary
to ensure the efficient reuse of the digital content. We aim to design and practically implement a
convenient mobile application that can be used independently or under the guidance of museum guides
to learn about the life and work of Ivan Puluj. A separate aspect of this work is the proper reproduction
of the historical context of the time based on archival photos from the Electronic Fund of Ivan Puluj.
The use of audio content makes it possible to improve the perception of detailed 3D models of
inventions and scanned sculptural art objects.
2. Object of Study
This article analyzes the features of implementing augmented reality technology when creating
a mobile application for a museum of scientific and cultural heritage. Marques and Costello [11]
emphasize that modern means of content development with elements of augmented reality have become
more accessible and technologically advanced. Currently, use of the AR applications are limited to
museum exhibition. They can be used more actively, in particular to supplement existing exhibitions.
One of the reasons for this is the limitation of development skills in the field of museum-related
augmented reality. Therefore, researchers suggest more active introduction of the innovative AR
elements into exhibition activities. This will contribute to the formation of positive impressions of the
exhibitions in users, the creation of a memorable emotional background from casual educational
activities, the improvement of communication links with the elements of the exposition and the
improvement of navigation in the exhibition halls.
Authors of [12] recommend forming a clear awareness of how the use of AR elements will enrich
the experience of museum visitors with new knowledge, feelings and impressions, when developing
specialized museum applications. It is advisable to design display of AR elements on the basis of a
mobile device perception as a kind of portal into a virtual interactive scientific and cultural space. Active
movements of the mobile device in all planes should cause changes of the spatial position of the
modeled entities and markers. It is necessary [13]:
to ensure a comfortable reception of the information content by the user;
to create intuitive instructions for working with the application;
to minimize the inflow of extra information.
Since a visitor can interact with the exposition individually or as a participant of a group tour, it
is necessary to provide different scenarios and levels of using the information content with the mobile
AR application. To ensure the safety of users interacting with augmented reality, special conditions
should be provided. In particular, the possibility of limiting the space due to active movements of the
limbs and warnings about the possible running into another visitor are to be ensured. The interface of
the museum application [12] should have a minimalist design and is to be as much intuitive as possible.
It is advisable to use icons to indicate functional elements. The maximum of the working area of the
mobile device display should be used to display AR content with high resolution and image refresh rate.
The flickering will cause an overload of the visual organs and, as a result, dizziness, so this defect is to
be fixed first of all. The vast majority of mobile users know how to tap, drag, and swipe across objects.
Therefore, it is advisable to use these interaction patterns in the user interface of the museum AR
application as well. The work [14] recommends the use of audio accompaniment, sound effects of
interaction with the virtual environment, and background music. After all, all this contributes to the
effect of immersion.
Mobile devices can successfully complement a visit to a museum with additional content adapted,
inter alia, for the educational purposes of the youngsters [15]. At the same time, it is possible to increase
the level of interaction between visitors and exhibits. Smartphones should help visitors to explore the
exhibition, but should not disperse attention on the interaction with the application itself. The
information system used by the mobile application should accumulate information about museum
exhibits and form connections between these entities and the virtual guide. Developing a mobile
application for a small museum is a non-trivial task, as it can be considered as a single-use interface
that users will interact with for a short period of time. Within the framework of the cultural and
educational paradigm, the museum application for pupils and students should be intuitive and visually
attractive. It is important that it does not cause emotional and physical fatigue during use. Therefore,
audiovisual content should be played for several minutes. Animated AR elements should be used to
attract attention or clearly instruct users [10].
3. Materials and Methods
At the previous stage of this research [16], a description of the reconstruction works regarding
the virtualization of the scientific and cultural heritage of the famous Ukrainian scientist Ivan Puluj was
presented (see Figure 1). A similar technique was used in [17] when analyzing the possibilities of using
augmented reality in the prototyping of cyber-physical systems to improve cyber security education.
Figure 1: Digital means of virtualization of the scientific and cultural heritage of Ivan Puluj
In particular, a promotional web portal and the development of a general concept of digitization
of archival materials were done, including using the capabilities of the Electronic Fund of Ivan Puluj
[18] in the institutional repository of the TNTU. This work describes the key stages of creating a mobile
application with AR elements. The use case diagram is presented in Figure 2. The designed mobile
application will be an integral element of the software complex of the digital museum of Ivan Puluj.
The photogrammetric method was used to create 3D models of the memorial sculptures of Ivan
Puluj. It is based on the processing of several hundreds of photos taken using digital camera with wide-
angle lens in high resolution, with partial overlap of image areas taken at different angles of view.
Separately, several circular passes at different heights relative to the scanning object and additional
pictures of important details were performed. On the basis of a set of photographs using specialized
algorithms, a dense point cloud, a depth map, a structured mesh of an object are created with the ability
to generate post-processing of the model (for example, texturing) and export in the required format (see
Figure 3).
Figure 2: Use case diagram of mobile application with AR elements of the Ivan Puluj Digital
Museum
Figure 3: Structural diagram of the photogrammetry process for art objects in the Ivan Puluj
Digital Museum
In the vast majority of cases, the Reality Capture photogrammetry software is used. This software
can process photo materials much faster than its competitors and operate with a large number of images
on a standard configuration computer. This creates opportunities for checking data right at the shooting
location. The flexible toolkit allows you to adjust the quality of the generated model, perform smoothing
and simplifying, as well as reduce the file size and increase the performance of the applications into
which the model is imported. However, there is a significant limitation – the production of a high-
polygonal mesh is possible only on a computer with a modern Nvidia discrete graphics card that
supports the software-hardware architecture of parallel computing CUDA. Also, Reality Capture has a
specific pricing and payment policies for created models. However, the export of the result is organized
according to the principle of PPI (Pay-Per-Input) – technology with payment for input, when the type
and size of uploaded photos are evaluated. After the license purchase, the input data may be reused at
no additional charge. The user can process all his data for free, get information about the evaluation of
the value of the 3D model. This allows optimizing financial costs for creating completed models.
During field work, it was established that at least 300-400 photographs are needed to create a
high-quality 3D model of the monument. Since direct sunlight creates an overexposed color gamut,
reflections appear, surface detail decreases and this directly reduces the quality of textures. Therefore,
uniform and diffused lighting is optimal.
The 3D content of the engineering and scientific heritage of Ivan Puluj includes vacuum tubes,
radiometers of original design, equipment for experiments with cathode rays and X-rays, accessories
for physical educational demonstrations. Autodesk 3ds Max and Autodesk Maya graphic editors were
used for graphics design purposes when creating a mobile AR application. In particular, their
capabilities for high-quality modeling of three-dimensional objects, animation and visualization are
well-known. The level of development of modern tools for creating virtual reality (VR) makes it
possible to use these information technologies to reproduce historical places that have undergone a
significant transformation or even ceased to exist.
Scholars of the scientific heritage of Ivan Puluj single out in the activity of Ivan Puluj a fruitful
period from 1884 till 1918, which were obtained, in particular, in the physics laboratory at the Prague
Polytechnic (for a review, see [19]). It was here that Puluj discovered the ionization property of X-rays,
previously unknown, during their passage through rarefied gases, investigated their place of origin and
spatial distribution. In addition, he managed to give a correct scientific explanation of the nature of X-
rays and their mechanism of occurrence, and the cathode tubes of their own design were of superior
quality. Archive photos of Ivan Puluj's laboratory, the detailed description of scientific and
demonstration equipment allows for their credible 3D reconstruction (see Figure 4).
Figure 4: Structural diagram of invention reconstruction process in the Ivan Puluj Digital
Museum
Figure 5: Screenshot of the Ivan Puluj laboratory's virtual space: (a) Archival historical photos
in the laboratory's reconstructed interior; (b) Ivan Puluj's inventions in the laboratory's reconstructed
interior.
In the virtual space of Ivan Puluj's laboratory, the interior and devices are reproduced on the basis
of historical photographs. Digital models of the Crooks radiometer and instruments designed by I. Puluj
namely the radiometer with a biplane cathode, the cathode lamp, the phosphorescent lamp with a cup-
shaped cathode, the incandescent lamp, the device for measuring the mechanical equivalent of heat,
phosphorescent lamps with mica plates, and the device for determining the acceleration of free fall can
be viewed from different distances and at different angles in the virtual interior of the laboratory (see
Figure 5).
To solve the tasks of AR support of the exhibition of the Ivan Puluj digital museum [https://puluj-
museum.tntu.edu.ua/], it is currently optimal choice to use the Unity software ecosystem. The flexibility
and effectiveness of this approach is confirmed by the experience of other research groups [8,9]. Unity
is a multi-platform environment for developing augmented reality spaces, virtual visualizations, training
simulators and games. It contains a fully integrated professional game engine that can provide rendering
functions, environment physics simulation and implementation of scene control scripting mechanisms.
Currently, Unity has a convenient, flexible and functional project porting mechanism for common OSs,
game consoles, VR and AR gadgets. Unity actively supports DirectX, OpenGL and WebGL
frameworks. Unity software provides opportunities for rapid prototyping and evaluation of interaction
processes with virtual or augmented reality objects. AR technology allows for real-time synchronization
between the entities of the virtual and real worlds. A significant advantage of the Unity environment
for modeling AR elements is the availability and relative simplicity of settings when creating a mobile
application using the Vuforia plugin from PTC. Vuforia includes AR-adapted modifications of
computer vision algorithms, real-time marker tracking for objects with flat surfaces and multiple sides
or images, the use of multi-targeting. In Vuforia, there are built-in functions for recognizing ready-made
3D models, scanned objects with rich surface texture and constant shape. Integrated image tracking
technology allows you to effectively place and orient virtual 3D objects, models or multimedia content
in space, associating them with real objects during viewing through mobile devices [20]. It is worth
noting that the virtual object is positioned on top of the real marker image so that the observer's sight
generates the illusion as if the virtual content is an integral part of the real world.
In Figure 6 the interface of interactive mobile application (requires Android 9 or newer) with
elements of augmented reality for the digital museum of Ivan Puluj is shown. The UI is based on
intuitive icons accompanied by textual explanations. For the convenience of the user, there is an option
to switch the language of information content display. The educational part of the application, which
instructs the visitor on the use of the AR functionality, is important to be read. Detailed information
about the project of the Digital Museum of Ivan Puluj, the possibility of viewing full-size detailed 3D
models of the scientist's inventions is provided elsewhere.
Figure 6: Several screenshots of Ivan Puluj digital museum mobile application: (a) The
welcome screenshot of the application; (b) AR-tutorial’s page; (c), (d) information pages
In this project the virtual panoramic tour has been adapted for mobile devices (see Figure 7). At
the same time, the possibility of a guided video tour or independent research of the virtual exposition
has been created. At the same time, each stand is accompanied by audiovisual support, and the user can
easily adjust the accompaniment and natural tactile navigation through the museum according to his
preferences. To demonstrate 3D models of the most remarkable inventions of Ivan Puluj, a set of twelve
cards was created, which have embedded markers of augmented reality (see Figure 8). For this, archival
drawings and photos of exhibit devices developed by a famous physicist and electrical engineer were
selected. When viewing the models (see Figure 9), it is possible to examine the design of the devices in
detail with audio accompaniment.
Figure 7: Several screenshots of Ivan Puluj digital museum mobile application: (a) the welcome
screenshot of the virtual tour of the Ivan Puluj digital museum; (b), (c) general view of the exposition
with integrated models of sculptures; (d) example of an exhibition stand with interactive content.
Figure 8: Examples of AR-cards from the Ivan Puluj Digital Museum set: (a) Puluj’s apparatus for
measuring the mechanical equivalent of heat; (b) Puluj’s incandescent lamp with carbonized filament;
(c) Puluj’s lamp with elliptical fluorescent plate.
Figure 9: Some models of Ivan Puluj’s devices as augmented reality objects: (a) Puluj’s
phosphorescent lamp with a cup-shaped cathode; (b) Puluj’s lamp with elliptical fluorescent plate; (c)
Puluj’s incandescent lamp with carbonized filament; (d) Puluj’s apparatus for measuring the
mechanical equivalent of heat; (e) X-rays experiment with Puluj’s cathode tube.
4. Conclusions
The use of augmented reality in museum mobile applications expands the possibilities of visitors'
general perception of information exhibitions, allows to involve the user in active interaction with
virtual elements. At the same time, an unforgettable experience of immersion in the amazing world of
scientific discoveries and cultural specificity of a specific historical era is formed. The use of 3D
modeling and photogrammetry makes the visual component more realistic and contributes to the
formation of a sustainable interest in augmented reality and the enrichment of the scientific worldview
of visitors, especially young people.
The created mobile application with elements of augmented reality can act as a hub. It allows not
only acquainting visitors with the assets of the museum exhibition, but also offers them the directions
of transitions-portals for a comprehensive own study of the scientific and cultural heritage of one of the
most famous Ukrainian scientists, physicist, electrical engineer and public figure Ivan Puluj. The
proposed format of the digital museum with elements of augmented reality has a significant potential
for integration of atypical and innovative means for the preservation of the national scientific and
cultural heritage, the tourist industry and scientific and educational activities.
5. Acknowledgements
The authors of the article and the Cyber-Physical Systems Laboratory are grateful to Ternopil Ivan Puluj
National Technical University for financial support of the project.
6. References
[1] S. D’Amico, V. Venuti (Eds.) Handbook of cultural heritage analysis, Part V. Monitoring of
Cultural Heritage, 3D Survey, Models, and GIS, Springer Cham, 2022. doi:
https://doi.org/10.1007/978-3-030-60016-7.
[2] M.K. Bekele, R.Pierdicca, E. Frontoni, et al., A Survey of Augmented, Virtual, and Mixed Reality
for Cultural Heritage, Journal on Computing and Cultural Heritage (JOCCH), 11 (2018), 1-36. doi:
http://dx.doi.org/10.1145/3145534.
[3] D. Ocon, Digitalising endangered cultural heritage in Southeast Asian cities: preserving or
replacing? Int. J. Herit. Stud., 27 (2021) 975. doi: https://doi.org/10.1080/
13527258.2021.1883711.
[4] S.Z. Salleh, A.R. Bushroa, Bibliometric and content analysis on publications in digitization
technology implementation in cultural heritage for recent five years (2016–2021), Digital
Applications in Archaeology and Cultural Heritage, 25 (2022) e0025 (1-7). doi:
https://doi.org/10.1016/j.daach.2022.e00225.
[5] A. Georgopoulos, CIPA’s Perspectives on Cultural Heritage, in: S. Münster, K. Friedrichs, F.
Niebling, A. Seidel-Grzesińska (Eds.), Digital Research and Education in Architectural Heritage,
Communications in Computer and Information Science, 817 (2018) 215-245. Springer Cham. doi:
https://doi.org/10.1007/978-3-319-76992-9_13.
[6] T. Caciora, G.V. Herman, A. Ilies et al, The Use of Virtual Reality to Promote Sustainable
Tourism: A Case Study of Wooden Churches Historical Monuments from Romania, Remote
Sensing 13 (2021) 1758. doi: https://doi.org/10.3390/rs13091758.
[7] J. Malpas, Cultural heritage in the age of new media, in: New Heritage. New Media and Cultural
Heritage; Y.E. Kalay, T. Kvan, J. Affleck, (Eds.), Routledge: London, UK, 2008; 13–26.
[8] P. Kyriakou, S. Hermon, Can I touch this? Using Natural Interaction in a Museum Augmented
Reality System, Digital Applications in Archaeology and Cultural Heritage, 12 (2019) e00088.
doi: https://doi.org/10.1016/j.daach.2018.e00088.
[9] R.G. Boboc, M. Duguleană, G.-D. Voinea et al, Mobile Augmented Reality for Cultural Heritage:
Following the Footsteps of Ovid among Different Locations in Europe, Sustainability, 11 (2019)
1167 (1-20). doi: https://doi.org/10.3390/su11041167.
[10] M. Hincapie, C. Diaz, M. I. Zapata-Cardenas et al. Augmented reality mobile apps for cultural
heritage reactivation, Computers & Electrical Engineering, 93 (2021) 107281 (1-9). doi:
https://doi.org/10.1016/j.compeleceng.2021.107281.
[11] D. Marques, R. Costello, Concerns and Challenges Developing Mobile Augmented Reality
Experiences for Museum Exhibitions. Curator, 61 (2018) 541-558. doi:
https://doi.org/10.1111/cura.12279
[12] A. Ejaz, S.A. Ali, M.Y. Ejaz, and F.A. Siddiqui, Graphic user interface design principles for
designing augmented reality applications, International Journal of Advanced Computer Science
and Applications, 10 (2019) 209-216.
[13] J. Cao, K.-Y. Lam, L.-H. Lee et al, Mobile Augmented Reality: User Interfaces, Frameworks, and
Intelligence. ACM Comput. Surv., (2022) 1-35. doi: https://doi.org/10.1145/3557999
[14] N. Babich. UX Design Principles for Augmented Reality, 2020. URL:
https://xd.adobe.com/ideas/principles/emerging-technology/ux-design-principles-for-augmented-
reality/
[15] V. Cesario, D. Petrelli, and V. Nisi, Teenage Visitor Experience: Classification of Behavioral
Dynamics in Museums, in: Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems. Association for Computing Machinery, NY, USA, paper 207 (2020) 1–13.
doi: https://doi.org/10.1145/3313831.3376334.
[16] O. Kramar, Yu. Skorenkyy, O. Rokitskyi, and T. Kramar, Application of virtual and augmented
reality technologies for creation of a digital museum of scientific and cultural heritage of Ivan
Puluj, 1st International Workshop on Information Technologies: Theoretical and Applied
Problems, ITTAP 2021, 3039 (2021), 285-293.
[17] Yu. Skorenkyy, R. Kozak, N. Zagorodna et al, Use of augmented reality-enabled prototyping of
cyber-physical systems for improving cyber-security education, J. Phys.: Conf. Ser. 1840 (2021)
012026. doi: https://doi.org/10.1088/1742-6596/1840/1/012026
[18] Electronic Fund of Ivan Puluj. URL: http://elartu.tntu.edu.ua/handle/123456789/329
[19] R. Gajda, R. Plyatsko, Ivan Puluj: life and works, Lviv, NTSh, 2019 [in Ukrainian].
[20] I. Pedersen, N. Gale, P. Mirza-Babaei, and S. Reid, More than meets the eye: the benefits of
augmented reality and holographic displays for digital cultural heritage, J. Comput. Cult. Herit. 10
(2017) 11 (1-15). doi: https://doi.org/10.1145/3051480.