A Mobile Augmented Reality Application for Museum Exhibitions Anastasiya Molchanova 1, Svitlana Kuznichenko 2, Iryna Buchynska 3 1, 2, 3 Odessa State Environmental University, 15 Lvivska Str, Odesa, 65016, Ukraine Abstract The article discusses software and architectural solutions for creating a mobile application «FindARt», which allows to recognize art objects using augmented reality technology. To implement AR technologies, the Vuforia Engine platform and the Vuforia Cloud Recognition recognition tool were used. It allows saving images and metadata in a Cloud database, and then recognizing them in the application. The mobile application «Find ARt» allows users to independently conduct individual excursions in museums in Odessa and contributes to expanding the creative horizons in matters of fine arts. Keywords 1 Mobile guide application, augmented reality, mobile augmented reality, Vuforia Engine. 1. Introduction visitors with useful information, navigation, guides and translations, thereby creating 2 conditions for the development of smart tourism Activities that allow us to learn something [6]. new with minimal personal contact using Despite the fact that there are a large number electronic devices are becoming increasingly of publications devoted to the creation of various popular in the world that has changed a lot in the AR mobile applications [7, 8, 9] the choice of last year. architecture and design solutions for each of them During the quarantine, many museums operate directly related to the task. Thus, architecture in a safe mode, improve their web-sites and create choice and development of a mobile application self-educational portals about art. More and more for the recognition of art objects using augmented of them organize virtual tours and online tours. reality technologies will be very relevant. Modern travel companies are actively using The offered mobile application «Find Art» will key digital technologies of Industry 4.0, for allow users to take personal tours in the museums, example, augmented reality (AR) technology [1]. recognize pictures on different objects and expand In recent studies, augmented reality has been their creative horizons to become more erudite. recognized as one of the most famous digital technologies that have enormous potential in tourism to make excursions more interactive, 2. The main research material convenient and fun [2, 3, 4]. AR apps can enhance the experience of Augmented reality (AR) is an environment museum visitors by overlaying digital information that complements the physical world with digital available through smartphone displays in a real- data in real time using appropriate devices and world environment. Three-dimensional virtual software. objects are integrated into a three-dimensional The specificity of augmented reality real environment in real time [5] to provide technology is that it programmatically visually III International Scientific And Practical Conference “Information Security And Information Technologies”, September 13–19, 2021, Odesa, Ukraine EMAIL: molchanowa1@gmail.com (A. 1); skuznichenko@gmail.com (A. 2); buchiskayira@gmail.com (A. 3) ORCID: 0000-0001-7982-1298 (A. 1); 0000-0001-7982-1298 (A. 2); 0000-0002-0393-2781 (A. 3) ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) connects two initially independent spaces: the A smartphone, tablet or smart glasses with a world of real objects and the virtual world video camera and appropriate software are used to reproduced on a computer. The new virtual work with AR. If the camera lens is aimed at an environment is created by superimposing object, the software recognizes it either by a pre- programmed virtual objects on top of the video set marker or after analyzing the shape of the signal from the camera and becomes interactive object. After recognizing the object, the software by using special markers. AR can also be defined can connect to a three-dimensional digital as a system that performs three main functions: a duplicate of the object, which is located on the combination of real and virtual worlds, real-time server or in the cloud. Then the AR device interaction, and accurate 3D recording of virtual downloads the required information and overlays and real objects. Overlaid sensory information can it on the object image. As a result, the screen (or be constructive (i.e., an additive to the natural glasses) displays partly physical, partly digital environment) or destructive (i.e., to mask the reality. In this case, different observers, looking at natural environment). The basis of augmented one object can see different augmented reality, reality technology is an optical tracking system. according to the functions performed. The The camera recognizes markers in the real world, repairman can see the operating time or operating "transfers" them to the virtual environment, temperature of the unit he is servicing. The AR imposes one layer of reality on another and thus can help the operator control the subject with a creates a world of augmented reality [10]. touch screen, voice or gestures. As the observer The hardware of the augmented reality device moves, the size and orientation of the AR display consists of processors, displays, various sensors are automatically adjusted, unnecessary and input devices. Modern mobile devices, such information disappears, and new information as smartphones or tablets, have all these elements, appears. as well as a camera and MEMS (accelerometer, A digital model of an object is usually created GPS, digital compass), which make them suitable at the stage of object development with the help for use as a platform for AR. This in turn of CAD or by digitization. This digital duplicate contributes to the spread of augmented reality and collects information about the state of the object, the growing popularity of technology among which is obtained from itself, information systems users. or external sources. With its help, augmented reality software scales and accurately places the 2.1. Basic principles of augmented actual data on the object image or around it. reality technology 2.2. Features of using Vuforia The general scheme of creating augmented Engine platform reality in all cases is as follows: the camera of the augmented reality device captures the image of a The Vuforia Engine platform, one of the most real object; the device software identifies the popular SDKs for developing augmented reality received image, selects or calculates the applications, was used to recognize objects and appropriate visual complement, combines the real work with augmented reality elements.. image with its addition and outputs the final image Vuforia is a comprehensive, scalable to the visualization device [11]. The scheme is enterprise augmented reality platform and graphically presented in Fig. 1. augmented reality software developer toolkit for mobile devices developed by Qualcomm. Vuforia uses computer vision technology, as well as tracking flat images and simple three-dimensional real objects (such as cubic) in real time, recognizes text and cylindrical markers. Vuforia supports various types of targets, including unmarked Image Targets, 3D Multi-Targets, and markers that highlight objects in the scene for recognition. Additional features allow users to Figure 1: General scheme of creating augmented avoid the effect of masking (occlusion) of objects reality using «Virtual Buttons», provide a selection of objects and the ability to programmatically create Android Studio, Unity was chosen. Unity is a and modify them. cross-platform application development With PTC developer tools, Vuforia platform environment developed by the American can be used to create integrated applications, real- company Unity Technologies. Android Studio is time applications, or develop graphical user not focused on developing applications using interfaces. Vuforia provides application augmented reality (AR) technologies, but the programming interfaces (APIs) in C ++, Java, leading platforms for working with AR (such as Objective-C ++ and .NET through extensions for Hololens, Oculus, Vuforia, etc.) are designed for the Unity game engine. Thus, the SDK supports Unity. To use Vuforia in Unity, the Vuforia both development for iOS, Android, and UWP, Engine AR package needs to be installed in the and also allows to develop AR-applications in project through the Package Manager. We use Unity, which are easy to transfer to different Vuforia Engine version 8.1.12, which is the platforms. The Vuforia Cloud Recognition approved version for Unity 2019.4. After service was used in the development of the Find installing the package, an ARCamera object needs ARt mobile application. It is part of the Vuforia to be added to the appropriate scene. ARCamera platform, which allows to store images and is a Unity gaming camera feature that includes metadata in a cloud databasea and then to VuforiaBehavior to support augmented reality recognize downloaded images and receive applications for both portable devices and digital information about them in the application [12]. glasses. The default camera can be deleted as The diagram in fig. 2 provides an overview of ARCamera contains its own Camera component. the application development process using the Vuforia offers three types of databases for Vuforia platform. It consists of Vuforia Engine, a storing target images: Device, Cloud, and target management system hosted on the VuMark. Due to the need to store a large number developer portal (Target Manager), and a cloud or of target images for the application and constantly local database [13]. add data, it was decided to use Cloud database to store the application targets. To do this, a Cloud database needs to be created on the Vuforia developer portal through the Target Manager. CloudRecognition and ImageTarget objects are nedded to be added to the scan scene. ImageTarget is an image that Vuforia Engine can detect and track. A C# script needs to be attached to the CloudRecognition object. This script initializes a CloudRecoBehavior object. This is a basic behavior class that encapsulates Cloud Recognition behavior. It will initialize the target Figure 2: Development of a mobile application finder and will wait for new results. Status using the Vuforia platform changes and new recognition results are tracked using the OnNewSearchResult() method. Vuforia Cloud Recognition is ideal for The developer can add target images to the applications that use many targets or targets that cloud database through the Target Manager on the need to be updated frequently, as is the case with Vuforia Engine developer portal. Target images museum exhibits. The free Vuforia license allows are detected based on natural features that are to use Cloud Recognition and recognize up to a extracted from the target image and then thousand targets each month. compared to the camera image in real time. Target ratings can range from 1 to 5 stars. For best 2.3. Implementation of the mobile results, it's best to use targets with 4 or 5 stars. application «Find ARt» Each target cloud image may additionally have associated metadata. Target metadata is user- defined data that can be associated with a target The mobile application «Find ARt» is and populated with special information up to the designed for devices based on Android OS. permitted limits (up to 2 MB per target). Metadata Despite the fact that the official integrated can contain a simple text message that will be development environment (IDE) for Android is displayed on the device screen when a target is detected; a simple URL string that points to a Vuforia Engine is considered to be the leading special network location where other content, tool in this field and the most complete SDK with such as a 3D model, video, image, or any other a wide range of features for AR applications: user data, is stored; some special text that the identification and tracking of target images, program can process and use to perform certain English texts and 3D objects in real time; actions, such as presenting an object in JSON placement of virtual objects, such as 3D models, format. Metadata can be loaded with the target in a real environment, etc. image when creating a target in the Cloud database, or developer can update the metadata of an existing goal later through the goal manager, but this can take up to several minutes. Therefore, it was decided to store image data not as target image metadata in the Vuforia cloud database, but in the Firebase Database cloud database. Firebase Realtime Database stores data in JSON format [14–17]. The database contains data about paintings (title, author ID, year of writing, museum ID, additional information if necessary, total number of recognitions, etc.), authors, museums and users of the application (recognition statistics and ID of paintings added to Favorites). In our case using a NoSql database is more convenient than relational databases. When storing data in JSON format, you can not worry about determining the complete structure of each element or the interaction of the stored data types. Figure 3: «Find ARt» interface For example, not all Artwork objects may have a description field. Materials or sizes of the original Due to the need to store a large number of may be unknown. The year of creation can be target images for the application and constantly specified in numbers or in text, for example, add data, it was decided to use Cloud database to «before 1715». store the application targets. The implemented software allows to recognize Due to the need to store a large number of paintings and immediately displays brief target images for the application and constantly information about them on the screen. In the add data, the Vuforia cloud database was used to message that appears, user can click "Details" and store the FindARt application targets. Vuforia go to the page with full information about the Cloud Recognition allows to store images and painting, which contains the name, author, size, metadata in a cloud database and then recognize year of creation, materials used, a brief them in the application. This service is free, but description and the museum in which it is located. has some limitations. There is the ability to log in to save user’s favorite pictures to the "Favorites" and share information 4. References about them with friends through social networks or messengers. From the main page user can view the expositions of the museums of Odessa and the [1] Loureiro S.M.C. (2021) The Use of works of popular artists. The application has a Augmented Reality to Expand the minimalist light design (fig. 3). Experience of Museum Visitors. In: Geroimenko V. (eds) Augmented Reality in Tourism, Museums and Heritage. Springer 3. Conclusions Series on Cultural Computing. Springer, Cham. https://doi.org/10.1007/978-3-030- In this work, a mobile application «Find ARt» 70198-7_8 was developed for convenient and quick obtaining [2] Han, D. I., Jung, T., & Gibson, A. (2013). information about objects of art using augmented Dublin AR: Implementing augmented reality reality (AR) technologies. The Vuforia Engine in tourism. In Information and platform was used to implement AR technologies. communication technologies in tourism 2014 [14] Firebase Documentation. URL: (pp. 511-523). Springer, Cham. https://firebase.google.com/docs/guides. [3] Claudia tom Dieck, M., & Jung, T. (2018). A [15] M. Karpinski, S. Kuznichenko, N. Kazakova, theoretical model of mobile augmented O. Fraze-Frazenko, D. Jancarczyk.. reality acceptance in urban heritage tourism. Geospatial Assessment of the Territorial Current Issues in Tourism, 21(2), 154-174. Road Network by Fractal Method. Future [4] Siang, Tan & Ab. Aziz, Kamarulzaman & Internet. 12. 201 (2020). Ahmad, Zauwiyah & Suhaifi, Syazani. 10.3390/fi12110201. (2019). Augmented Reality Mobile [16] Shvorov S., Pasichnyk N., Kuznichenko S., Application for Museum: A Technology Tolok I., Lienkov S., Komarova L. (2019). Acceptance Study. 1-6. Using UAV During Planned Harvesting by 10.1109/ICRIIS48246.2019.9073457. Unmanned Combines. 252-257. [5] Azuma, R. T. (1997). A survey of augmented 10.1109/APUAVD47061.2019.8943842. reality. Presence: Teleoperators & Virtual [17] Kuznichenko, S.,Buchynska, I.,Kovalenko, Environments, 6(4), 355-385. L.,Tereshchenko, T. Integrated information [6] Boes K, Buhalis D, Inversini A (2016) Smart system for regional flood monitoring using tourism destinations: ecosystems for tourism internet of things. CEUR Workshop destination competitiveness. Int J Tour Cities Proceedings, 2019, 2683, стр. 1–5 2(2):108– 124. https://doi.org/10.1108/IJTC-12-2015- 0032 [7] Procházka, David & Stencl, Michael & Popelka, Ondřej & Stastny, Jiri. (2011). Mobile Augmented Reality Applications. Mendel. [8] Carmigniani, Julie et al. “Augmented reality technologies, systems and applications.” Multimedia Tools and Applications 51 (2010): 341-377. [9] Abou El-Seoud, Samir; Taj-Eddin, Islam. An Android Augmented Reality Application for Retail Fashion Shopping. International Journal of Interactive Mobile Technologies (iJIM), [S.l.], v. 13, n. 01, p. pp. 4-19, jan. 2019. ISSN 1865-7923. doi:http://dx.doi.org/10.3991/ijim.v13i01.98 98. [10] Dorfmüller K. (1999) An Optical Tracking System for VR/AR-Applications. In: Gervautz M., Schmalstieg D., Hildebrand A. (eds) Virtual Environments ’99. Eurographics. Springer, Vienna. https://doi.org/10.1007/978-3-7091-6805- 9_4 [11] Augmented reality (AR). URL: https://www.it.ua/ru/knowledge- base/technology-innovation/dopolnennaja- realnost-ar. [12] Cloud Recognition Documentation. URL: https://library.vuforia.com/articles/Training/ Cloud-Recognition-Guide.html. [13] Developing with Vuforia. URL: https://developer.vuforia.com/resources/dev -guide/getting-started.