The article discusses software and architectural solutions for creating a mobile application «FindARt», which allows to recognize art objects using augmented reality technology. To implement AR technologies, the Vuforia Engine platform and the Vuforia Cloud Recognition recognition tool were used. It allows saving images and metadata in a Cloud database, and then recognizing them in the application. The mobile application «Find ARt» allows users to independently conduct individual excursions in museums in Odessa and contributes to expanding the creative horizons in matters of fine arts.
2 Activities that allow us to learn something new with minimal personal contact using electronic devices are becoming increasingly popular in the world that has changed a lot in the last year.
During the quarantine, many museums operate in a safe mode, improve their web-sites and create self-educational portals about art. More and more of them organize virtual tours and online tours.
Modern travel companies are actively using
key digital technologies of Industry 4.0, for
example, augmented reality (AR) technology [
AR apps can enhance the experience of
museum visitors by overlaying digital information
available through smartphone displays in a
realworld environment. Three-dimensional virtual
objects are integrated into a three-dimensional
real environment in real time [
Despite the fact that there are a large number
of publications devoted to the creation of various
AR mobile applications [
The offered mobile application «Find Art» will allow users to take personal tours in the museums, recognize pictures on different objects and expand their creative horizons to become more erudite.
Augmented reality (AR) is an environment that complements the physical world with digital data in real time using appropriate devices and software.
The specificity of augmented reality
technology is that it programmatically visually
connects two initially independent spaces: the
world of real objects and the virtual world
reproduced on a computer. The new virtual
environment is created by superimposing
programmed virtual objects on top of the video
signal from the camera and becomes interactive
by using special markers. AR can also be defined
as a system that performs three main functions: a
combination of real and virtual worlds, real-time
interaction, and accurate 3D recording of virtual
and real objects. Overlaid sensory information can
be constructive (i.e., an additive to the natural
environment) or destructive (i.e., to mask the
natural environment). The basis of augmented
reality technology is an optical tracking system.
The camera recognizes markers in the real world,
"transfers" them to the virtual environment,
imposes one layer of reality on another and thus
creates a world of augmented reality [
The hardware of the augmented reality device consists of processors, displays, various sensors and input devices. Modern mobile devices, such as smartphones or tablets, have all these elements, as well as a camera and MEMS (accelerometer, GPS, digital compass), which make them suitable for use as a platform for AR. This in turn contributes to the spread of augmented reality and the growing popularity of technology among users.
The general scheme of creating augmented
reality in all cases is as follows: the camera of the
augmented reality device captures the image of a
real object; the device software identifies the
received image, selects or calculates the
appropriate visual complement, combines the real
image with its addition and outputs the final image
to the visualization device [
A smartphone, tablet or smart glasses with a video camera and appropriate software are used to work with AR. If the camera lens is aimed at an object, the software recognizes it either by a preset marker or after analyzing the shape of the object. After recognizing the object, the software can connect to a three-dimensional digital duplicate of the object, which is located on the server or in the cloud. Then the AR device downloads the required information and overlays it on the object image. As a result, the screen (or glasses) displays partly physical, partly digital reality. In this case, different observers, looking at one object can see different augmented reality, according to the functions performed. The repairman can see the operating time or operating temperature of the unit he is servicing. The AR can help the operator control the subject with a touch screen, voice or gestures. As the observer moves, the size and orientation of the AR display are automatically adjusted, unnecessary information disappears, and new information appears.
A digital model of an object is usually created at the stage of object development with the help of CAD or by digitization. This digital duplicate collects information about the state of the object, which is obtained from itself, information systems or external sources. With its help, augmented reality software scales and accurately places the actual data on the object image or around it.
The Vuforia Engine platform, one of the most popular SDKs for developing augmented reality applications, was used to recognize objects and work with augmented reality elements..
Vuforia is a comprehensive, scalable enterprise augmented reality platform and augmented reality software developer toolkit for mobile devices developed by Qualcomm. Vuforia uses computer vision technology, as well as tracking flat images and simple three-dimensional real objects (such as cubic) in real time, recognizes text and cylindrical markers. Vuforia supports various types of targets, including unmarked Image Targets, 3D Multi-Targets, and markers that highlight objects in the scene for recognition. Additional features allow users to avoid the effect of masking (occlusion) of objects using «Virtual Buttons», provide a selection of objects and the ability to programmatically create and modify them.
With PTC developer tools, Vuforia platform
can be used to create integrated applications,
realtime applications, or develop graphical user
interfaces. Vuforia provides application
programming interfaces (APIs) in C ++, Java,
Objective-C ++ and .NET through extensions for
the Unity game engine. Thus, the SDK supports
both development for iOS, Android, and UWP,
and also allows to develop AR-applications in
Unity, which are easy to transfer to different
platforms. The Vuforia Cloud Recognition
service was used in the development of the Find
ARt mobile application. It is part of the Vuforia
platform, which allows to store images and
metadata in a cloud databasea and then to
recognize downloaded images and receive
information about them in the application [
The diagram in fig. 2 provides an overview of
the application development process using the
Vuforia platform. It consists of Vuforia Engine, a
target management system hosted on the
developer portal (Target Manager), and a cloud or
local database [
Vuforia Cloud Recognition is ideal for applications that use many targets or targets that need to be updated frequently, as is the case with museum exhibits. The free Vuforia license allows to use Cloud Recognition and recognize up to a thousand targets each month.
The mobile application «Find ARt» is designed for devices based on Android OS. Despite the fact that the official integrated development environment (IDE) for Android is Android Studio, Unity was chosen. Unity is a cross-platform application development environment developed by the American company Unity Technologies. Android Studio is not focused on developing applications using augmented reality (AR) technologies, but the leading platforms for working with AR (such as Hololens, Oculus, Vuforia, etc.) are designed for Unity. To use Vuforia in Unity, the Vuforia Engine AR package needs to be installed in the project through the Package Manager. We use Vuforia Engine version 8.1.12, which is the approved version for Unity 2019.4. After installing the package, an ARCamera object needs to be added to the appropriate scene. ARCamera is a Unity gaming camera feature that includes VuforiaBehavior to support augmented reality applications for both portable devices and digital glasses. The default camera can be deleted as ARCamera contains its own Camera component.
Vuforia offers three types of databases for storing target images: Device, Cloud, and VuMark. Due to the need to store a large number of target images for the application and constantly add data, it was decided to use Cloud database to store the application targets. To do this, a Cloud database needs to be created on the Vuforia developer portal through the Target Manager. CloudRecognition and ImageTarget objects are nedded to be added to the scan scene. ImageTarget is an image that Vuforia Engine can detect and track.
A C# script needs to be attached to the CloudRecognition object. This script initializes a CloudRecoBehavior object. This is a basic behavior class that encapsulates Cloud Recognition behavior. It will initialize the target finder and will wait for new results. Status changes and new recognition results are tracked using the OnNewSearchResult() method.
The developer can add target images to the cloud database through the Target Manager on the Vuforia Engine developer portal. Target images are detected based on natural features that are extracted from the target image and then compared to the camera image in real time. Target ratings can range from 1 to 5 stars. For best results, it's best to use targets with 4 or 5 stars.
Each target cloud image may additionally have associated metadata. Target metadata is userdefined data that can be associated with a target and populated with special information up to the permitted limits (up to 2 MB per target). Metadata can contain a simple text message that will be displayed on the device screen when a target is detected; a simple URL string that points to a special network location where other content, such as a 3D model, video, image, or any other user data, is stored; some special text that the program can process and use to perform certain actions, such as presenting an object in JSON format. Metadata can be loaded with the target image when creating a target in the Cloud database, or developer can update the metadata of an existing goal later through the goal manager, but this can take up to several minutes. Therefore, it was decided to store image data not as target image metadata in the Vuforia cloud database, but in the Firebase Database cloud database.
Firebase Realtime Database stores data in
JSON format [
The implemented software allows to recognize paintings and immediately displays brief information about them on the screen. In the message that appears, user can click "Details" and go to the page with full information about the painting, which contains the name, author, size, year of creation, materials used, a brief description and the museum in which it is located. There is the ability to log in to save user’s favorite pictures to the "Favorites" and share information about them with friends through social networks or messengers. From the main page user can view the expositions of the museums of Odessa and the works of popular artists. The application has a minimalist light design (fig. 3).
In this work, a mobile application «Find ARt» was developed for convenient and quick obtaining information about objects of art using augmented reality (AR) technologies. The Vuforia Engine platform was used to implement AR technologies.
Vuforia Engine is considered to be the leading tool in this field and the most complete SDK with a wide range of features for AR applications: identification and tracking of target images, English texts and 3D objects in real time; placement of virtual objects, such as 3D models, in a real environment, etc.
Due to the need to store a large number of target images for the application and constantly add data, it was decided to use Cloud database to store the application targets.
Due to the need to store a large number of target images for the application and constantly add data, the Vuforia cloud database was used to store the FindARt application targets. Vuforia Cloud Recognition allows to store images and metadata in a cloud database and then recognize them in the application. This service is free, but has some limitations. 4.