The relevance of the introduction of augmented reality technology in the production process is substantiated. It is developed the structure for the newest system on the base of AR technology, which will increase the motivation of employees and increase the level of information assimilations. The research used modern development technologies, such as ASP.NET Core 2 technology and Unity 3d environment. To demonstrate augmented reality, the Vuforia platform was used and virtual models of equipment have been created in the Solid Works simulation environment. The Factory Maintance Visualisation (FMV) software has been developed, witch consists of four scenes and includes all the features described above. Two models of equipment were created for the demonstration: the Cable 6-Axis Robot Arm and the EXCT 100 movement system.
elements from the virtual world into the real, complementing the things we are able to
see, hear, or even feel. It complements the physical world with digital data provided
by computer devices (smartphones, tablets and AR glasses) in real time. AR uses an
environment around the human (as opposed to virtual reality, which "absorbs" a
person) and simply imposes a piece of virtual information over it, such as graphics,
sounds and reaction to the touch. As virtual and real world coexist harmoniously,
users with augmented reality experiences have the opportunity to try a whole new,
improved world, where virtual information is used as an additional useful tool that
provides assistance in daily activities. Thus, the augmented reality is a concept that
describes the process of supplementing the existing reality with virtual objects,
allowing visualization of information [
The impact of this technology on social development can be comparable to the effect of internet appearance. 2
The relevance of introduction of augmented reality technology into the production process is that the use of such a newest system will undoubtedly increase the motivation of employees, as well as increase the level of information assimilation due to the diversity and interactivity of its visual representation.
Purpose of work is to develop a simple and understandable application for employees of the automated ("smart") factory. Such software will facilitate the diagnostics of the equipment at the factory and prevent breakage. Also, this application demonstrates equipment through augmented reality technologies, which allows new employees to master their skills faster with any type of equipment. 3
Augmented Reality is a term that represents all projects aimed at supplementing the
reality with any virtual elements [
In other words, it is a technology of overlaying information in the form of text, graphics, audio and other virtual objects on real objects in real time. It is the interaction of computing devices with the real world picture distinguishes augmented reality from virtual.
Even in the middle of the TWENTIETH century, for military pilots began to make
helmets with a built-in additional display. With it's help, the pilot received important
information: he saw, for example, how much fuel was left in the tank of the aircraft
and what course the moving machine [
1999 year deserves special attention: this year, "ARToolKit" was released — an open source library for developing applications with augmented reality elements. A few years later, the smartphone era has led a new wave of AR development, this time with a lot of mobile applications for a wide range of consumers.
The very first program of this kind was Wikitude that represents something similar
to Wikipedia. The application is available in the version for Google Android from the
end of 2008, for the iPhone since October 2009 and recently for Symbian OS.
EligoVision was the first to implement augmented reality systems in 2008 [
Augmented reality is divided into several types:
1. Marker based AR [
2. AR without markers [
3. Projection-based AR [
4.Superimposition-based AR [
Many modern devices already support augmented reality. From smartphones and
tablets to devices such as Google Glass or portable devices, these technologies
continue to evolve. In order to process and design AR devices and hardware must first
meet the following requirements, such as sensors, cameras, accelerometer, gyroscope,
digital compass, GPS, processor, displays, etc. [
Augmented reality devices are classified into the following categories [
Manufacturers such as Samsung and Sony have announced the development of AR lenses. Accordingly, Samsung is working on lenses as a smartphone accessory, while Sony is developing lenses as standalone AR devices (with features such as photography or data storage). Virtual retina displays (VRD), creates images by projecting laser light into the human eye The goal is to detect bright, high-contrast, high-resolution images.
Security [
Boeing introduced the Enhanced Vision System (EVS) back in 2007, designed to increase pilots' awareness of approach to the airfield, landing, take-off and steering. Using an infrared camera and sensors, it shows the environment in a transparent panel between the pilot and the windshield.
Industry [
Advertising and promotions [
What if the task is not to sell the product but the emotions associated with that product? For example, using face recognition and emotion technology can make people smile. As Unilever did, they created an ice cream vending machine that gave out free ice cream for a smile.
Medicine [
Games. (Box! Open Me, AppTag, Mosquitos, SpecTrek) — an oasis for AR. The PlayStation Vita, for example, comes with a set of augmented reality maps.
Education [
For the comfortable development of the application [
2. Development of the server part based on ASP.NET Core 2 technology. The client part consists of the created four "scenes", namely: MainScene, EquipmentScene, EquipmentDetailsScene, ModelVisualisationScene. The development of the client environment used the capabilities of the Unity3d, a cross-platform gaming engine developed by Unity Technologies and augmented reality platform, including libraries and development tools kit (SDK) to create augmented reality applications.
For the creation of the server part used the capabilities of the Microsoft ASP.NET Core platform, which is designed to create a variety of web applications: from small websites to large web portals and web services, and the Entity Framework platform which is a set of ADO.NET technologies that provide development of applications related to data processing. It will also tell you about the FactoryManager database created for the program. 4.1
MainScene – this is the employee authorization page in the factory system. The purpose of this scene is to check the user's login and password, or they coincide with the existing in the database (Fig 1). Canvas was used to create it, which is the area of placing all elements for user interaction. This is a standard feature in Unity. Two input fields were added to it – LoginInputField, PassInputField, for login and password respectively.
All the business logic that relates to this scene is implemented in the MainSceneScript file. ChangeSceneObject was created to bind this file to MainScene. This is an empty game object, which serves as an intermediary between the appearance of the scene and its functionality. The query to the test server was based on the UnityEngine WWW method, which is a built-in Unity tool.
The response and user data validation was performed in the OnResponse() function. This was done using a third-party library – SimpleJSON. This library is used in all scenes.
If all is well and the data entered by the user coincide with the data in the database, then after clicking on the button Login worker will be forwarded to the stage EquipmentScene.
When an employee is successfully logged into the system, capture device starts to work. The program looks for a marker on which the image will be rendered. Once the necessary section of space is found, the user just has to click on the corresponding button. In this work used markerbassed AR approach, that means that to show something you need to find plane with corresponding symbols.
For the project, two markers were created (Fig 2). As you can see it consists of two parts. On the left is the plane where the models will be shown and the data is displayed. On the right is a virtual button that responds to the movement of the employee and shows information about the equipment (Fig. 3). The IVirtualButtonEventHandler interface has been connected in order to work with such button. It also has the OnButtonReleased method, but in this case it is not needed. For requests to the test server, as in the previous case, was used WWW, the UnityEngine class method.
In Fig. 4 demonstrated call procedure on server. The last parameter in the address strip corresponds to the id (identifier) of the device whose data was interesting to the employee. Also, all the data extraction logic is described in the OnResponse(request) function.
Plane, a standard Unity object, was created to display all this data. It contains three text boxes: characteristiValue1, characteristiValue2, characteristiValue3, which show the employee the current characteristics of the equipment. There is also an equipmentTitle text box that shows the name of the equipment.
There are also two additional buttons on this scene: View Model, View Details. Each button has its own functionality that switches the scene to ModelVisualisationScene and EquipmentDetailsScene respectively.
It should also be noted that the work described above is similar for all models created for the project.
ModelVisualisationScene – is a scene where a factory worker can get acquainted with a 3D model of the equipment. In order to switch to the equipment in the program there is a button – View Model. Pressing this button will redirect user to the scene described (Fig. 5). It is worth saying that in Fig. 6 depicts a linear gantry portal EXCT-100 provided by FESTO. This is the so-called Cartesian movement system. In Fig. 5 shows the Cable 6-Axis Robot Arm, sponsored by 3DMEISTER.
Just so these models do not display through Unity. They were originally compiled through Solid Works 2014 and saved in the IGES (Initial Graphics Exchange Specification) format. The next step was to open this file through Autodesk 3dmax. Due to such manipulations, most textures have disappeared from the models. From 3dmax environment, the models were exported in a Unity FBX (Filmbox) format.
Each equipment has its own characteristics and description. In the case of the program, this was implemented using Plane, which is a standard Unity tool. An image with all the characteristics of the object was added to it.
The employee also has the ability to rotate the model in arbitrary directions. A special controller was added for this purpose. This is a special module from the Unity Standard Assets suite, called MobileSingleStickControl. Also, to implement the movement, a Rigidbody component was imposed on the model. Its role is to add physics to the object.
EquipmentDeatailsScene – this is the scene where the detailed condition of the equipment is described (Fig. 7). The main purpose of this scene is to give the employee full information about the current state of the equipment, to show when and by whom the last technical inspection was performed. The user is also able to request an additional inspection. Data sent by the employee will be stored in the corresponding FactoryManager database table.
For convenience, it was decided not to make this scene in virtual form, that is, to make it in 2D format. This was done for ease of use.
As in the previous scenes, the OnResponse (request) function was created. The "body" function is similar to the previous one, the only difference is the object where the data is stored.
The second section is historyViewBlock. The history of equipment maintenance is displayed here. The New Request button is located. Clicking this button displays a new input box where the employee writes the message. After clicking Submit, the data is sent to the server and stored in the database. 4.2
The test server was created to simulate the operation of the present plant, at least the part related to the project (Fig. 8). To create the server, in Visual Studio 2017 development environment was created a project. In turn, it is divided into two subprojects. The first is responsible for creating a database based on a written data model (the so-called code-first approach). The second is all server-related logic, controllers, links, and configuration files.
DataAccess – a class library created to store data models of a future database. Here describes all classes that will create the corresponding tables in FactoryManager.
Employee is a class that describes the user fields, that is, the future columns of the corresponding table. Equipment – model of equipment data. History is a class that describes the fields in the table where the last maintenance of the equipment will be stored. As a result, the FactoryManager database is formed (Fig. 9).
Three controllers were created to receive and process requests from the client side. The controller is a regular class written with the C # programming language. It is inherited from the abstract class Microsoft.AspNetCore.Mvc.Controller.
Generally created: EmployeeController, EquipmentController, HistoryController. Each controller is responsible for its part. Each spelled out so-called CRUD operations (create update delete), these are the four basic functions of managing data creation, reading, modification and deletion. Due to these operations, the controller knows which data to insert into the database, which ones to update, which to return to the user.
In order to make request on test server, you must correctly create a URL (Uniform Resource Locator). The server runs on "http: // localhost: 57144". The following is a prefix that must be added to do this.
Depending on the needs, the name of the controller will be substituted instead of the controller.
EmployeeController — this controller is responsible for the application user data. That is, it passes through all operations to extract user data. In the case of the application, this is the login, password, first and last name.
This demonstrates a method that returns user data based on its id (identity document). Inquiries here come from MainScene.
EquipmentController — this controller is responsible for providing user with information about the relevant equipment. To display data on the EquipmentScene scene, it return the name of the equipment, its characteristics, and the current state. The part of controllers code is shown below.
HistoryController — is responsible for storing and returning data on the latest equipment inspection. The user is prompted by the date of the last inspection, the message and the author of the message.
The user can also create a new record in the corresponding database table. Controller is used to process requests from the EquipmentDetailScene scene.
So, Factory Maintance Visualization (FMV) software was developed in this work. It includes all the features described above. Also, two models of equipment were created for the demonstration: Cable 6-Axis Robot Arm, EXCT 100 relocation system. Markers, that attach to the equipment, have been developed to show the virtual version of the equipment as well as information about it. 5
In this article is substantiated the actuality of augmented reality technology and consider various directions of use of this technology in the production process. Substantiated necessity of development mobile educational system based on augmented reality technology in order to increase information assimilation level of production sphere employees. The use of such a training system will certainly simplify perception of the theoretical material, as well as allow to consolidate knowledge into practice – having simulated virtual objects in real time.
During this research, FactoryMaintanceVisualization software was developed. This application will be useful for the "smart" factory employees. Such application will facilitate the diagnostics of equipment at the factory and prevent breakage. Also, this application demonstrates equipment through augmented reality technologies, which allows new employees develop their skills faster with any type of equipment.
FactoryMaintanceVisualization складається із чотирьох сцен: MainScene — is responsible for logging the employee into the system; EquipmentScene — provides user with general equipment information; EquipmentDetailsScene — shows a detailed description of the equipment, displays inspection histories; ModelVisualisationScene — shows a 3D model of the equipment.
A test server was also created to simulate the work of the plant and a database called “FactoryManager” was developed for easy data sharing.
The Vuforia platform was explored to create this application as it is a powerful tool for creating augmented reality applications. The capabilities of the Unity 3d development environment and the Solid Works 2014 design environment were also exploited. The latest ASP.NET Core 2 technology was explored to create a server that could provide user with the necessary information.