=Paper=
{{Paper
|id=Vol-2416/paper55
|storemode=property
|title=Big Data, Internet of Things, Augmented Reality: technology convergence in visualization issues
|pdfUrl=https://ceur-ws.org/Vol-2416/paper55.pdf
|volume=Vol-2416
|authors=Daria Smuseva,Alexey Rolich,Leonid Voskov,Ivan Malakhov
}}
==Big Data, Internet of Things, Augmented Reality: technology convergence in visualization issues ==
https://ceur-ws.org/Vol-2416/paper55.pdf
Big Data, Internet of Things, Augmented Reality: technology
convergence in visualization issues
D A Smuseva1, A Y Rolich1, L S Voskov1 and I Y Malakhov1
1National Research University “Higher School of Economics”, Myasnitskaya str., 20,
Moscow, Russia, 101000
e-mail: dasmuseva@gmail.com, arolich@hse.ru, lvoskov@hse.ru
Abstract. The paper reviews the current situation of the Augmented Reality and Internet of
Things markets. The implementing possibilities of AR for Big Data visualization from IoT
devices are considered in this paper. The review and the analysis of methods, tools, products
and data system of the visualization are presented. The paper provides an overview of the
programs and devices of Augmented Reality, and an overview of development environments.
The paper presents the existing classifications of computerized data visualization tools and
proposes new classification, which takes into account interactive visualization, the purpose of
the tool, the type of software product, the availability of ready-made templates, and other
characteristics. The article proposes the architecture of the system for collecting data from
IoT endpoint devices based on the Heltec modules. Experiments based on the developed
experimental stand were carried out with Heltec devices of both versions to determine
the number of losses with increasing distance between the sending device and the
receiving device. The results of measuring the power consumption of these devices are
presented in two modes: in standby mode and when sending a message to the Heltec
endpoint device and in standby mode and when receiving a message for the base station.
These studies were conducted using various data transfer protocols (LoRa, Wi-Fi and
Bluetooth). The paper presents the result of the development of a digital twin of a university
building and the development of augmented reality software for receiving data from real-time
data collection devices.
1. Introduction
The introduction of new technologies is often focused on simplifying the routine actions of a person
in everyday life. An example is the introduction of the Internet of Things technology, which requires
an integrated approach to development that combines new technologies, skills and capabilities
supported by effective management and organization. Such technology requires the support of user
interaction and the digital ecosystem to enable the management of connected equipment. For this, it
is necessary that the program can analyze and visualize the data obtained from the sensors. The
relevance of this work lies in the fact that the two-dimensional visualization of data obtained from
the sensors of the Internet of things, is not enough.
In this regard, there is a big growth in services for analyzing and visualizing the data received from
the sensors. The novelty of the work lies in the fact that for the most part, such services represent
information in a two-dimensional form. The problem is that programs of this format cannot
ensure the full assimilation of information. With a large amount of data, the user may miss key points
and, for example, in production, do not notice breakages due to clutter of the interface.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019)
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
Fortunately, nowadays modern technologies allow solving this problem with the help of
augmented reality [1]. Consideration of the use of this technology in the field of IoT, it is possible to
present its use in enterprises for timely identification of problems, in the home for remote control
of equipment, in large production facilities for personnel training through the visualization of
processes and instructions. In this regard, the purpose of this work is to identify methods,
approaches and technologies for visualizing big data on the Internet of things using augmented
reality.
2. Big Data
Over the past two decades, technological advances have led to a huge increase in data volumes.
The term Big Data was coined for working with such data streams. This type of data is not only
characterized by a large volume, but also demonstrates other unique characteristics compared to
traditional data. For example, Big data is usually unstructured and may require real-time analysis.
That is why for working with big data many new systems were created for collecting, transmitting,
storing and processing large-scale data.
Due to the great popularity and diversity of Big Data technology, today there is no generally
accepted definition for this term. In essence, big data means not only a large amount of data, but other
functions that distinguish it from the concepts of “massive data” and “very large data”. In the
literature, you can find many definitions for Big data, consider three of them, which well explain the
essence of the term Big data.
One of the currently popular definitions appeared back in 2001, in the report of META Group
analyst (now Gartner) Dag Lani [2], who noted the equivalence of data management issues in three
aspects (3Vs):
Volume. Big data is very large, often in petabytes or more. Examples include user data on
Facebook, the flow of people on the subway or transactions using credit cards;
Velocity. Big data is often generated continuously, often even in real time. For example, Twitter
messages, cell phone locations, weather and air quality sensors;
Variety. Big data can be of any type consisting of numbers, text, photos, video, audio and other
data. Therefore, Big Data analysis must be flexible enough to deal with any type of data and be
able to combine them.
Big data has three varieties: structured, semi-structured, and unstructured. Traditional data is
usually structured and therefore easy to use and can be easily stored. However, the vast majority of
today's data from sources such as Facebook, Twitter, YouTube and other user-generated content is
unstructured. The high rate of receipt of large data means that data sets should be analyzed at a rate
corresponding to the speed of data production.
Today, Big data can be found in many areas [3, 4]. One of the most common ways to get big data
is to collect data from sensors and devices. The popular branch of the development of this technology
is the Internet of Things (IoT). It implies a global collection of all indicators from several
sensors simultaneously (often in real time), the compilation of these data and their analysis.
3. Internet of Things
The Internet of Things (IoT) provides a network for connecting people, things, programs and data
over the Internet to provide remote control and interactive implementation of services. Today, the
amount of mobile devices with Internet connection exceeds the number of people on Earth.
Moreover, in 2015, Gartner predicted [5] that by 2020 about 20.8 billion smart devices will be
connected to the IoT. In order to support the work of devices connected to the IoT, it is necessary to
have a special Internet of Things support service. Such IoT services need to collect, analyze and
process information about sensors, they must have an interface for the operational management of
information. The data collected from IoT devices can be impressive in size, since the number of
sensors on the Internet of Things is very large. Therefore, a platform that can collect and store a huge
amount of information from sensors is needed to
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 433
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
control IoT. Such a platform should include IoT databases where cloud computing support is needed.
Moreover, IoT data analysis and Big Data support is required.
The economic effect can be seen where the connection between the machines (hereinafter - M2M) is
becoming increasingly important. However, person-to-person (hereinafter - P2P), person-to-machine
(hereinafter - P2M) and machine to person (hereinafter - M2P) connections still remain the majority of
the economic value of IoT. In the table below you can see the comparison of M2M with M2P or P2M,
as well as with P2P technology.
IoT is used in various fields, consider some of them:
Security. In this area, IoT is used to monitor alarms, monitor and control surveillance systems
in real time and track people [6];
Transport sphere. Intelligent applications for transportation systems are used for fleet
management, road safety, emissions control, toll payments, real-time traffic monitoring, etc;
Health. In this area, IoT is used for e-health, which includes individual health systems based on
sensors for the human body;
Utility management. The Internet of Things allows you to measure, prepare, and bill for gas,
water, electricity, and more;
Production. Production control is carried out through the monitoring and automation of the
product chain;
The scope of services. Control of supply and distribution, vending machines can also be
monitored, and support can be provided in managing and providing services;
Management of funds. Campus, building and home automation can be reached using IoT
technology.
4. Augmented Reality
Augmented Reality (AR) is a technology that allows a user to real-time overlay virtual text or
computer-generated images onto objects in the real world. Recognition occurs, usually with the help
of a camera, and the output of augmented reality through the user's screen, telephone, monitor, helmet
or Augmented Reality glasses, window or windshield of the car. Virtual text or image can be
superimposed on the selected objects in the environment surrounding the user, to display additional
information about the selected object.
It is customary to use two basic definitions of augmented reality. In 1994, Paul Milgram described
the continuum “virtuality-reality” [7], i.e. the space between reality and virtuality, between which
are Augmented Reality and augmented virtuality (figure 1). The real environment and the
virtual environment unite in a mixed reality, where, in turn, augmented reality is based more on
reality, and augmented virtuality on virtuality. Thus, Augmented Reality is the result of adding virtual
objects to the number of objects perceived by man as elements of the real world.
Figure 1. Milgram definition.
Augmented reality is based on context-sensitive computations. Thanks to this technology for the user,
the real world and virtual objects coexist on the same view. For example, this technology can be applied
to museums. If you point the camera of a smartphone with a special application of augmented reality on
any exhibit, information about it will be displayed next to the exhibit. The user of augmented reality can
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 434
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
receive useful information about a location or objects and can interact with virtual content in the real
world.
5. Data visuialization methods
5.1. Classification
There are many computer visualization tools that allow you to convert large amounts of abstract data
into formats that are effective for perception, cognition and communication. For example, in the
work [8] the following classification of computer visualization tools is given:
1) Tool functionality: user (professional or beginner), teamwork support (how many people can
simultaneously develop a model), extensibility (possibility of making changes), scalability.
2) Methods (characteristics of the methods used): number of methods, method type.
3) Task (characteristics of the problem to be solved): type of information (quantitative,
qualitative), subject area (science, business, education)
4) Manufacturer (characteristics of the product): manufacturing-firm: (domestic, foreign), type of
product (commercial, training and demonstration, conditionally free)
This classification considers the functionality of the tool from the point of view of user experience:
the level of user skills, support for teamwork, extensibility and scalability. The availability of ready-
made templates, interactive visualization and other important characteristics are not taken into account
in this work. The division of instrument manufacturers into domestic and foreign ones can be
replaced by the multi-lingual nature of the proposed interfaces.
Consider the following classification of visualization and data analysis tools [9], the
following classification of visualization and data analysis tools is proposed:
1) Category: analysis and charting, PC application, geographic information systems (GIS),
library, network analysis, statistical analysis, time data analysis, data retrieval, data cleansing,
data conversion, application/service for visualization and analysis, word clouds.
2) Multiple user support: true or false.
3) Data display on the map: true or false.
4) Platform.
5) User skill level: users who can work with spreadsheets, partially advanced users who are not
afraid to spend two hours learning a new application, advanced users, users with programming
experience or specialized knowledge in the field of GIS or network analysis.
6) Storage and processing location: user device, external server, public external server.
7) Posting results to the web: yes, with plugin, as a picture, no.
The considered classification summarizes such phenomena as a PC application, library,
network analysis, statistical analysis and a word cloud. However, a PC application and a library are
types of software, network analysis and statistical analysis are the purpose of the tool, and the word
cloud is a data visualization method.
We have developed a new improved classification of computer visualization tools that expands
and improves the previously proposed options:
1) Purpose: general purpose; specialized (GIS, business intelligence tools, network analysis);
auxiliary (designed to prepare data for visualization).
2) User skill level: beginer; intermediate; advanced; user with programming experience or special
knowledge.
3) Software: programming language (R, Python); library (D3, ChartJS, Highcharts JS, Raphaël и
др.); application/service.
4) Platform.
5) Distribution: commercial; conditionally free; free.
6) Multilingual interface: yes; no.
7) Need to register: yes; no.
8) Multiplayer mode: yes; no.
9) Types of visualized data: qualitative; qualitative and quantitative; quantitative.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 435
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
10) Supported Databases and file formats.
11) Availability of ready-made templates: yes; no.
12) Interactive visualization: yes; no.
13) Storage and processing location: user device; external server; public external server.
14) Formats of export results.
This classification takes into account interactive visualization, the purpose of the tool, the type
of software product, the availability of ready-made templates, and other characteristics.
4.2. Analysis of Big Data visualization methods
The aggregated data from all devices connected to the IoT network must be correctly reproduced for
the decision maker. For these purposes, visualization is a suitable tool. Data visualization represents
data in some systematic form, including attributes and variables for a unit of information.
Visualization approaches are used to create tables, charts, images, and other intuitive ways to display
data. Big data visualization is not as simple as traditional small data sets [10]. The expansion of
traditional approaches to visualization has already appeared, but far from enough. With large-scale
data visualization, many researchers use feature extraction and geometric modeling to significantly
reduce the size of the data before actually rendering the data. Choosing the right data representation
is also very important when visualizing big data [11].
Often, standard methods of data visualization are used, which include such methods as a
table, histogram, scatter plot, line diagram, pie chart, block diagram, bubble chart, combination of
charts, time line, entity relationship diagram, etc. In addition, the standard methods of data
visualization include additional methods: parallel coordinates, treemap, cone-shaped tree, semantic
network, etc [10]. Parallel coordinates (figure 2) are used to construct individual data elements in
many dimensions. This method of data visualization is very useful when displaying multidimensional
data [12].
Figure 2. Parallel coordinates.
Treemap (figure 3) - an effective method of visualizing the hierarchy, which is a map, divided
into sections. The smallest rectangles represent one measure, while color is often used to represent
another measure of data. Figure 3 shows an example of a treemap collection set for streaming music
and video tracks in a social networking community [13].
A cone-shaped tree is another method that displays hierarchical data, such as an organizational body
in three dimensions. Branches grow in the form of a cone. The semantic network is a
graphical representation of the logical connection between different concepts. It generates a
directed graph, a combination of nodes or vertices, edges or arcs, and an inscription above each edge
[10]. Visualization is not only static, but also interactive, thanks to such approaches as scaling
(zooming in and out); review and detail; pan; as well as focus and context or fisheye [10]. Below are
the steps for interactive visualization:
Choice. This implies an interactive selection of data objects, a subset, parts of whole data, or
the entire data set in accordance with the interests of the user;
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 436
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
Linking: useful for linking information between several types of graphs. An example is shown
in figure 4;
Filtering: helps users adjust the amount of information to display. It reduces the amount of data
and focuses on information of interest;
Regrouping or reuse: since spatial layout is the most important visual display, rebuilding the
spatial layout of information is very effective for creating different ideas;
Figure 3. Analysis of musical compositions using a treemap.
Figure 4. Interactive linking between histogram graphs and a geographic map.
The scalability and dynamics inherent in big data are the main problems in visual analytics. Big
data visualization with diversity and heterogeneity (structured, semi-structured, and unstructured) is
a big problem. As described above, speed (Velocity) is one of the important factors for analyzing Big
Data. Cloud computing and an advanced graphical user interface can be combined with big data to
better manage the scalability of big data [14].
Big data often has an unstructured data format, so due to bandwidth limitations and power
requirements, the visualization must fit the data in order to efficiently get up-to-date information.
Because of the large size of the data, there is a need for parallel computations, the task of which is to
distribute computations for independent tasks that can be performed simultaneously [15].
The following problems exist in Big Data visualization [16]:
Visual noise. Most of the objects in the dataset are too relative to each other. Users cannot
separate them as separate objects on the screen;
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 437
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
Loss of information. To simplify the calculations, it is often necessary to reduce the data sets,
which leads to loss of information;
Complicated image perception. Data visualization methods are limited not only by the aspect
ratio and resolution of the device from which the visualization is viewed, but also by the physical
limitations of perception;
High speed image change. Users observe the data and cannot respond to the amount of data
changes or their intensity on the display;
High performance requirements.
Visualizing each data point can overwhelm the schedule and can suppress users' perceptions
and cognitive abilities, and reducing data by sampling or filtering can lead to a loss of interesting
emissions and deviations. Requesting large data warehouses can lead to high latency,
disrupting real-time interaction [17].
Ways of solving problems of Big Data visualization [18]:
1. Satisfying the need for speed. This implies improved hardware with increased memory and
powerful parallel processing,
2. Understanding the data. One solution is for people who receive data to have relevant
experience and knowledge for quick understanding,
3. Solving the problem of data quality. It is necessary to control that the data arriving at the
visualization were cleared from unnecessary information,
4. Synthesis of information. One way is to combine the data into a higher-level view, where
smaller groups of data are visible and the data can be effectively visualized,
5. Create exceptions for emissions. A possible solution is to remove outliers from the data or
create a separate chart for outliers.
Big data can be visualized using a number of approaches and methods that were analyzed and
classified (table 1) according to the following data criteria: large data support, support for different
types of data, and the capacity to dynamically display data changes [7].
Treemap: based on the visualization of spatial images of hierarchical data;
Circle Packing: A direct alternative to treemap. However, it uses circles as a primitive form,
which can also be included in circles from a higher level of hierarchy;
Sunburst: uses a treemap visualization and converts it into a polar coordinate system. The main
difference is that the variable parameters are not the width and height, but the radius and length
of the arc;
Parallel coordinates: extends visual analysis using a variety of data factors for different objects;
Streamgraph: this is a type of graph of a stack region that is offset around a central axis, which
results in a smooth and organic form;
Circular Network Diagram: The data object is placed around the circle and connected by curves
based on the level of their interconnections. Different line widths or color saturation are
commonly used to measure the coherence of an object.
Table 1. Analysis of big data visualization methods.
Large amount Variety Data
of data of data dynamics
Treemap + - -
Circle Packing + - -
Sunburst + - +
Many data visualization tools work on the Hadoop platform. Such modules in Hadoop are: Hadoop
Common, Hadoop Distributed File System (HDFS), Hadoop YARN, and Hadoop MapReduce. There
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 438
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
are also separate softwares, but they all represent a two-dimensional visualization of data, which is a
difficulty for the user to understand the information. The solution to this problem is to use augmented
reality as a visualization tool, which will facilitate the work of the user and allow you to visualize the
data in a convenient form.
6. Big Data, IoT and AR: technology convergence in visualization issues
In 1979, a major accident at a nuclear power plant occurred in the United States, the cause of the
accident was erroneous actions by personnel who did not understand the incoming data and did not
identify the problem in time [12]. The introduction of Augmented Reality in the visualization of data
collected from IoT-sensors, will make work in many areas more secure and effective. The use of this
technology will allow real-time monitoring of the indicators of the sensors, output the data in a
visual form, it is also possible remote user interaction with the equipment. Thus, not only the
working personnel who are in close proximity to the equipment can make an inspection, but also the
manager will be able to more accurately control the production situation with the help of installed
smart sensors in all rooms. If something goes wrong, workers can instantly fix the problem, or
transmit the data to engineers outside the building.
Also, this technology can be used for training and staff training. Companies can develop
interactive manuals in augmented reality, filling them with interactive 3D equipment models and
visual step-by-step instructions. For example, global manufacturer Caterpillar Inc. makes good use
of AR to train its engineers and makes them more efficient. Engineers can use AR glasses or a
smartphone to get virtual, step-by-step live instructions on how to efficiently perform machine repairs
and other maintenance tasks. Many examples can also be seen in the areas of health, education,
aviation and space, travel, real estate and other sectors.
The interaction of AR and IoT requires a large number of calculations; in addition, the volume
of processing of augmented reality directly affects the power consumption of the device and its
lifespan. That is why to work with AR on smartphones an additional cloud database is needed to
reduce the load, save energy and overcome the limitations of the phone’s memory. It is worth noting
that the use of cloud technologies should be controlled by adaptive management, since the transfer
of all computational processes to the cloud platform can lead to large delays in the operation of the
program. To use cloud resources efficiently, you need to monitor the cloud server load and network
congestion status so that they are balanced. It is necessary to monitor the state of the network, the
status of the cloud server, the energy status of the device and the quality of perception (QoE, Quality
of Experiance).
Using an AR application on smartphones, tablets, or corporate AR headsets, workers can
monitor connected equipment to identify problems and perform maintenance. AR headsets, such as
Google Glass Enterprise and Microsoft HoloLens, can also be useful in enterprises, displaying step-
by-step online training and tutorials for assembling and operating equipment that would otherwise
require reading thick technical manuals and instructions.
To create an Augmented Reality program with the ability to visualize data from the IoT sensors,
you can use the existing augmented reality SDKs: ARToolkit [19], EasyAR [20], OpenCV [21], Maxst
[22], Kudan [23], Catchoom On-Device Image Recognition SDK [24] and Vuforia SDK [25]. With
the help of such SDKs, only a part with augmented reality can be developed, but additional
programs will be needed to connect IoT devices and collect data.
For the productive implementation of programs, there are ready-made solutions that combine all
stages of program development, including connecting sensors and visualizing data. The main leaders
among the platforms for creating programs with AR for the Internet of Things are PTC,
Augmenta, Bosh, Reflect.
Currently, the main area of convergence of the listed technologies is the visualization of big data
coming from IoT-devices in real time on mobile devices with augmented reality function.
Conventionally, we can distinguish the following types of convergence:
1. Visualization of sensor data and object management.
2. Monitoring equipment performance and decision support systems. The information presented in
a visual form will allow timely troubleshooting equipment problems and improving the quality of
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 439
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
manufactured products. Ideally, this technology should be implemented at every stage of
production.
3. People will be able to interact with electronics using 3D interfaces, which raises new questions
about the energy efficiency of mobile devices, which are displays for data visualization
(smartphones, tablets, smart watches or AR helmets).
The current trends in the field of the Internet of Things are the creation of new hardware platforms,
the development of specialized operating systems, the development and optimization of application
layer protocols, and the development of models and methods for data visualization aimed at
improving the energy efficiency of the Internet of Things, including mobile devices, to increase
battery life. The lack of mathematical models, methods and tools for organizing dynamic
reconfiguration of graphical interfaces of IoT devices (including using the AR) is a serious obstacle
for further research, as well as for increasing the time for effective functioning of mobile devices
and networks of software systems with limited resources. In this regard, the task of research and
development of an integrated model that makes it possible to assess the efficiency of energy
consumption and other limited resources of mobile devices, methods that make it possible to
optimize the consumption of resources of a mobile device according to the criterion of
maximizing residual energy taking into account the limitations and requirements of users for
final applications and graphic interfaces, and means of organizing reconfiguration of graphic
interfaces of devices.
7. Experimet and technology convergence example
To demonstrate the process of convergence of technologies we used software for iOS. This software
allows you to visualize data from sensors of various devices, combined in a data collection system, using
augmented reality. The experimental data acquisition system (figure 5) consists of endpoint devices that
can collect temperature, humidity, light, noise and the presence of movement data. Each end device
sends data using one of the LoRa, Wi-Fi or Bluetooth protocols to the base station, which generates data
messages and sends them to the ThingWorx cloud platform. Heltec LoRa 32 (V2) [26] microcontrollers
were used as the main components for receiving and transmitting data from various sensors. Most of
them served as endpoint devices that processed and prepared for sending messages containing indicators
from the sensors. While the other part was used as a basic device for receiving messages from all end
sensors. For the base station organization, the Raspberry PI 3 single board computer was used. This
computer is a ready-made solution for working with low-resource programs. The approach that was
chosen to work with this computer allows you to use it as a full-fledged server without using external
interfaces and communicating and sending commands to the Raspberry only over an SSL connection.
With the help of this computer, it became possible to store data coming from devices with sensors on
the Raspberry, thus turning it into a machine with the ability to send data for services,
including applications, using a special network broker.
Mosquitto is used as an MQTT broker. It allows you to send mqtt messages to both cloud platforms
and directly to applications. This is done because MQTT is an inter-computer connection (M2M)
protocol, in other words, for the Internet of things. It was designed as an extremely easy way to
publish publish (subscribe) and subscribe. This is useful for connections to remote locations where a
small amount of code and high bandwidth are required. For example, it is widely used in sensors that
communicate with a broker through satellite communications, through dial-up connections with health
care providers, and in a number of home automation scenarios. It is also ideal for mobile applications
because of its small size, low power consumption, minimized data packets and efficient distribution of
information to one or several recipients. MQTT broker is used as follows: we form a request in Unity
MQTT, which is needed in order to get data from the cloud platform. When the data on the platform
changes, the broker sends the data with changes. MQTT broker and Unity are connected in publisher-
subscriber mode. The software subscribes to receive data from the broker's MQTT and polls it with a
given frequency.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 440
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
Figure 5. Data acquisition system architecture.
The developed experimental stand allowed us to analyze both versions of Heltec devices for the
number of losses when the distance between the sender device and the receiver device was increased, to
measure the power consumption of these devices in two modes: during standby and while sending a
message for the final Heltec, but also during waiting time and while receiving a message for the base
station. These studies were conducted using LoRa, Wi-Fi and Bluetooth, for each type of device, the
average operating time per battery charge will be calculated.
The first test involves the use of two devices, one receiver, it is in a stationary position, and
one transmitter, and it must be transferred, thereby increasing the distance between these devices. The
results of the experiment are presented in table 2. As can be seen from the information received,
regardless of the version and the Heltec clock frequency, up to 40 meters from each other, all devices
work with 100% packet delivery, which means that no packet was lost for the tested period. After
increasing the distance between devices by 10 meters more, the number of undelivered packages in all
cases began to grow, but the percentage of packets received is still very high, not a single value has
dropped below 95%. After the next 2 iterations of the increase in the distance between the Heltec
modules, the number of lost packets increases by about 3 and 4 times, respectively. At the same time, it
becomes obvious that devices of the second version have a preponderance of losses in the direction of
the lower frequency, 9 packets against 5 at 60 meters and 16 against 10 at 70 meters. With a slight
advance, the best packet delivery percentage is at Heltec LoRa v2, operating at a frequency of 868
MHz. Thus, it was revealed that, at long distances, priority is given to using the device second
versions with a clock frequency of 868 MHz.
In the second experiment, an analysis of the current consumption of the endpoint devices was
carried out using various data transfer protocols. This experiment allows you to check the actual
current consumption by the Heltec endpoint devices with the payload on the microcontroller in the
form of sending packets using various data transfer interfaces. There is no intentional distinction here
between the versions of the Heltec modules, since both versions consume almost the same amount of
current, so the separation of tests by versions has no scientific basis [27]. Table 3 presents data from
tests of current consumption by devices in two modes: waiting for polling and sending a packet. As
you can see from the data, as expected, the highest energy efficiency was achieved using the LoRa
interface. The second place in terms of efficiency of current consumption was taken by the
protocol for Bluetooth communication, while Wi-Fi was in last place in terms of current
consumption both in data transfer mode and in standby mode. In this case, the relationship between
the readings of the end device and the base station is traced, so the base station consumes on average
about 35% more current than the end device in standby mode (polling). In this case, the
difference between the energy indices for the transmission (reception) modes is about one tenth of
the consumption of the Heltec end device.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 441
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
Table 2. Dependence of the number of delivered Heltec packages on the distance.
Heltec LoRa v1 Heltec LoRa v2 Heltec LoRa v2
Distance / Frequency MHz MHz MHz
>40 meters 100% 100% 100%
50 meters 96,6% 95% 96,6%
60 meters 90% 85% 91,6%
70 meters 81,6% 73,3% 83,3%
To demonstrate the convergence process of the technology IoT and AR was used Vuforia. The key
factor in choosing was the maximum distance to capture and hold the marker. The most important
thing is that Vuforia has integration with the Thingworx cloud platform, in which the data
collection system is connected. The software has two modes of operation: identification of endppont
devices and data output in augmented reality (figure 6) or the output of a virtual digital twin of the
building in which the data acquisition system is located (figure 7), and data output in different parts of
this digital twin. All data that comes from the endpoint devices in the form of text comes to the 3D
rendering unit along with 3D models, in which information about the properties of the models,
textures, materials and illumination is processed. In the first mode of operation, the camera compares
the received frames with the base of markers. As soon as the similarity is found, a request for a cloud
platform is formed, a response from the broker's MQTT comes and then digital content is output. In
the second case, using the camera, it is necessary to scan the surrounding space to create a mesh grid
to which the virtual digital twin of the building can be attached.
Table 3. Dependence of power consumption of the endpoint data collection device and the base
station on the operating mode used.
Endpoint data collection
Base station
device
Protocol / Operating
Standby
mode
LoRa 80 mA 170 mA 130 mA 180 mA
Wi-Fi 170 mA 300 mA 240 mA 330 mA
Bluetooth 130 mA 230 mA 200 mA 270 mA
Figure 6. Endpoint device and data vusualization using AR.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 442
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
Figure 7. Digital twin of the building in which the data acquisition system is located.
8. Conclusion
The connenction of AR with the IoT ecosystem presents tremendous opportunities for various
industries. Using this approach can make machines smarter by integrating sensors and intelligent
systems that can monitor and produce data to solve problems in a timely manner and prevent downtime
and breakdowns. IoT platforms provide a scalable, flexible, and secure way to cost-effectively
extend AR applications. The use of such programs will allow companies to remotely diagnose and
fix problems, preventing equipment and system system failures before they occur. This predictive
service not only eliminates system downtime, but also optimizes energy use, which saves
money. This paper proposes a classification of computer-based data visualization tools that can
be used in developing a decision support system in the data visualization field.
To demonstrate the potential convergence of technology, an experimental data collection system
stand was developed. The proposed data collection system allowed us to conduct experiments to
determine the dependence of the data transmission distance in the building of the university on the use
of specific models of Heltec modules of various versions and at different frequencies. Experiments
were carried out to measure the energy consumption of data collection endpoint devices based on
Heltec modules. To visualize data coming in real time from elements of the data collection system,
software was developed using the Vuforia library, which allows visualizing the digital twinof
the university building and all the information from the data collection system.
9. References
[1] Agogi E 2011 Augmented Reality in Education Athens, Greece: EDEN-2011 Open Classroom
Conference 1-88
[2] Laney D 2001 3d data management: Controlling data volume, velocity and variety (Gartner,
Stamford, CT, USA, White Paper)
[3] Rytsarev I A, Kirsh D V and Kupriyanov A V 2018 Clustering of media content from social
networksusing bigdata technology Computer Optics 42(5) 921-927 DOI: 10.18287/
2412-6179-2018-42-5-921-927
[4] Agafonov A A, Yumaganov A S and Myasnikov V V 2018 Big data analysis in a
geoinformaticproblem of short-term traffic flow forecasting based on a k nearest neighbors method
Computer Optics 42(6) 1101-1111 DOI: 10.18287/2412-6179-2018-42-6-1101-1111
[5] van der Meulen R Gartner URL: https://www.gartner.com/newsroom/id/3165317 (14.04.2018)
[6] Evsutin O O, Kokurina A S and Meshcheryakov R V 2019 A review of methods of embedding
information in digital objects for security in the internet of things Computer Optics 43(1) 137-154
DOI: 10.18287/2412-6179-2019-43-1-137-154
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 443
Data Science
D A Smuseva, A Y Rolich, L S Voskov and I Y Malakhov
[7] Milgram P, Kishino A F 1994 Taxonomy of Mixed Reality Visual Displays IEICE Transactionson
Information Systems 1321-1329
[8] Gavrilova T A, Gulyakina N A 2008 Visual methods of working with knowledge: an attempt to
review Iskusstvennyj intellekt i prinyatie reshenij 1 15-21
[9] Machlis Sh Chart and image gallery: 30+ free tools for data visualization and analysis URL:
https://www.computerworld.com/article/2506820/business-intelligence/business-intelligence-chart-and
-image-gallery-30-free-tools-for-data-visualization-and-analysis.html (18.04.2018)
[10] Khan M, Khan S S 2011 Data and Information Visualization Methods and Interactive
Mechanisms: a Survey International Journal of Computer Applications 1-14
[11] Chen C L P, Zhang C Y 2014 Data-intensive applications Challenges, Techniques and
Technologies 314-347
[12] Porter B 2012 Visualizing Big Data in Drupal: Using Data Visualizations to Drive Knowledge
Discovery Report, University of Washington 1-38
[13] Keahey T A 2013 Using Big Data Technical Report 1-16
[14] Sucharitha V, Subash S R and Prakash P 2014 Visualization of Big Data: Its Tools and
Challenges International Journal of Applied Engineering Research 5277-5290
[15] Childs H, Geveci B, Meredith J, Moreland K, Sewell C, Bethel E W, Kuhlen T and Schroeder W
2013 Berkeley National Laboratory Joint Research Report, Oak Ridge National Laboratory Sandy
National Laboratories, Los Alamos National Laboratory, RWTH Aachen University 1-11
[16] Gorodov E Y, Gubarev V V 2013 Analytical Review of Data Visualization Methods in Big Data
Journal of Electrical and Computer Engineering 4 1-7
[17] Liu Z, Jiangz B and Heer J 2013 imMens: Real-time Visual Querying of Big Data Eurographics
Conference on Visualization (EuroVis) 421-430
[18] SAS Institute Inc 2013 Survey data analyzers Report 1-2
[19] Azuma R T 1997 A survey of augmented reality Presence: Teleoperators and virtual
environments 6(4) 355-385
[20] Open Source Augmented Reality SDK ARToolKit.org URL: https://artoolkit.org/ (22.04.2018)
[21] Easy AR SDK URL: http://www.easyar.com/ (12.05.2018)
[22] OpenCV library URL: http://opencv.org/ (15.05.2018)
[23] Maxst URL: http://maxst.com/image_ar/?_Lang=en (22.05.2018)
[24] Kudan Computer Vision Kudan limited URL: https://www.kudan.eu/ (22.04.2018)
[25] Catchoom - Image Recognition and Augmented Reality URL: https://catchoom.com (22.04.2018)
[26] Vuforia Augmented Reality URL: https://www.vuforia.com (22.04.2019)
[27] WiFi LoRa 32 (V2) URL: https://heltec.org/project/wifi-lora-32/ (22.04.2019)
[28] Fernández-Ahumada L M 2019 Proposal for the Design of Monitoring and Operating Irrigation
Networks Based on IoT, Cloud Computing and Free Hardware Technologies Sensors 19(10) 2318
Acknowledgments
The publication was prepared within the framework of the Academic Fund Program at the National
Research University Higher School of Economics (HSE) in 2019– 2020 (grant № 19-04-022) and by
the Russian Academic Excellence Project "5-100".
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 444