This article discusses the interaction of multimodal interfaces in a heterogeneous network environment of the Internet of Things. Methods are proposed for testing data exchange between devices using various data transmission protocols in various noisy environments of the studied network.
The Internet of Things is a concept of a network of physical objects interacting
with each other or with the external environment [
Heterogeneous networks have gained particular popularity and development
in the eld of IoT. IoT networks are not designed to stream data [
The use of various protocols and technologies in IoT networks leads to
incompatibility problems of some technical solutions [
In this paper, the problem of the interaction of devices (interfaces) in a
heterogeneous network in the concept of Machine-to-machine (M2M) [
The purpose of this work is to test the interaction of devices in the conditions of the tested data network of the Internet of things, to track information losses during streaming data and to consider the data integrity when transmitting in various noise conditions of a wireless communication channel. Conducting these tests will serve as a starting point for creating a system of user interaction through multimodal interfaces (visual, voice, and sensory). 2
In modern systems the Internet of Things for communication between the M2M devices use heterogeneous data: the audio/video data, and streaming simple data packages with wearable devices, including sensors/actuators.
The devices used in the IoT environment must provide su cient speed of transmission and processing of heterogeneous data. In this paper we consider the most common current models of network modules and single-Board computers used in the eld of the Internet of things: single-Board computers, arduino, STM32, sensors and actuator, and etc.
To test the streaming of audio / video data, it is necessary to pay attention not only to the technical characteristics of the test devices, but also to the most popular codecs that allow to transfer data most e ciently. Among them were chosen popular codecs used in computer vision. As a test, the transmission of video data in the resolution: 360p, 480p, 720p, 1080p formats will be used: To check the ow of audio data, the most popular codecs were selected, including those used in the operation of voice assistants devices. When working with video data, a channel bandwidth of at least 1 Mb/s is required, and when transmitting a high-quality video stream at least 12-20 Mb/s, depending on the selected codec. Audio data is less demanding on data transfer bandwidth. These characteristics are used to select network modules and build an IoT testing platform.
The development of data transmission technologies has improved the existing modules and optimized power consumption, which has a positive impact on the duration of various devices, including in the eld of IoT.
To begin, consider the technical characteristics of the selected networks, taking into account the latest standards IEEE 802.11b/g/ac, IEEE 802.15.1, IEEE 802.15.4d. 6LoWPAN IEEE 802.15.4d 0.864 - 0.865 0.115 - 0.25 before 4000 65536 Speci cations Standard Frequency GHz Bandwidth, Mbps Communication range, m Maximum number of 30 nodes in the network WiFi networks provide a high transfer rate and are the best option for working with video and audio data, but these WiFi modules have high power consumption and are most often used in stationary solutions with a constant power source.
The characteristics of BLE networks allow you to work with audio data and streaming data from devices. Low power consumption allows you to use these modules in wearable devices.
6LoWPAN networks are used to ensure maximum autonomy of remote
sensors, the exchange of small amounts of data over long distances. This technology
provides a high autonomy of the device when running on an AAA battery for
up to 3 years, but it has a low data transfer rate and low reliability due to the
lack of data validation. Widely used in IoT on various sensors, counters, widely
used in solutions Smart Home [
Transmission Device Modules. The most popular and stable modules for each considered network were reviewed and selected.
Devices using Wi-Fi and BLE protocols operate on the same frequency band, which can lead to poor communication quality and data transfer problems during simultaneous operation. Device parameter Standard Local Area Network (LAN). Today, one of the most common and reliable data networks. In the eld of the Internet of Things, it has received little development, since it does not provide su cient mobility, but it allows maintaining high bandwidth and using Power over Ethernet (PoE) technology to provide power to devices remote from the transmission network.
Power-line communication (PLC). Power-line communication (PLC) is used to transmit data over a power line, allowing you to power and control devices on your home network. This technology implements the IEEE 1901.2 standard. It is mainly used for Smart Home solutions, which is well suited for Internet of Things.
of the problem of data transmission in the Internet of Things environment, the
following test options are considered [
Connection Testing: wireless signal testing to monitor and optimize data exchange between devices.
Performance Testing: test the communication and computing capabilities of devices involved in testing.
Stress testing: to determine the number of simultaneous connections supported by the device.
Compatibility testing, veri es the correct operation of various protocols and con gurations.
Evaluation of the e ciency of data transmission is organized using the tools
of Wireshark, Tcpdump, OpenWSN [
It is worth noting that a large amount of tra c with di erent characteristics is generated in the IoT environment. Sensors and actuators, counters send a small packet of data at the time of activation, while streaming video requires a large amount of resources on the communication channel and its processing on the server.
Testing for "noise" is carried out by adding unrelated devices with a test bench operating at the same frequency to create natural interference. 4
The laboratory stand was assembled taking into account testing of heterogeneous networks and technologies used in the Internet of Thing environment.
The server used is a single-board computer Orange Pi RK3399 and Intel Nuc responsible for processing and analyzing tra c, including connecting external devices. Additional Orange Pi RK3399 is used to transfer streaming video and audio signal to the server using various codecs.
The sensor network is built using the Arduino Mega as a node (Node), external devices based on the Arduino Nano V3.0 are connected to it using WiFi, BLE, 6LowPAN modules. Each node (Node) collects and transmits information to the server.
The number of nodes in the network under the conditions of the testing task is limited to 5.
The listed devices are based on di erent architectures, such as ARM7, x86, Atmega328p. 5 5.1
Testing the transmission of streaming video data to a receiver (server) device in various noise conditions of the channel used. The architecture of data transmission in a heterogeneous network is presented in Figure 1.
A device with a connected camera sends data to the server using various codecs and data transfer protocols over a WiFi network. The distance between the transmitter and receiver varies according to experiment [1,5,10,15,20,25 meters]. At each step, the characteristics of data reception and transmission are measured, including the addition of additional devices operating in the same room at the frequency of the tested communication channel.
The number of devices transmitting streaming data at the tested frequency varies from 0 to 25 in increments of 5 for each stage of testing. The test takes place in the same room. The use of various technologies (standards) Wi-Fi, BLE, 6LoWPAN, as well as supported transport protocols in order to test the transfer of simple streaming data using various devices and protocols to a dedicated server in a heterogeneous network. The data transfer architecture is shown in Figure 2. We consider the loss, damage, delays in receiving data from devices on the network, depending on the number of requests sent per second and the number of devices and nodes in the network involved.
Testing the speed and synchronism of obtaining data from di erent sources in a heterogeneous network in response to a single triggering event. In this experiment are used: Node (node) - a device connecting nodes (protocol). One device (SD) |the device communicates with the server directly and transfers data. The server is a single-board computer with a Linux operating system that records the time and result of receiving data.
Each device sends a pre-created data packet indicating the number of the packet being sent, a xed set of data and network parameters. This is the example of data package structure: f g
Device: "Dev #"; Network: "Name network Node #"; Network node: "Number #"; Package number: "Number #";
Data set: "Test data:";
The event is initiated arti cially by giving a signal to all devices participating in the experiment. The test is carried out for all transport protocols supported by the device and the network. Each device in the network is responsible only for one data transfer protocol.
The test result allows you to determine the delay in receiving the packet, the time of desynchronization of the received data and their loss. This information will be used in the future to create multi-modal interfaces for user interaction with the system (Human-to-Device). 5.4
Collect statistics of power consumption of devices at all stages of the experiment. The data obtained are necessary for analysis and application in further work. 5.5
Testing the in uence exerted on network parameters of a heterogeneous network
connected to nodes. The testing model is based on the methodology for
determining the connectivity of nodes in a heterogeneous network [
During throughput testing, the number of bytes transferred per second is recorded.
Calculated throughput, as the ratio of the number of bits of information
successfully transmitted during the test to the time of testing [
The delay is calculated as follows:
D =
Pn i i=1 trecieved n i tsent where n is the number of transmitted packets; tirecieved - the time of receipt of the i-th packet; tisent is the departure time of the i-th packet.
Packet loss we can nd using formula:
T =
Nbits
t L = ndropped
nsent where B is the throughput, N is the average power of noise and interference in the passband.
To calculate the bitrate, it is necessary to calculate the values of the RSSI value - the power of the received radio signal, measured in dBm. This value is where ndropped is the number of lost packets; nsent is the number of packets sent.
The IEEE 802.11a and IEEE 802.11g standards specify the bitrate when two devices interact, depending on the distance between them using two onedimensional data arrays with one-to-one correspondence: each distance value has its own bitrate value.
Using the method of least squares, we can obtain the coe cients a and b of the linear function y = ax + b, which will describe the dependence of the bitrate on the distance with su cient accuracy.
To determine a similar dependency for Bluetooth, you need to use additional calculations and transformations to obtain a similar functional dependency based on data from the Bluetooth standard IEEE 802.15.1. To obtain the bitrate values, the Shannon-Hartley theorem is used:
C = B log2 1 +
RSSI
N
(1)
(2)
(3)
(4)
found using a formula based on the Friis transmission equation [
Next, you need to convert power values from dBm to W. This is done using the following formula:
PW = 10((PdBm 30)=10); (6) These mathematical models for WiFi and Bluetooth are used in the interaction of two devices.
Since the study conducted experiments with several transmitting devices, as well as the communication channel is noisy, the above mathematical model for WiFi and Bluetooth changes for more than two interacting devices. Based on the data obtained during the experiments, a new mathematical model is compiled, taking into account the number of devices, the noise of the communication channel and the distance from the data transmission nodes to the receiving server. The function of two variables is considered: b(d; n), where b is the bitrate, d is the distance, n is the number of devices. The function itself is obtained by approximation of experimental data. 7
In the course of the work, test cases were compiled and a test model of the Internet of Things data transmission was chosen. The results of this test will serve as a starting point for designing Multimodal User Interfaces based on voice recognition, image and gesture control using wearable things.
Also in the future it is planned to consider models of interoperability of devices based on semantic technologies.