In the device-edge-cloud continuum era, where e plethora of diferent devices, from simple sensors to drones, coexist and collaborate in the same system of systems, reliable and flexible communication plays a crucial role. This technical paper presents the Abinsula Multi-Sensor Gateway, a unified communication and management platform for cooperative embedded systems integrated in a system-of-systems environment. The Multi-Sensor Gateway is exploited in drone and rover based services, in the context of the COMP4DRONES ECSEL-JU project.
• agile and cost-efective compositional and modular design;
• eficient customization and integration of drone-based systems; • secure and safe autonomous decisions related to both individual and cooperative missions; • reliable drone to drone and ground-based drone communication.
Communication is one of the technological bricks of the COMP4DRONES framework. When
unmanned vehicles are involved two aspects are highly relevant: (1) the characteristics and the
availability of the link, and (2) the presence of unintentional or malicious attacks. In addition, it
is necessary to enable a certain degree of flexibility in the communication infrastructure, to
satisfy the needs of diferent heterogeneous devices that could need to be integrated in the
system. Indeed, according to the Metcalf’s Low, the value of an interconnected system and the
functions it supports are proportional to the number of its interconnections and its degree of
interaction with other systems, devices, people and resources [
Diferent commercial solutions are available that try to address a flexible communication.
For instance, the Mindtree’s Multiprotocol IoT gateway support multiple industrial protocols
and ofers two versions: 1) a Mesh Gateway to aggregate data coming from diferent sensors
and from Bluetooth Low Energy Mesh nodes, and 2) an Edge Gateway that communicate with
multiple Mesh Gateways and to the application server [
Other companies ofer more integrated platforms. The Bosh IoT Edge Services ofer
connectivity to a heterogeneous device landscape, through a large number of diferent IoT protocols and
open API. It also ofers data history, statistics and the tools to build a custom set of services [
All of above solutions are well developed products ready for the market, an so they do not meet the flexibility required for special customized applications such the ones in the COMP4DRONES context where the connected devices are not the typical devices that belong to the IoT world, being drones and rovers involved. Also, most state-of-the-art solutions are based on their own edge application which is specifically dependent on a particular hardware, thus lacking in ease of customization, applicability and in open source aspects.
The Multi-Sensor Gateway provides a unified management system, contributing to the secure and safe integration of a cooperative embedded system in a system-of-systems environment. It is a generic and modular component, based on an embedded Linux distribution, that works with all the main communication protocols and can be interfaced with elements of the system that communicate through user-specified data formats.
The Multi-Sensor Gateway enables a runtime configurable multi-context support by modifying data priority transmission upon the scenario requirements, acting as the orchestration actor of a heterogeneous network composed of a plethora of diferent nodes: sensors and actuators (valves, temperature, alarms, liquid and gas sensors), cameras, hardware accelerators, drones, rovers.
Figure 2 illustrates a schematic overview of the Multi-Sensor Gateway that provides: • a flexible and extensible Edge gateway, to interface diferent custom devices that talk through diferent communication protocols and user-defined interfaces; • a Cloud gateway with a user-friendly dashboard, to ofer a unified interface for monitoring and managing a complex system, composed of diferent heterogeneous devices; • Processing at the Edge, enabling the possibility of running custom applications and distribute the computation among diferent points in the communication infrastructure.
The Edge gateway is based on Ability, the Abinsula Embedded Linux distribution. It is designed to be flexible and scalable, with respect to the type and number of devices that can communicate with it, and transparent for the user, enabling transparent communication among the devices of the system and the Cloud gateway. Being possible a bidirectional communication, the Edge gateway receives inputs from devices in the system on one side, and from the Cloud gateway on the other side. Inputs from the devices are characterized by a custom user-specific interface. For each interface, a data converter module is available to normalize the data and convert them into standard JSON data that can be sent to the Cloud gateway (see Figure 2). Any data sent from the Cloud gateway to the system devices pass through the Edge gateway that adapt them according to the user-specific interface. New sensors can be added at any time without any particular registration, the gateway exploits the user ID to recognize it as a new module. The Edge gateway has been extended to support the diferent communication and physical layer protocols required by diferent devices, such as sensors, smart cameras, FPGA-based hardware accelerators, drones and rovers. Currently it supports several diferent communication protocols. Table 2.1 depicts an overview of the Edge gateway communication.
The Cloud gateway enables remote feedback and management of the system, through a dashboard that ofers a unified and user-friendly interface for the operator, not expert of the specific devices connected in the system, to monitor and manage the system, allowing for visualizing data, errors and for triggering actions. It is also meant to set up the configuration of the communication system. When new sensors are added, the dashboard can be easily updated for managing the newly received values. From the dashboard it is possible to: • check the system status; • receive real time data; • visualize video streams; • visualize alarms and events; • access to historical data;
• configure sensors and actuators and alarm events.
These are examples of the possible activities, but it can be extended according to the user request and the system needs. Furthermore, the dashboard integrates a series of scripts to simulate input from sensors, for verification and demonstration purposes. Any user can register to the Cloud gateway and receive the data with standard JSON format. Therefore, this open the possibility of integrating more components, as a Time Series Database, or to interface the Multi-Sensor Gateway with other systems.
Finally, the Multi-Sensor Gateway provides support for the integration of custom processing algorithms at the edge, for instance to allow for preliminary processing the data or implementing algorithm for secure communication (e.g., encryption and decryption).
At run-time, the Multi-Sensor Gateway orchestrates the reconfiguration of the network. The dashboard is the human-system interface, from which the operator can monitor the status of the system and intervene when necessary. The dashboard is meant to be the entry point in which to configure the reactions of the gateway according to the inputs. Upon reception of a specific trigger or event, the gateway can send to each sensor/device a sequence of commands to adjust the system configuration for the specific case.
Figure 3 illustrates a schematic overview of the Drone – Multi-Sensor Gateway communication.
HOME MAPS VIDEO REPORT EVENTS
STATISTICS { } "lng": 9.02739, "vz" : 0, "spd": 0.001, "alt": 6.01, "GPS": "GPS0", "lat": 39.29626
Gateway Dashboard
DRONE Name Drone 1 Drone 2 Drone 3
ROVER La tude 39,30 39,31 39.29
Generally speaking, the communication between a device and the Edge gateway, as well as the data format, are defined by the user. In this example, the drone sends data directly to the Edge gateway, using the MQTT communication protocol and the data format defined as follows:
For the user that manages the system, the device-dashboard communication is transparent. The user visualizes the system status and aggregated data from diferent devices and sensors into the dashboard, and does not need any knowledge of the communication mechanism behind it. Figure 4 illustrates an example of the dashboard in which the user selects the REPORT view and, among diferent options (DRONE, ROVER, FPGA, CAMERA), selects the DRONE tab to check data transmitted from diferent drones; similarly, in the VIDEO view the user could check the received video streams. Please, notice that the dashboard can be configured and customized according to the specific system and its application needs.
To test the Multi-Sensor Gateway it has been defined a set of procedures to be executed in sequence every time a new device is connected: 1. check the proper visualization of the data from the device, through a set of simulation scripts embedded in the dashboard; 2. check the proper reception of the data from the device, testing the communication link; 3. check the proper reception and visualization of the data from the newly added device, and from all the other devices connected through the communication system.
This set of procedures has been defined to be flexible with respect to diferent devices and scalable with respect to the number of devices and has been proved efective in the tests that have been executed with devices internal and external to the COMP4DRONES project. Integration with diferent devices (sensors, a Network Video Recorder (NVR), an FPGA-based hardware accelerator and a drone) proved that the modularity and flexibility of the Multi-Sensor Gateway facilitate the connection of diferent element through diferent protocols and user-specified interfaces: It also proves its efectiveness in managing diferent type of data (videos streams, telemetry data, alarms...) from diferent source and visualize them.
To evaluate the Multi-Sensor Gateway three key performance indicators (KPIs) has been defined in the context of the COMP4DRONES project: • KPI-1 - Ease of integration. • KPI-2 - Lightweight communication.
• KPI-3 - Robust communications.
KPI-1 has been measured through a qualitative evaluation from an experienced software engineer. An experienced software engineer that does not know the gateway, needs on average 3 days to get familiar with it and then from 4 to 12 hours (depending on the complexity of the interface and protocol) to implement the necessary changes to make it communicate with a new device and run the tests. Results, in terms of ease of integration, demonstrated the flexibility and modularity of the gateway and its usability in diferent contexts with limited set up efort.
The metric for KPI-2 is a boolean, since the gateway must be able to integrate with diferent devices, managing with diferent data from diferent sources, properly visualizing in the dashboard. Integration tests demonstrate the ability of the Multi-Sensor Gateway of acting as integration layer and orchestration actor of a system in which computation is distributed among diferent devices and sensors.
KPI-3 is evaluated through two metrics that refer to the capacity of the Multi-Sensor Gateway to be integrated with a plethora of diferent and heterogeneous devices: 1) The number of diferent communication protocols supported and 2) the number of diferent user-defined interfaces (data-interchange format) supported. Thanks to the integration with diferent devices the Multi-Sensor Gateway currently supports 6 communication protocols (REST, MQTT, UDP, TCP, TLS, GRAPHQL) and 5 user-defined interfaces. These numbers could easily increase according to further future integration needs. These results further contribute to demonstrate the ability of the Multi-Sensor Gateway of acting as integration layer and orchestration actor of a system, being able to integrate diferent and heterogeneous devices.
The Multi-Sensor Gateway has been assessed in real use-cases, mainly in the context of two
European Projects: the FitOptiVis ECSEL-JU Project [11] (concluded assessment) and the
COM4DRONES ECSEL-JU Projects [
In FitOptiVis, the use-case was a Water Supply Surveillance system including a SCADA system, a set of cameras (fixed cameras, smart cameras and a drone) and an NVR. From the point of view of the gateway, the challenge was the proper connection of multiple heterogeneous devices and the proper management of data received from them. From the dashboard (Figure 5) the operator could can see an overview of the system and check at any moment its status. The dashboard gave the possibility of selecting the diferent devices to check values, videos, alarms and trigger the reconfiguration of the smart cameras (simulated through the use of hardware reconfigurable accelerators). or videos. It is also possible to check the videos from the lateral menu, selecting the video source (camera or drone). A video presenting the Multi-Sensor Gateway in the context of FitOptiVis is available online2.
In Comp4Drones, the use-case is a Precision Agriculture that is meant to be the assessment ground for advanced observation and intervention methodologies that involve a combined usage of a drone and a rover [9]. Here, the challange focus to diferent processing related to three main task.
• On-ground non-realtime: this task concerns the processing of images previously acquired by a generic drone. It includes the creation of orthomosaic maps and related analysis. 2The Abinsula Multi-Sensor Gateway - https://youtu.be/hyBhZEb3Dwk
These images and the related data are then visualized in the dashboard, ofering an overview of how they change in diferent acquisitions. • On-ground realtime: this task concerns the realtime processing of a video acquired by a generic drone. The video is sent on-ground where it is processed at runtime and visualized on the dashbaord. • On-board realtime: this task includes a customized drone, which integrates a companion computer [10] with custom algorithms running on-board. Results of the processing are encrypted on-board, together with the drone telemetry data, before transmission and need to be decrypted in the gateway. Depending on the received data, a rover can be sent to the critical location. This opens the possibility to enable a feedback mechanism, in which the gateway can make decisions according to received data and send a command to reconfigure operation and it’s the base for more complex scenarios.
Current trends in transportation and other domains indicates the cameras as new fundamental sensors, and video streams as information to be fused with the one from other types of sensors. The strategy behind the evolution of this work is based on two pillars: communication and computation. The communication pillar is related to the integration of more heterogeneous nodes, from simple sensors, to drones, to other systems. Here we intend to integrate or study new protocols to increase the connectivity capabilities and explore more complex use cases.
The computation pillar is the key for consolidating the evolution of the Multi-Sensor Gateway, from a simple bridge among the sensors and the cloud, to a node able to make decisions and trigger the system reconfiguration. For this reason we are exploring more challenges algorithms and models and working on giving support for improving computation at the edge.
This work is carried out in the context of the FitOptiVis and COMP4DRONES Projects. FitOptiVis has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 783162. COMP4DRONES has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 826610. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Austria, Belgium, Czech Republic, France, Italy, Latvia, Netherlands. [7] Dell, EMC Edge Gateway Essentials: Everything You Want to Know but Were Afraid to Ask.
White Paper H19040. January 2022 [8] Eurotech, Everyware IoT. Online (6th July, 2022): https://www.eurotech.com/en/products/iot [9] COMP4DRONES, System Under test requirements and Test system requirements v2. Deliverable
D1.3. Online: https://www.comp4drones.eu/downloads/deliverables/ [10] Daniel Madroñal et al., Unmanned Vehicles in Smart Farming: a Survey and a Glance at Future Horizons. In Proceedings of the 2021 Drone Systems Engineering and Rapid Simulation and Performance Evaluation: Methods and Tools Proceedings (DroneSE and RAPIDO ’21). [11] Carlo Sau et al., Design and management of image processing pipelines within CPS: Acquired experience towards the end of the FitOptiVis ECSEL Project. Microprocessors and Microsystems, Volume 87, 2021.