The Multi-Sensor Gateway, a unified communication scheme and orchestration actor for heterogeneous systems - Technical Paper⋆ Tiziana Fanni1 , Giuseppe Meloni1 , Marco Melis1 , Antonio Solinas1 and Maria Katiuscia Zedda1 1 Abinsula Srl, Sassari, Italy Abstract In the device-edge-cloud continuum era, where e plethora of different devices, from simple sensors to drones, coexist and collaborate in the same system of systems, reliable and flexible communication plays a crucial role. This technical paper presents the Abinsula Multi-Sensor Gateway, a unified communica- tion and management platform for cooperative embedded systems integrated in a system-of-systems environment. The Multi-Sensor Gateway is exploited in drone and rover based services, in the context of the COMP4DRONES ECSEL-JU project. Keywords IoT, Embedded system, Edge Gateway, Device-Edge-Cloud 1. Context and Objectives The diffusion of edge computing, internet of things (IoT) technologies and the access to high- speed internet contributed to a shift toward distributed device-edge-cloud computing continuum. More and more objects are embedded with sensors and connected to exchange information, and data can be processed in different points of the system, according to different needs. This enabled the possibility of complex systems, composed of heterogeneous objects (spanning from simple sensors to objects that are small systems themselves), that are able to operate (semi-)autonomously with little or no involvement of human operator (see Figure 1. In such a context, the COMP4DRONES ECSEL-JU project1 focuses on services based on unmanned aircraft vehicles (referred to as drones) and unmanned ground vehicles (referred to as rovers), which are driven by increased levels of connectivity and automation. In particular, the main goal of COMP4DRONES is to provide a framework of key enabling technologies for safe and autonomous drones [1]. It proposes a holistically designed ecosystem, based on a tightly integrated multi-vendor and compositional drone architecture solution and supported by a chain of tools, that sustains: • agile and cost-effective compositional and modular design; Woodstock’22: Symposium on the irreproducible science, June 07–11, 2022, Woodstock, NY $ tiziana.fanni@abinsula.com (T. Fanni)  0000-0002-4301-6497 (T. Fanni) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop CEUR Workshop Proceedings (CEUR-WS.org) Proceedings http://ceur-ws.org ISSN 1613-0073 1 https://www.comp4drones.eu/ Figure 1: IoT systems are not standalone anymore, but they are often interconnected systems, that can include not only simple sensors, but also more complex devices. • efficient customization and integration of drone-based systems; • secure and safe autonomous decisions related to both individual and cooperative missions; • reliable drone to drone and ground-based drone communication. Communication is one of the technological bricks of the COMP4DRONES framework. When unmanned vehicles are involved two aspects are highly relevant: (1) the characteristics and the availability of the link, and (2) the presence of unintentional or malicious attacks. In addition, it is necessary to enable a certain degree of flexibility in the communication infrastructure, to satisfy the needs of different heterogeneous devices that could need to be integrated in the system. Indeed, according to the Metcalf’s Low, the value of an interconnected system and the functions it supports are proportional to the number of its interconnections and its degree of interaction with other systems, devices, people and resources [2]. Different commercial solutions are available that try to address a flexible communication. For instance, the Mindtree’s Multiprotocol IoT gateway support multiple industrial protocols and offers two versions: 1) a Mesh Gateway to aggregate data coming from different sensors and from Bluetooth Low Energy Mesh nodes, and 2) an Edge Gateway that communicate with multiple Mesh Gateways and to the application server [3]. And the SWYCS Industrial IoT Gateway is a multi-protocol gateway with an open operating system that can be adapted to the application needs [4]. However, these solutions do not offer any integrated platform for data visualization and management. Other companies offer more integrated platforms. The Bosh IoT Edge Services offer connectiv- ity to a heterogeneous device landscape, through a large number of different IoT protocols and open API. It also offers data history, statistics and the tools to build a custom set of services [5]. The ADLINK’s IoT Gateway platform offers edge-to-cloud connection. Its edge gateways are based on Intel Atom processors, they collect data from several edge devices and transmit it to the cloud over wireless communication [6]. Dell developed IoT solutions ranging from edge controllers, gateways and cloud platforms. The hardware gateways of the 5200 series ninth generation are based on Intel Core and support the 20.04 LTS and Microsoft Windows 10 IoT Enterprise LTSC operating systems [7]. The Eurotech Everyware IoT is an open-source based integrated platform which provides an IoT gateways, an open edge framework and a modular cloud infrastructure. Hardware gateways are Intel Atom/NXP I.MX 8M -based and include certified Industrial, Automotive, Railway product [8]. All of above solutions are well developed products ready for the market, an so they do not meet the flexibility required for special customized applications such the ones in the COMP4DRONES context where the connected devices are not the typical devices that belong to the IoT world, being drones and rovers involved. Also, most state-of-the-art solutions are based on their own edge application which is specifically dependent on a particular hardware, thus lacking in ease of customization, applicability and in open source aspects. 2. The Multi-Sensor Gateway The Multi-Sensor Gateway provides a unified management system, contributing to the secure and safe integration of a cooperative embedded system in a system-of-systems environment. It is a generic and modular component, based on an embedded Linux distribution, that works with all the main communication protocols and can be interfaced with elements of the system that communicate through user-specified data formats. 2.1. Overview The Multi-Sensor Gateway enables a runtime configurable multi-context support by modifying data priority transmission upon the scenario requirements, acting as the orchestration actor of a heterogeneous network composed of a plethora of different nodes: sensors and actuators (valves, temperature, alarms, liquid and gas sensors), cameras, hardware accelerators, drones, rovers. Figure 2 illustrates a schematic overview of the Multi-Sensor Gateway that provides: • a flexible and extensible Edge gateway, to interface different custom devices that talk through different communication protocols and user-defined interfaces; • a Cloud gateway with a user-friendly dashboard, to offer a unified interface for monitoring and managing a complex system, composed of different heterogeneous devices; • Processing at the Edge, enabling the possibility of running custom applications and dis- tribute the computation among different points in the communication infrastructure. The Edge gateway is based on Ability, the Abinsula Embedded Linux distribution. It is designed to be flexible and scalable, with respect to the type and number of devices that can communicate with it, and transparent for the user, enabling transparent communication among Figure 2: Multi-Sensor Gateway - overview. the devices of the system and the Cloud gateway. Being possible a bidirectional communication, the Edge gateway receives inputs from devices in the system on one side, and from the Cloud gateway on the other side. Inputs from the devices are characterized by a custom user-specific interface. For each interface, a data converter module is available to normalize the data and convert them into standard JSON data that can be sent to the Cloud gateway (see Figure 2). Any data sent from the Cloud gateway to the system devices pass through the Edge gateway that adapt them according to the user-specific interface. New sensors can be added at any time without any particular registration, the gateway exploits the user ID to recognize it as a new module. The Edge gateway has been extended to support the different communication and physical layer protocols required by different devices, such as sensors, smart cameras, FPGA-based hardware accelerators, drones and rovers. Currently it supports several different communication protocols. Table 2.1 depicts an overview of the Edge gateway communication. The Cloud gateway enables remote feedback and management of the system, through a dashboard that offers a unified and user-friendly interface for the operator, not expert of the specific devices connected in the system, to monitor and manage the system, allowing for visualizing data, errors and for triggering actions. It is also meant to set up the configuration of the communication system. When new sensors are added, the dashboard can be easily updated for managing the newly received values. From the dashboard it is possible to: • check the system status; • receive real time data; • visualize video streams; • visualize alarms and events; • access to historical data; Table 1 Edge gateway Communication. Inputs From device User-custom interface From Cloud gateway JSON data Outputs To device User-custom interface To Cloud gateway JSON data Communication protocol With device User specified (REST, MQTT, UDP, TCP, GRAPHQL. . . ) With Cloud gateway MQTT Physical layer protocols The ethernet is the only one currently implemented, but it can be expanded to support other different physical layer protocols i.e. Zigbee, ZWave, WiMAX LoRaWAN. Figure 3: Example of a Drone - Multi-Sensor Gateway communication. • configure sensors and actuators and alarm events. These are examples of the possible activities, but it can be extended according to the user request and the system needs. Furthermore, the dashboard integrates a series of scripts to simulate input from sensors, for verification and demonstration purposes. Any user can register to the Cloud gateway and receive the data with standard JSON format. Therefore, this open the possibility of integrating more components, as a Time Series Database, or to interface the Multi-Sensor Gateway with other systems. Finally, the Multi-Sensor Gateway provides support for the integration of custom processing algorithms at the edge, for instance to allow for preliminary processing the data or implementing algorithm for secure communication (e.g., encryption and decryption). 2.2. Run-time support At run-time, the Multi-Sensor Gateway orchestrates the reconfiguration of the network. The dashboard is the human-system interface, from which the operator can monitor the status of the system and intervene when necessary. The dashboard is meant to be the entry point in which to configure the reactions of the gateway according to the inputs. Upon reception of a specific trigger or event, the gateway can send to each sensor/device a sequence of commands to adjust the system configuration for the specific case. Figure 3 illustrates a schematic overview of the Drone – Multi-Sensor Gateway communication. Gateway Dashboard LOGOUT HOME MAPS DRONE ROVER FPGA CAMERA VIDEO Name La tude Longitude Variometer Speed Al tude GPS REPORT Drone 1 39,30 9.007 0 0.001 6.01 GPS0 Drone 2 39,31 9.022 1 0.002 15.07 GPS0 EVENTS Drone 3 39.29 9.027 2 0.003 11.21 GPS0 STATISTICS Figure 4: Example of the Dashboard. Generally speaking, the communication between a device and the Edge gateway, as well as the data format, are defined by the user. In this example, the drone sends data directly to the Edge gateway, using the MQTT communication protocol and the data format defined as follows: { "lng": 9.02739, "vz" : 0, "spd": 0.001, "alt": 6.01, "GPS": "GPS0", "lat": 39.29626 } For the user that manages the system, the device-dashboard communication is transparent. The user visualizes the system status and aggregated data from different devices and sensors into the dashboard, and does not need any knowledge of the communication mechanism behind it. Figure 4 illustrates an example of the dashboard in which the user selects the REPORT view and, among different options (DRONE, ROVER, FPGA, CAMERA), selects the DRONE tab to check data transmitted from different drones; similarly, in the VIDEO view the user could check the received video streams. Please, notice that the dashboard can be configured and customized according to the specific system and its application needs. 3. Assessment and Evaluation To test the Multi-Sensor Gateway it has been defined a set of procedures to be executed in sequence every time a new device is connected: 1. check the proper visualization of the data from the device, through a set of simulation scripts embedded in the dashboard; 2. check the proper reception of the data from the device, testing the communication link; 3. check the proper reception and visualization of the data from the newly added device, and from all the other devices connected through the communication system. This set of procedures has been defined to be flexible with respect to different devices and scalable with respect to the number of devices and has been proved effective in the tests that have been executed with devices internal and external to the COMP4DRONES project. Integration with different devices (sensors, a Network Video Recorder (NVR), an FPGA-based hardware accelerator and a drone) proved that the modularity and flexibility of the Multi-Sensor Gateway facilitate the connection of different element through different protocols and user-specified interfaces: It also proves its effectiveness in managing different type of data (videos streams, telemetry data, alarms...) from different source and visualize them. To evaluate the Multi-Sensor Gateway three key performance indicators (KPIs) has been defined in the context of the COMP4DRONES project: • KPI-1 - Ease of integration. • KPI-2 - Lightweight communication. • KPI-3 - Robust communications. KPI-1 has been measured through a qualitative evaluation from an experienced software engineer. An experienced software engineer that does not know the gateway, needs on average 3 days to get familiar with it and then from 4 to 12 hours (depending on the complexity of the interface and protocol) to implement the necessary changes to make it communicate with a new device and run the tests. Results, in terms of ease of integration, demonstrated the flexibility and modularity of the gateway and its usability in different contexts with limited set up effort. The metric for KPI-2 is a boolean, since the gateway must be able to integrate with differ- ent devices, managing with different data from different sources, properly visualizing in the dashboard. Integration tests demonstrate the ability of the Multi-Sensor Gateway of acting as integration layer and orchestration actor of a system in which computation is distributed among different devices and sensors. KPI-3 is evaluated through two metrics that refer to the capacity of the Multi-Sensor Gateway to be integrated with a plethora of different and heterogeneous devices: 1) The number of different communication protocols supported and 2) the number of different user-defined interfaces (data-interchange format) supported. Thanks to the integration with different devices the Multi-Sensor Gateway currently supports 6 communication protocols (REST, MQTT, UDP, TCP, TLS, GRAPHQL) and 5 user-defined interfaces. These numbers could easily increase according to further future integration needs. These results further contribute to demonstrate the ability of the Multi-Sensor Gateway of acting as integration layer and orchestration actor of a system, being able to integrate different and heterogeneous devices. Figure 5: Overview of the dashboard in the FitOptiVis project. 3.1. The use cases The Multi-Sensor Gateway has been assessed in real use-cases, mainly in the context of two European Projects: the FitOptiVis ECSEL-JU Project [11] (concluded assessment) and the COM4DRONES ECSEL-JU Projects [1] (ongoing assessment). In FitOptiVis, the use-case was a Water Supply Surveillance system including a SCADA system, a set of cameras (fixed cameras, smart cameras and a drone) and an NVR. From the point of view of the gateway, the challenge was the proper connection of multiple heterogeneous devices and the proper management of data received from them. From the dashboard (Figure 5) the operator could can see an overview of the system and check at any moment its status. The dashboard gave the possibility of selecting the different devices to check values, videos, alarms and trigger the reconfiguration of the smart cameras (simulated through the use of hardware reconfigurable accelerators). or videos. It is also possible to check the videos from the lateral menu, selecting the video source (camera or drone). A video presenting the Multi-Sensor Gateway in the context of FitOptiVis is available online2 . In Comp4Drones, the use-case is a Precision Agriculture that is meant to be the assessment ground for advanced observation and intervention methodologies that involve a combined usage of a drone and a rover [9]. Here, the challange focus to different processing related to three main task. • On-ground non-realtime: this task concerns the processing of images previously acquired by a generic drone. It includes the creation of orthomosaic maps and related analysis. 2 The Abinsula Multi-Sensor Gateway - https://youtu.be/hyBhZEb3Dwk Figure 6: Overview of the dashboard in the COMP4DRONES project. These images and the related data are then visualized in the dashboard, offering an overview of how they change in different acquisitions. • On-ground realtime: this task concerns the realtime processing of a video acquired by a generic drone. The video is sent on-ground where it is processed at runtime and visualized on the dashbaord. • On-board realtime: this task includes a customized drone, which integrates a companion computer [10] with custom algorithms running on-board. Results of the processing are encrypted on-board, together with the drone telemetry data, before transmission and need to be decrypted in the gateway. Depending on the received data, a rover can be sent to the critical location. This opens the possibility to enable a feedback mechanism, in which the gateway can make decisions according to received data and send a command to reconfigure operation and it’s the base for more complex scenarios. Figure 6 illustrates the dashboard in the context of the on-board realtime task, depicting data received fro, the drone and the encryption algorithm. 3.2. The evolution Current trends in transportation and other domains indicates the cameras as new fundamental sensors, and video streams as information to be fused with the one from other types of sensors. The strategy behind the evolution of this work is based on two pillars: communication and computation. The communication pillar is related to the integration of more heterogeneous nodes, from simple sensors, to drones, to other systems. Here we intend to integrate or study new protocols to increase the connectivity capabilities and explore more complex use cases. Figure 7: The strategy behind the evolution of the Multi-Sensor Gateway. The computation pillar is the key for consolidating the evolution of the Multi-Sensor Gateway, from a simple bridge among the sensors and the cloud, to a node able to make decisions and trigger the system reconfiguration. For this reason we are exploring more challenges algorithms and models and working on giving support for improving computation at the edge. Acknowledgements This work is carried out in the context of the FitOptiVis and COMP4DRONES Projects. FitOptiVis has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 783162. COMP4DRONES has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 826610. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Austria, Belgium, Czech Republic, France, Italy, Latvia, Netherlands. References [1] Réda Nouacer et al., Towards a framework of key technologies for drones. Microprocessors and Microsystems, Volume 77, 2020. [2] B. Metcalfe: Metcalfe’s Law: A network becomes more valuable as it reaches more users. Info-world, Oct. 2, 1995. [3] Mindtree, Multiprotocol IoT Gateway. Online (6th July, 2022): https://www.mindtree.com/services/customer-success/product-engineering- services/smart-lighting-solutions/multiprotocol-iot-gateway. [4] SWYCS, Industrial IoT Gateway. Online (6th July, 2022): https://landing.swycs.com/swycs- iot-industrial-gateway/ [5] Bosch, Bosch IoT Edge Services. Online (6th July, 2022): https://bosch-iot- suite.com/service/bosch-iot-edge-services/ [6] ADLINK, IoT Gateway. Online (6th July, 2022): https://www.adlinktech.com/en/Industrial_IoT_and_Cloud_solutions_IoT_Gateway. [7] Dell, EMC Edge Gateway Essentials: Everything You Want to Know but Were Afraid to Ask. White Paper H19040. January 2022 [8] Eurotech, Everyware IoT. Online (6th July, 2022): https://www.eurotech.com/en/products/iot [9] COMP4DRONES, System Under test requirements and Test system requirements v2. Deliverable D1.3. Online: https://www.comp4drones.eu/downloads/deliverables/ [10] Daniel Madroñal et al., Unmanned Vehicles in Smart Farming: a Survey and a Glance at Future Horizons. In Proceedings of the 2021 Drone Systems Engineering and Rapid Simulation and Performance Evaluation: Methods and Tools Proceedings (DroneSE and RAPIDO ’21). [11] Carlo Sau et al., Design and management of image processing pipelines within CPS: Acquired experience towards the end of the FitOptiVis ECSEL Project. Microprocessors and Microsystems, Volume 87, 2021.