Towards IoT Diversity via Automated Fleet Management Rustem Dautov Hui Song SINTEF Digital SINTEF Digital Oslo, Norway Oslo, Norway Email: rustem.dautov@sintef.no Email: hui.song@sintef.no Abstract—Large-scale Internet of Things (IoT) systems are continuously evolve during system operation. Examples of characterised by an increased level of heterogeneity, both in such changes may include security patches, user-specific terms of hardware and software caused by varying device configurations, upgrades to a new version of a software functionality, capabilities and performance. Moreover, since library, or an introduction of a new feature. As a result of this agile business requirements force IoT vendors to continuously software evolution, initially uniform devices constituting a modify the software components deployed at the Edge, even common IoT system might end up running software stacks, initially uniform devices constituting a common IoT ecosystem differing in their individual components and/or configura- might end up running software differing in individual compo- tions. Going beyond the traditional notion of a technique nents and/or configurations. The amount of effort required to for increasing system security and fault-tolerance, we refer maintain and operate such an increasingly diverse ecosystem to this phenomenon as software diversity to describe a wide – i.e. to perform fleet management – grows proportionally to range of fine-grained software modifications, applied by an the size and complexity of the IoT fleet, and is especially IoT vendor and driven by emerging business requirements. important for IoT vendors aiming to achieve economies of At the same time, it becomes challenging to operate scale. To address this challenge, this paper proposes a model- increasingly diverse systems and react to emerging require- based diversity engineering approach to enable automated fleet ments in an agile manner. With the recent advances in hard- management. Based on a model of an IoT system with fine- ware, networking, and containerisation technologies, this grained modifications to be applied, the proposed approach challenge, known as fleet management, is partially addressed by some IoT cloud platforms, which made it possible to is able to diversify and manage large-scale IoT systems at remotely access individual Edge devices, deploy container- run-time. As a proof of concept, the proposed approach was ised applications in a standardised automated manner, and implemented on top of the Azure IoT Hub fleet management monitor an IoT system at run-time. Such cloud platforms, facilities, and validated on a Remote Patient Monitoring use however, only provide some basic tools and lack an in- case scenario. telligent mechanism that would take into account business Index Terms—Internet of Things, Edge Computing, Diversity requirements and drive the software diversification process Engineering, Fleet Management, Azure IoT Hub. in an agile, fine-grained, and secure manner. To this end, this paper proposes a model-based approach to diversity-oriented fleet management in the context of 1. Introduction large-scale IoT deployments. The approach is inspired by and further extends the concept of software diversity, which The Internet of Things (IoT) facilitates creation of smart acts as a reference technique to enable multiple fleet man- spaces by converting existing environments into sensor-rich agement use cases. Accordingly, the main contribution of cyber-physical systems. As IoT ecosystems grow in size the paper is three-fold: i) a model-based, diversity-oriented and complexity, they become increasingly heterogeneous, approach to fleet management, ii) a diversity-oriented meta- especially at the very bottom layer, constituted by Edge model, and iii) a proof-of-concept implementation of the infrastructures. At this layer, Edge devices, albeit belonging fleet management system in the context of a Smart Health- to a common IoT ecosystem, might considerably differ both care scenario. in terms of their hardware (CPU type, networking interfaces, The rest of the paper is organised as follows. Section 2 available sensors/actuators, etc.) and software (operating explains the motivation behind the proposed research using systems, programming languages, libraries, communication a Smart Healthcare scenario. Section 3 provides background protocols, etc.) stacks. The former is relatively static, since information on IoT fleet management and relevant IoT new devices, once deployed and connected, are typically not cloud platforms, as well as software diversity. It also briefly expected to update their hardware configuration at run-time. describes the current state of the art and existing gaps. On contrary, the software configuration of devices con- Section 4 presents the overall approach with its conceptual stituting the very Edge of an IoT network is expected to architecture, and further elaborates on it with the details Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 47 of model-based diversity engineering for fleet management. Section 5 puts theory into practice by explaining how the proposed approach is applied to the previously introduced Smart Healthcare scenario. Section 6 concludes the paper and outlines some directions for future work. 2. Motivating Example: Remote Patient Mon- itoring Healthcare is one of the many domains, continuously improved by the pervasive penetration of IoT technologies, which are used to support core functions of healthcare institutions. Traditional hospitals and private houses are con- verted into smart digital environments, extensively making use of interconnected sensor systems to continuously collect Figure 1. Remote Patient Monitoring system. and transfer biometric data for timely analysis. Such smart telemedicine relies on a powerful infrastructure stack that includes sensor-enabled ‘leaf’ IoT devices, Edge devices and fault-tolerance [2] and is motivated by several other (e.g. gateways or other network nodes close to IoT devices), requirements. More specifically, taking the RPM scenario as wired/wireless networks, Cloud platforms, etc. [1]. a reference, the following diversity aspects can be observed: In this context, a simple, yet realistic scenario highlight- • User-tailored configuration: different users may have ing the challenges associated with software diversity at the different preferences with regard to the functionality, Edge is Remote Patient Monitoring (RPM). As depicted in performance, or security/privacy settings. Therefore, the Fig. 1, an RPM system, to which we will refer throughout same front-end application, when deployed on different this paper, assumes that a single installation in a residential premises, may be accordingly configured/customised by building involves multiple sensor-enabled microcontrollers the vendor, the end user, or a third party. (i.e. IoT devices) for monitoring patients and collecting • Hardware-specific configuration: large-scale IoT systems biometric/environmental data. There is also a Linux-based are highly heterogeneous in terms of underlying hardware Edge device that acts as a hub for collecting and pre- infrastructure (different CPU types, networking interfaces, processing raw sensor data, and further transferring them available sensors/actuators, etc.), which requires software to a Cloud-based back-end application. The Edge device to be accordingly configured/customised. Also, in some is equipped with several networking interfaces to interact cases, relatively powerful Edge device may be tasked with IoT devices and the Cloud. It also provides an inter- with local data pre-processing, whereas less powerful ones face for end users to configure the installation according are configured to only push data to the Cloud, which to individual requirements. Particular sets of IoT devices also results in uneven distribution of software components differ from one installation to another – i.e. some houses among Edge devices. might be equipped with fall detection sensors, whereas the • Targeted software updates: as part of a continuous soft- others are suited for sleep tracking. Through a Cloud-based ware development process, it is a common practice to management console, the system administrator is able to maintain more than one running version in production. monitor the status of each installation and push software For example, a widely used technique is the so-called updates to Edge devices. The RPM software comprises two A/B testing, when a sub-set of users is selected to try parts: new features, while the majority of users still keeps using • Front-end application runs on IoT and Edge devices and a previous stable release. is in charge of collecting and transferring data from the • Asynchronous updates: Edge devices can unpredictably surrounding physical environment (including humans), in- lose connectivity (e.g. due to unstable wireless connec- teracting with end users, and acting on the environment tion or limited power). During such (unpredictably long) via actuators when necessary. outages, new features or security patches may be applied • Back-end application runs on the Cloud and is in charge to the software. As a result, a disconnected device may of data analytics and decision making based on the data possibly miss several rounds of updates and thus run an received from the front-end application. obsolete version of software, thus introducing an acciden- To remain competitive on the market and offer a user- tal complexity in the operation of the system. tailored RPM solution, the IoT vendor has to react to new • Synthetic diversity: vendors may use various techniques to business requirements in an agile and flexible manner. In deliberately diversify their software for improving security these circumstances, it is expected that devices, initially uni- (i.e. implementing a ‘moving target’ for security attacks) form both in terms of hardware and software, will become or fault-tolerance (i.e. recovering a failed system to a increasingly diverse in terms of software. This software different version not to suffer from the same fault again). diversity, introduced by the vendor, goes beyond the tradi- Each of these aspects requires the IoT vendor to diversify tional notion of a technique for improving system security their IoT system by deploying and maintaining multiple 48 software versions for their fleets of IoT/Edge devices. Fur- IP addresses, constrained hardware/software capabilities of thermore, the vendor needs to match software configura- devices, limited connectivity, etc. In such circumstances, tions/customisations with target systems, and push software agile software development for IoT/Edge devices in a truly updates to relevant devices in a timely and guaranteed DevOps-compliant manner is hardly possible, as there is manner. This increases the complexity of IoT operation, an inevitable element of manual work required to deploy challenging the agility, reliability and economy of scale. software on a remote device. Admittedly, addressing this challenge in the context of large- This situation is changing with the recent advances in the scale and highly-distributed IoT deployments goes beyond networking and containerisation technologies. The former manual capabilities of the IoT vendor and requires an au- enabled accessing and interacting with remote devices by tomated fleet management approach to enable remote, se- establishing virtual private networks, while the latter allowed cure, and reliable support for multiple operations, including deploying and running light-weight and isolated software deployment, upgrade, monitoring, and troubleshooting of components in a platform-agnostic manner. These advances software components. have also been underpinned by the continuously increasing computing and networking capabilities of Edge/IoT devices. 3. Background and Related Work As a result, existing IoT platforms are now able to extend their functionality with tools for remote deployment of software components (packaged as Docker containers) and 3.1. Fleet Management through Cloud Platforms management of IoT devices. Fleet management is a cross-cutting multi-faceted con- cern for IoT vendors, especially when scaling from local 3.2. Software Diversity installations to global distributed ecosystems. In this paper, Traditionally, the scope of software diversity has been we focus on the control part, i.e. the deployment and upgrad- limited to the two main fields – namely, cyber-security and ing of software components of the front-end applications fault-tolerance [2]. The former case relies on applying var- on the IoT/Edge devices. In this respect, we identify the ious randomisation techniques at different system levels to following key requirements for effective and efficient fleet make software less vulnerable to generic threats by becom- management: ing a ‘moving target’ for them. In the latter case, software • Agility: Software deployment or update of the entire fleet systems with diversified functions and elements are expected must be done within an acceptable time frame. In the con- to handle various failures either by completely avoiding text of widely adopted DevOps practices, when vendors them, or by recovering an affected system to a different, aim to continuously introduce new features or patch vul- diversified version, thus lowering the risk of suffering from nerabilities, such duration is measured with hours or even the same fault again. minutes. Admittedly, achieving such short-term agility Currently, the concept of software diversity appears as implies having an automated solution able to remotely a rich and manifold notion with multiple facets, such as the access the managed system. goal of diversity, the diversification techniques, the scale of • Reliability: IoT vendors must be ensured at all times diversity, the application domain, when it is applied, etc. [2]. that the designated software is eventually deployed to It is also common to distinguish between natural (emerging) target end users, even in the presence of unstable network diversity, which appears spontaneously from the software connectivity and varying physical conditions of the Edge development process and run-time operation, and results in environment. different software versions, yet with similar functionality, • Economy of scale: As the number of end users increases, and artificial (synthetic or automated) diversity, which is average operation costs per installation should decrease, a result of explicit diversification actions taken by the IoT so that the vendor can benefit from the growth. vendor and applied to the system. These requirements are partially addressed by IoT cloud platforms – an emerging family of cloud solutions, which, 3.3. Existing solutions and Related Work apart from offering the traditional computing and storage resources, also provide an IoT-specific management layer for As of July 2019, there are at least three IoT cloud device monitoring, data flow design, data visualisation, etc. platforms already offering support for container-based fleet These existing platforms [3], [4] enable IoT developers to management as part of their portfolio. As a prerequisite, discover and integrate devices, monitor and diagnose system these solutions require Edge devices to be pre-installed with operation, as well as to collect, process, store, and visualise a software agent to communicate with the Cloud counterpart telemetry data. This typically assumes that Edge devices and a container engine to run containerised software compo- are deployed and manually configured to push collected nents. Among the available alternatives, Azure IoT Hub1 ap- data to a centralised back-end Cloud service in a ‘vertical’, pears as the most advanced option. On the one hand, it offers unidirectional manner [5]. Information exchange in the op- a rich ecosystem of various tools and services through its posite direction (e.g. actuation commands, re-configuration, marketplace (e.g. users can benefit from an existing Docker software/firmware updates) is not straight-forward and easily implemented due to network barriers, absence of static 1 https://azure.microsoft.com/services/iot-hub/ 49 image repository, an automatic device provisioning service, 4. Proposed Approach or a certificate-based authentication and access control), and, unlike AWS IoT Greengrass,2 supports full-featured In this paper, we propose a model-based approach to containers, not just serverless functions. On the other hand, diversity-oriented fleet management of IoT systems, and it outperforms Balena Cloud,3 which also provides similar implement a prototype fleet management tool named Di- full-featured support for container management, but is not vEnact.9 We primarily focus on the connection between yet mature and developed enough to offer an extensive IoT fleet management and software diversity, regardless of collection of of tools and services via a marketplace. whether it naturally emerges or is synthesised at design- and The community-driven Eclipse hawkBit4 is another rel- development-time. Driven by a diversity-oriented model, the evant framework aiming to automate IoT software updates proposed DivEnact solution is able to provide automatic run- at scale. Unlike the proprietary Cloud-based solutions, it is time management to deploy the diversified software onto agnostic to specific underlying technologies and can be inte- individual IoT/Edge devices. grated with many third-party components (albeit at the cost of increased manual integration and configuration effort). 4.1. Conceptual Architecture Despite the advanced built-in functionality offered by all these frameworks, they still lack another policy-driven control layer that would enable flexible, fine-grained and diversity-oriented software management and address the pre- viously outlined fleet management requirements. Potential solutions, albeit not primarily tailored to the purposes of IoT fleet management, already exist. More specifically, for some years now, multiple tools have been available on the market to support the deployment and configuration of software systems, e.g. Puppet,5 Chef,6 and new tools emerged for deployment of cloud-based systems such as CloudMF [6], OpenTOSCA [7], and Brooklyn.7 In addition, similar tools focus on the management and orchestration of containers, such as Kubernetes.8 When dealing with large- scale systems, these approaches focus on scaling out, i.e. duplicating identical components for load balancing. Figure 2. System architecture. As far as IoT fleet management is concerned, a sys- tematic literature review of 17 prominent approaches for Fig. 2 illustrates the overall architecture of the DivEnact orchestration and deployment for the IoT was conducted approach, which we conceptually split into three main parts. in [8]. All the surveyed approaches focus on the automatic 1) At the Edge, there is a fleet of multiple IoT installations. deployment of one or many IoT systems according to a Each installation serves an end user and has one central single deployment specification, without support for man- Edge device, which uses local wireless channels (e.g. aging a diverse fleet of IoT systems according to multiple WiFi and Bluetooth) to control and communicate with specifications. coupled IoT devices belonging to the same installation. With the proposed approach, we aim to bridge this For clarity purposes, each installation is represented only identified gap by enhancing the existing Cloud-based tools by a corresponding Edge device, while the underlying for fleet management with support for intelligent diversity- IoT devices are omitted. It is assumed that each Edge aware deployment and maintenance of software compo- device hosts a copy of the front-end application, which nents. To validate the proposed approach, we will take Azure consists of a set of software components running as IoT Hub as the baseline, which currently appears to be the Docker containers (depicted as multicoloured boxes). most mature and feature-rich (yet open-source and free for Under this assumption, fleet management can be sim- research purposes) offering. It is, nevertheless, expected that plified to a problem of managing containers running on the rest of the IoT cloud market will soon catch up with the Edge devices. fleet management trend by offering similar container-based 2) The edge devices constituting the fleet are registered functionality. to a central Cloud Hub and, through publish/subscribe messaging, are able to continuously report on their cur- 2 rent status and receive management instructions. The https://aws.amazon.com/greengrass/ 3 initial registration and further interaction with the Hub https://www.balena.io/cloud/ 4 https://www.eclipse.org/hawkbit/ is undertaken by a device-side software agent, which 5 https://puppet.com/ communicates to the following two services provided by 6 https://www.chef.io/chef/ the Cloud Hub: 7 https://brooklyn.apache.org/ 8 9 https://kubernetes.io/ Diversity Enactment for IoT fleets. 50 • Life-cycle Monitor collects reported information from Edge devices (e.g. connection status, deployed com- Deployment Module Variable Module Template instanceOf Instance ponents, system up-time, etc.) and keeps track of the name: string image-url: string fleet’s state. pre-condition module variable • Deployment Manager decides what components template deployedModule Deployment should be deployed on each Edge device, and enacts Variation Version derives Plan appliesTo Device this deployment on the device. It maintains a repos- value: string id: string id: string connection: string priority: integer itory of deployment specifications, each of which is Boolean tags defined with a scope of applicable Edge devices. For Expression operand Condition Tag devices that fall into the scope of a specification, the and or not Deployment Manager will enforce the devices to run refersTo the containers as specified, but sending corresponding Device Tag Capability Environment instructions. Docker container images are assumed to Condition Condition temperature production deviceID: string pressure preview be stored in a publicly available repository, such as humidity Docker Hub.10 refersTo 3) The DivEnact component interacts with the Cloud Hub through a REST API for querying and configuring Edge Figure 3. Meta-model for diversity-oriented IoT Fleet Management. device properties, and for manipulating deployment spec- ifications. Based on this API, the Diversity Controller implements diversity-oriented fleet management at run- Edge device in an IoT installation, and the latter represents time. It maintains a Diversity Pool – a collection of all a software component running on the device as a Docker potential variants of the front-end application used to container. A device is identified by its id and a unique generate deployment specifications for the Deployment connection string used for registering to the Cloud Hub. Manager. The Diversity Controller also maps deployment A device may be annotated by a Tag, which is a user- specifications to specific devices by matching device defined value used to limit the target scope of a deployment properties with specification scopes. These potential di- specification. For example, two sample Tag enumerations versity variants are either manually synthesised by the shown in Fig. 3 are Capability and Environment, which vendor, or automatically generated at design-time. The reflect hardware capabilities of a device and its application design-time diversity generation, at both code and ar- environment, respectively. The relationship between devices chitecture levels, is included in the overall architecture, and modules is derived – i.e. the IoT vendor cannot directly but goes beyond the scope of this paper. Further details assign modules to an Edge device, but rather they will on the model-based underpinnings of the Diversity Con- be specified with an application scope in a deployment troller are discussed below. specification. To express deployment specification, the IoT vendor 4.2. Model-Based Diversity Engineering defines a set of DeploymentPlans, each of which contains a set of Modules. Briefly, a Module serves to define how to At the core of the proposed DivEnact approach is the deploy a piece of software on a device. For example, as de- Diversity Controller, which maintains a run-time model of picted in Fig. 3, a Module may specify a URL of a container the IoT fleet and a diversity configuration to be applied. image, as well as parameters to instantiate the container Following the models@runtime pattern [9], the model is from this image. If target conditions of a DeploymentPlan dynamically synchronised with the running system, so that are satisfied by a device, then each Module defined in this the IoT fleet can be monitored and manipulated by reading plan will have a corresponding ModuleInstance deployed and writing the model, either manually or programmatically. on the device. Accordingly, the appliesTo relationship Fig. 3 illustrates key concepts and relationships, which between a DeploymentPlan and a Device will be derived constitute a meta-model for expressing a diversity-oriented from the Conditions of the plan. IoT fleet management scenario. Bold lines represent rela- A Condition is defined using either a device ID (i.e. tionships directly editable by the IoT vendor, while thin lines DeviceCondition) or Tags (i.e. TagCondition) attached indicate the relationships derived at run-time by the Diver- to devices. The former only applies to a device with this sity controller. The IoT vendor can control the system by specific ID, whereas the latter applies to multiple devices adding/removing model elements, changing their attributes, annotated with this specific Tag. A composite condition can or editing relationships between them. The DivEnact run- include several expressions, connected by Boolean operators time engine will automatically maintain the derived rela- and, or, and not, as illustrated by the code snippet below, tionships based on the edits, and apply the changes to the limiting the scope of a DeploymentPlan to devices in a underlying system, i.e. the IoT fleet under management. preview environment and either equipped with a temperature The concepts that directly represent the IoT fleet are sensor or having an empty ID (used for testing purposes). Device and ModuleInstance. The former represents an tags.Environment == preview and ( tags.Capability == temperature or 10 https://hub.docker.com/ ID == EmptyForTesting ) 51 The IoT vendor can create a DeploymentPlan man- the API provided by the Cloud Hub. The Diversity Con- ually, setting up its conditions and device tags to define troller will first invoke the API to create a deployment the scope of the deployment. However, when the size plan with the same ID in the Hub, and use a series of the fleet and the number of diverse deployment plans of subsequent API invocations to create the modules increase, such manual control would be time-consuming and insert them into the newly created deployment plan. and error-prone. We introduce the diversity generation part Similarly, when the vendor tags a device, the engine will into the meta-model to handle the automated creation of get its ID call the corresponding API method passing the deployment specifications. The vendor can start by defining tag value and the device ID. a DeploymentTemplate, which aggregates a number of 3) Enactment: we reuse the management features provided DeploymentPlans, and defines a common part shared by by the Deployment Manager to enact the changes on these plans. The varying parts of the plans are defined Edge devices. When a deployment plan is created or up- as Variables in the template. From each template, the dated, for each device falling into the scope, the Deploy- vendor can define several Versions, each of which contains ment Manager will evaluate if the desired modules (the Variations that assign concrete values to Variables. A ones defined in the deployment plan) are already running template is essentially a text document with a deployment on the device by querying the Life-cycle Monitor. If specification, expressed in YAML following the widely not, it will instruct the device to download and deploy adopted ‘Infrastructure as Code’ trend. the desired module. In case a device is not currently connected to the Hub, the pending deployment action 4.3. Run-time Model Synchronisation will be triggered immediately after the device appears online again. Following the models@runtime pattern, the IoT vendor is able to manipulate the fleet management model (e.g. 5. Proof of Concept11 add new elements, change attributes, and edit relationships), and the Diversity Controller will automatically apply these The proof of concept is currently implemented on top changes to the running devices. Depending on the changes, of Azure IoT Hub, which offers wide integration opportu- the synchronisation happens in three different levels. nities and rich built-in functionality to be re-used. Azure 1) In-model synchronisation, happening after changes on IoT Hub already provides its built-in Life-cycle Manager, parts of models for deployment templates and version, Deployment Manager, and a software agent to be installed performed by the diversity controller. The controller on Edge devices, which exempts us from ‘re-inventing the collects the user-input changes and directly change other wheel’ and focus on implementing the required DivEnact parts of models to maintain the consistency. functionality. 2) Model-to-hub synchronisation, happening after changes on the deployment plans and modules, performed also 5.1. Experimental Setup by diversity controller. The controller collects the model changes and transforms them into invocations to the REST API provided by the IoT Hub, in order to update We now re-visit the reference RPM scenario to demon- the deployment plans maintained by the hub. strate the viability of the proposed approach. This simplified 3) Hub-to-device synchronisation, happening after changes demonstration uses three Raspberry Pi boards acting as on deployment plan and device, performed by the IoT Edge devices and three SenseHat12 shields acting as sensor- Hub. Such synchronisation is triggered automatically enabled IoT devices. Each SenseHat shield is equipped after the API invocations to the IoT Hub, and will with a LED matrix and three sensors (humidity, tempera- eventually result in containers deployed on the devices. ture, and pressure) that can be used to collect information about the surrounding environmental conditions. Table 5.1 Typically, the synchronisation happens as a sequence in the summarises the available capabilities of each Raspberry Pi following three steps, corresponding to the three levels. board. The front-end application deployed on Raspberry Pi 1) Creation: this step takes place when the vendor creates boards can interact with the SenseHat shields to light up a deployment plan from one of the templates from the the LED matrix, collect sensor data and transfer them to the Diversity Pool. Instead of manually defining modules back-end application, which is assumed to evaluate whether for a deployment plan, the vendor can instead create an current conditions in a patient’s house are acceptable. empty plan and link it to a version. The synchronisation The fleet of these three Edge devices and a sample engine will load the template used by this version, and deployment plan are modelled in Fig. 4. Each Edge device resolve template variables to generate module definitions. is running in a production environment and is tagged with These modules will then be inserted into the deployment its capabilities (as summarised in Table 5.1). The model also plan. If the template contains pre-conditions, the engine contains a deployment plan with two modules (i.e. Docker will also pass them into the deployment plan. 2) Instantiation: when these model elements are created, 11 Source code related to this implementation can be found in removed, or edited, the Diversity Controller synchronises https://github.com/SINTEF-9012/divenact the changes with the Deployment Manager by invoking 12 https://www.raspberrypi.org/products/sense-hat/ 52 TABLE 1. T ESTBED SETUP. with two different versions frequency-1 and frequency-10 with frequencies of 1 and 10 readings Edge Device Temperature Pressure Humidity LED Matrix per second respectively, which can be retrieved by RPi1 Yes Yes - Yes specifying the variable FREQUENCY. The template also RPi2 Yes - Yes Yes has a composite pre-condition tags.Capability= RPi3 - Yes Yes Yes temperature and tags.Capability=led-matrix and tags.Environment=production. As a result, the appliesTo generated deployment plan only applies to RPi1 and RPi2 , mi1: but not RPi3 (which is not equipped with a temperature module1: RPi1: temp-sensor instanceOf ModuleInstance sensor). As a result, the module defined in the deployment Device image-url: temp-sensor deployedOn tag.Environment=production plan will only be instantiated on RPi1 and RPi2 . instanceOf tag.Capability=temperature deployedOn tag.Capaility=pressure module mi2: tag.Capability=led-matrix ModuleInstance 5.2. Sample Diversity Use Cases plan1: deployedOn RPi2: DeploymentPlan Device priority: 1 We now demonstrate how the Diversity Controller is able tag.Environment=production condition: tag.Environment=production mi3: tag.Capability=temperature to support some of the diversity use cases outlined in Section and tag.Capability=temperature ModuleInstance tag.Capaility=humidity and tag.Capability=led-matrix deployedOn tag.Capability=led-matrix 2. These diversity use cases are intended to demonstrate how module instanceOf RPi3: the model-based approach simplifies automated IoT fleet module2: mi4: Device management. Each use case is implemented as a simple, yet led-matrix instanceOf ModuleInstance tag.Environment=production efficient script, which manipulates the model accordingly. tag.Capability=humidity image-url: led-matrix tag.Capaility=pressure tag.Capability=led-matrix appliesTo 5.2.1. Pushing a new version to production. The IoT vendor has released a new version frequency-100, and Figure 4. Sample deployment plan. wants it to be the only production version for all Edge devices equipped with temperature sensors. DivEnact can implement this as the following script: images) led-matrix and temp-sensor that contain cor- for d in Device: responding application logic, while the latter is also able to d.tags.Environment = production set up the sampling frequency of a temperature sensor. The p = DeploymentPlan("production") p.version = frequency-100 plan is generated from the following template: p.condition = "tags.Environment==production" and p.version.template.precondition templates: divenact-template: temp-sensor: Briefly, the script sets all the devices into production settings: environment, and then create a deployment plan p from the image: songhui/temp-sensor:{{FREQUENCY}} version frequency-100. The condition of p is a two-fold: createOptions: "{"HostConfig": i) the plan works for all devices tagged with the production {"Privileged": true }}" environment, and ii) the plan inherits a pre-condition from type: docker led-matrix: the template, which limits its scope to devices equipped with settings: a temperature sensor. image: songhui/led-matrix createOptions: 5.2.2. Previewing a staging version. The IoT vendor has "{"HostConfig": {"Privileged": true }}" developed a new version frequency-100, but before re- type: docker leasing it to production wants to test it on a limited number pre-condition: tags.Capability=temperature and tags.Capability=led-matrix of selected users, while the rest of the users will still run and tags.Environment=production the current production version. Extending the previous use versions: case’s model, the following script will accomplish this: frequency-1: template: divenact-template frequency-100 = Version(’frequency-100’) parameter: temp-devices = [d for d in Devices FREQUENCY: 1 if eval(frequency-100.template.precondition, d)] frequency-10: for d in shuffle(temp-devices)[0:2]: template: divenact-template d.tags.Environment = preview parameter: p = DeploymentPlan(’preview’) FREQUENCY: 10 p.version = frequency-100 p.condition = ’tags.Environment==preview’ The code snippet above shows a sample deployment and p.version.template.precondition template with two modules named temp-sensor The script first finds all devices that satisfy the pre- and led-matrix, based on corresponding Docker condition of the version frequency-100, i.e. the devices images.13 The temp-sensor image is registered that are tagged with Capability=temperature. After 13 Please note the configuration parameter Privileged set to true, that, it randomly picks 2 devices and tags them with the which is required to enable communication with a SenseHat shield through preview environment. Finally, it creates a new deployment low-level GPIO pins. plan from this version, similar to the previous use case. 53 5.2.3. Shuffling versions. Given the three available ver- communicate with the Cloud. However, another common sions for the temperature sensor module (frequency-1, source of diversity comes from ‘leaf’ IoT devices, which frequency-10, and frequency-100) and one version for are typically equipped with micro-controllers, run simple the LED matrix (led-matrix), the IoT vendor wants to ar- firmware written in C, and only communicate with a cen- tificially synthesise diversity for increased security and fault- tral Edge hub using a wireless channel. Admittedly, there tolerance by deploying several diversified, but functionally are many situations, when IoT vendors might also require equivalent versions onto the fleet. remotely updating such device firmware in an agile manner. group1 = [frequency-1, frequency-10, frequency-100] In this light, a primary direction for future work to extend group2 = [led-matrix] the current approach with our IoT deployment tool named v2c = {} GeneSIS [10], which supports the ‘last-mile deployment’ for group in [group1, group2]: devices = [d for d in Devices if from Edge devices to IoT devices. The idea is to deploy a eval(group[0].template.precondition, d)] GeneSIS engine as a container on an Edge device; the engine index = 0 for d in shuffle(devices): will then interact with the coupled IoT devices and push re- v = group[index % len(group)] quired updates using available communication channels. By v2c[v] = "id = " + d.id extending the proposed meta-model with relevant concepts, for v in group1 + group2: p = DeploymentPlan(v.id) it will be possible to model IoT devices and diversification p.version = v and apply these modifications to the running system. As p.condition = "or". join(v2c[v]) a result, we expect to provide a holistic fleet management solution covering both Edge and IoT devices. The script divides the available versions into two groups according to their relation to the temperature sensor or the LED matrix, respectively. For each group, it finds all devices Acknowledgments that satisfy the common pre-condition of this group. Next, This work is supported by the H2020 programme under for each device it assigns a version from this group in the grant agreement #780351 (ENACT). a circular order, and generates a device-based condition. Finally, it creates a deployment plan for each version, and References the condition of this plan is a conjunction of all device-based conditions that we created for the corresponding version. [1] R. Dautov, S. Distefano, and R. Buyya, “Hierarchical data fusion for In all the experiments in the current implementation, the Smart Healthcare,” Journal of Big Data, vol. 6, no. 1, p. 19, 2019. time frame between the moment when a deployment plan [2] B. Baudry and M. Monperrus, “The multiple facets of software is first generated and the moment when an Edge device diversity: Recent developments in year 2000 and beyond,” ACM Computing Surveys (CSUR), vol. 48, no. 1, p. 16, 2015. starts running new software ranges from 30 seconds to two [3] P. P. Ray, “A survey of IoT cloud platforms,” Future Computing and minutes. This primarily depends on the size of a Docker Informatics Journal, vol. 1, no. 1-2, pp. 35–46, 2016. image to be downloaded and launched, which remained rel- [4] T. Pflanzner and A. Kertész, “A survey of IoT cloud providers,” in atively small given the simplified setup of the experiments. 2016 39th International Convention on Information and Communica- In practise, downloadable software components are expected tion Technology, Electronics and Microelectronics (MIPRO). IEEE, to be somewhat larger. Nevertheless, we still expect the op- 2016, pp. 730–735. eration of the proposed system fulfil the agile requirements [5] R. Dautov, S. Distefano, D. Bruneo, F. Longo, G. Merlino, and A. Pu- liafito, “Pushing Intelligence to the Edge with a Stream Processing of DevOps practices. Architecture,” in 2017 IEEE International Conference on Internet of Things (iThings). IEEE, 2017, pp. 792–799. 6. Conclusion and Future Work [6] N. Ferry, F. Chauvel, H. Song, A. Rossini, M. Lushpenko, and A. Solberg, “CloudMF: Model-Driven Management of Multi-Cloud Applications,” ACM Transactions on Internet Technology, vol. 18, With this paper, we addressed the automated, diversity- no. 2, p. 16, 2018. oriented fleet management in large-scale IoT deployments [7] A. C. F. da Silva, U. Breitenbücher, K. Képes, O. Kopp, and through a model-based approach. Based on a software diver- F. Leymann, “OpenTOSCA for IoT: Automating the Deployment sity model, the proposed approach allows dynamically man- of IoT Applications Based on the Mosquitto Message Broker,” in Proceedings of the 6th International Conference on the Internet of aging containerised software components on Edge devices in Things. ACM, 2016, pp. 181–182. IoT systems ranging from local deployments to large-scale, [8] P. H. Nguyen, N. Ferry, G. Erdogan, H. Song, S. Lavirotte, J.-Y. geographically distributed ecosystems. Characterised by an Tigli, and A. Solberg, “Advances in deployment and orchestration increasing degree of software diversity, such systems are approaches for iot - a systematic review,” in IEEE International required to be managed in a reliable and automated manner Congress On Internet of Things (ICIOT). IEEE, 2019. so as to achieve economies of scale. As a first step towards [9] N. Bencomo, R. B. France, B. H. Cheng, and U. Aßmann, Mod- the validation of the approach, the paper described a simple, els@run.time: Foundations, Applications, and Roadmaps. Springer, 2014, vol. 8378. yet realistic smart IoT scenario, where software components [10] N. Ferry, P. Nguyen, H. Song, P.-E. Novac, S. Lavirotte, J.-Y. Tigli, were pushed to Edge devices based on a diversity-oriented and A. Solberg, “GeneSIS: Continuous Orchestration and Deploy- model continuously synchronised at run-time. ment of Smart IoT Systems,” in 2019 IEEE 43rd Annual Computer The proposed approach assumes that target Edge de- Software and Applications Conference (COMPSAC), vol. 1. IEEE, vices are sufficiently capable to run a container engine and 2019, pp. 870–875. 54