<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Towards IoT Diversity via Automated Fleet Management</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Rustem Dautov</string-name>
          <email>rustem.dautov@sintef.no</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Hui Song</string-name>
          <email>hui.song@sintef.no</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>SINTEF Digital</institution>
          ,
          <addr-line>Oslo</addr-line>
          ,
          <country country="NO">Norway</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2019</year>
      </pub-date>
      <fpage>47</fpage>
      <lpage>54</lpage>
      <abstract>
        <p>-Large-scale Internet of Things (IoT) systems are characterised by an increased level of heterogeneity, both in terms of hardware and software caused by varying device functionality, capabilities and performance. Moreover, since agile business requirements force IoT vendors to continuously modify the software components deployed at the Edge, even initially uniform devices constituting a common IoT ecosystem might end up running software differing in individual components and/or configurations. The amount of effort required to maintain and operate such an increasingly diverse ecosystem - i.e. to perform fleet management - grows proportionally to the size and complexity of the IoT fleet, and is especially important for IoT vendors aiming to achieve economies of scale. To address this challenge, this paper proposes a modelbased diversity engineering approach to enable automated fleet management. Based on a model of an IoT system with finegrained modifications to be applied, the proposed approach is able to diversify and manage large-scale IoT systems at run-time. As a proof of concept, the proposed approach was implemented on top of the Azure IoT Hub fleet management facilities, and validated on a Remote Patient Monitoring use case scenario.</p>
      </abstract>
      <kwd-group>
        <kwd>Internet of Things</kwd>
        <kwd>Edge Computing</kwd>
        <kwd>Diversity Engineering</kwd>
        <kwd>Fleet Management</kwd>
        <kwd>Azure IoT Hub</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>The Internet of Things (IoT) facilitates creation of smart
spaces by converting existing environments into sensor-rich
cyber-physical systems. As IoT ecosystems grow in size
and complexity, they become increasingly heterogeneous,
especially at the very bottom layer, constituted by Edge
infrastructures. At this layer, Edge devices, albeit belonging
to a common IoT ecosystem, might considerably differ both
in terms of their hardware (CPU type, networking interfaces,
available sensors/actuators, etc.) and software (operating
systems, programming languages, libraries, communication
protocols, etc.) stacks. The former is relatively static, since
new devices, once deployed and connected, are typically not
expected to update their hardware configuration at run-time.</p>
      <p>On contrary, the software configuration of devices
constituting the very Edge of an IoT network is expected to
continuously evolve during system operation. Examples of
such changes may include security patches, user-specific
configurations, upgrades to a new version of a software
library, or an introduction of a new feature. As a result of this
software evolution, initially uniform devices constituting a
common IoT system might end up running software stacks,
differing in their individual components and/or
configurations. Going beyond the traditional notion of a technique
for increasing system security and fault-tolerance, we refer
to this phenomenon as software diversity to describe a wide
range of fine-grained software modifications, applied by an
IoT vendor and driven by emerging business requirements.</p>
      <p>At the same time, it becomes challenging to operate
increasingly diverse systems and react to emerging
requirements in an agile manner. With the recent advances in
hardware, networking, and containerisation technologies, this
challenge, known as fleet management, is partially addressed
by some IoT cloud platforms, which made it possible to
remotely access individual Edge devices, deploy
containerised applications in a standardised automated manner, and
monitor an IoT system at run-time. Such cloud platforms,
however, only provide some basic tools and lack an
intelligent mechanism that would take into account business
requirements and drive the software diversification process
in an agile, fine-grained, and secure manner.</p>
      <p>To this end, this paper proposes a model-based approach
to diversity-oriented fleet management in the context of
large-scale IoT deployments. The approach is inspired by
and further extends the concept of software diversity, which
acts as a reference technique to enable multiple fleet
management use cases. Accordingly, the main contribution of
the paper is three-fold: i) a model-based, diversity-oriented
approach to fleet management, ii) a diversity-oriented
metamodel, and iii) a proof-of-concept implementation of the
fleet management system in the context of a Smart
Healthcare scenario.</p>
      <p>The rest of the paper is organised as follows. Section 2
explains the motivation behind the proposed research using
a Smart Healthcare scenario. Section 3 provides background
information on IoT fleet management and relevant IoT
cloud platforms, as well as software diversity. It also briefly
describes the current state of the art and existing gaps.
Section 4 presents the overall approach with its conceptual
architecture, and further elaborates on it with the details
of model-based diversity engineering for fleet management.
Section 5 puts theory into practice by explaining how the
proposed approach is applied to the previously introduced
Smart Healthcare scenario. Section 6 concludes the paper
and outlines some directions for future work.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Motivating Example: Remote Patient Monitoring</title>
      <p>
        Healthcare is one of the many domains, continuously
improved by the pervasive penetration of IoT technologies,
which are used to support core functions of healthcare
institutions. Traditional hospitals and private houses are
converted into smart digital environments, extensively making
use of interconnected sensor systems to continuously collect
and transfer biometric data for timely analysis. Such smart
telemedicine relies on a powerful infrastructure stack that
includes sensor-enabled ‘leaf’ IoT devices, Edge devices
(e.g. gateways or other network nodes close to IoT devices),
wired/wireless networks, Cloud platforms, etc. [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
      </p>
      <p>In this context, a simple, yet realistic scenario
highlighting the challenges associated with software diversity at the
Edge is Remote Patient Monitoring (RPM). As depicted in
Fig. 1, an RPM system, to which we will refer throughout
this paper, assumes that a single installation in a residential
building involves multiple sensor-enabled microcontrollers
(i.e. IoT devices) for monitoring patients and collecting
biometric/environmental data. There is also a Linux-based
Edge device that acts as a hub for collecting and
preprocessing raw sensor data, and further transferring them
to a Cloud-based back-end application. The Edge device
is equipped with several networking interfaces to interact
with IoT devices and the Cloud. It also provides an
interface for end users to configure the installation according
to individual requirements. Particular sets of IoT devices
differ from one installation to another – i.e. some houses
might be equipped with fall detection sensors, whereas the
others are suited for sleep tracking. Through a Cloud-based
management console, the system administrator is able to
monitor the status of each installation and push software
updates to Edge devices. The RPM software comprises two
parts:</p>
      <p>Front-end application runs on IoT and Edge devices and
is in charge of collecting and transferring data from the
surrounding physical environment (including humans),
interacting with end users, and acting on the environment
via actuators when necessary.</p>
      <p>Back-end application runs on the Cloud and is in charge
of data analytics and decision making based on the data
received from the front-end application.</p>
      <p>
        To remain competitive on the market and offer a
usertailored RPM solution, the IoT vendor has to react to new
business requirements in an agile and flexible manner. In
these circumstances, it is expected that devices, initially
uniform both in terms of hardware and software, will become
increasingly diverse in terms of software. This software
diversity, introduced by the vendor, goes beyond the
traditional notion of a technique for improving system security
and fault-tolerance [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] and is motivated by several other
requirements. More specifically, taking the RPM scenario as
a reference, the following diversity aspects can be observed:
User-tailored configuration: different users may have
different preferences with regard to the functionality,
performance, or security/privacy settings. Therefore, the
same front-end application, when deployed on different
premises, may be accordingly configured/customised by
the vendor, the end user, or a third party.
      </p>
      <p>Hardware-specific configuration: large-scale IoT systems
are highly heterogeneous in terms of underlying hardware
infrastructure (different CPU types, networking interfaces,
available sensors/actuators, etc.), which requires software
to be accordingly configured/customised. Also, in some
cases, relatively powerful Edge device may be tasked
with local data pre-processing, whereas less powerful ones
are configured to only push data to the Cloud, which
also results in uneven distribution of software components
among Edge devices.</p>
      <p>Targeted software updates: as part of a continuous
software development process, it is a common practice to
maintain more than one running version in production.
For example, a widely used technique is the so-called
A/B testing, when a sub-set of users is selected to try
new features, while the majority of users still keeps using
a previous stable release.</p>
      <p>Asynchronous updates: Edge devices can unpredictably
lose connectivity (e.g. due to unstable wireless
connection or limited power). During such (unpredictably long)
outages, new features or security patches may be applied
to the software. As a result, a disconnected device may
possibly miss several rounds of updates and thus run an
obsolete version of software, thus introducing an
accidental complexity in the operation of the system.</p>
      <p>Synthetic diversity: vendors may use various techniques to
deliberately diversify their software for improving security
(i.e. implementing a ‘moving target’ for security attacks)
or fault-tolerance (i.e. recovering a failed system to a
different version not to suffer from the same fault again).</p>
      <p>Each of these aspects requires the IoT vendor to diversify
their IoT system by deploying and maintaining multiple
software versions for their fleets of IoT/Edge devices.
Furthermore, the vendor needs to match software
configurations/customisations with target systems, and push software
updates to relevant devices in a timely and guaranteed
manner. This increases the complexity of IoT operation,
challenging the agility, reliability and economy of scale.
Admittedly, addressing this challenge in the context of
largescale and highly-distributed IoT deployments goes beyond
manual capabilities of the IoT vendor and requires an
automated fleet management approach to enable remote,
secure, and reliable support for multiple operations, including
deployment, upgrade, monitoring, and troubleshooting of
software components.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Background and Related</title>
    </sec>
    <sec id="sec-4">
      <title>Work</title>
      <sec id="sec-4-1">
        <title>3.1. Fleet Management through Cloud Platforms</title>
        <p>Fleet management is a cross-cutting multi-faceted
concern for IoT vendors, especially when scaling from local
installations to global distributed ecosystems. In this paper,
we focus on the control part, i.e. the deployment and
upgrading of software components of the front-end applications
on the IoT/Edge devices. In this respect, we identify the
following key requirements for effective and efficient fleet
management:</p>
        <p>Agility: Software deployment or update of the entire fleet
must be done within an acceptable time frame. In the
context of widely adopted DevOps practices, when vendors
aim to continuously introduce new features or patch
vulnerabilities, such duration is measured with hours or even
minutes. Admittedly, achieving such short-term agility
implies having an automated solution able to remotely
access the managed system.</p>
        <p>Reliability: IoT vendors must be ensured at all times
that the designated software is eventually deployed to
target end users, even in the presence of unstable network
connectivity and varying physical conditions of the Edge
environment.</p>
        <p>Economy of scale: As the number of end users increases,
average operation costs per installation should decrease,
so that the vendor can benefit from the growth.</p>
        <p>
          These requirements are partially addressed by IoT cloud
platforms – an emerging family of cloud solutions, which,
apart from offering the traditional computing and storage
resources, also provide an IoT-specific management layer for
device monitoring, data flow design, data visualisation, etc.
These existing platforms [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ], [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ] enable IoT developers to
discover and integrate devices, monitor and diagnose system
operation, as well as to collect, process, store, and visualise
telemetry data. This typically assumes that Edge devices
are deployed and manually configured to push collected
data to a centralised back-end Cloud service in a ‘vertical’,
unidirectional manner [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ]. Information exchange in the
opposite direction (e.g. actuation commands, re-configuration,
software/firmware updates) is not straight-forward and easily
implemented due to network barriers, absence of static
        </p>
        <p>IP addresses, constrained hardware/software capabilities of
devices, limited connectivity, etc. In such circumstances,
agile software development for IoT/Edge devices in a truly
DevOps-compliant manner is hardly possible, as there is
an inevitable element of manual work required to deploy
software on a remote device.</p>
        <p>This situation is changing with the recent advances in the
networking and containerisation technologies. The former
enabled accessing and interacting with remote devices by
establishing virtual private networks, while the latter allowed
deploying and running light-weight and isolated software
components in a platform-agnostic manner. These advances
have also been underpinned by the continuously increasing
computing and networking capabilities of Edge/IoT devices.
As a result, existing IoT platforms are now able to extend
their functionality with tools for remote deployment of
software components (packaged as Docker containers) and
management of IoT devices.</p>
      </sec>
      <sec id="sec-4-2">
        <title>3.2. Software Diversity</title>
        <p>
          Traditionally, the scope of software diversity has been
limited to the two main fields – namely, cyber-security and
fault-tolerance [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ]. The former case relies on applying
various randomisation techniques at different system levels to
make software less vulnerable to generic threats by
becoming a ‘moving target’ for them. In the latter case, software
systems with diversified functions and elements are expected
to handle various failures either by completely avoiding
them, or by recovering an affected system to a different,
diversified version, thus lowering the risk of suffering from
the same fault again.
        </p>
        <p>
          Currently, the concept of software diversity appears as
a rich and manifold notion with multiple facets, such as the
goal of diversity, the diversification techniques, the scale of
diversity, the application domain, when it is applied, etc. [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ].
It is also common to distinguish between natural (emerging)
diversity, which appears spontaneously from the software
development process and run-time operation, and results in
different software versions, yet with similar functionality,
and artificial (synthetic or automated) diversity, which is
a result of explicit diversification actions taken by the IoT
vendor and applied to the system.
        </p>
      </sec>
      <sec id="sec-4-3">
        <title>3.3. Existing solutions and Related Work</title>
        <p>As of July 2019, there are at least three IoT cloud
platforms already offering support for container-based fleet
management as part of their portfolio. As a prerequisite,
these solutions require Edge devices to be pre-installed with
a software agent to communicate with the Cloud counterpart
and a container engine to run containerised software
components. Among the available alternatives, Azure IoT Hub1
appears as the most advanced option. On the one hand, it offers
a rich ecosystem of various tools and services through its
marketplace (e.g. users can benefit from an existing Docker
1https://azure.microsoft.com/services/iot-hub/
image repository, an automatic device provisioning service,
or a certificate-based authentication and access control),
and, unlike AWS IoT Greengrass,2 supports full-featured
containers, not just serverless functions. On the other hand,
it outperforms Balena Cloud,3 which also provides similar
full-featured support for container management, but is not
yet mature and developed enough to offer an extensive
collection of of tools and services via a marketplace.</p>
        <p>The community-driven Eclipse hawkBit4 is another
relevant framework aiming to automate IoT software updates
at scale. Unlike the proprietary Cloud-based solutions, it is
agnostic to specific underlying technologies and can be
integrated with many third-party components (albeit at the cost
of increased manual integration and configuration effort).</p>
        <p>
          Despite the advanced built-in functionality offered by
all these frameworks, they still lack another policy-driven
control layer that would enable flexible, fine-grained and
diversity-oriented software management and address the
previously outlined fleet management requirements. Potential
solutions, albeit not primarily tailored to the purposes of
IoT fleet management, already exist. More specifically, for
some years now, multiple tools have been available on
the market to support the deployment and configuration
of software systems, e.g. Puppet,5 Chef,6 and new tools
emerged for deployment of cloud-based systems such as
CloudMF [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ], OpenTOSCA [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ], and Brooklyn.7 In addition,
similar tools focus on the management and orchestration of
containers, such as Kubernetes.8 When dealing with
largescale systems, these approaches focus on scaling out, i.e.
duplicating identical components for load balancing.
        </p>
        <p>
          As far as IoT fleet management is concerned, a
systematic literature review of 17 prominent approaches for
orchestration and deployment for the IoT was conducted
in [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ]. All the surveyed approaches focus on the automatic
deployment of one or many IoT systems according to a
single deployment specification, without support for
managing a diverse fleet of IoT systems according to multiple
specifications.
        </p>
        <p>With the proposed approach, we aim to bridge this
identified gap by enhancing the existing Cloud-based tools
for fleet management with support for intelligent
diversityaware deployment and maintenance of software
components. To validate the proposed approach, we will take Azure
IoT Hub as the baseline, which currently appears to be the
most mature and feature-rich (yet open-source and free for
research purposes) offering. It is, nevertheless, expected that
the rest of the IoT cloud market will soon catch up with the
fleet management trend by offering similar container-based
functionality.</p>
        <p>2https://aws.amazon.com/greengrass/
3https://www.balena.io/cloud/
4https://www.eclipse.org/hawkbit/
5https://puppet.com/
6https://www.chef.io/chef/
7https://brooklyn.apache.org/
8https://kubernetes.io/</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>4. Proposed Approach</title>
      <p>In this paper, we propose a model-based approach to
diversity-oriented fleet management of IoT systems, and
implement a prototype fleet management tool named
DivEnact.9 We primarily focus on the connection between
IoT fleet management and software diversity, regardless of
whether it naturally emerges or is synthesised at design- and
development-time. Driven by a diversity-oriented model, the
proposed DivEnact solution is able to provide automatic
runtime management to deploy the diversified software onto
individual IoT/Edge devices.</p>
      <sec id="sec-5-1">
        <title>4.1. Conceptual Architecture</title>
        <p>Life-cycle Monitor collects reported information from
Edge devices (e.g. connection status, deployed
components, system up-time, etc.) and keeps track of the
fleet’s state.</p>
        <p>Deployment Manager decides what components
should be deployed on each Edge device, and enacts
this deployment on the device. It maintains a
repository of deployment specifications, each of which is
defined with a scope of applicable Edge devices. For
devices that fall into the scope of a specification, the
Deployment Manager will enforce the devices to run
the containers as specified, but sending corresponding
instructions. Docker container images are assumed to
be stored in a publicly available repository, such as
Docker Hub.10
3) The DivEnact component interacts with the Cloud Hub
through a REST API for querying and configuring Edge
device properties, and for manipulating deployment
specifications. Based on this API, the Diversity Controller
implements diversity-oriented fleet management at
runtime. It maintains a Diversity Pool – a collection of all
potential variants of the front-end application used to
generate deployment specifications for the Deployment
Manager. The Diversity Controller also maps deployment
specifications to specific devices by matching device
properties with specification scopes. These potential
diversity variants are either manually synthesised by the
vendor, or automatically generated at design-time. The
design-time diversity generation, at both code and
architecture levels, is included in the overall architecture,
but goes beyond the scope of this paper. Further details
on the model-based underpinnings of the Diversity
Controller are discussed below.</p>
      </sec>
      <sec id="sec-5-2">
        <title>4.2. Model-Based Diversity Engineering</title>
        <p>
          At the core of the proposed DivEnact approach is the
Diversity Controller, which maintains a run-time model of
the IoT fleet and a diversity configuration to be applied.
Following the models@runtime pattern [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ], the model is
dynamically synchronised with the running system, so that
the IoT fleet can be monitored and manipulated by reading
and writing the model, either manually or programmatically.
        </p>
        <p>Fig. 3 illustrates key concepts and relationships, which
constitute a meta-model for expressing a diversity-oriented
IoT fleet management scenario. Bold lines represent
relationships directly editable by the IoT vendor, while thin lines
indicate the relationships derived at run-time by the
Diversity controller. The IoT vendor can control the system by
adding/removing model elements, changing their attributes,
or editing relationships between them. The DivEnact
runtime engine will automatically maintain the derived
relationships based on the edits, and apply the changes to the
underlying system, i.e. the IoT fleet under management.</p>
        <p>The concepts that directly represent the IoT fleet are
Device and ModuleInstance. The former represents an</p>
        <p>Variable
name: string</p>
        <p>variable
Variation
value: string</p>
        <p>Boolean
Expression
and
or
not
template
Version
pre-condition</p>
        <p>derives
operand</p>
        <p>Condition
Device
Condition
deviceID: string</p>
        <p>Tag
Condition</p>
        <p>Module
image-url: string
deployedModule</p>
        <p>Device
appliesTo id: string
connection: string
tags</p>
        <p>Tag
Capability Environment
thperumemspsideuirrtaeyture pprreovdiuecwtion
Edge device in an IoT installation, and the latter represents
a software component running on the device as a Docker
container. A device is identified by its id and a unique
connection string used for registering to the Cloud Hub.
A device may be annotated by a Tag, which is a
userdefined value used to limit the target scope of a deployment
specification. For example, two sample Tag enumerations
shown in Fig. 3 are Capability and Environment, which
reflect hardware capabilities of a device and its application
environment, respectively. The relationship between devices
and modules is derived – i.e. the IoT vendor cannot directly
assign modules to an Edge device, but rather they will
be specified with an application scope in a deployment
specification.</p>
        <p>To express deployment specification, the IoT vendor
defines a set of DeploymentPlans, each of which contains
a set of Modules. Briefly, a Module serves to define how to
deploy a piece of software on a device. For example, as
depicted in Fig. 3, a Module may specify a URL of a container
image, as well as parameters to instantiate the container
from this image. If target conditions of a DeploymentPlan
are satisfied by a device, then each Module defined in this
plan will have a corresponding ModuleInstance deployed
on the device. Accordingly, the appliesTo relationship
between a DeploymentPlan and a Device will be derived
from the Conditions of the plan.</p>
        <p>A Condition is defined using either a device ID (i.e.
DeviceCondition) or Tags (i.e. TagCondition) attached
to devices. The former only applies to a device with this
specific ID, whereas the latter applies to multiple devices
annotated with this specific Tag. A composite condition can
include several expressions, connected by Boolean operators
and, or, and not, as illustrated by the code snippet below,
limiting the scope of a DeploymentPlan to devices in a
preview environment and either equipped with a temperature
sensor or having an empty ID (used for testing purposes).
tags.Environment == preview and
( tags.Capability == temperature or</p>
        <p>ID == EmptyForTesting )</p>
        <p>The IoT vendor can create a DeploymentPlan
manually, setting up its conditions and device tags to define
the scope of the deployment. However, when the size
of the fleet and the number of diverse deployment plans
increase, such manual control would be time-consuming
and error-prone. We introduce the diversity generation part
into the meta-model to handle the automated creation of
deployment specifications. The vendor can start by defining
a DeploymentTemplate, which aggregates a number of
DeploymentPlans, and defines a common part shared by
these plans. The varying parts of the plans are defined
as Variables in the template. From each template, the
vendor can define several Versions, each of which contains
Variations that assign concrete values to Variables. A
template is essentially a text document with a deployment
specification, expressed in YAML following the widely
adopted ‘Infrastructure as Code’ trend.</p>
      </sec>
      <sec id="sec-5-3">
        <title>4.3. Run-time Model Synchronisation</title>
        <p>Following the models@runtime pattern, the IoT vendor
is able to manipulate the fleet management model (e.g.
add new elements, change attributes, and edit relationships),
and the Diversity Controller will automatically apply these
changes to the running devices. Depending on the changes,
the synchronisation happens in three different levels.
1) In-model synchronisation, happening after changes on
parts of models for deployment templates and version,
performed by the diversity controller. The controller
collects the user-input changes and directly change other
parts of models to maintain the consistency.
2) Model-to-hub synchronisation, happening after changes
on the deployment plans and modules, performed also
by diversity controller. The controller collects the model
changes and transforms them into invocations to the
REST API provided by the IoT Hub, in order to update
the deployment plans maintained by the hub.
3) Hub-to-device synchronisation, happening after changes
on deployment plan and device, performed by the IoT
Hub. Such synchronisation is triggered automatically
after the API invocations to the IoT Hub, and will
eventually result in containers deployed on the devices.
Typically, the synchronisation happens as a sequence in the
following three steps, corresponding to the three levels.
1) Creation: this step takes place when the vendor creates
a deployment plan from one of the templates from the
Diversity Pool. Instead of manually defining modules
for a deployment plan, the vendor can instead create an
empty plan and link it to a version. The synchronisation
engine will load the template used by this version, and
resolve template variables to generate module definitions.
These modules will then be inserted into the deployment
plan. If the template contains pre-conditions, the engine
will also pass them into the deployment plan.
2) Instantiation: when these model elements are created,
removed, or edited, the Diversity Controller synchronises
the changes with the Deployment Manager by invoking
the API provided by the Cloud Hub. The Diversity
Controller will first invoke the API to create a deployment
plan with the same ID in the Hub, and use a series
of subsequent API invocations to create the modules
and insert them into the newly created deployment plan.
Similarly, when the vendor tags a device, the engine will
get its ID call the corresponding API method passing the
tag value and the device ID.
3) Enactment: we reuse the management features provided
by the Deployment Manager to enact the changes on
Edge devices. When a deployment plan is created or
updated, for each device falling into the scope, the
Deployment Manager will evaluate if the desired modules (the
ones defined in the deployment plan) are already running
on the device by querying the Life-cycle Monitor. If
not, it will instruct the device to download and deploy
the desired module. In case a device is not currently
connected to the Hub, the pending deployment action
will be triggered immediately after the device appears
online again.
5. Proof of Concept11</p>
        <p>The proof of concept is currently implemented on top
of Azure IoT Hub, which offers wide integration
opportunities and rich built-in functionality to be re-used. Azure
IoT Hub already provides its built-in Life-cycle Manager,
Deployment Manager, and a software agent to be installed
on Edge devices, which exempts us from ‘re-inventing the
wheel’ and focus on implementing the required DivEnact
functionality.</p>
      </sec>
      <sec id="sec-5-4">
        <title>5.1. Experimental Setup</title>
        <p>We now re-visit the reference RPM scenario to
demonstrate the viability of the proposed approach. This simplified
demonstration uses three Raspberry Pi boards acting as
Edge devices and three SenseHat12 shields acting as
sensorenabled IoT devices. Each SenseHat shield is equipped
with a LED matrix and three sensors (humidity,
temperature, and pressure) that can be used to collect information
about the surrounding environmental conditions. Table 5.1
summarises the available capabilities of each Raspberry Pi
board. The front-end application deployed on Raspberry Pi
boards can interact with the SenseHat shields to light up
the LED matrix, collect sensor data and transfer them to the
back-end application, which is assumed to evaluate whether
current conditions in a patient’s house are acceptable.</p>
        <p>The fleet of these three Edge devices and a sample
deployment plan are modelled in Fig. 4. Each Edge device
is running in a production environment and is tagged with
its capabilities (as summarised in Table 5.1). The model also
contains a deployment plan with two modules (i.e. Docker
11Source code related to this implementation can be found in
https://github.com/SINTEF-9012/divenact
12https://www.raspberrypi.org/products/sense-hat/
images) led-matrix and temp-sensor that contain
corresponding application logic, while the latter is also able to
set up the sampling frequency of a temperature sensor. The
plan is generated from the following template:
templates:
divenact-template:
temp-sensor:
settings:
image: songhui/temp-sensor:{{FREQUENCY}}
createOptions:
"{"HostConfig":</p>
        <p>{"Privileged": true }}"
type: docker
led-matrix:
settings:
image: songhui/led-matrix
createOptions:
"{"HostConfig":</p>
        <p>{"Privileged": true }}"
type: docker
pre-condition: tags.Capability=temperature
and tags.Capability=led-matrix
and tags.Environment=production
versions:
frequency-1:
template: divenact-template
parameter:</p>
        <p>FREQUENCY: 1
frequency-10:
template: divenact-template
parameter:</p>
        <p>FREQUENCY: 10</p>
        <p>The code snippet above shows a sample deployment
template with two modules named temp-sensor
and led-matrix, based on corresponding Docker
images.13 The temp-sensor image is registered
13Please note the configuration parameter Privileged set to true,
which is required to enable communication with a SenseHat shield through
low-level GPIO pins.
with two different versions frequency-1 and
frequency-10 with frequencies of 1 and 10 readings
per second respectively, which can be retrieved by
specifying the variable FREQUENCY. The template also
has a composite pre-condition tags.Capability=
temperature and tags.Capability=led-matrix
and tags.Environment=production. As a result, the
generated deployment plan only applies to RPi1 and RPi2,
but not RPi3 (which is not equipped with a temperature
sensor). As a result, the module defined in the deployment
plan will only be instantiated on RPi1 and RPi2.</p>
      </sec>
      <sec id="sec-5-5">
        <title>5.2. Sample Diversity Use Cases</title>
        <p>We now demonstrate how the Diversity Controller is able
to support some of the diversity use cases outlined in Section
2. These diversity use cases are intended to demonstrate how
the model-based approach simplifies automated IoT fleet
management. Each use case is implemented as a simple, yet
efficient script, which manipulates the model accordingly.
5.2.1. Pushing a new version to production. The IoT
vendor has released a new version frequency-100, and
wants it to be the only production version for all Edge
devices equipped with temperature sensors. DivEnact can
implement this as the following script:
for d in Device:</p>
        <p>d.tags.Environment = production
p = DeploymentPlan("production")
p.version = frequency-100
p.condition = "tags.Environment==production"
and p.version.template.precondition</p>
        <p>Briefly, the script sets all the devices into production
environment, and then create a deployment plan p from the
version frequency-100. The condition of p is a two-fold:
i) the plan works for all devices tagged with the production
environment, and ii) the plan inherits a pre-condition from
the template, which limits its scope to devices equipped with
a temperature sensor.
5.2.2. Previewing a staging version. The IoT vendor has
developed a new version frequency-100, but before
releasing it to production wants to test it on a limited number
of selected users, while the rest of the users will still run
the current production version. Extending the previous use
case’s model, the following script will accomplish this:
frequency-100 = Version(’frequency-100’)
temp-devices = [d for d in Devices</p>
        <p>if eval(frequency-100.template.precondition, d)]
for d in shuffle(temp-devices)[0:2]:</p>
        <p>d.tags.Environment = preview
p = DeploymentPlan(’preview’)
p.version = frequency-100
p.condition = ’tags.Environment==preview’
and p.version.template.precondition</p>
        <p>The script first finds all devices that satisfy the
precondition of the version frequency-100, i.e. the devices
that are tagged with Capability=temperature. After
that, it randomly picks 2 devices and tags them with the
preview environment. Finally, it creates a new deployment
plan from this version, similar to the previous use case.
5.2.3. Shuffling versions. Given the three available
versions for the temperature sensor module (frequency-1,
frequency-10, and frequency-100) and one version for
the LED matrix (led-matrix), the IoT vendor wants to
artificially synthesise diversity for increased security and
faulttolerance by deploying several diversified, but functionally
equivalent versions onto the fleet.
group1 = [frequency-1, frequency-10, frequency-100]
group2 = [led-matrix]
v2c = {}
for group in [group1, group2]:
devices = [d for d in Devices if</p>
        <p>eval(group[0].template.precondition, d)]
index = 0
for d in shuffle(devices):
v = group[index % len(group)]
v2c[v] = "id = " + d.id
for v in group1 + group2:
p = DeploymentPlan(v.id)
p.version = v
p.condition = "or". join(v2c[v])</p>
        <p>The script divides the available versions into two groups
according to their relation to the temperature sensor or the
LED matrix, respectively. For each group, it finds all devices
that satisfy the common pre-condition of this group. Next,
for each device it assigns a version from this group in
a circular order, and generates a device-based condition.
Finally, it creates a deployment plan for each version, and
the condition of this plan is a conjunction of all device-based
conditions that we created for the corresponding version.</p>
        <p>In all the experiments in the current implementation, the
time frame between the moment when a deployment plan
is first generated and the moment when an Edge device
starts running new software ranges from 30 seconds to two
minutes. This primarily depends on the size of a Docker
image to be downloaded and launched, which remained
relatively small given the simplified setup of the experiments.
In practise, downloadable software components are expected
to be somewhat larger. Nevertheless, we still expect the
operation of the proposed system fulfil the agile requirements
of DevOps practices.</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>6. Conclusion and Future Work</title>
      <p>With this paper, we addressed the automated,
diversityoriented fleet management in large-scale IoT deployments
through a model-based approach. Based on a software
diversity model, the proposed approach allows dynamically
managing containerised software components on Edge devices in
IoT systems ranging from local deployments to large-scale,
geographically distributed ecosystems. Characterised by an
increasing degree of software diversity, such systems are
required to be managed in a reliable and automated manner
so as to achieve economies of scale. As a first step towards
the validation of the approach, the paper described a simple,
yet realistic smart IoT scenario, where software components
were pushed to Edge devices based on a diversity-oriented
model continuously synchronised at run-time.</p>
      <p>
        The proposed approach assumes that target Edge
devices are sufficiently capable to run a container engine and
communicate with the Cloud. However, another common
source of diversity comes from ‘leaf’ IoT devices, which
are typically equipped with micro-controllers, run simple
firmware written in C, and only communicate with a
central Edge hub using a wireless channel. Admittedly, there
are many situations, when IoT vendors might also require
remotely updating such device firmware in an agile manner.
In this light, a primary direction for future work to extend
the current approach with our IoT deployment tool named
GeneSIS [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ], which supports the ‘last-mile deployment’
from Edge devices to IoT devices. The idea is to deploy a
GeneSIS engine as a container on an Edge device; the engine
will then interact with the coupled IoT devices and push
required updates using available communication channels. By
extending the proposed meta-model with relevant concepts,
it will be possible to model IoT devices and diversification
and apply these modifications to the running system. As
a result, we expect to provide a holistic fleet management
solution covering both Edge and IoT devices.
      </p>
    </sec>
    <sec id="sec-7">
      <title>Acknowledgments</title>
      <p>This work is supported by the H2020 programme under
the grant agreement #780351 (ENACT).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>R.</given-names>
            <surname>Dautov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Distefano</surname>
          </string-name>
          , and
          <string-name>
            <given-names>R.</given-names>
            <surname>Buyya</surname>
          </string-name>
          , “
          <article-title>Hierarchical data fusion for Smart Healthcare</article-title>
          ,
          <source>” Journal of Big Data</source>
          , vol.
          <volume>6</volume>
          , no.
          <issue>1</issue>
          , p.
          <fpage>19</fpage>
          ,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>B.</given-names>
            <surname>Baudry</surname>
          </string-name>
          and
          <string-name>
            <given-names>M.</given-names>
            <surname>Monperrus</surname>
          </string-name>
          , “
          <article-title>The multiple facets of software diversity: Recent developments in year 2000 and beyond</article-title>
          ,
          <source>” ACM Computing Surveys (CSUR)</source>
          , vol.
          <volume>48</volume>
          , no.
          <issue>1</issue>
          , p.
          <fpage>16</fpage>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>P. P.</given-names>
            <surname>Ray</surname>
          </string-name>
          , “
          <article-title>A survey of IoT cloud platforms</article-title>
          ,
          <source>” Future Computing and Informatics Journal</source>
          , vol.
          <volume>1</volume>
          , no.
          <issue>1-2</issue>
          , pp.
          <fpage>35</fpage>
          -
          <lpage>46</lpage>
          ,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>T.</given-names>
            <surname>Pflanzner</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Kerte</surname>
          </string-name>
          <article-title>´sz, “A survey of IoT cloud providers,” in 2016 39th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)</article-title>
          . IEEE,
          <year>2016</year>
          , pp.
          <fpage>730</fpage>
          -
          <lpage>735</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>R.</given-names>
            <surname>Dautov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Distefano</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Bruneo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Longo</surname>
          </string-name>
          ,
          <string-name>
            <surname>G.</surname>
          </string-name>
          <article-title>Merlino, and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Puliafito</surname>
          </string-name>
          , “
          <article-title>Pushing Intelligence to the Edge with a Stream Processing Architecture</article-title>
          ,” in
          <source>2017 IEEE International Conference on Internet of Things (iThings)</source>
          . IEEE,
          <year>2017</year>
          , pp.
          <fpage>792</fpage>
          -
          <lpage>799</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>N.</given-names>
            <surname>Ferry</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Chauvel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Song</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Rossini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Lushpenko</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Solberg</surname>
          </string-name>
          , “
          <article-title>CloudMF: Model-Driven Management of Multi-Cloud Applications,”</article-title>
          <source>ACM Transactions on Internet Technology</source>
          , vol.
          <volume>18</volume>
          , no.
          <issue>2</issue>
          , p.
          <fpage>16</fpage>
          ,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>A. C. F.</given-names>
            da
            <surname>Silva</surname>
          </string-name>
          , U. Breitenbu¨cher, K. Ke´pes,
          <string-name>
            <given-names>O.</given-names>
            <surname>Kopp</surname>
          </string-name>
          , and
          <string-name>
            <given-names>F.</given-names>
            <surname>Leymann</surname>
          </string-name>
          , “
          <article-title>OpenTOSCA for IoT: Automating the Deployment of IoT Applications Based on the Mosquitto Message Broker</article-title>
          ,”
          <source>in Proceedings of the 6th International Conference on the Internet of Things. ACM</source>
          ,
          <year>2016</year>
          , pp.
          <fpage>181</fpage>
          -
          <lpage>182</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>P. H.</given-names>
            <surname>Nguyen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Ferry</surname>
          </string-name>
          , G. Erdogan,
          <string-name>
            <given-names>H.</given-names>
            <surname>Song</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Lavirotte</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.-Y.</given-names>
            <surname>Tigli</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Solberg</surname>
          </string-name>
          , “
          <article-title>Advances in deployment and orchestration approaches for iot - a systematic review,” in IEEE International Congress On Internet of Things (ICIOT)</article-title>
          . IEEE,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>N.</given-names>
            <surname>Bencomo</surname>
          </string-name>
          , R. B. France,
          <string-name>
            <given-names>B. H.</given-names>
            <surname>Cheng</surname>
          </string-name>
          , and U. Aßmann, Models@run.time: Foundations, Applications, and Roadmaps. Springer,
          <year>2014</year>
          , vol.
          <volume>8378</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>N.</given-names>
            <surname>Ferry</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Nguyen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Song</surname>
          </string-name>
          , P.-E. Novac,
          <string-name>
            <given-names>S.</given-names>
            <surname>Lavirotte</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.-Y.</given-names>
            <surname>Tigli</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Solberg</surname>
          </string-name>
          , “
          <article-title>GeneSIS: Continuous Orchestration and Deployment of Smart IoT Systems</article-title>
          ,” in
          <source>2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC)</source>
          ,
          <source>vol. 1</source>
          . IEEE,
          <year>2019</year>
          , pp.
          <fpage>870</fpage>
          -
          <lpage>875</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>