Challenges of DevOps ready IoT Testbed 1st Janis Judvaitis 2nd Krisjanis Nesenbergs Institute of Electronics and Computer Science Institute of Electronics and Computer Science Riga, Latvia Riga, Latvia janis.judvaitis@edi.lv krisjanis.nesenbergs@edi.lv 3rd Rihards Balass 4th Modris Greitans Institute of Electronics and Computer Science Institute of Electronics and Computer Science Riga, Latvia Riga, Latvia rihards.balass@edi.lv modris.greitans@edi.lv Abstract—Developing, deploying, testing and validating com- plex software systems is complicated and as a result security, privacy and quality in general often suffer due to limited resources and rapid development cycles. In the case of IoT system development there are additional levels of complexity and risks. In this paper we present our ongoing effort towards providing and validating a solution for these problems as a toolset/TestBed implementing previously established ENACT IoT DevOps concepts and Framework, aimed at ensuring continued Quality of Service and application of best practices during development cycle of IoT systems. Index Terms—IoT TestBed; DevOps; Secure IoT; Fig. 1. Product life-cycle stages3 I. I NTRODUCTION Forces behind the rapid growth of the Internet-of-Things (IoT) market both in number of active devices1 and market facilitate creation and operation of trustworthy Smart IoT Sys- value2 are at direct odds with security and privacy of said tems [3]. This framework defines related research challenges, devices and their users. Companies rushing to carve out their and proposes solutions in form of novel IoT platform enablers own IoT market share are faced with the complex, dynamic bundled in several toolkits for the whole Life-Cycle of Smart and challenging realities of developing, testing and debugging IoT System development (see Section II). these devices [1]. In this paper we describe our work towards one part of this In order to counteract potential lack of such expensive framework - an IoT development TestBed implementing the development aspects as security, privacy and quality in general EDO approach - from the scope and requirements to practical driven by these market forces an appropriate toolset is required concerns and challenges. for implementing best practices in development and related The rest of the paper is structured as follows: Section II operations (DevOps). Such software engineering best practices describes the Preliminary work and State of the Art, including and tools to ensure Quality of Service and ease of use while general EDO Framework concepts and previous attempts at allowing continuous evolution of complex systems in an agile TestBed development by our team, Section III reveals specific fashion with rapid innovation cycles are at the heart of the DevOps TestBed related requirements, and Section IV con- DevOps movement [2], and have a stable place in classic cludes with future challenges and conclusions of the ongoing software development. work to implement this DevOps enabling IoT TestBed. As such a solution for IoT was previously unavailable, ENACT DevOps (EDO) concept and Framework was born to II. P RELIMINARY WORK AND BACKGROUND An in-depth description of the research challenges in IoT The research leading to these results has received funding from the Euro- and how a specifically designed IoT DevOps framework might pean Commissions H2020 Programme, grant agreement no. 780351 (ENACT) Copyright ©2019 for this paper by its authors. Use permitted under Creative be beneficial has been previously published by Ferry et al. Commons License Attribution 4.0 International (CC BY 4.0). [3] presenting EDO Framework for development, operation 1 7B devices mid 2018, 10B projected by 2020 - https://iot- and quality assurance of trustworthy Smart IoT Systems. A analytics.com/product/state-of-the-iot-2018/ 2 235B USD in 2017, 520B USD projected by 2021 - brief overview of the main points in this work follows, laying https://www.bain.com/insights/unlocking-opportunities-in-the-internet-of- things/ 3 http://www.bestdevops.com/has-devops-changed-the-role-of-a-tester/ 3 ground for current ENACT project, followed by our previous IoT TestBed development work for context. A. ENACT DevOps (EDO) framework DevOps as a concept is is well established as an approach to ensure a rapid and efficient value delivery to market through tight collaboration between the developers (Dev) and the teams that deploy and operate the software systems (Ops), in essence to decrease the gap between product design and its operation by automation and continuous processes, supported by different tools at various stages of the product life-cycle. In the EDO Framework a next step is taken to formalize this process for application in trustworthy Smart IoT Systems in spite of such IoT research challenges identified by Ferry et al. as: (i) Support for Continuous Delivery of trustworthy smart Fig. 2. EDI TestBed hardware architecture IoT systems; (ii) support for Agile Operation of trustworthy smart IoT systems; (iii) support for Continuous Quality As- surance strengthening trustworthiness of Smart IoT Systems; and (iv) leveraging capabilities of existing IoT platforms and fully exploiting legacy, proprietary and off-the-shelf software components and devices. Afterwards they present the EDO enablers relating to 8 stages of IoT development life-cycle as shown in Figure 1 and separated into three toolkits as follows: ENACT Continuous Delivery Toolkit for agile and contin- uous evolution of IoT systems with early detection of issues in development process, consisting of (i) Orchestration and Fig. 3. EDI TestBed software architecture Continuous Deployment Enabler and (ii) Test, Emulation and Simulation Enabler. ENACT Agile Operation Toolkit for ensuring automated B. EDI TestBed operation of the developed systems and ensuring their trust- EDI TestBed [6] is a set of debugging tools that help worthiness during operation, consisting of (i) Context-Aware increase the development speed of IoT and WSN systems. Self-adaptation Enabler, (ii) Root Cause Analysis Enabler and The TestBed is comprised of 100+ nodes that are distributed (iii) Context Monitoring and Actuation Conflict Management around a 7 floor building for validation and research in sensor Enabler. network and wireless network protocols. The current hardware ENACT Trustworthiness Toolkit for crosscutting trustwor- and software architectures of EDI TestBed can be seen in thiness concerns of IoT systems (e.g. robustness, security and Figures 2 and 3. privacy), consisting of (i) Robustness and Resilience Enabler, Currently the EDI TestBed has no tools assisting with (ii) Risk Management Enabler, and (iii) Security and Privacy DevOps integration and there is only very basic support Monitoring and Control Enabler. functionality required from a DevOps ready TestBed: (i) These enablers are designed to be loosely coupled and in support for heterogeneous IoT network, (ii) basic logging this paper we describe ongoing ENACT project work on pro- functionality, (iii) remote device access for reprogramming viding a subset of these enablers to be integrated with existing and bi-directional serial communication(UART), (iv) remote IoT platforms in the form of a DevOps ready IoT TestBed. power supply control for controlled power consumption mea- Specifically, existing TestBed is enriched and applied in use surements and emulation of supply voltage changes, (v) analog case for Intelligent Transport Systems, to assess the feasibility interface simulation and emulation and (vi) mobile TestBed of IoT services in the domain of train integrity control for the workstations. logistics and maintenance of the rolling stock and on-track To develop a DevOps ready IoT testbed on this basis, a set equipment.4 The first iteration of on-board IoT system for train of requirements are defined in the next section. integrity control [4] was developed and demonstrated using EDI TestBed in DEWI project [5]. Afterwards, in ENACT III. D EVO PS T EST B ED REQUIREMENTS project, a rework of DEWI ITS on-board IoT system was done A. Testbed role in EDO framework introducing on-track IoT system satisfying EDO Framework needs. Although an IoT testbed as such could potentially provide a wide range of services and features, the emphasis must 4 https://www.enact-project.eu/ucs.php be on the tools provided to support efficient development 4 and operations of trustworthy smart IoT systems. Thus, the components must be transparently integrated in the system envisioned DevOps ready IoT development TestBed is aimed allowing like-a-local work flow with hardware; to implement the first of three EDO toolkits described in Available remote micro controller debugging capabilities Section II-A - the ENACT Continuous Delivery Toolkit. This - development and testing requires low level debugging facili- means, that it must be an enabling technology in orchestration ties, which are often unavailable in remote abstracted systems. and continuous deployment as well as testing, emulation and A quality DevOps enabled Testbed needs such capabilities to simulation of smart IoT systems during their development. support agile and continuous evolution and to make it easy to identify the source of many problems. B. Continuous delivery toolkit related requirements (iii) Easy integration with existing tool chains - in the Even though there are requirements related to specific most basic abstraction TestBed should be one of many tools enablers within the toolkit described in the next subsections, used in trustworthy smart IoT system DevOps life cycle, so there are some non-enabler-specific requirements as well, accordingly it should provide the users with interface of a specifically: basic tool set, which can be easily integrated in any existing (i) Toolkit modularity, supported by a robust messaging work flow as necessary, including: backbone and API - as all of the ENACT toolkits must Well rounded command line interface (CLI) - interactive contain loosely coupled enablers, the related DevOps services web based interface of current TestBed version is not easily provided by the TestBed must also be modular, interchange- integrated into existing development pipelines, thus users able and easy to extend in the future. Specifically: cannot deploy their code to IoT nodes, debug, test, validate and Message brokering system - the current TestBed infras- script them in continuous manner using classic DevOps tools tructure is a monolith system with multiple task specific and their robust CLI interfaces minimizing the gap between communication channels, leading to DevOps problems with trustworthy IoT and traditional system development life cycles. the Testbed infrastructure itself and limiting the rate at which Thus high level integrated development environments (IDE) new needed modules can be developed. A messaging system could interact with planned TestBed CLI tools via plugins for and broker can take care of this decoupling, and provide complete IoT smart system life cycle integration; an easy plug-and-play ability to add such critical modules Seamless remote back-end tools - to make the EDI TestBed as complete system state logging, target IoT device output more DevOps friendly a background daemon must be in- logging as well as promote easy scaling of the TestBed system troduced, which provides full EDI TestBed functionality to for use in development and testing of larger IoT networks; local machine or network for use on actual work space, Standardized low level API - the messaging system on supporting not only CLI, but also local-like interaction with its own can manage passing of required messages between the remote hardware, e.g. reverse control through GPIO for the modular components of the TestBed, such as hardware event detection and precise timing, ground truth data such as testing endpoints, logging servers or user stations, but the precise time synchronization, possible radio channel routes, types of messages must still be declared in a strong API, so and logic analyzer functionality for advanced IoT device that introduction of new modules does not require a complete behaviour analysis and debugging, protocol decoders for most rework of the existing communications protocols. popular protocols used in IoT (SPI, I2C UART). (ii) Hardware invariant abstraction, supporting hetero- geneity - the final toolkit needs to support development and C. Orchestration and Continuous Deployment enabler related testing of complex networks consisting of different types of requirements node hardware, as well as different types of TestBed endpoint TestBed should conform to several requirements related hardware (e.g. static endpoint, mobile endpoint, endpoint with to facilitating engineering and continuous development in a software defined radio, endpoint with extremely precise energy decentralized way: measurement capabilities etc.) and for the DevOps processes Remote simultaneous reprogramming of target IoT to be usable, the end user must not be burdened with the devices - the most basic requirement of a TestBed enabling technical differences in this hardware, but instead should work continuous delivery is to allow centralized mass control and with standardized abstractions. Specifically: reprogramming of the target IoT network. This must be TestBed architecture with heterogeneity built in from the supported by a robust API as mentioned above. start - all other requirements must be considered in the lens Automated hardware unit testing support - for early of heterogeneity with reliable and expandable detection of detection of issues in software development process, easy devices, API with gracious fall-backs in case of unavailable testing is required - ability to run tests on end nodes each time specific hardware features etc., leading to capability to encour- they are reprogrammed, comparing the results to some baseline age development of more complex heterogeneous trustworthy and reporting errors/inconsistencies together with relevant data IoT networks; about this specific node. Seamless bi-directional serial communications (UART) Precise power measurements - as IoT devices use battery support - UART protocol is ubiquitous in IoT devices and power it is critical to develop them with optimal power should be a first-class citizen in the TestBed hardware environ- consumption in mind. To develop and continually improve ment and UART messaging to and form all TestBed hardware the power consumption of the system under development, 5 precise power consumption data must be available in real- time resulting in ability to differentiate between the costs of ”smart” and ”trustworthy” in terms of power consumption to make better trade-off decisions. D. Test, Emulation and Simulation enabler related require- ments To assess the behaviour and trustworthiness of the system during its life-cycle TestBed must provide: System testing support - like with hardware unit testing in the case of continuous deployment, automated system tests should also be supported with plug-in type smart test result analysis highlighting the differences between different runs of tests or a specified baseline. Fig. 4. EDI TestBed next version architecture blocks Testing and verification continuum - to ease the gradual migration from test to operational environment the TestBed • Automated hardware unit testing, including (i) automated should provide the functionality to seamlessly transfer IoT system tests on specific events like reprogramming and devices from local tests/validation to target location using (ii) smart test result analysis. mobile TestBed workstations using identical work flow on • Analog and digital interface simulation, including (i) au- local nodes and remote nodes. tomated basic security attacks and (ii) simulated possible Real-Virtual radio interface - for performance assessment security threat actors. while the system is (a) not complete or (b) completely located • Easy local interaction with remote TestBed hardware, in TestBed, special endpoints (workstations) with software including (i) reverse control through GPIO, (ii) ground defined radio (SDR) [7] should be added to TestBed, providing truth data, (iii) direct UART link and (iv) logic analyzer, radio connectivity (a) between real and simulated nodes and with these functions: (i) automated protocol decoding and (b) between nodes located in different physical locations (e.g. (ii) precise event timing between different workstations. local TestBed and forest etc.). This enables decentralized Even though much of the TestBed hardware can be reused, processing through SDR channel allowing part of unoptimized the software part must be completely rebuilt - the planned algorithms to be run on servers while developing IoT devices. system schematic can be seen in figure 4. The work of Precise power control - to test IoT systems in close to real implementing this TestBed as an ENACT Continuous Delivery physical environment TestBed should provide functionality of Toolkit is currently underway and the applicability of the power supply control simulating battery discharge or power result to enabling DevOps for trustworthy smart IoT system drop due to ambient temperature changes etc. development will be evaluated using ENACT ITS use case and Automated recorded, simulated and physical security expanded as necessary based on these results. tests - IoT system trustworthiness testing through automated basic security attacks, simulated threat actors and physical R EFERENCES access attacks (both analog and digital) should be provided for [1] A. Taivalsaari and T. Mikkonen, “A roadmap to the programmable world: advanced security analysis of IoT device as a separate entity. software challenges in the iot era,” IEEE Software, vol. 34, no. 1, pp. 72– 80, 2017. IV. C ONCLUSION [2] C. Ebert, G. Gallardo, J. Hernantes, and N. Serrano, “Devops,” IEEE Software, vol. 33, no. 3, pp. 94–100, 2016. In this paper we have provided a road map and related [3] N. Ferry, A. Solberg, H. Song, S. Lavirotte, J.-Y. Tigli, T. Winter, practical concerns and challenges in development of a DevOps V. Muntés-Mulero, A. Metzger, E. R. Velasco, and A. C. Aguirre, “Enact: Development, operation, and quality assurance of trustworthy smart iot ready IoT testbed, that conforms to EDO approach using the systems,” in International Workshop on Software Engineering Aspects of existing EDI TestBed as baseline. Continuous Development and New Paradigms of Software Production and Summary of features to be added include: Deployment. Springer, 2018, pp. 112–127. [4] N. Barkovskis, A. Salmins, K. Ozols, M. A. M. Garcı́a, and F. P. • Robust messaging backbone (using messaging broker like Ayuso, “Wsn based on accelerometer, gps and rssi measurements for train MQTT); integrity monitoring,” in 2017 4th International Conference on Control, Decision and Information Technologies (CoDIT). IEEE, 2017, pp. 0662– • Standardized TestBed low-level API providing seamless 0667. access to back-end tools; [5] W. Rom, P. Priller, J. Koivusaari, M. Komi, R. Robles, L. Dominguez, • Well rounded CLI for integration in existing tool-chains; J. Rivilla, and W. Van Driel, “Dewi–wirelessly into the future,” in 2015 Euromicro Conference on Digital System Design. IEEE, 2015, pp. 730– • Complete system state and output logging through plug- 739. ins; [6] R. Ruskuls, D. Lapsa, and L. Selavo, “Edi wsn testbed: Multifunctional, • Remote micro controller debugging (GDB like) function- 3d wireless sensor network testbed,” in Advances in Wireless and Optical Communications (RTUWO), 2015. IEEE, 2015, pp. 50–53. ality; [7] F. K. Jondral, “Software-defined radio: basics and evolution to cognitive • Hardware update with SDR nodes and related software radio,” EURASIP journal on wireless communications and networking, for real to virtual communication; vol. 2005, no. 3, pp. 275–283, 2005. 6