Graceful degradation is an established concept to improve the resilience of systems, especially when other resilience mechanisms have failed. Its implementation is often heavily tied to the application code and, thus, cumbersome and error prone. As IoT systems get not only ubiquitous but also critical, reliable graceful degradation would be ideal. In this paper, we present the Morpheus framework that provides a TypeScript-internal DSL to enable a systematic development of degradable IoT systems. The design of the framework is based on the concept of separation of concerns by providing distinct yet linked languages to specify hierarchical components and their connections; the components' operating modes and transfer functions between them; as well as state machines for the specification of the components' behaviour in each operating mode. The operating modes for each component serve as degradation levels. Automatic degradation of a component is triggered in case of failures of connected components. With recovery from underlying failures, the component is automatically upgraded back to a higher level. We illustrate our framework using a simplified prototype of an entrance barrier of a parking garage.
The Internet of Things (IoT) has made huge progress from a generic vision to concrete realisations. IoT systems are becoming more and more ubiquitous and pervasive. Simultaneously, our daily lives depend even more on them, e.g., smart homes, smart cities as well as smart transportation, delivery, and manufacturing processes. By their nature, IoT systems can directly influence the real world and as a consequence their failures can cause harm and even fatalities. Thus, it is important that they are highly resilient and particularly, never enter undefined states.
While IoT systems are conceptually close to distributed applications that have been planned, implemented, and operated for decades, significant diferences exist: (i) IoT systems span nEvelop-O large geographical regions; (ii) they consist of a large number of heterogeneous, even batterypowered, devices with diferent capabilities, network bandwidths, and failure characteristics; (iii) depending on the region, replacing failed devices and patching broken software may be a matter of days and weeks instead of minutes and hours as with Web-like distributed systems.
Graceful degradation is a known approach to reduce a system’s capabilities in a controlled
manner [
To overcome these limitations, we propose Morpheus, a degradation framework for IoT
systems as extension of our previous work [
Section 2 discusses related work, while Section 3 introduces previous work and a running example used throughout the text. Section 4 introduces Morpheus’ concepts and illustrates their use via the running example. Section 5 concludes and presents future work.
With graceful degradation, an IoT system is able to sustain some functionality when faults can
no longer be tolerated by other resilience mechanisms [
Graceful degradation approaches are well known for design and operation of distributed
embedded systems [
A degraded system may automatically switch back to the originally desired service level,
or at least to a better level, if the cause of the degradation has disappeared or changed. This
graceful upgrade is addressed in only few related works [
Degradation is supported by various modelling approaches such as the Architecture Analysis
& Design Language (AADL) and its error annex [
Morpheus builds upon our previous work—an internal Domain Specific Language [
Each atomic component contains internal state consisting of a discrete finite state as in
automata and local data structures (similar to the state in abstract state machines [
Our experience as presented in our previous paper [
In order to demonstrate the capabilities of Morpheus, we implemented a physical prototype1 of a Smart Entrance Barrier (SEB) that could be deployed in a parking garage. Customers of the garage reserve a parking spot through an online reservation system using credit card details and license plate number. The SEB automatically recognises customers on entrance and exit. Our SEB prototype is composed of multiple dependent sub-components: a barrier unit B U , a barrier sensor B S , a car sensor C S , a card reader C R , and a camera C A M . Further, it uses two external software components: a plate recognition service P R S and a parking management system P M S .
When a car approaches the S E B , the car sensor detects its presence and the camera scans the license plate. The image is then sent to the P R S , which extracts the license plate number in text format. This information is used to query the P M S , and if a valid reservation is found, the S E B will open. In this seamless authentication process with multiple interacting components, a faulty component shall not prevent a car from passing the barrier. The prototype is equipped with 1The prototype uses an ESP8266-based microcontroller and a Raspberry Pi Zero-based camera OctoCam. degradation functionality: Morpheus adjusts the S E B ’s functionality based on the availability of internal and external components as detailed in Section 4.
PRS(:PEyxthtoenrn-baalCseodmCpVo)nent camera: ExternalComponent barrierUnit: ExternalComponent
barrierMCU barrierSensor: ExternalComponent barrierMCU cardReader: ExternalComponent barrierMCU carSensor: ExternalComponent
barrierMCU smartEntranceBarrier: Component parkManagementSystem: Component
Connections
MQTT REST (internal) REST (external)
In Morpheus, an operating mode defines a component’s functionality. By default, each component implements the operating mode O f f , which specifies that the component is currently unable to provide any reasonable functionality. This operating mode is entered if essential sub-components are unreachable or have failed, or the component itself sufers from internal faults. It is required that each component is equipped with at least one additional non-trivial operating mode, which implements the component’s intended feature set. Developers can now define further operating modes that provide a graduated subset of the component’s capabilities. In order to ensure that the specified functionality can actually be provided, it has to be defined which dependant sub-components have to be available.
Our example uses the following operating modes for the S E B : • O f f : The S E B does not provide any reasonable functionality. • B l o c k e d : The S E B is locked, no car can enter or leave, e.g., to lock the garage. • O p e n : The S E B is open and allows entrance and exit of cars, e.g., for maintenance. • M a n u a l : The S E B requires manual authentication with a credit card for every car. • A u t o m a t i c : The S E B automatically recognises cars by their licence plates, authenticates the user, and authorises entrance. operation modes O n and O f f , which means that they are either fully functional or not available at all. Further, D C (don’t care) entries indicate that the operating mode of the given component is irrelevant. A u t o m a t i c requires all other components to be O n except the card reader, as it is not used. In case the camera C A M , the car sensor C S or the plate recognition service P R S are faulty, the automatic authentication procedure is unavailable and therefore the SEB should be operated in M a n u a l . If further the card reader C S is also unavailable, the S E B can not provide reasonable service and should be operated in B l o c k e d , which efectively closes the gate to the garage. If the parking garage utilises multiple entrance barriers, the garage could proceed to operate with degraded performance. The operating mode O p e n is intended for maintenance or free parking campaigns, because the barrier will remain open until it is manually reconfigured. In case either the barrier unit BU or the barrier sensor BS sufer from a mechanical failure, none of the pre-conditions can be fulfilled which leads to the operating mode O f f . In this case only the physical state of the barrier is unknown and has to be inspected by a technician, but the application logic is still in a well-defined state.
Implementation – Framework: A component can be switched into one of the available
operating modes either by human operators, software logic, or Morpheus. To realise these
features, we first extend the basic type A b s t r a c t S t a t e [
We further introduced a new type called D e p d e n c y F u n c t i o n that is used to define the preconditions for each operating mode. Essentially, this is a boolean function that takes the components current state and the sub-component’s operating modes into account. These are called Shadow Modes since it can not be reliably determined wether an external component is unavailable due to a hardware failure or because of an interrupted network connection. We choose to realize this as a generic function, which evaluates declarative conditions like Table 4.1, while still providing great freedom, e.g., to perform arbitrary calculations.
Listing 1: The datatypes that are used to store a component’s state. 1 t y p e D e p e n d e n c y F u n c t i o n < S , E , P , D > = ( 2 s t a t e : D e g r a d a b l e S t a t e < S , E , P , D > , 3 s h a d o w M a p : M a p < s t r i n g , s t r i n g > 4 ) = > b o o l e a n ;
Listing 2: The D e p e n d e n c y F u n c t i o n is used to define pre-conditions for an operating mode.
shown in Listing 3, which provides a snippet of the dependency map implementation for the S E B . In essence, this map stores the pre-conditions for each operating mode. The snippet shows an example implementation for the operating mode A u t o m a t i c . 1 e n u m M o d e s { L 0 = ” O F F ” , L 1 = ” B L O C K E D ” , L 2 = ” O P E N ” , L 3 = ” M A N U A L ” , L 4 = ” A U T O M A T I C ” } ; 2 e n u m S h a d o w M o d e s { O F F = ” O F F ” , O N = ” O N ” } ; 3 e n u m S u b C o m p { C A M = ” C A M ” , C S = ” C S ” , B U = ” B U ” , B S = ” B S ” , C R = ” C R ” , P R S = ” P R S ” , P M S = ” P M S ” } ; 4 c o n s t d e p e n d e n c y M a p = n e w M a p < M o d e s , D e p e n d e n c y F u n c t i o n < a n y , E v e n t s , P o r t s , M o d e s » ( [ 5 [ M o d e s . L 4 , ( s t a t e , s h a d o w M a p ) = > { 6 i f ( s h a d o w M a p . g e t ( S u b C o m p . C A M ) = = = S h a d o w M o d e s . O N & & 7 s h a d o w M a p . g e t ( S u b C o m p . C S ) = = = S h a d o w M o d e s . O N & & 8 s h a d o w M a p . g e t ( S u b C o m p . B U ) = = = S h a d o w M o d e s . O N & & 9 s h a d o w M a p . g e t ( S u b C o m p . C R ) = = = S h a d o w M o d e s . O N & & 10 s h a d o w M a p . g e t ( S u b C o m p . P R S ) = = = S h a d o w M o d e s . O N & & 11 s h a d o w M a p . g e t ( S u b C o m p . P M S ) = = = S h a d o w M o d e s . O N ) { 12 r e t u r n t r u e ; 13 } 14 r e t u r n f a l s e ; 15 } ] , / / f u r t h e r D e p e n d e n c y F u n c t i o n s f o l l o w h e r e 16 ] ) ;
Listing 3: Implementation of a D e p e n d e n c y F u n c t i o n for the operating mode A u t o m a t i c .
At any given time, each component has a target operation mode. Under normal circumstances this is the operating mode that provides the full intended feature set (A u t o m a t i c in our example). Yet, due to external dependencies to other components, may not be able to provide this ideal mode. In this case, Morpheus picks another operation mode for for which all dependencies are met. We refer to this operation mode as ’s degradation level. For enabling Morpheus to select an appropriate degradation level Morpheus requires a developer to specify a directed acyclic graph (DAG) for each degradable component. The graph connects the component’s implemented operating modes, such that each of them is directly or indirectly degradable to operating mode O f f . Morpheus selects the next degradation level along the graph that fulfils its preconditions, and executes adjacent transitions. Operating modes that are connected through a path that leads to the operating mode O f f should provide a graduated subset of the component’s functionality. Whenever the operating mode of a subcomponent changes, Morpheus first checks if the current operating mode/degradation level can be kept and if not selects a new operation mode to degrade/upgrade to. Further, based on the DAG, it determines the path to reach that state and iteratively degrades/updates along the path from one operation mode to the next until the selected mode has been reached.
Implementation – Framework: For each operation mode associated with its own state machine, a mechanism is needed to not only switch between modes, but also to transfer the state of one state-machine to the state of another. This functionality can be implemented using the T r a n s f e r F u n c t i o n whose signature is provided in Listing 4. While the main task of a transfer functions is to take one operating mode as an input and output another one, it can be used to perform additional tasks. First, it is possible to save a component’s internal state on a stack, such that it can be restored if the operating mode is reached later again. Second, the component’s degradation path can be recorded and analysed in order to make fine-grained adjustments to the component state. 1 t y p e T r a n s f e r F u n c t i o n < S , E , P , D > = 2 ( c u r r e n t : D e g r a d a b l e S t a t e < S , E , P , D > ) = > D e g r a d a b l e S t a t e < S , E , P , D > ; Listing 4: The T r a n s f e r F u n c t i o n is used for the state transfer between two operating modes.
Morpheus now provides the u p d a t e O p e r a t i n g M o d e function, which automatically selects the optimal operating mode for the component based on the defined pre-conditions, the degradation DAG and the availability of the dependent subcomponents. The implementation of this function is provided in Listing 5. Internally, it evaluates the implemented D e p e n d e n c y F u n c t i o n s in descending order. If a function returns true, it is checked whether the degradation level of the assigned operating mode is higher or lower than the degradation level of the current operating mode. If the degradation level is lower, the internal d e g r a d e function is executed, which performs a depth-first search on the degradation DAG, that produces an array of T r a n s f e r F u n c t i o n s . This array of T r a n s f e r F u n c t i o n s is then iteratively processed, and the function finally returns the degraded state. In case no path could be found the state is returned unaltered. The implementation of the u p g r a d e function is quite similar. This function is executed if the degradation level of the new operating mode is higher than the degradation level of the current operating mode. However, it additionally guarantees that an upgrade path is only executed if it leads to the component’s target operating mode. This property is especially important if only a partial upgrade can be performed, i.e. if the component can be upgraded, but the target operating mode can not be reached yet. Implementation – Running Example: Figure 2 shows an example DAG for the S E B , based on the conditions from Table 4.1. If the S E B ’s current operating mode is A u t o m a t i c and the conditions are no longer true, e.g., the P R S switched to mode O f f , then the next possible mode is considered in the DAG. Concretely, operating mode M a n u a l is the first degradation level for target operating mode A u t o m a t i c . Further, the operating mode O p e n can only be reached through manual reconfiguration, if it is assumed that A u t o m a t i c is the standard target operating mode. Listing 6 provides a snippet that shows how the T r a n s f e r F u n c t i o n can be used to implement the transition from A u t o m a t i c to M a n u a l . Here, we utilise the d e g r a d a t i o n H i s t o r y to record the component’s current operating mode and internal state.
In this paper we presented Morpheus, a framework that aims to assist developers in realising
graceful degradation and automatic recovery. Morpheus provides lightweight integration into
our previously presented internal DSL based on operating modes of interconnected components.
In our previous internal DSL [
The authors acknowledge the financial support by the Federal Ministry of Education and Research of Germany (BMBF grant nr. 01IS18068, SORRIR). Further, this work was partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 435878599 ; 453895475.