[Context and motivation] In the last years, a considerable number of solutions were developed for self-adaptive systems (SAS). These solutions are based on a feedback loop (MAPE loop), but employ di erent approaches to realize the di erent facets of the feedback loop. For example, the models used for the analysis step di er (e.g., goalmodels, feature-models). [Question/problem] In order to build a selfadaptive system for a particular application, a researcher or practitioner has not only to implement the application itself, but also the feedback loop. That is, one has rst to select an appropriate solution from the literature and second to implement and evaluate the solution in the environment at hand. [Principal ideas/results] The goal of our work is to reduce the e ort for developing a SAS by providing a framework for prototyping and evaluation. This framework comprises (1) a set of open source implementations of selected approaches from which an engineer can choose, (2) a case study to execute the implementations, and (3) a set of metrics for data collection during execution. [Contribution] The engineer can use the framework as a sandbox to compare several approaches in order to make a more informed decision which approach to choose. Possibly, the prototypical implementation can also be used in the nal SAS implementation. This is a research preview that reports rst results about the implementation of selected approaches (1), setting up a case study (2), and it outlines concepts regarding evaluation (3).
Following the ITU [10], the Internet of Things (IoT) is a global infrastructure for the information society, enabling advanced services by interconnecting (physical and virtual) things based on existing and evolving interoperable information and communication technologies. A thing is an object of the physical world (physical things) or the information world (virtual things), which is capable of being identi ed and integrated into communication networks. A device is a piece of equipment with the mandatory capabilities of communication and the Copyright c 2016 for this paper by its authors. Copying permitted for private and academic purposes. optional capabilities of sensing, actuation, data capture, data storage, and data processing. One key concern of systems operating in the IoT is to dynamically adapt to changing environments, due uncertainties during requirements-, design-, and run-time (see [12]).
A self-adaptive system (SAS) contains a feedback loop to react on
uncertainties in the environment. This loop is called MAPE loop by IBM [
A considerable number of concepts for self-adaptive systems has been developed. From a practitioner's perspective, open implementations of the MAPE feedback loop for prototyping purposes are missing. This is a show-stopper in practice, as a practitioner would need to work through research papers in order to build such a prototype. A researcher who is interested in the comparison, extension and/or application of existing solutions to a new domain is in a similar situation.
In this research preview we suggest a prototyping and evaluation framework for self-adaptive systems. The rst goal of the framework is to ease the prototyping (and possibly development) of self-con guring systems. For this purpose, the framework o ers implementations of selected approaches to SAS based on the MAPE loop (e.g., based on feature models as suggest by Pascual et al. [11]).
The second goal of the framework is to ease the evaluation of self-con guring systems, to allow for instance benchmarks between di erent approaches. For this purpose, the framework provides a case study drawn from the smart city domain, and a set of metrics for evaluation. The framework is able to collect data on a subset of the metrics at runtime about overall quality, e ort, and cost.
The framework can be used in a number of ways on the given case study: { to understand a particular SAS approach, { to optimize the application of an approach, or { to compare approaches in a particular target environment.
The following section reports on the related work regarding frameworks for construction and approaches to evaluation of self-adaptive systems. Section 3 describes the design of our framework for prototyping and evaluation, the current extent of implementation and rst experiences. Section 4 outlines the next steps that we scheduled to complete the framework, and Section 5 concludes the paper by describing one envisioned application scenario in more detail. 2
This section presents previous approaches to prototyping and evaluation of
selfadaptive systems. Several implementations of the MAPE loop have been
published [
With respect to evaluation, metrics have been published. Some authors
proposed metrics to compare SAS, for example McCann et al. [
Gjorven et al. [
We surveyed the literature on metrics that are actually used to evaluate selfadaptive and self-con guring systems and identi ed the following metrics: (3) the number of needed recon gurations vs optimal number of recon gurations, (4) resource usage (a) power consumption, (b) memory size, (c) execution time of the optimization process or parts like initializations, (5) average satisfaction of elements over a time interval, (6) impact on the target system (a) average lines of code per module that must be change (b) average response time of the target system.
We observed that an approach is validated with an average of two metrics out of the list above. Furthermore, the di erent approaches vary in the chosen metrics. These facts make it di cult or even impossible to compare current approaches given the information provided in the publications.
Our framework follows previous work in that it supports to build a SAS. The di erence lies in our goal also to evaluate and compare competing solutions. 3
The framework bases on experiences we made with the implementation of a
goaloriented (i*-based) approach to self-adaptive systems [
Design. Three important paradigms for self-con guring and adaptive systems
can be distinguished:
{ Goal Oriented Approaches e.g. Zanshin from Mylopoulos et al. [13]
1 Autonomic Computing Toolkit Home page: http://www.ibm.com/developerworks/
autonomic/r3/overview.html
{ Dynamic Decision Networks by Bencomo et al. [2]
{ Feature Models e.g. by Garcia-Galan et al. [
The focus of these approaches mostly lies on a better recon guration of the system and also on a better decision about the need of a new con guration. This means to identify the best con guration for the current system situation.
The largest commonality is the architecture of the approaches to self-adaptive systems that we currently analyzed. All of them are based on a feedback loop (MAPE) that consists of the parts monitor (monitoring con guration), analyze (on a Requirements model), plan (decision algorithm & decision model or elements) and execute. The di erences are in the way in which the di erent activities of the MAPE feedback cycle are carried out: 1. An indicator for an adaption can be Quality of Service for service-oriented systems or a variable (input, output, parameter) for embedded systems. 2. A monitor con guration can be a simple event-condition-action (ECA), an assertion, or a domain assumption. A monitor con guration describes a fact of the running systems that should be monitored because it can lead to a recon guration or adaption. 3. Di erent models are used for the analyze activity such as goal models or feature models. These models allow to represent requirements with alternative realization strategies. 4. The decision making is based on quality values for di erent realization strategies. For example in goal models, softgoals and contribution links are used to compute quality values, which in turn are used generate a recon guration plan. 5. The recon guration plan can be to increase or decrease values of variables or activate or deactivate components, modules, or functions.
The implementation of the framework presented in the following section is based on this analysis of commonalities and di erences.
Implementation. The prototyping framework is implemented in the Python programming language as this language allows for a wide range of deployments and is frequently used for prototyping (yet not limited to that purpose). The framework contains components to implement the di erent activities of the MAPE loop as well as further functions that are needed to build and run an adaptive system.
Fig. 1 shows the concept of the prototyping framework. Fundamentally, we separate development time and runtime artifacts. The development time artifacts represent additional aspects to be captured in the requirements phase. Runtime artifacts are components required to build the SAS. To easy prototyping, some runtime artifacts are generated from the development time artifacts. This process is described in more detail in the subsection "Generator" on Page 6.
We start the detailed discussion of the framework with the development time artifacts. Self-con guring Requirements are requirements with changing indicator values and also a status label that shows if the requirements is active or disabled for a system situation. Indicators are measurable elements that are gathered by hardware agents or software agents depending on the system. A Monitor Con guration describe how an indicator is measured and how an assured element change if the indicator change. A simple example is if the WLAN connection (indicator) of a system fails a realization strategy that needs this connection must be disabled. A Requirements Model contains the di erent realization strategies of a system. The model must be validated to verify that a con guration for a given situation is valid or an adaption has to be performed. The implementation of a Validation Algorithm di ers because of the previous mentioned di erent requirements models. The last component of the framework is a Decision Model, which is used to nd the best con guration for the current system situation. It contains all information to compute the value for a realization strategy. The computing is done by the Decision Algorithm. The result of the decision algorithm is a recon guration plan for the application.
The prototyping framework provides a set of components from which can be chosen to develop a self-adaptive system. At the time of writing it contains a feature-oriented approach with utility functions to compute a con guration with a genetic algorithm (see Pascual et al. [11]).
The framework can be populated with further approaches. In order to give an impression of the required e ort, we provide programming e ort data of the feature-oriented approach mentioned above. The extension has taken about 3 man-week's and 1500 LoC: { ECA's are used for monitoring the system (400 LoC 3-4 Days ) { Feature model is used as requirements model and representation of alternative realization strategies (600 LoC 3-5 days ) { Utility Table and - function are used to compute a con guration (200 LoC 3 Days) { A genetic algorithm is used to choose the \best" con guration (100 LoC 2
Days) { Con guration example (150 LoC 1/2 day ) { Simple Speci cation (1 textual requirement and feature model with 19 features) (200 LoC 1/2 day ) Generator. In order to ease the development of a prototype, a self-adaptive system is partially generated. That is, during development time (see upper part of Fig. 1), only the essential information for building the SAS is speci ed by the user. When using the feature-oriented approach this is: indicators (variables of the systems with a type), a feature model (di erent features of the systems and variation points), ECA rules (indicator change event, boolean condition, change action on a element, e.g. feature), utility function (table with utility values of the di erent variations of the feature model).
Probes for an application and the whole self-con guring framework (with integrated MAPE loop), see bottom part Fig. 1 can now be generated out of a speci cation. The indicators are used to generate probes for an application. These probes are used by the self-con guring framework to monitor the indicators and check the related ECA rules. The ECA description is used to generate the rules of the ECA rule engine. An application that include the generated probes can use the self-con guring systems to adapt the behavior to the changing context.
If the speci cation changes because new elements like indicators, ECA rules,
features or utility elements are added or elements are re ned for example an ECA
rule change, only the generation process is re-triggered and the new prototype
can be used for the system which contains the probes. Further details about the
generator concept are given in [
The next steps on our roadmap to complete the framework are described in the following.
Extension of Framework. Our next step is to re-implement our goal-oriented
approach to self-adaptive systems based on i* described in [
Case Study. Within the Internet of Things, we have identi ed smart cities as an interesting domain for a case study. It is accessible to many readers, it is complex, and comprises many di erent facets. A hardware demonstrator of a smart public lighting systems is under development that includes dynamic street lights which adapt there light color and light intensity to its environment and also communicate with other systems for example to display free parking-lots in navigation systems.
Evaluation. A component for logging and monitoring metrics is under development. Given the overall structure of our framework shown in Fig. 1, the data on the following metrics can be automatically collected: { Cost : Some possible indicators are memory size, CPU or power consumption. { Failure Avoidance: how much time elapse until a failure is xed. { Human in the loop/ Degree of Autonomy : count of the interaction with a user over a speci ed time interval. { Optimal Adaption: collect the satisfaction of con gurations and count of adaptations during a speci ed time interval. Build the average of these values and compare the values with a de ned ideal state. { Adaption/ Reaction Time: execution time of an adaption process. { Safe Adaption: count of failures (e.g. real-time constrains of an embedded system) that are produced because of an adaption. { Stabilization: that is the time taken from a change in the systems environment to stabilize the operations of a system.
The evaluation is limited to the metrics that can be captured at runtime. 5
Under the assumption that a researcher has developed a SAS prototype (using feature models, ECAs, and a utility function as described before), we sketch in this section a scenario for evaluation.
Measurement is an established discipline in software engineering and there is a large consensus that measurement has to conducted in an goal-oriented fashion. The Goal-Question-Metric paradigm [1] is one approach to systematically derive metrics from measurement goals.
A goal for a researcher could be to analyze the decision process of the MAPE loop for the purpose to understand with the focus on performance of the decision process from the perspective of a researcher in the context of an SAS implementation on a Raspberry Pi. Following the GQM approach, questions are derived to capture quality factors and factors that are hypothesized to in uence the quality factors. Questions are further re ned into metrics. Table 1 shows a possible GQM measurement model for this evaluation. 6
In this research preview we outlined a prototyping and evaluation framework for self-adaptive systems (SAS) and sketched its application. The architecture of the framework is based on the MAPE loop and it will o er a number of components implementing di erent approaches to SAS.
One major bene t of the framework is to ease the development of selfadaptive systems by a code generator and components that can be used out Metrics Question Metrics Question Metrics M1 M2 Q2 M3 M4 M5 Q3 M6 M7
Understand the performance of decision process.
What is the current adaption and decision processing performance Average execution time Average execution time of the decision process What resources are currently used by the decision process related to the adaption process Average memory size of addeacipstiioonnpprroocceessss 100 Average power consumption of addeacipstiioonnpprroocceessss 100 Average CPU load of addeacipstiioonnpprroocceessss 100 What is the current quality of the adaption process Average utility
Average number of adaptations of the box. That is, the user of the framework is able to focus on the application itself and on the de nition of the adaptation requirements in the selected formalism (e.g., feature models or goal models). The framework may also serve as a basis for 3rd party implementations.
The other major bene t is to provide a common basis for evaluation of approaches to SAS. For this purpose, a component is currently added the framework that automates data collection at execution time. The metrics that could be collected with the help of the framework were outlined.
As this paper is a research preview we have not paid much attention to validation. Yet, one major question to answer is how much e ort the framework actually saves in prototyping a SAS.
Finally, we believe that commonly available, open-source implementations of self-adaptive systems help to promote the eld. Therefore, it is planed to make the framework open source and publicly available using GitHub, when it has passed a rst validation.