The basic idea of developing future 'lifelike' systems is to transfer qualities in technical utilisation that go beyond wellestablished mechanisms such as self-adaptation, learning, and robustness. In this paper, we argue that the resulting systems will need components to self-explain their behaviour - if we want to avoid acceptance issues that result from surprising and irritating system behaviour. Such self-explaining behaviour needs to answer the questions of when, what and how explanations should be provided to the user. We review existing metrics, outline a concept of how to address the 'when' question and identify corresponding research challenges towards an automated generation of explanations.
Recent trends in information and communication
technology entailed increasingly autonomous systems that adapt
their own behaviour and try to optimise it over time –
resulting in so-called self-adaptive and self-organising (SASO)
systems. Initiatives such as Organic (
Even though SASO technology already had its roots
in cybernetics and has been drastically strengthened again
in the last two decades (perhaps starting from
Tennenhouse’s Proactive Computing, see (
In this article, we note that in addition to the obvious lifelike properties such as evolution and continuous adaptation to the ’living space’ or ’environmental niche’ as well as focused response mechanisms (to name only the obvious examples), another prominent challenge comes to the fore in the acceptance of such systems by the human users (or better: stakeholders or influenced persons). This raises the question of how such systems can explain their behaviour to humans in an automated way, which in turn leads directly to the two crucial questions: When are explanations of behaviour necessary? And: What needs to be explained.
From a developer’s point of view, this primarily means that we need a concept to answers the question of ’when’, which then enables us to answer the ’what’. Therefore, this article explains a concept to measure system behaviour, whereupon abnormal behaviour of these measurements will then serve as an answer to the question ’when’.
Building on recent work, we present a measurement framework for system behaviour that forms the basis for such an explanation framework (Section II). In addition, we discuss possible further variables that can be relevant for lifelike behaviour and can therefore be integrated into the framework. On this basis, we discuss a concept to automatically detect events that serve as triggers for selfexplanations, which is combined with a principled, possible use to answer the question ’what’ (Section III). Since this is intended as a first concept, we highlight the most urgent research challenges to automatically generate the resulting self-explanations. The article concludes with a summary and an outlook on how the defined concepts can be explored and implemented.
The basis of our approach to self-explanatory mechanisms of lifelike technical systems is the possibility of quantifying system behaviour by means of (external) observation. To this end, in this section, we first present our system model, which we currently assume - and which can form the basis for future lifelike systems. Using this system model, we then explain existing and potential approaches for quantifying system properties.
In this article, we refer to a technical system S as a collection A of autonomous subsystems ai that are able to adapt their behaviour based on self-awareness of the internal and external conditions. We further assume that such a subsystem is an entity that interacts with other entities, i.e., other systems, including hardware, software, humans, and the physical world with its natural phenomena. These other entities are referred to as the environment of the given system. The system boundary is the common frontier between the system and its environment.
Each ai 2 A is equipped with sensors and actuators (both,
physical or virtual). Internally, each ai consists of two parts:
The productive system part PS, which is responsible for the
basic purpose of the system, and the control mechanism CM,
which controls the behaviour of the PS (i.e., performs
selfadaptation) and decides about relations to other subsystems.
In comparison to other system models, this corresponds to
the separation of concerns between System under
Observation and Control (SuOC) and Observer/Controller tandem
(
At each point in time, the productive system of each ai is configured using a vector ci. This vector contains a specific value for each control variable that can be altered to steer the behaviour, independently of the particular realisation of the parameter (e.g., as real value, boolean/flag, integer or categorical variable). Each subsystem has its own configuration space, i.e. an n-dimensional space defining all possible realisations of the configuration vector. The combination of the current configuration vectors of all contained subsystems of the overall system S defines the joint configuration of S. We assume that modifications of the configuration vectors are done by the different CM only, i.e. locally at each subsystem, and are the result of the self-adaptation process of the CM .
This system model describes an approach based on the current state-of-the-art in the field of self-adaptive and selforganising systems. We assume that ongoing research towards more lifelike systems will shift the boundaries in terms of the underlying technology as well as the possibility to alter higher-levelled design decisions – but it will most likely not result in entirely new design concepts. In turn, we assume that fundamental questions will arise about how the CM evolves according to the characteristics of its ’environmental niche’, for instance, but the separation of concerns between CM and PS remain visible.
Traditionally, the success and the behaviour of a technical
system is quantified using the primary purpose of the
application. In the first place, this directly refers to the system
goal. Based on the categorisation proposed by McGeoch in
These overarching aspects of system behaviour are
augmented with a more theoretical analysis of the
applicability. In particular, given techniques such as the O(n)
notation, runtime and memory complexity are quantified. This
can be extended with verification of processes, i.e.,
guarantees that may be quantified in terms of coverage or degree
of guarantee-able behaviour. As an alternative, the
’restoreinvariant approach’ by Nafz et al.
In addition to these considerations, the robustness and
resilience of systems, as well as their behaviour, can be
quantified using specific metrics. One recent example from the
domain of Organic Computing systems can be found in
(
Due to the shift in responsibility as visualised by our system
model depicted in Figure 1 – a CM is added to the PS that
performs (semi-)autonomous decisions – research on SASO
systems entailed augmenting the measurement framework
by several SASO-specific metrics. For instance, Kaddoum et
al. (
In addition to these goal- and effort-based metrics, several
further measurements indicate a macro-level behaviour of a
set of autonomous subsystems. The most important are:
a) Emergence is basically described as the emergence
of macroscopic behaviour from microscopic interactions of
self-organised entities (
b) Self-organisation can be expressed as a degree to
which the autonomous subsystems forming an overall SASO
system decide about the system’s structure without
external control, where the structure is expressed as
interaction/cooperation/relation among individual subsystems
(
c) Self-adaptation refers to the ability of systems to
change their behaviour according to environmental
conditions, typically with the goal to increase a utility
function. The degree of adaptivity can be measured using static
(
d) Scalability is a property that defines how far the underlying mechanisms are still promising if the number of participants grows strongly. Quantitatively, this can be expressed as an exponent for the control overhead, for instance.
e) Stability is to a certain degree a meta-measure applied,
e.g., to the degrees of self-adaptation and self-organisation.
It determines how far the metrics are static allowing for
standard changes and identifying deviations from the expected
behaviour. An example can be found in (
f) Variability or Heterogeneity are terms referring to
population of individual subsystems as they focus on the
differences in the behaviour, the capabilities or the strategies
followed by the subsystems. Examples can be found in
(
g) Mutual influences among distributed autonomous
subsystems indicate that the decisions of one have an impact
(e.g., on the degree of utility achievement) of another
subsystem (
Considering the concept of lifelike technical systems and their desired capabilities, the set of existing metrics is probably not sufficient enough to cover the entire behaviour. In particular, we will have to investigate novel measurement techniques that explicitly cover lifelike attributes. Although there is currently no exact definition of what lifelike computing systems are, we can approach the question of what is missing in the measurement framework by considering ’qualities of life’ that we aim to transfer to technical usage and that go beyond the SASO-based scalability, adaptation, organisation, or robustness questions. In particular, we identified the following aspects as primary options based on the considerations of how we consider lifelike systems outlined in Section I:
First, lifelike system will evolve over time which may
include an adaptation of its primary usage. Consequently, a
first measure should aim at quantifying the evolution
behaviour itself and a second one the coverage of the primary
purpose. The latter case continues the ideas formulated in
the Organic Computing initiative when defining the property
of ’flexibility’, i.e. how far a SASO system can react
appropriately to changing goal functions (
Second, this evolution corresponds to an adaptation to the niche in which the system survives. This may be expressed with a measure of ’fitness in the niche’ or ’degree of niche appropriateness’.
Third, such an evolution implies that the system is somehow converted (or better: converts itself). Besides the description of this process of time, a more static measure based on the design can aim at determining a ’degree of convertability’, i.e. the freedom to which the system can evolve during operation.
Fourth, this may include a transfer to an entirely different niche, or in other words to another problem domain. This can be expressed in a static manner by comparing the current problem space with the initial one or in a dynamic manner by a degree of transfer that the system has undergone.
Fifth, such a lifelike, evolutionary behaviour is done in
the context of the environmental conditions, which includes
the presence of other subsystems in open system
constellations. As a result, parts of the decisions of a lifelike system
are about the current integration into such a constellation,
resulting in ’self-improving system integration’ (
Finally, such a lifelike character obviously has implications on the way we design and operate systems. In contrast to current practices that take design-related decisions and provide corridors of freedom for the self-* mechanisms, design-time decisions themselves need to become reversible or changeable by the systems, resulting in a degree of reversibility (in a static manner) or changes (in the sense of how strong the design has already been altered).
Obviously, this list is not meant to be complete. It illustrates the need for further techniques that are suitable to quantify the lifelike-based properties. We are convinced that a necessary path in lifelike research is to fill this gap with an integrated measurement framework that provides a basis for comparison and assessment of the observed runtime behaviour.
Within the last year, several contributions proposed steps
towards a self-explanation of technical systems, particularly
focusing on aspects of self-adaptation. The most prominent
examples can be found in
In contrast to these approaches, we propose to develop
a self-explanation component for lifelike technical systems
that builds upon the metrics outlined above and establishes
an observation and explanation loop. The idea of such a
selfexplanation is that this should cover the following aspects:
• It should only be provided if the system recognises
unanticipated behaviour or abrupt shifts that are perceived by
humans that interact with the system (otherwise we face
an attention problem of users)
• The explanations should contain information about what
changed and why this change happens, which includes the
triggers that have been identified as root causes (e.g. a
failure of a component, abnormal external effects or
behaviour change of other systems)
• This may be augmented with an estimation of the impact
and severity as well as a prediction of the upcoming
developments.
• Further, the self-explanation should come in a
humanunderstandable format, i.e. using human-interpretable
terminology (e.g., ’Device X became too hot due to overload
that was caused by new component Y’)
• Finally, these explanations have to be generated in a
timely and accurate manner and become subject to a
learning process that optimises the self-explanation per
user. In particular, this can consider direct (i.e., approval
or intervention at goal level) and indirect (i.e., recognition
and no following action by the user) feedback for
optimisation purposes. The result will then be a user-specific
degree of explanation behaviour.
1. An observation loop is established at the macro-level that
gathers all externally visible variables of the contained
subsystems. We aim at the maxro- or system-wide level
for explanations as we consider the autonomous
subsystems as components for the overall functionality.
However, this system boundary choice depends on the purpose
and the perception of the user.
2. The resulting data is pre-processed, brought into an
appropriate representation and analysed to determine the
key figures. This includes static and dynamic indicators.
3. Based on novelty/anomaly/change detection such as
Please note that the integrated quantification framework to assess the system behaviour and the corresponding explanation loop is assumed to work at the macro-level (i.e., without any insight on the specific mechanisms and representations of the individual subsystems), since some of the metrics only occur at macro-level (e.g. emergence). However, this can be turned into a hybrid system approach, where each subsystem cooperates with the system-wide loop to filter, augment, and customise the explanations.
Considering this envisioned process towards automated self-explanations in lifelike systems, we face several research challenges. In the following paragraphs, we outline the most urgent ones and provide first ideas on how to solve them.
Challenge 1 - Metrics: The first challenge is concerned with the metrics briefly summarised in Section II. In particular, we have to answer the questions, which of these metrics is relevant? This includes answers to the question of what do metrics for the quantification of lifelike qualities look like? Based on this, we have to define a mechanism to pre-process, and represent the incoming data – which defines a standard time-series analysis problem.
Challenge 2 - Types of metrics and availability: As outlined above, we distinguish between static and dynamic measures. Considering the inherent heterogeneity, we have to find the concept of how to fuse the measures by integrating both, static and dynamic measures. This further results in questions of how to augment the pure scores, i.e. if predictions of upcoming behaviour are required and in which resolution to allow for proactive actions.
Challenge 3 - Anomaly/Novelty detection: The core of
our concept lies in a sophisticated detection of triggers,
which is defined as unexpected behaviour of the key
indicators. Technically, this should be realised in terms of
anomaly, novelty or change detection techniques.
Consequently, the questions arises which techniques are most
appropriate and how we can provide online methods. Here, we
can make use of approaches from the field of self-integrating
systems (
Challenge 4 - Root cause detection: Given that a trigger for self-explanation is detected, we have to identify the root causes that have been responsible for the observed behaviour. This means to provide techniques that are able to detect possible root causes (also as sequences of interconnected events and not just as isolated events) and rank them? Based on such an approach, we have to investigate how to select the most likely root cause or set/sequence of root causes that explain the behaviour.
Challenge 5 - Definition of explanations: Above, we already mentioned which information an explanation to the user should contain. This needs to be formalised and investigated in detail. In particular, this results in the challenge of how to generate explanations and which aspects they should comprise.
Challenge 6 - Presentation of explanations: Finally,
explanations have to be automatically presented to the user in a
human-understandable manner. This implies that we have to
develop concept of combining human-based terms (such as
cold/warm or fast/slow) with system-based measures.
Possible concepts could establish joint input spaces and use
human feedback to learn the resulting mapping. Using such
a joint representation, concepts from deliberative abductive
reasoning (
In this paper, we outlined our notion of what lifelike technical systems should be - or better which qualities of life we aim at imitating in technical systems that go beyond the well-established field of self-adaptive and self-organising systems. Based on this, we reviewed approaches to quantify system behaviour mainly at the macro-level using a standard system model for self-adaptation. This review also included an identification of possible measurement approaches that close the gap for observation and behaviour assessment of future lifelike systems.
In our notion, an important property of these lifelike systems will be to allow for self-explanation of decisions and resulting behaviour, otherwise the acceptability of even more autonomous and evolving systems will most likely face acceptance problems. We outlined that such selfexplanation has to answer two major questions: i) When to provide self-explanations to the user and ii) what is explained (including ’how’). This paper proposed to address the first question by using a measurement framework.
Future work will investigate possible metrics to quantify especially the evolution aspects of lifelike behaviour, including the properties of reversibility or transferability of the system purpose. By using selected applications as use cases, we aim at analysing how the identification of events or conditions that need explanations to the users can be established. Following this, the final goal of this research is to provide mechanisms and techniques that actually derive human-understandable self-explanations.