=Paper=
{{Paper
|id=Vol-2019/mrt_1
|storemode=property
|title=Active System Integrity Fences
|pdfUrl=https://ceur-ws.org/Vol-2019/mrt_1.pdf
|volume=Vol-2019
|authors=Christopher Landauer
|dblpUrl=https://dblp.org/rec/conf/models/Landauer17
}}
==Active System Integrity Fences==
https://ceur-ws.org/Vol-2019/mrt_1.pdf
Active System Integrity Fences
Christopher Landauer
Topcy House Consulting, Thousand Oaks, California
Email: topcycal@gmail.com
Abstract—This paper describes the architecture of a Wrapping- 1.1. Example: Active System Integrity Fences
Based Self-Modeling System that includes all of the models
and processes we have mentioned in our previous papers on In this Subsection, we describe our application example:
the subject of methods for mitigating the effects of the Get “active system integrity fences”, which are modules charged
Stuck theorems. To make the description more concrete, we with a single simple (if difficult) goal: explore system be-
have selected a particular application domain example: “active havior and look for anomalies.
system integrity fences”, that protect and defend a complex We use the term “fences” intentionally, to emphasize a
computing system or network (called the host system) from all distinction in purpose: guards keep things out, wardens keep
enemies, foreign and domestic. things in, watchers and monitors do neither, and fences do
An “active integrity fence” sits on an interface between both.
two system components, with an explicit notion of the char- We consider complex engineered systems that are man-
acteristics expected in the traffic (in both directions), derived aged or controlled by software, but that have essential
from expectations provided at design time and observations
hardware components driven by the software. These sys-
collected at run time. We do not describe a particular host
tems may be distributed, and operate in environments that
are too large, too remote, too hazardous, or too rapid for
system, since our primary interest is in the protector system,
direct human operation. They therefore need a great deal of
and we expect that it applies to any host system for which we
autonomy, and even local adaptivity.
can specify the expected internal interactions.
These systems are not entirely software. Hardware in
We define the relevant models and processes, and how they systems is usually accompanied by very specific operating
interact with each other and with the host system. We also conditions, provided by the manufacturer. Complex hard-
provide a short description of the Wrapping infrastructure ware often has a large and diverse set of commands that
that holds the system together and organizes and executes all it interprets, and part of the role of the active fences here
of its behavior, both protective and mission. is to guarantee that the command streams do not drive the
Keywords: Self-Aware Systems, Self-Adaptive Systems, hardware outside its operating envelope without a certain
Self-Modeling Systems, Get Stuck Theorems, Behavior Min- level of authorized over-ride (there are frequently multiple
ing, Dynamic Knowledge Management, Wrapping Integration levels of envelopes for any hardware component ranging
Infrastructure from “not recommended” to “this will break it”.
The purpose of an active system integrity fence is to
protect and defend a complex computing system or network
1. Introduction to the Problem (called the host system) from all enemies, foreign and
domestic.
This paper is a further continuation of and companion An active integrity fence sits on an interface between two
to previous papers on Self-Modeling Systems [31] [26], system components (or at the common entry interface of one
concerning new and extended methods for mitigating the component), with an explicit notion of the characteristics
effects of the “Get Stuck” Theorems (see [31] for a more re- expected in the traffic (in both directions, both temporal
cent explanation. and [26] for a discussion of issues). These behavior and semantics), derived from expectations provided
theorems basically say that the system needs to be able to re- at design time and observations collected at run time.
arrange its own knowledge structures for compactness and In the simplest case, it can throttle the data volume to an
efficiency, if it expects to survive for long periods of time acceptable level (by silently ignoring or pointedly rejecting
in a demanding environment [30] [31]. The mechanism for input), but it can also use constraints on the contents of data
doing that must include comparison of design-time expec- to make similar decisions (e.g., for routing, acceptance, and
tation models with run-time observations of behavior. Self- even explicit rejection or tacit dropping of data).
Modeling Systems will do these comparisons themselves, The criteria are a combination of “type” constraints on
as much as feasible. Though, strictly speaking, this is a the content characteristics of the data and associated “al-
way to decide on adaptations, not an adaptation itself; lowed data volumes” (more complex examples have explicit
the adaptation processes we use fall out of the Wrapping state-machine protocol identifiers and constraints on their
infrastructure. allowed transition volume).
The main purpose is to identify unexpected conse- The basic idea starts with the “Problem Posing” inter-
quences and prevent them from damaging system perfor- pretation of programs [27], which replaces explicit invoca-
mance: for example, even after a system password compro- tion of computationsl resources with an implicit request to
mise, the notion that certain external entities can access the address a problem.
system is a failure that should be rejected. Thus, programs interpreted in this style do not “call
What are the likely dangerous unexpected conse- functions”, “issue commands”, or “send messages”; they
quences? “pose problems” (these are information service requests).
• system leaks (provides data content it should not) Program fragments are not written as “functions”, “mod-
• system overloads ules”, or “methods” that do things; they are written as
• system thrashes “resources” that can be “applied” to problems (these are
• system forgets (loses data) information service providers).
• system breaks (still runs, but wrong answers) Because we separate the problems from the applicable
• system stops resources, we can use more flexible mechanisms for con-
necting them than simply using the same name.
Available computational resources dictate whether the We have shown that this approach leads to some interest-
fence is continuous (checking whenever anything in the ing flexibilities, when combined with the “meta-reasoning”
interface changes) or sporadic (occasional, periodic, or other approach of Wrappings [5], including such properties as
event based activity rule). software reuse without source code modification, delaying
Our focus here is on the computational mechanisms that language semantics to run-time, and system upgrades by
enable this kind of protection. incremental migration instead of version based replacement.
We can also consider mobile fences, moving around in
We specifically want to make the mapping from prob-
the system or network, either transferring from one machine
lems to resources explicit, because implicit mechanisms are
to another, or just changing focus on different interfaces
hard to study, so for Wrappings we use a Knowledge Base.
(which clearly needs a map of the interfaces to define the
space of movement. The Wrapping integration infrastructure is defined by
its two complementary aspects, the Wrapping Knowledge
Bases and the Problem Managers.
1.2. Structure of Rest of Paper
The Wrapping Knowledge Bases (WKBs) contain the
The structure of the rest of the paper is as follows. In Wrappings that map problems to resources in context. They
Section2, we provide some background on Wrappings, to define the entire set of problems that the system knows how
make the paper more self-contained. to treat (there are usually also default problems that catch the
In Section3, we provide some comments on why we ones otherwise not recognized). The mappings are problem-,
consider self-modeling systems, and how they relate to problem parameter-, and context-dependent.
the self- world. We also briefly introduce the “Get Stuck” The Problem Managers (PMs) are the programs that
theorems, which motivate this mitigation approach. read WKBs and select and apply resources to problems. The
In Section 4, we provide a notional system architecture, meta-recursion follows because the PMs are also resources,
based on Wrappings, that is sufficiently flexible to allow and are Wrapped in exactly the same way as other resources,
the “behind-the-scenes” knowledge management that is per- and are therefore available for the same flexible integration
formed by our mitigation processes, and within which the as any resources. These systems therefore have no privileged
mitigation models interact. resource; anything can be replaced. Default PMs are pro-
In Section 5, we provide a description of the models vided with any Wrapping implementation, but the defaults
used in our application and in the mitigation processes, and can be superseded in the same way as any other resource.
describe the mitigation processes in more detail, These are the processes that replace the implicit invocation
Finally, in Section 6, we present our conclusions and process, allowing arbitrary processes to be inserted in the
prospects for further advances. middle of the resource invocation process. This choice leads
to very flexible systems [33] [34].
2. Background on Wrapping The basic notion is the interaction of one very simple
loop, called the “Coordination Manager”, and a very simple
We provide a short description of Wrappings in this planner, called the “Study Manager”.
Section, since there are many other more detailed descrip- The default Coordination Manager (CM) is responsible
tions elsewhere [27] [25] [6], and especially the tutorials for keeping the system going. It has only three repeated
[33] [34]. The Wrapping integration infrastructure is our steps, after an initial FC = Find Context step as shown in
approach to run-time flexibility, with its run-time context- Figure 1.
aware decision processes and computational resources. The To “Find Context” means to establish a context for
basic idea is that Wrappings are Knowledge-Based inter- problem study, possibly by requesting a selection from a
faces to the uses of computational resources in context, user, but more often getting it explicitly or implicitly from
and they are interpreted by processes that are themselves the system invocation. It is our placeholder for conversions
resources. from that part of the system’s invocation environment that
Find Context means to apply the resource to the problem in the current
context; and “Assess Results”, which means to evaluate the
Assimilate Results result of applying the resource, and possibly posing new
problems. We further subdivide problem interpretation into
CM five steps, which organize it into a sequence of basic steps
that we believe represent a fundamental part of problem
Pose Problem Study Problem study and solution. These are implemented in the default
Study Manager (SM).
Match Resources To “Match Resources” is to find a set of resources that
might apply to the current problem in the current context. It
Resolve Resources is intended to allow a superficial first pass through a possibly
SM large collection of Wrapping Knowledge Bases.
Select Resource
To “Resolve Resources” is to eliminate those that do not
Adapt Resource apply. It is intended to allow negotiations between the posed
problem and each Wrapping of the resource to determine
Advise Poser whether or not it can be applied, and make some initial
bindings of formal parameters of resources that still apply.
This step invokes Apply Resource To “Select Resource” is simply to make a choice of
the resource to do which of the remaining candidate resources (if any) to use.
whatever it does Assess Results To “Adapt Resource” is to set it up for the current prob-
lem and problem context, including finishing all required
Figure 1. CM and SM Steps bindings.
To “Advise Poser” is to tell the problem poser (who
could be a user or another part of the system) what is about
is necessary for the system to represent to whatever internal to happen, i.e., what resource was chosen and how it was
context structures are used by the system. set up to be applied.
To “Pose Problem” means to get a problem to study from To “Apply Resource” is to use the resource for its
the problem poser (a user or the system), which includes a information service, which either does something, presents
problem name and some problem data, and to convert it into something, or makes some information or service available.
whatever kind of problem structure is used by the system To “Assess Results” is to determine whether the appli-
(we expect this is mainly by parsing of some kind). cation succeeded or failed, and to help decide what to do
To “Study Problem” means to use an SM and the Wrap- next.
pings to study the given problem in the given context, and Finally, we insist that every step in the above sequences
to “Assimilate Results” means to use the result to affect is actually a posed problem, and is treated in exactly the
the current context, which may mean to tell the poser what same way as any other, which makes these sequences
happened. Each step is a problem posed to the system by “meta”-recursive [1]. That means that if we have any knowl-
the CM, which then uses the default SM to manage the edge at all that a different planner may be more appropriate
system’s response to the problem. The first problem, “Find for the context and application at hand, we can use it (after
Context”, is posed by the CM in the initial context of “no defining the appropriate context conditions), either to replace
context yet”, or in some default context determined by the the default SM when it is applicable, or to replace individual
invocation style of the program. steps of the SM, according to that context (which can be
The main purpose of the default CM is cycling through selected at run time).
the other three problems, which are posed by the CM in Of course, we also have to have something to replace
the context found by the first step. This way of providing or supersede. We have therefore provided default resources
context and tasking for the SM is familiar from many for each of the CM and SM steps, to be used when no
interactive programming environments: the “Find context” other is selected to supersede it (as the above SM is the
part is usually left implicit, and the rest is exactly analogous default resource for the problem “Study Problem”). A simple
to LISP’s “read-eval-print” loop, though with very different complication occurs with the default among many possible
processing at each step, mediated by one of the SMs. In resources for the “Select Resource” problem: we want to
this sense, this CM is a kind of “heartbeat” that keeps the allow other resources to be used, so we insist that the default
system moving. resource (which otherwise might just pick the first resource
If the Coordination Manager is the basic cyclic program on the list) not pick itself if there is another choice when it
heartbeat, then the Study Manager is a planner that organizes is addressing the “Select Resource” problem.
the resource applications. The CM and SM interact as shown In addition, since the resources that read the WKB are
schematically in Figure 1. selected in context as is any other, the WKB can be het-
We have divided the “Study Problem” process into three erogeneous, with context determining which reader is used
main steps: “Interpret Problem”, which means to find a for which format of Knowledge Base. This helps greatly for
resource to apply to the problem; “Apply Resource”, which implementing improvements to programs, since the new and
old formats can exist simultaneously, unti the old format is We are especially enamored of Self-Modeling Systems
no longer needed. [28] [29], which have models of their own behavior, derived
We have used these algorithms many times to explain from original specifications of intent (from the designers),
and implement autonomous and reflective agents and sys- as modified by observation of actual behavior, because (in
tems [28] [29], and shown that they provide the appropriate this author’s paraphrase):
level of manageable flexibility and auditable integration. No plan ... extends with any certainty beyond the
The advantage in flexibility this approach provides over first contact with ... [reality], reference [39], p.92
other activity loops that have been proposed is that the SM These models are interpreted to produce the system’s be-
and CM steps are “meta”-steps, with posed problems for havior. That is to say, the behavior, sometimes including
the activities, allowing one further level of abstraction and the interpreter itself, is the interpretation of the models (this
indirection when it is useful. There are a number of other is not as hard as it seems [42] [28] [31]).
activity loops that we have seen described in various places The reason for self-modeling systems is to retain as
[5], especially the popular MAPE-K loop of autonomic much flexibility in the operational system as possible, and to
computing [21] [24] [3] [40], and we have shown that our allow the system to depart wildly from its original design
CM / SM meta-recursive interaction subsumes all of them. specifications (under carefully controlled or appropriately
The meta-interpretation style [1] of Wrappings [27] can of identified situations, of course). Additionally, it allows the
course be applied to any of them to make them much more system to examine itself, looking for anomalies:
flexible. O wad some Power the giftie gie us To see oursels
We have implemented several different kinds of CMs as ithers see us! [10]
in addition to the simple default CM defined above. There This architectural approach also supports the processes
are CMs that short cut the reflection by calling the default that mitigate the effects of the “Get Stuck” Theorems, which
step resources directly, and fully recursive versions that have essentially say that any software-intensive system that makes
extra levels of problem posing. Some of them are described models of its environment and behavior will eventually need
in other papers in the references. to reorganize its knowledge structures. To that end, we
We have also used different SMs, beyond the default defined several mitigation processes in [26] [32], and this
one that tries only one resource: one SM tries all applicable paper is a description of a notional architecture in which to
resources and returns with the first success, another tries implement the processes.
them all and evaluates them to return the best success,
and one collects all successes and summarizes. There are
4. Architecture
also different kinds of SM steps. The Match and Resolve
resources that read XML WKBs are different from the ones In this Section, we describe an architectural context for
that read text only WKBs. A different Match or Resolve the mitigation processes, using a Wrapping infrastructure to
might invoke a more sophisticated planner if there are no provide the flexibility of operations that we want. In the next
matches. A different Select might choose all compatible Section, we describe the processes in a little more detail and
resources, then negotiate among them. Different versions of show how they might interact.
apply, beyond the default function call, might send a request A notional picture of the architecture under considera-
message, or invoke an interpreter or other process. Another tion is in Figure 2.
one might simply add the resource to a configuration, instead The top part of the picture is the usual Wrapping CM /
of invoking it. SM loop, accessing the WKB in context to apply resources
Wrapping-based systems support run-time decisions to posed problems, possibly also adjusting the context (and
about which resources to apply in the current context, both allowing the context to be changed by the system environ-
at the application level (the resources that perform the ment). This is the standard behavior of a Wrapping-based
task at hand) and at the meta-level (the resources that are system.
used to select and organize the application level resources). The other processes in the main system, that is, the
This flexibility does come with a cost, but there are also ones in the application domain, are invoked by the SM, and
mechanisms based on partial evaluation [13] [20] [27] [41] build and maintain the domain knowledge bases that are the
[15] for removing any decisions that will be made the same focus of the mitigation effort (though, of course, the same
way every time, thus leaving the costs where the variabilities mitigation processes apply equally well to the Wrapping
need to be. Knowledge Base. due to reflection).
The mitigation processes BM, KnRef, and DKM collect
3. Self-Modeling Systems information from the choices made in the SM and adjust
the WKB, and they collect information from the domain
Our approach is related to the architectures of robots knowledge bases and adjust them also.
and autonomous vehicles [2] [37], but we add some features
not computationally feasible at the beginning. Our systems 5. Description of Models Used
are Self-Adaptive [6] [9] [23], which means that they can
observe their own behavior, reason about it, and use the There are three classes of models (processes): the in-
results to adjust the behavior. frastructure models, such as the CM, SM, and other PMs;
or just randomly spot check); this process clearly
CM
needs a map of system interfaces;
• examination: build models of the data crossing the
SM
interface (in both directions), infer actual protocol
context with performance measurements; in the simplest
case, just measuring close approaches to acceptabil-
ity boundaries is enough model;
domain behavior • retention: store these models in a system domain
processes knowledge base;
• communication: announce current status and ob-
served behavior, trends and variability distributions;
interact with other fences to discover and isolate
WKB domain resources problems; announce to the system that there is an
knowledge issue in certain places (so that the context can be
chand to avoid them);
• cooperation: interact with other fences to discover
KnRef and isolate problems; apply a requested data throttle
DKM at an interface;
• escalation: when a problem gets dangerous or large
or widespread enough (criteria supplied by the de-
BM signers). ask for help.
Each of these processes has relatively simple inputs, and
only the examination process has any difficult algorithms,
Figure 2. Notional Architecture Most of the difficulty lies in the criteria for data extraction,
modeling , and escalation.
the application models that perform the system’s objective 5.3. Mitigation Models
activity, and the mitigation models that sit behind that
activity and keep it running. These are the ones that do the mitigations [31] [26]:
5.1. Infrastructure Models • Behavior Mining
• Model Deficiency Analysis
The infrastructure has been described before, in Sec- • Knowledge Refactoring
tion 2. It contains the default PMs, which are the various • Dynamic Knowledge Management
flavors of SM (default is straight line plan, others are try all • Constructive Forgetting
until success, try all and combine, and others. These can be • Continual Contemplation
fully recursive or not. We describe the intent of each of these processes and how
It contains the various flavors of CM used in the system,
they might interact.
such as the default simple loop and the Mud CM for
distributed components, and these can be recursive or not.
5.3.1. Behavior Mining. This process examines every de-
It contains the WKB and the context knowledge base, and
cision in context, recording the following data:
the reflective nature of Wrappings allows both of them to
be heterogeneous, since the knowledge base readers are also • problem with parameters,
resources, selected according to posed problems in the usual • relevant context,
way. • resource application selected and constructed.
The relevant context is the set of context conditions that
5.2. Fences Application Models were examined and succeeded for the selection. In fancier
cases, this will also include resources not selected for this
These are the models that do the fences application
problem, and why (according to the context conditions that
work. Each fence is responsible for one class of interfaces
eliminated them).
(often just one interface).
In general, problems are stratified into layers, based
• instrumentation: record summaries of data types and on their resolution (in time, space, or concept). Strictly
volume for all (or just representative) crossing the speaking, this should be called semiotic content, but that
interface in either direction; in certain cases, the discussion leads beyond the scope of this paper.
content matters also; The BM process also examines changes to the domain
• movement: for mobile fences, decisions about when knowledge base, looking for infelicities, which can be sta-
and where to move (perhaps to track down a problem tistical or even esthetic (unbalanced trees, different amounts
of detail in different areas, unique or very low frequency to the esthetic criteria in the BM process: if we consider
references, which often results from specification errors). In any knowledge representation mechanism as a graph with
this case, the models are “plausibility” models, since there labeled nodes and directed labeled edges, then nodes with
is no available basis to declare them correct or incorrect. an excessive number of edges may be too general, and nodes
The BM process then feeds its results to the MDA with too few edges may be too specific.
process for resolution. This process is also sensitive to the access patterns
of the knowledge elements. We want elements that are
5.3.2. Model Deficiency Analysis. This process attempts accessed very often to have shorter access paths, which can
to determine where models have gone wrong (this is a distort any nice a priori ontology (the goal in this case is
retrospective analysis, not predictive). The simplest form of not readability; it is efficiency; other semantics-preserving
this begins with a behavioral assertion about model effects refactorings may be needed for readability).
that has been violated. This process is intended to be transparent to the users
We expect the assertions to be provided initially by of the knowledge bases, in the sense that their access mech-
the designers, since models are expected to provide some anisms remain exactly the same; only the resulting search
information service, and the model creators decide what that performance is affected.
is. After all, there was a purpose for creating the model in
the first place, and these assertions define the designers’ 5.3.4. Dynamic Knowledge Management. This process
expectations for it. organizes the knowledge for quick reasoning; it subsumes
Every behavioral assertion involves some of the vari- Constructive Forgetting (a mitigation from [31] [26]). We
ables within the model (or some performance parameters). called that one out explicitly because it is not normally
In the most intrusive case, every change to any of those acceptable to throw knowledge away, but we have shown
variables causes the assertion to be (it is possible to reduce that it is inevitable.
this a little bit if if can be proven that the change cannot The idea here is that there are different frequencies of
make an assertion go fro true to false). access for different parts of the knowledge base, and re-
When an assertion fails, the hard part begins: the assign- arranging the knowledge base can take that into account.
ment of blame, that is, how can the system decide where The simplest version of this is to pushd the Least
the failure is and who did it. This is especially difficult Recently Used (LRU) knowledge elements off to longer
for assertions in the usual kinds of first-order logic, since access paths (this is much the same process as defining
mathematically there is no such culprit. Huffman codes [19] [36] [14]). This measure of recency
However, the assertions are not the only information can be weighted by importance of consequences and gath-
available to the MDA process. It also has access to the ered by relevant context (big context change implies much
component behavior models constructed by the BM process, knowledge re-ordering). There are also other relevant factors
and sometimes it can use those to decide which part of an that will be different for each different application.
assertion is less likely to be wrong. This assessment can
often be done by maintaining a reliability “reputation” for 5.3.5. Continual Contemplation. This process is the gen-
each of the assertions, each of its components and each of eral term for all the background concurrent examination
the processes that produce the variables tha occur in those processes that examine everything: some examine as-is mod-
components. The reputation of an assertion component is els (from Behavior Mining), and compare with as-designed
enhanced by its success frequency (tempered by a measure models (from system definition). These functions are de-
of how well the input data to each variable producer fits the scribed earlier in this Section.
input assumptions).
In addition, the MDA process gets warnings from the
BM process about models that may be incorrect due to 6. Conclusions and Prospects
statistical or esthetic considerations. It tries to decide when
a strange structure is an error and when it is just strange. We have argued that the flexibility of self-modeling
Of course, it can’t really do that, mainly for undecideability systems make them well-suited to handle remote, complex,
reasons, but it can discover certain kinds of problems and and / or hostile environments, but that flexibility comes with
announce the others to the system monitors to try to get the obvious cost in performance, and a less obvious cost
help. If no help is forthcoming, then the issue is simply in organization. How much of this infrastructure machinery
recorded as a problem that the MDA process cannot solve, is used in any given application is an application-specific
and if enough instances of those occur, it escalates the issue engineering decision that depends on the expected level of
to a deficiency warning. hazard and the required level of performance.
We have shown that the Wrapping infrastructure sup-
5.3.3. Knowledge Refactoring. This process is partly ports a very flexible interweaving of domain resources and
housekeeping: re-arranging knowledge for efficiencies: it infrastructure resources, and can now include processes that
can be applied to the context conditions for the same mitigate the effects of the “Get Stuck” Theorems, which
problem (e.g., to get the shortest average time for decision, limit the lifetime of any autonomous system that builds and
given the time distribution of conditions). It is also related maintains models.
One of the most exciting prospects is that the system [6] Kirstie L. Bellman, Christopher Landauer, Phyllis Nelson, Nelly
has enough knowledge about its own behavior that it can Bencomo, Sebastian Götz, Peter Lewis and Lukas Esterle, “Self-
modeling and Self-awareness”, Chapter 9, pp. 279-304 in [22]
explain what it is doing, or what it is about to do, and why,
and most particularly, what it is not doing and why (the SM [7] Leopoldo E. Bertossi, Anthony Hunter, Torsten Schaub (eds.), In-
manages the selection, so it has the data to explain what it consistency Tolerance, Springer Lecture Notes in Computer Science,
Volume 3300, Springer Verlag (2004)
does not select). This requires the system to reason about
the situation it is in [4], so they can describe it. [8] Jean-Yves Beziau, Walter Carnielli and Dov Gabbay (eds.), Hand-
book of Paraconsistency, King’s College (2007)
A key advance in the state of the reasoning processes
would be to provide tools for them to reason about incom- [9] Robert Birke, Javier Cámara, Lydia Y. Chen, Lukas Esterle, Kurt
plete and inconsistent information [7] [8] [12], since that Geihs, Erol Gelenbe, Holger Giese, Anders Robertsson and Xiaoyun
Zhu, “Self-aware Computing Systems: Open Challenges and Future
describes essentially all of the system’s knowledge. Simi- Research Directions”, Chapter 26, pp. 709-722 in [22]
larly, various kinds of advanced learning methods [11] [38]
[10] Robert Burns, “To a Louse” (1768); Robert Burns in Your Pocket,
[18] [17] could be applied in the model building processes, Waverley Press (2009); along with many other collections and web
to avoid needing to specify a model type or structure in sites
advance. These could be of great use to the model building
[11] Jaime G. Carbonell, “Learning by Analogy: Formulating and Gen-
processes in these systems. erating Plans from Past Experience”, pp. 137-161 in [38]
However, learning methods such as XCS [43] [44] have
[12] Walter A. Carnielli, M.E. Coniglio and J. Marcos, “Logics of Formal
not much place here, since the behaviors in these systems Inconsistency”, pp. 15-107 in [16]
are not well modeled by MDP (Markov Decision Processes)
or even POMDP (Partially Observable MDP) in most cases, [13] C. Consel, O. Danvy, “Tutorial Notes on Partial Evaluation”, p.493-
501 in Proceedings PoPL 1993: The 20th ACM Symposium on Prin-
and these methods are not model-free; they assume a state ciples of Programming Languages, 10-13 January 1993, Charleston,
transition model that can be described as a POMDP. SC (January 1993)
We also described mobile fences, moving around in the [14] Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and
system or network, with the responsibility to explore, detect, Clifford Stein, Introduction to Algorithms, MIT Press (1990) Second
decide and act or escalate. If these fences are also knowl- Edition, McGraw-Hill (2001), Section 16.3, p.385392
edgeable about more of the system behavioral expectations, [15] Marcus Denker, Orla Greevy, Michele Lanza, “Higher Abstractions
then they should be able to detect certain software errors, for Dynamic Analysis”, pp.32-38 in Proceedings PCODA’2006: the
in addition to external anomalies. 2nd International Workshop on Program Comprehension through
We can also imagine some physical existence for “touch Dynamic Analysis, Technical report 2006-11 (2006)
points” (like the little blue police / watchman boxes for [16] D. Gabbay, F. Guenthner (eds.), Handbook of Philosophical Logic,
periodic checking in), so the fence can monitor and examine vol. 14, Reidel (2007)
its own progress. [17] Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning,
We think that these systems can be made much safer MIT (2016)
than they are now, but that requires an engineering judg- [18] Trevor Hastie, Robert Tibshirani, Jerome Friedman, The Elements
ment based choice of how much protective infrastructure to of Statistical Learning: Data Mining, Inference, and Prediction,
include. Springer (2009), 2nd ed. Springer (2016)
[19] David A. Huffman, “A Method for the Construction of Minimum-
Redundancy Codes”, Proceedings of the IRE, v.40, no.9, p.1098-
References 1101 (1952)
[20] N. D. Jones, “Partial Evaluation”, Computing Surveys, Volume 28,
[1] Harold Abelson, Gerald Sussman, with Julie Sussman, The Structure No. 3 (September 1996)
and Interpretation of Computer Programs, Bradford Books, now
MIT (1985) [21] J. Kephart, “Feedback on feedback in autonomic computing sys-
tems”, in Proceedings FC 2012: the 7th International Workshop on
[2] James S. Albus, Alexander M. Meystel, Engineering of Mind: An Feedback Computing, San Jose, California (2012)
Introduction to the Science of Intelligent Systems, Wiley (2001)
[22] Samuel Kounev, Jeffrey O. Kephart, Aleksandar Milenkoski, Xi-
[3] Paolo Arcaini, Elvinia Riccobene, Patrizia Scandurra, “Modeling aoyun Zhu (eds.), Self-Aware Computing Systems, Springer (2017)
and Analyzing MAPE-K Feedback Loops for Self-Adaptation”,
Proceedings SEAMS 2015: The 2015 IEEE/ACM 10th Interna- [23] Samuel Kounev, Peter Lewis, Kirstie L. Bellman, Nelly Bencomo,
tional Symposium on Software Engineering for Adaptive and Self- Javier Cámara, Ada Diaconescu, Lukas Esterle, Kurt Geihs, Holger
Managing Systems, 18-19 May 2015, Florence, Italy (2015) Giese, Sebastian Götz, Paola Inverardi, Jeffrey O. Kephart and
Andrea Zisman, “The Notion of Self-aware Computing”, Chapter
[4] Jon Barwise, The Situation in Logic, CSLI Lecture Notes No. 17, 1, pp. 3-16 in [22]
Center for the Study of Language and Information, Stanford U.
(1989) [24] Philippe Lalanda, Julie A. McCann, and Ada Diaconescu, Autonomic
Computing: Principles, Design and Implementation, Undergraduate
[5] Dr. Kirstie L. Bellman, Dr. Christopher Landauer, Dr. Phyllis R. Topics in Computer Science Series, Springer (2013)
Nelson, “Managing Variable and Cooperative Time Behavior”, Pro-
ceedings SORT 2010: The First IEEE Workshop on Self-Organizing [25] Christopher Landauer, “Infrastructure for Studying Infrastructure”,
Real-Time Systems, 05 May, part of ISORC 2010: The 13th IEEE In- Proceedings ESOS 2013: Workshop on Embedded Self-Organizing
ternational Symposium on Object/component/service-oriented Real- Systems, 25 June 2013, San Jose, California; part of 2013 USENIX
time distributed Computing, 05-06 May 2010, Carmona, Spain Federated Conference Week, 24-28 June 2013, San Jose, California
(2010) (2013)
[26] Christopher Landauer, “Mitigating the Inevitable Failure of Knowl- [35] Dr. Christopher Landauer, Dr. Kirstie L. Bellman, Dr. Phyllis R.
edge Representation”, Proceedings M@RT@ICAC 2017: The 2nd Nelson, “Modeling Spaces for Real-Time Embedded Systems”,
International Workshop on Models@run.time for Self-aware Com- Proceedings SORT 2013: The Fourth IEEE Workshop on Self-
puting Systems, Part of ICAC2017: The 14th International Confer- Organizing Real-Time Systems, 20 June 2013, part of ISORC 2013:
ence on Autonomic Computing, 17-21 July 2017, Columbus, Ohio The 16th IEEE International Symposium on Object / component /
(2017) service-oriented Real-time distributed Computing, 19-21 Jun 2013,
Paderborn, Germany (2013)
[27] Christopher Landauer, Kirstie L. Bellman, “Generic Programming,
Partial Evaluation, and a New Programming Paradigm”, Chapter 8, [36] Jan Van Leeuwen, “On the construction of Huffman trees”, p.382-
pp.108-154 in Gene McGuire (ed.), Software Process Improvement, 410 in Proceedings ICALP 1976: the Third International Collo-
Idea Group Publishing (1999) quium on Automata, Languages and Programming, 20-23 July 1976,
Edinburgh (1976)
[28] Christopher Landauer, Kirstie L. Bellman, “Self-Modeling Sys-
tems”, pp.238-256 in R. Laddaga, H. Shrobe (eds.), “Self-Adaptive [37] Alexander M. Meystel, James S. Albus, Intelligent Systems: Archi-
Software”, Springer Lecture Notes in Computer Science, vol.2614 tecture, Design, and Control, Wiley (2002)
(2002)
[38] Ryszard S. Michalski, Jaime G. Carbonell, Tom M. Mitchell (eds.),
[29] Christopher Landauer, Kirstie L. Bellman, “Managing Self- Machine Learning: An Artificial Intelligence Approach, Tioga Press,
Modeling Systems”, in R. Laddaga, H. Shrobe (eds.), Proceedings Palo Alto, CA (1983)
Third International Workshop on Self-Adaptive Software, 09-11 Jun
[39] Helmuth Karl Bernhard Graf von Moltke, On Strategy (in German),
2003, Arlington, VA (2003)
translated in Daniel J. Hughes and Harry Bell, Moltke on the Art of
[30] Christopher Landauer, Kirstie L. Bellman, “Model-Based Coopera- War: Selected Writings, Presidio Press (1993); paperback Presidio
tive System Engineering and Integration”, Proceedings SiSSy 2016: Press (1995)
3rd Workshop on Self-Improving System Integration, 19 July 2016,
[40] E. Rutten, “Feedback Control as MAPE-K loop in Autonomic
part of ICAC2016: 13th IEEE International Conference on Auto-
Computing”, Research Report RR-8827, INRIA Sophia Antipolis
nomic Computing, 19-22 July 2016, Wuerzburg, Germany (2016)
- Méditerranée, INRIA Grenoble - Rhône-Alpes (10 Dec 2015)
[31] Christopher Landauer, Kirstie L. Bellman, “Self-Modeling Systems
[41] Gregory T. Sullivan, “Dynamic Partial Evaluation”, Proceedings
Need Models at Run Time”, Proceedings M@RT 2016: the 11th
PADO II: Second Symposium on Programs as Data Objects, 21-
International Workshop on Models@run.time, 04 October 2016,
23 May 2001, Aarhus, Denmark (2001)
Part of ACM/IEEE 19th International Conference on Model Driven
Engineering Languages and Systems. 02-07 October 2016, Palais du [42] Ken Thompson, “Reflections on Trusting Trust”, Comm. of the ACM,
Grand Large, Saint Malo, Brittany, France (2016) vol.27, no.8, pp.761-763 (Aug 1984), http://dl.acm.org/citation.cfm?
id=358210 (availability last checked 03 Apr 2017); see also the
[32] Christopher Landauer, Kirstie L. Bellman, “An Architecture for Self-
“back door” entry of “The Jargon File”, widely available on the
Awareness Experiments”, Proceedings SeAC 2017: 2nd Workshop
Web, and other comments findable by searching for “back door Ken
on Self-Aware Computing, Part of ICAC2017: The 14th International
Thompson moby hack” (availability last checked 03 Apr 2017)
Conference on Autonomic Computing, 17-21 July 2017, Columbus,
Ohio (2017) [43] Stewart W. Wilson, “Classifier Fitness Based on Accuracy”, Evolu-
tionary Computation, v.3, no.2, p.149175 (1995)
[33] Dr. Christopher Landauer, Dr. Kirstie L. Bellman, Dr. Phyllis R.
Nelson, “Wrapping Tutorial: How to Build Self-Modeling Sys- [44] Stewart W. Wilson, “Generalization in the XCS Classifier System”,
tems”, Proceedings SASO 2012: The 6th IEEE Intern. Conf. on p.665674 in John R. Koza et al. (eds.), Proceedings GP 1998:
Self-Adaptive and Self-Organizing Systems, 10-14 Sep 2012, Lyon, the Third Annual Conference on Genetic Programming, 22-25 July
France (2012) 1998, University of Wisconsin, Madison, Wisconsin (1998)
[34] Dr. Christopher Landauer, Dr. Kirstie L. Bellman, Dr. Phyllis R.
Nelson, “Wrapping Tutorial: How to Build Self-Modeling Systems”,
Proceedings CogSIMA 2013: 2013 IEEE Intern. Inter-Disciplinary
Conf. Cognitive Methods for Situation Awareness and Decision
Support, 25-28 February 2013, San Diego, California (2013)