Cyber-physical systems are being deployed in a wide variety of applications, creating a highly-capable infrastructure of networked \smart" systems that utilize coordinated computational and physical resources to perform complicated tasks either autonomously or in cooperation with humans. The design and testing of these systems using current methods, while time-consuming and costly, is not necessarily su cient to guarantee appropriate and trustworthy behavior, especially under unanticipated operational conditions. Biological systems o er possible examples of strategies for autonomous self-improvement, of which we explore one: active experimentation. The combined use of active experimentation driven by internal processes in the system itself and computational re ection (examining and modifying behavior and structure during operation) is proposed as an approach for developing trustworthy and adaptable complex systems. Examples are provided of implementation of these approaches in our CARS testbed. The potential for applying these approaches to improve the performance and trustworthyness of mission-critical systems of systems is explored.
Complex systems of systems (SoS), especially those that include cyber-physical systems (CPS), are now being deployed in critical infrastructure applications such as the electrical grid, health care, manufacturing, transportation, commerce, law enforcement and defense. We bet our lives, or at least our livelihoods, that these systems will function as anticipated. Yet, as they become increasingly complex and interconnected (networked), developing the systems engineering methods to ensure that these SoS will be trustworthy has become its own technical challenge.
For example, space systems (which term includes not only the satellites, but also the ground control and dissemination systems and the launch systems that put them up there) are simply the most complex engineered systems that humans build that work (and they do work almost always and often far beyond their projected design life). They typically involve hundreds of organizations, thousands of people, tens of thousands of components, millions of pages of documentation, and they are expected to last sometimes for decades. (The development process does usually last for decades even when the satellites are not expected to). It has been clear for some time that these systems exceed our ability to understand them, and that they only work by dint of what we have heard called \heroic engineering", but even that approach is now regularly exceeded by current and planned systems.
Successfully planning and implementing the integration of such complex
constructs would, in principle, require detailed knowledge of hundreds of thousands
(or more) of components, how they are connected into subsystems, and all of
the possible interactions between components, subsystems, and the environment.
From a practical perspective, it is exactly the lack of this detailed knowledge that
leads us to characterize a system as complex. [
Systems of systems are built from systems that themselves have been
developed and tested, often for a di erent application. The integration challenge,
then, concerns most importantly the necessity of reconciling the multiple and
sometimes con icting operation and control strategies of these systems with
respect to a new SoS purpose or goal. [
Complex systems of systems challenge established systems engineering practices in several ways.
{ Managing the complexity is a fundamental technical challenge in itself, independent of the particular system or application. { Updates and upgrades mean that the SoS evolves during its operational life. { The capability and value of a system / component / device leads us to repurpose it for applications that were never envisioned by its original designers rather than developing a completely new device. { Instances of the system are often unique, although there may be other, similar instances (i.e. Amtrak's reservation system, a segment of the electrical power grid, a space system including all ground and launch resources). { Ubiquitous wired- and wireless communications networks mean that the boundaries of the SoS and its possible states are probably not completely de nable. { Self-x capabilities mean that the system is never fully designed.
Component, subsystem, and system design and test currently utilize a variety of models at di ering levels of detail, together with a set of \goodness" measures linked to a priori requirements, as inputs to computational processes (often optimization) that evaluate candidate strategies. However, as complexity increases, \emergent" behaviors become increasingly likely. Such self-organized, coherent actions that were not planned or anticipated by the designers often occur through interactions that are not present in the models of components, processes and interactions used in design, integration and test. CPS SoS design and testing is further complicated by the re-purposing of legacy hardware and software which may not have been designed in accordance with current procedures, standards and interfaces, thus requiring specialized adaptations. New approaches are needed for design, veri cation and validation of complex cyberphysical SoS to better ensure their trustworthiness. The most desirable of these approaches will also address the spiraling cost of implementing and testing these complex SoS. 3
Biological systems provide a rich source of inspiration for engineering complex
SoS both because, in spite of their obvious complexity, they achieve remarkable
robustness, and also because of the extreme degree of interconnection of their
various components and subsystems. [
Control in biological systems occurs through the combined operation of many processes and actions, with desired behaviors being achieved by small changes in relative strengths. A web of overlapping monitoring and regulatory processes that maintain appropriate conditions at all levels of complexity is critical to the success of this paradigm. For example, the actions and processes used to achieve the top-level goal of walking over rough terrain are achieved by many instances of humans in spite of signi cant di erences in their structure, strength and ability. That is, biological systems rely not on uniformity of structure, but on the ability to adjust similar structures and generic patterns of actions based on a high degree of monitoring of local conditions in order to accomplish a behavior that is adequate for the current context and goal.
Controlling a SoS with a complex web of balanced interactions is strikingly di erent from the traditional block-diagram approach to engineering design that focuses on building a few strong and well-understood interactions between components while striving to nullify all other interactions. We suggest that the assumption that small interactions can be neglected, together with implementation of this assumption throughout the modeling process, is one important reason that emergent behaviors are often not predicted by simulations. In contrast, the biological style does not deprecate interactions, but instead achieves a \balance of forces" form of control based on extensive overlapping webs of monitoring and regulation at all levels of the hierarchy of complexity. We propose that implementing this style in strategic portions of engineered systems could mitigate the challenges posed by unmodeled interactions.
The biological design approach leads to a \permissive" style in which, while actions, states, conditions, and processes may vary from one instance to another, overall performance goals are achieved by adjustments in their relative intensities. This permissive style is in clear contrast to the restrictive control approach of traditional engineered systems, in which adjustments of a few inputs achieve all of the desired actions or processes through clearly de ned pathways. However, there is promising similarity between the dense web of interactions in biological systems and the challenge of managing the many unknown or unmodeled interactions in a complex SoS, again suggesting that a more biological approach to design, operation, and integration may be useful provided that the appropriate information about actual interactions can be discovered.
The admirable robustness of biological systems is due in part to their ability
to learn to accomplish the same goal using a variety of strategies, although not
necessarily equally e ciently. For example, if you break your right arm you are
still able to accomplish most of the tasks of daily life by substituting your left
arm or accommodating to the reduced motion allowed by a cast. This broad
ability to nd a way to accomplish a goal in spite of changes in capability or
con guration is exactly the type of robustness and reliability that we would
like to have in engineered SoS, and to understand and utilize during design
and integration. New ways of acting can take place through the recruitment of
existing structures and processes in new combinations to address a new context,
purpose or goal. [
The success of the biological control-through-balance style rests on experience with the available processes, structures and patterns, as well as of their limits of capability and their applicability to situations similar to the present one, either through evolutionary selection or from the experience of a speci c individual. This knowledge is not necessarily innate in a biological system, just as there is important knowledge lacking in models of SoS.
Biological systems use excess resources to actively experiment. By doing so,
they discover and re ne models of their capabilities, limitations, and possible
interactions with their surroundings that include consideration of both
internal state (hungry, cold, tired, etc.) and external conditions. Signi cantly, such
experimentation also enables the grouping of collections of useful resources,
processes and capabilities into generic pre-patterned templates with simpli ed
control mechanisms. Such templates can be easily shaped to t a speci c current
context. [
The biological analogy suggests that, if it were possible for a complex SoS or some of its components and subsystems to engage in active experimentation, the existence of con icts between the existing operation and control strategies of a repurposed subsystem and the overall SoS purpose or goal could be identi ed and modeled before such a con ict gives rise to failure or to disruption of the service provided by the overall SoS. In addition, e cient strategies for accomplishing common purposes and goals could be discovered and collected into templates for accomplishing similar operations. Such templates could then be reviewed for correctness either by the system itself using further active experimentation, or supplied to designers, integrators and operators for evaluation.
Our existing systems have not been built with the capabilities required in order to engage in active experimentation. Thus, an important research challenge is to implement such processes while preventing the resulting experiments from damaging some part of the SoS or compromising the service it provides. Implementation is particularly challenging when, as with space systems or the electrical grid, there is only one operating instance of the entire SoS.
In following sections we discuss several possible approaches for introducing active experimentation into engineered complex cyber-physical SoS. However, rst we list the additional capabilities required of such implementations. They are: { instrumentation at all levels of the hierarchy of complexity to measure what happens. { models that relate what is measured to properties or symbols that are local but have meaning that can be communicated to other parts of the SoS. { models that relate what is measured locally to higher-level purposes, goals and constraints. { the capability to retain the information produced by these measurements and models. { a hypothesis-generating engine that can propose possible actions (experiments). { a predictive capability to project and analyze the potential consequences of a proposed future action.
{ the ability to engage in a proposed action.
Taken together, these resources and capabilities would create an engineered SoS
able to reason about itself (its resources, capabilities, and limitations) in the
context of its current environment, purposes and goals, and also to both propose
and implement a course of action based on that reasoning rather than on
preprogrammed control strategies. [
These capabilities required for achieving active experimentation, taken
together, constitute computational re ection. [
The crucial question is, of course, how to implement this biologically-inspired
approach of active experimentation coupled with computational re ection to
improve and extend existing SoS, as well as to design, develop, integrate and
test new ones. We suggest two complementary strategies, both of which leverage
the capabilities we have listed above. One approach, which we are following
in our own work, is to build testbeds [
The other, and more advanced, strategy is to implement portions of the required capabilities locally in an already-operating system and monitor the proposed courses of action for compatibility with known \concepts of operations" (CONOPS), which are the di erent styles of use intended for the system. Since the cases of most interest are also SoS providing important services that cannot be interrupted, we suggest that, after testing at the subsystem level, such modi cations could be implemented during planned maintenance, update, or upgrade periods for the a ected portion of the SoS. We note that all critical systems have methods for implementing such planned modi cations. Addition of re ective capabilities and active experimentation could be implemented one step at a time, starting with re ection, but trapping the proposed modi cations instead of implementing them. Multiple periods of testing and review could be accomplished during successive maintenance periods, carrying out all of the necessary processes for implementation except executing the proposed actions. This strategy allows a period during which the proposed actions can be compared with known CONOPS for consistency throughout the entire SoS, providing a basis for veri cation and validation of the expected operation of the entire SoS once the new capabilities are allowed to a ect operation.
In a SoS that supplies a mission-critical service, we do not have the ability to isolate the whole system (with new incoming systems or capabilities and legacy systems) from its ongoing requirements within its true operational context. And yet, it is arguably even more critical that SoS, which are dynamic, which have many unknowns, which have constantly new combinations of legacy systems / components and new systems / components, have some \safe" places within which to actively try out component con gurations and to reason about and record / learn the impacts of such con gurations in matching their requirements and operational constraints.
Most SoS are modular and utilize redundancy to achieve robustness so that sections can go down without bringing the rest of the system down, and also can be routinely taken o ine for necessary check-out, maintenance, and upgrades. To leverage redundancy and maintenance periods for evaluation of the e ectiveness of new capabilities such as re ection and active experimentation, one would have to devise a simulation that would mimic the current operational settings. Combining emulation / simulation and protected operation are currently done for checking out space vehicles and their subsystems and components, as well as other similarly expensive systems that require testing within very realistic operational conditions. These operational simulations could be used to test the new combinations of components, capabilities and system integrations by a human system engineer using a set of pre-designed tests. Certainly this would have great advantages over the current practices in developing and testing SoS, changing it from a certi cation process into one of continual veri cation and validation.
We now discuss our testbed, how it enables us to implement both active experimentation and computational re ection, and how we can apply what we learn to the cases of complex SoS. 6
CARS (Computational Architectures for Re ective Systems) is a testbed that we
have been developing as an ongoing student project at California State
Polytechnic University, Pomona. [
A series of benchmark tasks are utilized for evaluation of CARS that span a broad range of sometimes con icting strategies: independent or multi-agent, cooperative or competitive, asynchronous or synchronous. Speci cally, we use the \games" follow-the-leader, tag, soccer practice (bump a ball into a designated goal), and push-the-box (move a large, heavy object that cannot be moved by any individual agent to a designated goal). We use Wrappings to implement computational re ection and self-modeling. Wrappings grew out of work on conceptual design environments for space systems, and has been in continuous development since its inception in 1989. [4, 13{15]
Some of the important characteristics of CARS are { The cost of each robotic motion platform (< $50) means that, unlike most deployed systems, the investment in any part of the system is relatively small. (A new agent can be prepared in less than a day from COTS hardware and the electronics of a damaged agent.) { The robotic components are relatively crude, requiring more modeling and self-re nement of generic models than better hardware. { The performance of the SoS for any task can be evaluated from recorded video of the \ eld of play." { The tasks and the appropriate performance measures are easy to express in everyday language. { Use of Wrappings frees experimenters from many of the detailed programming tasks normally associated with adding or modifying a process, model, or sensor interface.
What Wrappings provides here is the ability for the system to have multiple
alternative resources for any given problem, and to select them according to
their operational context at the time of use. Because the process that make
those selections are also resources, and are also selected just like any other,
these systems have a very strong kind of computational re ection [
We now speculate on the applicability of both the CARS testbed and the incremental approach as strategies for eventually implementing active experimentation and computational re ection in mission-critical SoS. 7
In the CARS testbed, we have the luxury of allowing the system and its agents in the true operational environment to practice, make mistakes, learn its characteristics (e.g., turning ratio, speed on di erent surfaces etc.), and even damage an agent without dire consequences to itself or to the rest of the testbed, somewhat as children learn their capabilities and the constraints of their various environments through play. However, in addition to pre-de ned test sets, we speculate that in fact the style of self-modeling, learning, and subsequent recording of new rules and constraints that we have advocated for the CARS testbed could become very useful for o ine testing and progressive integration of parts of a SoS. In our approach, each component and subsystem of the CARS is constantly developing better and better rules and constraints on its behavior and its allowable operational envelope. The result is that because the \experimentation" is being developed in parallel from the point of view of many di erent types of components playing their diverse roles, the system is very likely to discover much more about potential problems than a test set developed by even a knowledgeable and experienced system engineering team.
This kind of exploratory behavior is an extension of exploring the system's external environment to exploring the space of potential behaviors. Since this space is enormous, some very powerful directive constraint mechanisms will be needed to keep the system within some reasonable expectations, and some very powerful veri cation and validation methods will be needed to assure us that the system will accede to any safety- and mission- critical constraints we may choose to impose.
Eventually, we can envision a situation in which the components themselves when faced with a novel component interface or con guration or operational setting can request a time out, a voluntary removal of themselves to maintenance / self-examination / hypothesis generation and testing mode in a simulation. Imagine that in addition to the meta-knowledge normally provided to a SoS broker, each component / system has strong self-models at multiple time and space resolutions that are being continually re ned with interaction with other components and environments. Initially, as a new con guration of components is brought together with the top level descriptions provided to the broker in the Wrappings, there will now be a deeper process of negotiation among the components as their self-models now compare constraints, expectatins, rules for best practice, and other behavior modi cation and constraint conditions. If a component is now faced with either an unknown situation (a new condition for which it has no rules or constraints) or a partially violated constraint (whose priority might not be that high), it can request that the system allow it to temporarily go into maintenance mode.
Of course, to be able to entertain this type of negotiation will take more information in the self-model about that component's expected CONOPS, in addition to its expected environment. The system will either have some type of holding action it can take or it might request the broker to provide it a new component and go on. Meanwhile the o ine component now starts a set of experiments in the safe simulation, with current operational setting values and conditions and with either the other relevant software components (clones) and / or emulated hardware components. If its experiments go well enough (measured by seriousness of system use), then that component can go out of maintenance mode and back online. At that point, it tracks and records all the real results of its interactions in this new use or con guration for future rules and constraints. If the results of the experimentation are equivocal, then human intervention may be requested for further experiments.
To summarize then, we want to develop methods that allow and even encourage processes that continually improve the performance of a system through better use of its existing resources, the correct incorporation of new component resources, and the most appropriate integration among resources given the current operational context and CONOPS. 8
A study of biological systems suggests possible strategies for creating robust adaptive responses of a complex SoS to changing or even unanticipated conditions. In this paper, we focused on one such strategy: active experimentation. We have shown that successful active experimentation requires several specialized capabilities that, taken together, amount to computational re ection. We then proposed various strategies for implementing active experimentation and computational re ection in mission-critical systems of systems.
We have suggested that having testbeds like CARS allows components, subsystems, and systems to build ever-improving self-models based on active experimentation. The active experimentation coupled with the re ective reasoning processes allows these components / systems to develop and re ne rules and constraints with speci c details about di erent operating conditions and other components or systems.
We have then speculated that some of these approaches could be applied to mission-critical SoS by taking advantage of the o ine maintenance mode allowed for most SoS components / subsystems. This second best case is to have during maintenance, some way of setting up a safe operational environment (set of simulations and emulations) for the o ine components to actively experiment performing new behaviors, joining in novel con gurations of components, or experiencing new operational settings. This experimentation would help re ne the current self-models to take into account these new conditions.
The last case is to develop a new style of negotiation where components are out tted not only with their own constraints and behavioral rules, but also CONOPS that helps explicitly de ne the expectations for how this component is expected to be used under di erent circumstances. This negotiation would be going on in parallel across layers of systems and components, allowing many lines and types of detailed interactions to be analyzed by the self-modeling processes. In this last most speculative case, based on this negotiation, individual components would request being put into study mode (o ine maintenance mode and into operational simulation mode) in order to follow up on any con icts with current constraints or lack of information on requested behaviors.
We are hoping this paper will stimulate a community wide discussion into many di erent ways one could create safe places for self-modeling and experimentation resulting in better system integration, system validation, and system performance.