We present two learning methods for designing Cyber-Physical Systems with Learning-Enabled Components: one is imitation learning of feedback controllers, and the other is learning of temporal specifications.
Artificial intelligence (AI) and data sciences are revolutionizing information systems used for control and supervision of various devices (e.g., sensors, robots, IoT devices) with higher levels of autonomy in uncertain and dynamically changing environments. Among such information systems are Cyber-Physical Systems (CPS) from which emerges a new generation of AI-intensive Cyber-Physical Systems. There are a number of challenges in designing such systems. On one hand, the existing CPS design methodologies, relying on rather fixed models, face a fundamental problem because these systems containing learning-enabled components are supposed to learn from experience and interactions with the environment, and adapt their behaviors accordingly. It is thus imperative to ensure that their learning-enabled components work correctly. On the other hand, AI techniques are “unpredictable” due to a lack of formal framework to provide safety guarantees. In general, the outcomes of learning activities in AI components (e.g., deep neural nets) are not well-understood and interpretable. When coupling CPS with AI, the increased heterogeneity in dynamics and behaviors can aggravate the reliability and explainability issues, if the learning activities are not properly formalized.
In this work we present our recent results in this direction, namely making design of
learningenabled components in AI-intensive CPS more formal. More concretely, we first describe a
method for imitation learning of feedback controllers satisfying temporal specifications. We
then describe an approach for learning temporal specifications from labelled data. In both of
these learning problems, temporal specifications are given in Signal Temporal Logic (STL) [
We focus on the problem of training a neural network (NN) (playing the role of a learner)
to imitate a complex controller (playing the role of an expert). The ultimate goal is to replace
this complex controller with a trained NN. NNs have long been used to control dynamical
systems from inverted pendulums to quadcopters, learning from scratch to control the plant
by maximizing an expected reward, e.g. [
To make such an imitation learning framework more formal and more eficient, we add the following novel features: (i) a formalization of performance evaluation for both the learner’s and expert’s policies using their abilities to satisfy requirements specified by temporal logic, (ii) a leverage of the power of existing temporal logic property falsification tools to create correct training data, (iii) a new method of data aggregation in order to guarantee a good performance of NN in terms of imitation and generalization.
To explain these features, we point out some major dificulties in this problem. Training a NN to imitate a feedback controller is more complex than the problem of approximating a function using pairs of input and output values, since feedback controllers can themselves be stateful dynamical systems. We identify the following dificulties in data generation by executing the nominal controller in closed loop: • Infinite behavior space. The behavior space is not only large but can also be infinite. It is thus important to define a coverage measure to quantify how representative the generated training data is. • Non-uniform accuracy. Depending on the control requirement, the NN may need to be very precise around some region of the state space while in other regions a rough approximation is acceptable.
Formal control requirements. The problem of non-uniform accuracy is particularly
pronounced when the requirement depends on time or sequences of events. This is frequently the
case in control applications, where properties such as rise time, settling time, and overshoot are
typical. We consider complex properties that can include not only time but also causal
relationships. They can be conveniently described in Signal Temporal Logic (STL) [
Additionally, in order to allow some quantitative behavioral flexibility, we use parametric STL (PSTL) to specify the expected properties of a nominal controller. Then, we want to train a neural network-based controller achieving performance comparable to that of the nominal controller, as measured by valid parameters of the PSTL requirement. Parametric STL (PSTL) is a variant of STL which makes it possible to replace numeric constants in an STL formula with symbolic variables or parameters. For instance, the formula = □ [0, ](‖()‖ < ) with two parameters and expresses the requirement that during seconds, the norm of signal should be less than . For example, (2) defines a formula requiring that if the system is not stabilizing (that is, the formula st is not satisfied), then it should eventually stabilize, i.e., after at most tr seconds, st should remain true for at least st seconds.
st := ‖()‖ < st
st := ¬ st ⇒ ♢ [0, tr]□ [0, st] st
(1)
(2)
Control policy performance measure and imitation learning quality. In imitation
learning, it is essential to have an appropriate measure of performance of control policies,
especially when it is unclear what reward function is being optimized [
Finally our state feedback controller imitation learning can be formally stated as follows. Given a plant , a nominal controller such that the closed-loop system || satisfies a PSTL specification Φ , our problem is to learn a neural network controller to imitate such that the closed-loop system || satisfies Φ , and the performance similarity ,Φ(, ) is as small as possible.
Learning guidance. The learning guidance here is provided using positive examples, i.e. good behaviors, generated by the nominal controller which already satisfies the desired requirement.
While observing closed-loop behaviors may reduce the number of behaviors to be sampled,
we still need to find good training samples that are relevant to an STL property. To do this, we
ifnd counter-examples that are closed-loop behaviors violating this property by leveraging the
existing falsification tools [
Finally, we demonstrate our approach on a robotic case study where a NN controller is designed to imitate a model-predictive controller.
The success of learning in AI has also impacted the field of formal modelling and specification.
We focus here on supervised learning, and use parametric temporal logics (PSTL) [
Parametric Pattern Predictors (PPP) can be defined as specifications, for a given value of
parameters, take an observation and produce a Boolean signal which is true where a pattern
is predicted an false elsewhere [