Training acting agents for the goal of controlling complex live systems on the system itself is often an unfeasible task, either due to the high cost or the potential dangers that might arise. In this paper, we take a step towards identifying ways to evaluate the transferability of models for the class of constrained Reinforcement Learning problems. Furthermore, we present an approach based on free-energy advantage functions to improve adaptability and in turn transferability for constrained Reinforcement Learning problems and subsequently manage to increase the performance of a baseline algorithm, CPO, with regard to safety constraints in noisy environments.
CEUR ceur-ws.org
AI systems can have significant real-world impact, and if not designed and deployed with safety
in mind, they can cause harm to individuals, organizations, or society as a whole. Ensuring
safety is crucial to prevent accidents, unintended consequences, or malicious uses of AI. When
deploying trained models to large-scale industrial applications, unstable live systems can cause
damage of economic or other nature. Because of the high complexity, cost, and potential danger
of training live systems from scratch, usually, these models are trained on historical or simulation
data, which may or may not accurately reflect the actual use case environment. Specifically,
in some instances, knowledge of the actual environment dynamics is only partially available,
and algorithms need to be able to handle situations where there is a degree of uncertainty.
Classically, in control environments, robustness can be achieved with Model Predictive Control
approaches ([
Reinforcement Learning (RL) is a machine learning paradigm that includes a variety of algorithmic approaches, foremost in sequential decision-making environments. Recently, RL has become a promising way to solve sequential decision-making tasks such as in marketing, gaming, and control tasks, such as robotics and autonomous cars, where the aspect of safety and trustworthiness in the agent is an important factor.
We argue that in real-world applications that require safety guarantees, RL methods that transfer well could improve upon satisfying certain thresholds. CEUR Workshop Proceedings
Transfer learning is an established concept in areas such as image classification and natural
language processing ([
In this section, we will introduce the mathematical framework for the problem setting.
Reinforcement Learning problems can typically be modeled with the help of a Markov Decision
Process (MDP) = (, , , , )
function ∶ × × → [
with a state space , an action space , a transition probability
, a discount factor ∈ [
To extend this to safety critical problems, one possibility is to introduce a constraint cost function ∶ × → ℝ
analogue to the reward function and a safety threshold ∈ ℝ . We define a Constrained Markov Decision Process (from now on referred to as CMDP) = . We can calculate a weighted return value for constrained problems with
with ∈ Π for the set of all policies Π and a be the set of policies that satisfy the constraint . Then we can (, , , , , , ) ( ) = ∼
∞ [∑=0 ( , )] of a policy ∶ → trajectory = ( 0, 0, 1, 1, … ).
Let Π = { ∈ Π ∶ ( ) ≤ } calculate the optimal policy ∗ = arg max ( ) .
∈Π
In real-life applications of Reinforcement Learning, environment dynamics, especially state transitions, can be unknown. Therefore, we introduce a generalization of the MDP model by assuming transition probabilities ,⋆ ∈ Δ for finite states and actions and probability simplex Δ ⊂ ℝ+. A common way to learn the objective under the assumption of unknown transition probabilities is to maximize a lower bound on the return. be written as ∗ = arg
∼ ,∼
[
In a mathematical sense, given a source domain S and a target domain T, Transfer Learning (TL) is used to learn an optimal policy ∗ for T by incorporating both external information from the source ℐS and internal information ℐT gathered from T. The optimal policy can (, )]
for initial set of states . Taylor and Stone [
In literature ([
In TL, domains can be described by MDPs, and any parts of it can have diferences between the source and target domain. Consider state spaces S and T. Any of these relations might be true, depending on the problem: S ⊂ T, S ≡ T or S ⊂ T. Diferences for the action spaces S and T are analogs. Since both state and action spaces can difer, reward functions can also be defined diferently for both domains. Ultimately, trajectories can difer for problems where reaching a goal can be achieved diferently (e.g., path-finding tasks).
This can be further extended to safety critical applications. Difering state spaces can be the result of failed sensors, difering action spaces are the result of hard constraints implemented by the system. Additionally, reward functions might yield diferent values in cases where sensors supply noisy data. In the case of CMDPs, for similar reasons, diferences can be found in both constrained cost function and safety threshold.
On the topic of which kind of knowledge is transferable, we can define multiple forms. The
transfer of trajectories is the main subject of LfD. Furthermore, the transfer of model dynamics
is possible when an approximation by ofline learning algorithms trained on historical data,
and before getting transferred to an online system, is feasible. Ofline RL algorithms usually
mitigate the impact of the gap between real and estimated values by adding a pessimism factor
to these learned values ([
The transfer of policies has been discussed by [
Transfer by starting from prior distributions has been explored by [
In real-world applications, such as robotics, it can be hard to separate signals from noise, especially at the early stages after deploying a learned strategy. We consider a scenario where there is a cost to receiving state data from an actor, e.g., sensor data from a robot’s joints. Since we are considering the case of a SimToReal-transfer, we assume the existence of priors learned from simulation interactions. In this section, we propose the use of an advantage function over the simulation priors based on the free-energy principle to improve the agent’s robustness.
Free-energy functions are fundamental concepts in thermodynamics and statistical mechanics that describe the energy available to do work in a system while accounting for both its internal energy and its entropy.
Rubin et al. [
With the help of information theory, we can quantify the expected cost of executing a policy in state ∈ as Δ () =
∑ T() log( T() ) with Δ ( ) = 0 for a terminal state . With this, we are able to measure the relative entropy between the source policy S and target policy T. The source policy is used by the agent in the absence of information from its new noisy environment. For any state , Δ () describes the minimal number of bits required to describe the outcome, or action sampled, of the random variable ∼ measure for the cost of control. Similar to the value function ( 0), we can define the total control information involved in executing policy starting from the initial state 0: T. In our case, it serves as a S() ( 0) = lim [Δ ( )] →∞ →∞ →∞ = lim [ ∑ log = lim [ log
T( ) S( )
] (
0, 1, … , −1 | 0, T) ( 0, 1, … , −1 | 0, S) ] Here, the optimal target policy ∗,T should minimize the control information cost and at the same time maximize the reward while respecting environmental constraints.
arg max ( ) makes sure that the new policy is within a so-called trust region of the previous one: +1 =
) ≤ . Here, Π ⊂ Π denotes a -parameterized policy
subset that filters for relevant parameters. Trust region algorithms for reinforcement learning
([
)[]), and > 0 is the step size.
The advantage functions calculates the expected reward gain along a trajectory and is given by: (, ) = (, ) −
() = ∼ [( )| 0 = , 0 = ] − ∼ [( )| 0 = ] The trust region is then defined by the set { ∈ Π ∶ ( ‖ )}.
CPO solves the CMDP problem approximately by calculating the update ( ‖ ) ≤ (2) (3) (4) (5) (6)
We define
We aim to use a free-energy function to derive optimal policies while balancing the tradeof between value and information during exploring.
Early works ([
(, ) = log T(|) . The expected information cost of the target policy T can be written as [ T( , |)] = ( T ‖ S). Considering the dynamics induced by the transition probabilities ( +1 | , ) of the underlying MDP, we can now consider the total discounted expected information cost for the target policy:
S(|) T() = ∞ ∑
=0 T(, ) = T() +
T ( ,
S ‖ , ).
1 T() as a -weighted free-energy function with controlling the tradeof between value and information. From this we get a state-action free-energy function T(, ) = [|, ] + [ T ( ′)|, ].
Now, we define the free-energy advantage function as: T(, ) = T(, ) − T() T
= ∼ T[( ) +
( +1 , +1 )| 0 = , 0 = ] − ∼ T[( )| 0 = ] Here, ( )
represents the cumulative sum of constraint costs along the trajectory .
Finally, we can calculate the free-energy advantage transfer policy update: (, )
In this section, we will present the evaluation framework, metrics and results.
In this section we will first evaluate the performance of the Constrained Policy Optimization
algorithm [17] for constrained RL problems. CPO yields better performance on constrained
tasks than methods such as Trust Region Policy Optimization or Primal-Dual Optimization
([
1. We use a continuous action space with = [
For safety-critical applications at any scale, the best direct improvement of TL would generally be starting from accurate prior distributions, because we can expect a reduced exploratory period. While this is expected to reduce training time, prevention of constraint violations is not necessarily guaranteed. Having reliable algorithms should also make it possible to train an agent in a simulation and then transfer the model to safety-critical applications in the real world without violating constraints imposed by the task. We, therefore, extend the list by the following measurements: • The ratio of total constraint cost accumulated by the transfer learner and total constraint cost accumulated by the non-transfer learner or between diferent transfer learners. • The sum of constraint violations committed by the transfer learner compared to the non-transfer learner (or between multiple transfer learners) above a specified threshold. Note that we hypothesize that measuring the robustness gained by simultaneously learning system dynamics ([19]) could a valid metric, which we intend to examine in the future.
We hence compare the CPO algorithm with and without free-energy advantage policy transfer (FEAT) on noisy environments with a noise factor ∼ (1, ) for every state variable index ∈ {1, … , ||} by evaluation the post-transfer performance according to the formerly proposed criteria. In all experiments, we first pre-train an agent with an implementation of the CPO algorithm in a simulated environment without noise for 2500 iterations. After the final iteration, the agent is able to control the HalfCheetah at a satisfactory level.
For the criterion of constraint violations, we define a constraint threshold . Like above, we train the agents for a total of = 1000 iterations. In a noisy environment with = 0.1 , we evaluate both agents with a strict safety threshold of = 0.02 . Here, the value for means that the HalfCheetah is not allowed to show signs of falling over. While CPO without FEAT violates the threshold 7.2% of the time, CPO with added FEAT evaluates at only 3.5%. For = 0.2 , we chose a higher threshold of = 0.15 (the agent is allowed to appear unstable, but is not allowed to fall over). CPO without FEAT violates the threshold in 86.7% of iterations, while CPO with FEAT is significantly lower, with only 32.3% violations. Unfortunately, both algorithms still lack the necessary robustness to guarantee safety for environments with higher noise levels.
In this paper, we highlighted how Transfer Learning can be interpreted in the context of constrained Reinforcement Learning and proposed a way that transferability can be evaluated. The experiments indicate that our approach improves the transferability of policies for constrained problems in the specific case of the Constrained Policy Optimization algorithm.
In the future, we aim to research further how this approach is applicable for similar policy based RL algorithms and extended this to a more general case. Furthermore, to reflect real-world problems more accurately, we plan to add further restrictions to the actor’s perception of the environment, such as partial observability.
This research has been funded by the Federal Ministry of Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence. New operators for reinforcement learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016. [16] R. Fox, A. Pakman, N. Tishby, Taming the noise in reinforcement learning via soft updates, arXiv preprint arXiv:1512.08562 (2015). [17] J. Achiam, D. Held, A. Tamar, P. Abbeel, Constrained policy optimization, in: International conference on machine learning, PMLR, 2017, pp. 22–31. [18] Y. Chow, M. Ghavamzadeh, L. Janson, M. Pavone, Risk-constrained reinforcement learning with percentile risk criteria, The Journal of Machine Learning Research 18 (2017) 6070–6120. [19] P. G. Sessa, I. Bogunovic, M. Kamgarpour, A. Krause, Mixed strategies for robust optimization of unknown objectives, in: International Conference on Artificial Intelligence and Statistics, PMLR, 2020, pp. 2970–2980.