Traditional reinforcement learning methods optimize agents without considering safety, potentially resulting in unintended consequences. In this paper, we propose an optimal actor-free policy that optimizes a risk-sensitive criterion based on the conditional value at risk. The risk-sensitive objective function is modeled using an input-convex neural network ensuring convexity with respect to the actions and enabling the identification of globally optimal actions through simple gradientfollowing methods. Experimental results demonstrate the eficacy of our approach in maintaining efective risk control.
Over the past decade, reinforcement learning (RL) has
In this study, we focus on learning a risk-sensitive
actorachieved notable advancements [
In this paper, we explore this approach by training
a policy that optimizes a risk-sensitive criterion based
on the conditional value at risk (CVaR). Traditional RL
algorithms typically aim to maximize the expected value
over all future cost-returns [
Additionally, we propose an actor-free architecture
in which the action is implicitly defined as the
solution to a convex optimization problem approximating
the risk-sensitive criterion. This eliminates the need for
incremental actor learning, which often necessitates
hyperparameter tuning and tricks to stabilize the training
process. With our actor-free approach, the policy aligns
optimally with the approximated criterion. This deviates
from prior research on CVaR-based safe RL [
We model the RL agent and its environment as a constrained Markov Decision Process (CMDP), represented by a tuple (, , , , , , ) where ∈ R is the × → : × → × → safety threshold and
R is the action space, : × R is the probabilistic transition function,
R is the immediate reward function, : R is the immediate cost function, ∈ R is the ∈ (0.0, 1.0) is the discount factor.
The goal of the agent under the CMDP framework is to learn a policy that maximizes the expected return given an upper bound on the (safety violation) cost, maximize E∼ ,∼ () ∑︁ (, ) ≤ .
]︃ ]︃ (1) where is the stationary distribution over the state at Risk
In CMDP, the safety violation costs are usually the
(in)finite-horizon discounted future cost-return as shown
in (1). However, only considering the expected value is
inmethods prefer a policy with lower cost, but also higher
variance, over a policy with slightly higher cost but much
lower variance. In other words, since higher variance
amounts to higher risk, the policy is not risk-averse. To
incorporate risk, we replace the expectation in the safety
violation cost with the Conditional Value at Risk (CVaR)
[
∈ (0, 1] is used to define the risk level, is
Calculating the CVaR measure directly for lengthy time
horizons using, for example, sampling would be
excessively costly [
︀( (, ), 2 (, ))︀ where (, ) =
=
E [∑︀∞
future cost-return and 2 (, ) its variance. This
Gaus − (, ) | = , = ] is the expected
sian distribution leads to a closed-form CVaR measure of
future cost-return [
problem is a nonsmooth exact penalty formulation of the constrained problem in (4). It is well-known that for suficiently large
the two problems have the same solution [10]. Moreover, since this problem is convex, ifnding a globally optimal action is tractable.
We first define the PICNN over state-action pairs (, ; ) where is convex in action but not convex in state . Figure 1 illustrates the simple convex net(2) work structure used in our paper. As shown in the figure, output 3 can be calculated by forwarding the network, 0 = , +1 = ˜(˜ + ˜), 1 = 0(0 + 0 + 0), = 0, 1 +1 = (
+ + ), = 1, 2 (6)
where are weight matrices, are bias terms, is the
nonlinear activation function and 3 is the output of
the network which is made convex in the input by
restricting the weight matrices 0 and to be
nonnegative and the activation function to be convex and
non-decreasing, e.g. a rectified linear unit (ReLU).
where (, ) = E [∑︀∞
=
− (, )| = , =
] is the expected future return. To learn the mean and
variance of cost-to-go, a distributional critic is learned
with 2-Wasserstein distance as the loss function [
Policy gradient algorithms typically feature an
actorcritic structure, utilizing two distinct neural networks
known as the actor and the critic [
* = arg min − (, ) + max{0, Γ(, ) − } (5) approximate − (, ) and another to estimate the cost- to the distance between the reference signal and the s a ˜ W
1 Wzs 0 Wza 0 Wza 1 Wza 2 u
1 z 1 ˜ W
2 Wzu 0 Wzz 1 u
2 z 2
Wzu 1 Wzz 2 z 3
The above method was evaluated in simulation on a continuous control task: cascade water tank level control.
As depicted in Figure 2a, the task is to maintain a specific water level in the lower tank by changing the input signal represents the voltage to the pump. The state of the system includes the heights of the two tanks, () = [︀ 1()
2()]︀ and the output is () = 2().
The reward function contains two parts, one related output, and the other to the cost incurred by a critical event. To introduce a risk element, we define a “critical” event as the level of the upper tank exceeding crit = 10 cm. Thus the reward and cost functions are defined as (a) The water tank (b) Training cost-return (c) Training return (d) Evaluation (, ) = −| 2 − |, and (, ) = 1 − crit, respec- by a Gaussian distribution in order to get a closed-from tively. The long-term safety threshold is set to − 250. of CVaR. Future research could explore a more general The system was discretized using the Euler method in the distribution. simulation, with a sampling period of 2 seconds. Unlike a real-world tank, there are no upper bounds for 1, 2 in the simulation. References
Our learning algorithm to update reward critic and
safety critic is based on Twin Delayed Deep
Deterministic policy gradient algorithm (TD3) [
We compare our method with its actor-critic version, CVaR-TD3, using a standard neural network instead of the PICNNs for actor, critic, and safety critic. The training results are illustrated in Figure 2 and Figure 2b with two metrics, average episodic returns, and average episodic cost-returns. It can be observed that CVaR-TD3 has a much higher variance, potentially attributed to the neural network-structured actor getting stuck in poor local minima, while our approach can identify the globally optimal action, aided by the PICNNs. Careful tuning of the hyperparameters could potentially mitigate this. Figure 2d shows the evaluation of the learned policy. To highlight the diferences between the methods, the state 1, 2 is clipped with a maximum value 20. We observe that our method with = 0.1 exhibits a more conservative behavior, striving to maintain a safe distance from the critical level (dashed red). Consequently, our approach is also slightly farther from the goal level (dashed blue), which is close to the critical level.