Safe Reinforcement Learning via Probabilistic Logic Shields

Abstract

Safe Reinforcement learning (Safe RL) aims at learning optimal policies while ensuring that the agent stays safe. A popular solution to Safe RL is shielding, which uses a logical safety specification to prevent an RL agent from taking unsafe actions. Traditional rejection-based shielding techniques provide rigorous safety guarantees, however, they are dificult to integrate with continuous, end-to-end deep RL methods for three reasons. 1. Previous shielding approaches have been limited to symbolic state spaces. 2. Rejection-based shields are deterministic, assuming that any action is either safe or unsafe in a particular state. However, this is an unrealistic assumption as the world is inherently uncertain, and safety is often a matter of degree rather than an absolute concept.

Keywords

• probabilistic shields
• statistical relational AI
• probabilistic logic programming
• safe reinforcement learning

-

Italy

https://wenchiyang.github.io/ (W. Yang) program, as shown in Fig. 1. To ensure safety, we consider a set of noisy sensors surrounding the agent. These sensor are represented in the probabilistic logic program as neural predicates, which take an image as input and generate probabilities indicating the presence of obstacles or potential dangers. The program can then be automatically compiled into a diferentiable structure, allowing for the optimization of a single loss function through the shield, enforcing safety directly in the neural policy. Therefore, PLS can be seamlessly applied to any policy gradient algorithm while still providing the same convergence guarantees. Overall, probabilistic logic shields have the following benefits compared to rejection-based shields. • Realistic Safety Function. PLSs allow for risk control by using probabilistic safety measure. • Simpler Model. PLSs use a simpler safety model that only represents internal safety-related properties, which is less demanding than many model-based approaches that require the full MDP (e.g. [ 1 ]). • End-to-end Deep RL. PLSs are diferentiable and can be seamlessly applied to any modelfree RL agent such as PPO, TRPO, A2C, etc.

• Convergence. PLSs in deep RL provide convergence guarantees.

Our experiments show that applying PLS to policy gradients leads to safer and more rewarding policies compared to other state-of-the-art shielding techniques in diferent discrete and continuous Atari domains. The sources are available on https://github.com/wenchiyang/pls.

This work has been accepted at IJCAI 2023 Main Track. A preprint can be found on https: //arxiv.org/abs/2303.03226.

References

[1] N. Hunt , N. Fulton , S. Magliacane , T. N. Hoang , S. Das , A. Solar-Lezama , Verifiably safe exploration for end-to-end reinforcement learning , in: Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control , HSCC '21 , 2021 . URL: https: //doi.org/10.1145/3447928.3456653. doi:1 0 . 1 1 4 5 / 3 4 4 7 9 2 8 . 3 4 5 6 6 5 3 .