This paper discusses the challenges of applying reinforcement techniques to real-world environmental monitoring problems and proposes innovative solutions to overcome them. In particular, we focus on safety, a fundamental problem in RL that arises when it is applied to domains involving humans or hazardous uncertain situations. We propose to use deep neural networks, formal verification, and online refinement of domain knowledge to improve the transparency and eficiency of the learning process, as well as the quality of the final policies. We present two case studies, specifically (i) autonomous water monitoring and (ii) smart control of air quality indoors. In particular, we discuss the challenges and solutions to these problems, addressing crucial issues such as anomaly detection and prevention, real-time control, and online learning. We believe that the proposed techniques can be used to overcome some limitations of RL, providing safe and eficient solutions to complex and urgent problems.
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License ature; however, our drone is modular and diferent types Attribution 4.0 International (CC BY 4.0). 1. Introduction In recent years, Reinforcement Learning (RL) has emerged as a powerful technique to solve complex problems in a variety of applications, reaching and often outperforming classical algorithms and humans. Successful stories include chess and Go [1], video games [2], and more [3, 4]. One fundamental trend of research in RL addresses the problem of safety, i.e., the application of
tion with humans, hazardous situations and expensive hardware. This is the case, for instance, of robotics and autonomous driving [5]. In fact, standard model-free RL approaches just aim at maximizing a given reward signal, computing an optimal strategy or policy for the task.
guarantees [6]. whelming requirements in terms of data availability, storage, and power, ultimately afecting environmental susespecially in systems with limited hardware resources.
In this paper, we propose techniques to address the problem of safe RL with limited resources, leveraging symbolic artificial intelligence, formal verification tools, and online refinement of domain knowledge to improve the transparency and eficiency of the learning process, 2.
Water Monitoring we address in this paper, the autonomous water monitoring. This task presents diferent challenges that range we used for our experiments, a diferential drive platform equipped with sensors and actuators that allows the boat to navigate autonomously in the environment and collect data on the water’s quality. In more detail, the platform is based on a hull equipped with two in-water propellers that allow navigation in shallow water with a max velocity of 3/ . The set of sensors for localization includes a GPS, compass, accelerometer, and gyroscope, with the addition of a sonar and a stereo camera for collision avoidance. Regarding water quality, the base set includes PH, dissolved oxygen, conductivity, and temper
Moreover, a crucial limitation of RL lies in the over- In this section, we introduce one of the two problems tainability and large-scale adoption of such techniques, from hardware to software [ 7]. Fig. 1 shows the drone we first consider the following dynamic model of the agent: =̇ cos( ) =̇
sin( ) =̇ − =̇
(1) where , are the boat position in earth frame, is the motors thrust, is the boat mass, is the angular velocity in body frame and is the water drag coeficient.
Let now = [, , , ]
the state vector of the system.
During DRL training phase, the DRL agent explores different actions ⟨ ⟩ , which are the input to a feedback inverse dynamic control loop incorporating the drone model. In this way, the low-level commands at the drive of the drone are ultimately sent by a model-based conobtains a new dynamically correct state vector with the associated reward value for the specific action. For the reward signal, we adapted a well-known function in the context of mapless navigation, formally:
1 = {−1 reaches the end collision with an obstacle
(2) ( −1 − ) ⋅ −
otherwise where is the distance from the target position at time , is a normalization factor used to guarantee the stability of the gradient, and is a fixed value, decreased at each time-step, resulting in a total penalty proportional to the length of the path.
of interactions between an agent and its environment. Training the agent directly on the actual platform is often unfeasible, particularly in a robotic scenario where a failure during the early stages of training cannot be tolerated. To overcome this challenge, we have developed a realistic simulator using the Unity3D engine. Leveraging the built-in physics engine, our simulator can accurately simulate water, waves, and other weather phenomena (Fig. 1 shows a screenshot of our simulator).
Our drone is designed to be easy to deploy for non- troller, guaranteeing their feasibility. The DRL agent then (a) Training simulator
(b) Real aquatic drone expert users; therefore, a high level of autonomy is required. The main objective is to provide a platform to be released into a reservoir, able to navigate and monitor water parameters and potential sources of pollution with very limited human intervention [7]. However, autonomous navigation in the aquatic scenario is challenging due to the non-stationarity of the environmental conditions such as waves, wind, light reflections, moving obstacles (both above and below the surface), and more [8]. Hence, safety is of utmost importance to avoid catastrophic failures, e.g., collisions causing water to seep into the hull and severely damaging electrical components [9].
In the following sections, we will propose techniques from safe RL and particularly safe DRL, i.e., RL based on
tions, we propose to exploit DRL to generate a DNNbased real-time controller for the water monitoring drone.
DRL is a unique paradigm for the training of DNN. Its fundamental characteristic is that it does not require a set of labeled data; in contrast, an agent (the drone in our case) interacts with the environment, with a trial-anderror process, learning from its mistakes and improving the policy (i.e., the strategy to solve the task), driven only by a reward function that it aims to maximize [11]. Using model-free DRL to learn low-level control commands for robots is known to be a challenging problem, also due to electrical and mechanical hardware limitations posing safety constraints [12]. Hence, we integrate DRL with a low-level controller for , , respectively the linear velocity and the heading (yaw) angle of the drone. Specifically, (a state-of-the-art approach), called GPPO and PEPPO, stream from onboard sensors and find relevant patterns that exploit methodologies based genetic approaches to useful for decision-making and anomaly recovery or preenhance the training phase, showing promising results on vention. However, interpreting a large amount of heteroour navigation tasks [7]. Fig. 2 shows a plot of the results geneous data and handcrafting efective reward specifiobtained with our approach; even though reward-wise cations for RL accordingly is often very challenging [ 22], the performance is good, one limitation of the standard especially in complex, uncertain, and dynamic environDRL approaches is that they are oriented only on the ments or when the reward is sparse [23]. performance, often failing to provide guarantees on high- We have addressed the problem of online anomaly delevel safety criteria, which are crucial for water drones. A tection for water drones using hidden Markov models possible solution is to exploit constrained reinforcement combined with normalized Hellinger distance [24], relearning approaches, such as Lagrangian-PPO and CPO sulting in robust anomaly recognition when conducting [14]. In the following section, we propose an approach exploration campaigns over multiple days. The anomaly for ex-post verification of safety after training. signal can be used to provide an early reward in RL and prevent possible faults.
Formal Verification of DNNs One limitation of purely data-driven methods for anomaly detection is the lack of interpretability. To adDNNs have shown impressive performance in various dress this problem, we have started to investigate the use tasks. However, the vulnerability of these models to of Satisfiability Modulo Theory (SMT) [ 25] and Inductive adversarial inputs is a well-documented phenomenon Logic Programming (ILP) [26] to automatically identify observed across various applications [15]. Formal Ver- high-level logical patterns in data. ILP allows to easily ification (FV) of DNNs uses mathematical methods to incorporate domain knowledge, increasing data and comrigorously prove that a neural network meets certain putational eficiency. Moreover, it can provide situational safety and reliability requirements expressed as input- and policy explanations [27], which can be useful to more output relationships [16]. In more detail, the goal of efectively communicate with on-coast crew in case of the FV is to provide guarantees that a DNN will behave emergency. We have preliminarily explored the use of correctly and predictably under all possible inputs and ILP to detect behavioral patterns in paradigmatic simuconditions. This is especially important in safety-critical lation tasks involving an autonomous agent operating applications such as autonomous driving or medical pro- in an uncertain environment with sparse rewards [28]. cedures, where any error or unexpected behavior of these Starting from the definition of high-level commonsense models could have catastrophic consequences. The FV concepts about the domain of interest, we have collected of DNNs is still a challenging and active area of research. traces of normal executions (i.e., state-action pairs) of One of the main challenges is the computational com- the agent and identified logical rules matching actions plexity of verifying very large networks [17]. Hence, and environmental concepts. Rules have proven useful to researchers are actively developing new techniques and guide the exploration process in unexperienced settings, tools to make formal verification more practical for real- achieving an improvement in computational time and world applications [16, 18, 19]. In particular, we devel- the final reward. Our approach can be extended to RL, usoped an interval propagation-based method called ing rules learned from normal drone executions to guide [20]. Our algorithm is more computationally eficient optimal policy search and early identifying anomalous resince it performs parallel verification on sub-intervals of gions of the state-action space. Moreover, ILP can be used the input space. Based on the results obtained with , ofline to generate explanations for registered anomalous we extended the classic DNN-Verification problem to a behaviors and provide useful contrastive explanations new type of verification called #DNN-Verification or quan- [29], as well as to discover high-level safety specifications titative verification [21], which aims not only to discover implicitly embedded in the nominal behavior. a potentially single unsafe configuration but to count (or enumerate) all the unsafe areas in a particular region of the input space expressed by the safety property. This 3. Air Quality Management type of verification allows estimating the probability that the agent violates specific safety properties and selecting fully safe models before final deployment.
managing air salubrity to ensure comfort and safety in closed environments. Air quality and thermal comfort control are important features of modern Heating, VentiAnomaly Detection and Recovery lation and Air Conditioning (HVAC) systems. The spreadWater drones may face unpredictable challenges, e.g., bat- ing of the SARS-Cov-2 pandemic highlighted the importery drainage, unexpected obstacles, and adverse weather tance of air quality conditions in indoor environments conditions. Thus, it is crucial to analyze the real-time since it has a positive impact in reducing virus spread.
We consider the example of a company meeting room that must be booked in advance for a given number of people. The room is equipped with sensors for measuring concentrations of CO2 and Volatile Organic Compounds (VOCs), the indoor temperature and the outdoor temperature (from weather forecast). Moreover, actuators to open/close windows and turn on/of vents and sanitiz- aFpigpulyrieng4:thDeetsatirlaotengtyhpereofedcutcoefdthbey aocutruaRtLorapcopnrotraoclhparonfdileby ers are available. We assume to have a daily occupation an expert on VOC concentration. schedule of the room (i.e. number of persons present in the room at each hour of the day).
In the following, we first assume to have an approximate model of the scenario (e.g., provided by experts), actions suggested by the expert (on the right). MCTS expressed as a Markov Decision Process (MDP), and use obtains better performance than the expert by leveraging Monte Carlo Tree Search (MCTS) [30] to compute the its simulation-based nature. Moreover, as highlighted policy. Then, we propose a methodology for data-driven by green boxes, we can see that the activation of highSafe Policy Improvement (SPI) online, exploiting large intensity ventilation and the sanitization process (third amounts of data available from environmental sensors row) causes a sudden decrease in VOCs concentration [31, 32], with the potential to deal with model inaccura- (second row). cies. An overview of our approach to air quality management is depicted in Figure 3. Safe Policy Improvement
Monte Carlo planning Algorithms for safely improving policies are important to MCTS is a popular algorithm for optimal decision making deploy reinforcement learning approaches in real-world in large state / observation spaces. It performs forward scenarios (e.g., autonomous cars, drones, or industrial simulations from the current state of the environment, in plants) where safety, robustness, and reliability of conorder to compute the best action at each step maximizing trol policies are crucial issues. Safe RL investigates how the expected reward. Our research deals with probabilis- these issues can be addressed by learning policies that tic planning and reinforcement learning methods based maximize expected return while ensuring minimal peron MCTS in both completely observable and partially ob- formance level or respecting safety constraints. servable [33, 34] environments, and related applications We focus on Safe Policy Improvement (SPI) [35] for to real-world problems. MDPs =< , , , , > , where the agent is pro
Figure 4 shows, for instance, a detail of an example of vided with a baseline policy 0 and dataset of trajeca control profile generated by MCTS (on the left) and by tories =< , , , ′ >∈[1, ] collected running in the an expert (on the right) and their efects on an environ- real environment. SPI computes a new policy that mental variable, i.e., VOCs concentration. On the first outperforms the baseline 0 safely with an admissible row, we can observe the room occupancy profile, that is, performance loss ∈ ℝ + and confidence level 1 − , with the number of people (blue line) in the room at a given 0 ≤ ≤ 1 and loss ( 0, ) − ( , ) . time of the day. In the second row, we represent the Safe Policy Improvement with Baseline Bootstrapping evolution of VOCs concentration, and finally in the third (SPIBB) [36, 37] is a state-of-the-art method that conrow, the actions that are performed. As we can see in the siders the worst-case scenario reformulating the persecond row, VOCs concentration is always significantly centile criterion [38] to make the search for an eficient below the maximum threshold (orange line) by apply- and provably-safe policy tractable. SPIBB splits stateing the policy produced by MCTS (on the left), while it action pairs into two subsets: the bootstrapped subreaches the maximum threshold value by performing the set ℬ = {(, ) ∶ (, ) < ∧} is the set of stateaction pairs that occur less than ∧ times in ; the non- [2] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, bootstrapped set ℬ = {(, ) ∶ (, ) ≥ ∧} is the set I. Antonoglou, D. Wierstra, M. Riedmiller, Playof state-action pairs that occur at least ∧ times in . ing Atari with Deep Reinforcement Learning, 2013. The approach guarantees that is a -approximate Technical Report. https://arxiv.org/abs/1312.5602. safe policy improvement of the baseline 0 with high [3] A. Pore, D. Corsi, E. Marchesini, D. Dall’Alba, probability 1− , where depends on ∧ and . A. Casals, A. Farinelli, P. Fiorini, Safe Reinforce
Our current research deals with extending state-of-the- ment Learning using Formal Verification for Tisart safe policy improvement algorithms to enable their sue Retraction in Autonomous Robotic-Assisted applicability to real world problems. This manily requires Surgery, in: Proc. IEEE/RSJ Int. Conf. on Intellithe scaling of the algorithms to very large state-spaces. gent Robots and Systems (IROS), 2021. We are currently evaluating diferent methodologies for [4] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, improving the scaling capabilities of safe policy improve- O. Ronneberger, K. Tunyasuvunakool, R. Bates, ment methods and trying to evaluate the capabilities of A. Žídek, A. Potapenko, et al., Highly accurate such approaches to work on real-world domains with protein structure prediction with alphafold, Nature several states, domains having partially observable states (2021). and domains with multiple agents. [5] A. Sallab, M. Abdou, E. Perot, S. Yogamani, Deep reinforcement learning framework for autonomous driving, Electronic Imaging (2017). 4. Final Remarks [6] D. Corsi, E. Marchesini, A. Farinelli, Formal Verification of Neural Networks for Safety-Critical Tasks in In this paper, we tackled the pressing issue of exploit- Deep Reinforcement Learning, in: Proc. 37th Conf. ing AI, and particularly DRL, to realize advanced solu- on Uncertainty in Artificial Intelligence (UAI), 2021. tions for sustainability. Specifically, we examined two [7] E. Marchesini, D. Corsi, A. Farinelli, Benchmarking complex scenarios: water monitoring with autonomous safe deep reinforcement learning in aquatic navigadrones and automatic air quality management indoors. tion, in: 2021 IEEE/RSJ International Conference We proposed methodologies prioritizing safety as a cru- on Intelligent Robots and Systems (IROS), 2021. cial requirement for cyber-physical systems operating in [8] L. Steccanella, D. D. Bloisi, A. Castellini, A. Farinelli, real environments in the presence of humans. We com- Waterline and obstacle detection in images from bined model-free DRL with tools from formal verification, low-cost autonomous boats for environmental monstandard control theory, and model-based planning. itoring, Robotics and Autonomous Systems 124
Our current and future research is devoted to opti- (2020) 103346. mizing and further testing our algorithms in real-world [9] R. S. Sutton, D. McAllester, S. Singh, Y. Mansour, contexts, and exploring more trustable and transparent Policy gradient methods for reinforcement learning interaction with humans exploiting techniques from ex- with function approximation, Advances in neural plainable AI, e.g., ILP and logics. Our ultimate goal is to information processing systems (1999). deploy advanced and sustainable systems for the safe- [10] R. S. Sutton, A. G. Barto, Reinforcement learning: guard of both human beings and the environment. An introduction, MIT press, 2018. [11] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, Acknowledgments G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, D. Silver, Rainbow: Combining improveThe research has been partially supported by the fol- ments in deep reinforcement learning, in: 32nd lowing projects: SAFE PLACE, POR-FESR 2014-2020, Re- AAAI conference on artificial intelligence, 2018. gione del Veneto; PON “Ricerca e Innovazione 2014-2020”, [12] L. Brunke, M. Greef, A. W. Hall, Z. Yuan, S. Zhou, Italian Ministry for University and Research (grant agree- J. Panerati, A. P. Schoellig, Safe learning in robotics: ment No. 40-G-14702-1); “INEST Interconnected Nord- From learning-based control to safe reinforcement Est Innovation”, Piano Nazionale della Ricerca (PNR). learning, Annual Review of Control, Robotics, and Autonomous Systems (2022). [13] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, References O. Klimov, Proximal Policy Optimization Algorithms, 2017. Technical Report. http://arxiv.org/abs/ [1] D. Silver, J. Schrittwieser, K. Simonyan, 1707.06347.
I. Antonoglou, A. Huang, A. Guez, T. Hu- [14] D. Corsi, R. Yerushalmi, G. Amir, A. Farinelli, bert, L. Baker, M. Lai, A. Bolton, et al., Mastering D. Harel, G. Katz, Constrained reinforcement learnthe game of go without human knowledge, Nature ing for robotics via scenario-based programming, (2017). arXiv preprint arXiv:2206.09603 (2022). [15] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Er- ence on Trust, Privacy and Security in Intelligent han, I. Goodfellow, R. Fergus, Intriguing properties Systems and Applications, IEEE, 2021, pp. 269–278. of neural networks, arXiv:1312.6199 (2013). [28] G. Mazzi, D. Meli, A. Castellini, A. Farinelli, Learn[16] C. Liu, T. Arnon, C. Lazarus, C. Strong, C. Barrett, ing logic specifications for soft policy guidance in M. J. Kochenderfer, Algorithms for verifying deep pomcp, in: Proceedings of the 22nd Conference neural networks, Foundations and Trends® in Op- on Autonomous Agents and MultiAgent Systems, timization (2021). 2023, p. in publication. [17] G. Katz, C. Barrett, D. L. Dill, K. Julian, M. J. Kochen- [29] J. Rabold, M. Siebers, U. Schmid, Generating conderfer, Reluplex: An eficient smt solver for veri- trastive explanations for inductive logic programfying deep neural networks, in: International con- ming based on a near miss approach, Machine ference on computer aided verification, Springer, Learning 111 (2022) 1799–1820.
2017, pp. 97–117. [30] C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, [18] S. Wang, H. Zhang, K. Xu, X. Lin, S. Jana, C.-J. Hsieh, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, J. Z. Kolter, Beta-CROWN: Eficient bound propaga- S. Samothrakis, S. Colton, A survey of monte carlo tion with per-neuron split constraints for complete tree search methods, IEEE Transactions on Compuand incomplete neural network verification, Ad- tational Intelligence and AI in Games 4 (2012) 1–43. vances in Neural Information Processing Systems doi:10.1109/TCIAIG.2012.2186810. 34 (2021). [31] A. Zanella, N. Bui, A. Castellani, L. Vangelista, [19] P. Henriksen, et al., Deepsplit: An eficient splitting M. Zorzi, Internet of things for smart cities, IEEE method for neural network verification via indirect Internet of Things Journal 1 (2014) 22–32. doi:10. efect analysis., in: IJCAI, 2021, pp. 2549–2555. 1109/JIOT.2014.2306328. [20] D. Corsi, E. Marchesini, A. Farinelli, Formal verifi- [32] M. Capuzzo, A. Zanella, M. Zuccotto, F. Cunico, cation of neural networks for safety-critical tasks M. Cristani, A. Castellini, A. Farinelli, L. Gamberini, in deep reinforcement learning, in: Conference on Iot systems for healthy and safe life environments, Uncertainty in Artificial Intelligence (UAI), 2021. in: 7th Forum on Research and Technologies for [21] L. Marzari, D. Corsi, F. Cicalese, A. Farinelli, The Society and Industry Innovation (RTSI), 2022. #dnn-verification problem: Counting unsafe inputs [33] A. Castellini, G. Chalkiadakis, A. Farinelli, Influfor deep neural networks, ArXiv abs/2301.07068 ence of State-Variable Constraints on Partially Ob(2023). servable Monte Carlo Planning, in: IJCAI 2019, [22] H. Zou, T. Ren, D. Yan, H. Su, J. Zhu, Learning task- Macao, China, August 10-16, 2019, ijcai.org, 2019, distribution reward shaping with meta-learning, in: pp. 5540–5546.
Proceedings of the AAAI Conference on Artificial [34] M. Zuccotto, A. Castellini, A. Farinelli, Learning Intelligence, volume 35, 2021, pp. 11210–11218. state-variable relationships for improving pomcp [23] M. Grześ, Reward shaping in episodic reinforce- performance, in: Proceedings of the 37th ACM/SIment learning, in: Proceedings of the 16th Con- GAPP Symposium on Applied Computing, SAC ’22, ference on Autonomous Agents and MultiAgent Association for Computing Machinery, New York, Systems, 2017, pp. 565–573. NY, USA, 2022, p. 739–747. [24] D. Azzalini, A. Castellini, M. Luperto, A. Farinelli, [35] P. S. Thomas, G. Theocharous, M. Ghavamzadeh, F. Amigoni, Hmms for anomaly detection in au- High confidence policy improvement, in: Proceedtonomous robots, in: Proceedings of the 19th Inter- ings of the 32nd International Conference on International Conference on Autonomous Agents and national Conference on Machine Learning (ICML MultiAgent Systems, 2020, pp. 105–113. 2015), JMLR.org, 2015, pp. 2380–2388. [25] G. Mazzi, A. Castellini, A. Farinelli, Identification of [36] R. Laroche, P. Trichelair, R. Tachet Des Combes, unexpected decisions in partially observable monte- Safe policy improvement with baseline bootstrapcarlo planning: A rule-based approach, in: Pro- ping, in: Proceedings of the 36th International ceedings of the 20th International Conference on Conference on Machine Learning (ICML 2019), volAutonomous Agents and MultiAgent Systems (AA- ume 97 of Proceedings of Machine Learning Research, MAS), International Foundation for Autonomous PMLR, 2019, pp. 3652–3661.
Agents and Multiagent Systems, 2021, p. 889–897. [37] T. D. Simão, R. Laroche, R. Tachet des Combes, Safe [26] S. Muggleton, L. De Raedt, Inductive logic program- Policy Improvement with an Estimated Baseline ming: Theory and methods, The Journal of Logic Policy, in: Proc. AAMAS, IFAAMAS, 2020, pp.
Programming 19 (1994) 629–679. 1269–1277. [27] A. Drozdov, M. Law, J. Lobo, A. Russo, M. W. Don, [38] E. Delage, S. Mannor, Percentile optimization for Online symbolic learning of policies for explainable Markov Decision Processes with parameter uncersecurity, in: 2021 Third IEEE International Confer- tainty, Operations Research 58 (2010) 203–213.