Responsible artificial intelligence is the next challenge of research to foster the deployment of autonomous systems in the real world. In this paper, we focus on safe and explainable design and deployment of autonomous agents, e.g., robots. In particular, we present our recent contributions to: i) safe and explainable planning, leveraging on safe Reinforcement Learning (RL) and neurosymbolic planning; ii) efective deployment of RL policies via model-based control; iii) formal verification of the safety of deep RL policies; and iv) explainable anomaly detection of complex real systems.
are represented as a Markov Decision Process (MDP)
= ⟨, , , ⟩, defining respectively the state space,
the action space, the transition map, and the reward map.
The first approach is based on Safe Policy Improvement
(SPI) [
Artificial Intelligence (AI) and robotics are pervading
everyday activities, from industrial automation [
In this paper, we summarize our main contributions
in the field of responsible AI. We focus on autonomous
agents, e.g., robots, and present our approach to
responsible autonomy at diferent developmental stages. We first
describe our solutions for safe and explainable planning
in autonomous agents, via safe Reinforcement Learning
(RL) and neurosymbolic approaches. We also analyze
the problem of safe and compliant transfer of a planned
policy on a physical robotic system, combining RL with
model-based control. We then investigate how to provide
formal guarantees of safety for black-box policies, e.g.,
trajectories (belief-action pairs) of POMDP executions
collected ofline. Specifically, given a set of task-related
concepts provided by the user to describe the belief
space, ofline trajectories are converted to a logical
formalism, where logical predicates encode concepts in .
b) As an example, consider the paradigmatic POMDP
rocksample scenario depicted in Figure 2a, where a robotic
agent must collect valuable rocks (green dots) avoiding
worthless ones (red dots) in a grid world. The state of
the POMDP includes information about the position of
agents and rocks, and the probability (belief) of rocks
to be valuable. The state can be translated to a logical
representation in terms of the following concepts in :
Figure 1: Safe Policy Improvement: a) Comparison of per- the Manhattan distance D between the agent and each
formance among SPI algorithms; b) Scalability comparison rock R dist(R,D) and the probability P of a rock R to
between MCTS-SPIBB and SPIBB [
To this aim, in [
Trained RL policies, especially model-free RL policies encoded in a Deep Neural Network (DNN), do not guarantee to provably meet the safety standards required in the real world. For instance, DNNs are vulnerable to the so-called adversarial inputs, i.e., minimal input variations that fool the system to output an undesired value (or action) [24]. Consequently, in recent years, Formal Verification (FV) of DNNs (aka DNN-Verification ) has been developed to provide formal guarantees on the behavior of these systems [25]. In particular, given a predefined safety property, the goal of DNN-Verification is to assert whether at least one input configuration exists that violates the property. However, given the non-convex and non-linear nature of DNNs, verifying safety properties in the worst case has been shown to be an NP-complete problem [26]. Moreover, the standard binary response of DNN-verification (safe vs. unsafe) does not provide suficient information to compare the safety of diferent DNNs. with respect to other existing Safe DRL methodologies in the context of autonomous navigation, promoting safer behaviors while maintaining similar or better returns.
To overcome these limitations, in [27], we proposed a quired to reliably work over long periods of time (Long novel quantitative formulation of the DNN-verification Term Autonomy, LTA) under changing and unpredictable problem, allowing to enumerate all unsafe regions for a environmental conditions. In this context, anomaly detecgiven domain of interest and thus rank the models on the tion is crucial to promptly identify situations that diverge portion of unsafe regions they may have. However, we from the desired behaviour. Specifically, unsupervised showed that this problem turns out to be #P-hard. Hence, anomaly detection aims to idenfity anomalies related to in [28] we proposed -ProVe. Exploiting a controllable the global behavior of the system [34, 35, 36], monitorunderestimation of the output reachable sets obtained ing multivariate time series generated from sensors and via statistical prediction of tolerance limits [29], the al- actuators and starting from the only knowledge of the gorithm provides a tight —with provable probabilistic nominal (i.e., anomaly-free) behavior. guarantees— lower estimate of the (un)safe areas. We recently proposed two contributions in this area.
We validated DNN-Verification in realistic robotic namely, an online approach for detecting anomalous besafety-critical scenarios. In particular, in [30], we showed haviors of robotic systems involved in complex LTA scethat DNN-Verification can be used to rank diferent suc- narios (HHAD) [37], and an adversarial data augmencessful DNN models according to the level of safety, tation and retraining approach (HHAD-AUG) [38]. In verifying collision avoidance in robotic mapless navi- HHAD [37], we use Hidden Markov Models (HMMs) gation. We then applied a similar pipeline in a more to represent the nominal behavior of a robot. We then safety-critical domain, namely autonomous colonoscopy evaluate online the dissimilarity between the probability navigation for colorectal detection with deep RL [31] distribution of multivariate sensor time series in a sliding (Figure 5). In particular, we trained an agent to navigate window and the emission probability of the related HMM the endoscope in patient-specific colon models based on hidden states. We adopt the Hellinger distance [39] as a endoscopic images, using Constrained RL (CRL) to im- distance measure since it is bounded (thus it lends itself pose a safety cost for the agent to touch colon walls at to simpler interpretation and thresholding) and it is less the training stage. Nevertheless, due to the Lagrangian noisy, hence more informative and discriminative. relaxation implemented by CRL to perform constrained In HHAD-AUG [38], we address the usual lack (or optimization, safety may not be guaranteed. Hence, we paucity) of anomalous examples and the noise that charadopted a model selection strategy that harnesses FV to acterizes time series of real systems. We propose a data evaluate the safety of a vast pool of trained policies to augmentation method based on perturbed (adversarial) select the one the meets all the behavioral preferences time series [40], having the advantage of not requiring specified. The results of our study are reported in Ta- any prior knowledge about the application domain and ble 1 over 300 trained models, finding 3 completely safe data conformation. We generate adversarial examples models that provably meet the safety requirements. only for nominal points, optimizing a loss function based
Finally, to address the necessity of running the FV on the Hellinger distance between the observed and the process only after training due to its computational com- expected data distributions. plexity, in [32] we proposed an unconstrained DRL frame- We evaluate our data augmentation and re-training apwork that leverages a novel sample-based method to ap- proach on several public datasets, plus one collected from proximate local violations of input-output conditions to our aquatic drones developed in the EU H2020 project foster the learning of safer behaviors inside the training INTCATCH [41]. Results show that (i) the adversarial loop. However, such conditions are typically hard-coded generation algorithms can generate meaningful adversarand require task-level knowledge, making their applica- ial examples for HHAD, employing them to significantly tion intractable in challenging safety-critical tasks. To improve the performance of HHAD; (ii) our data augmenthis end, in [33], we introduced the Collection and Refine- tation method yields higher performance than examples ment of Online Properties (CROP) framework to collect generated by state-of-the-art augmentation methods; (iii) and refine safety properties during training. The combi- adversarial examples generated considering the Hellinger nation of CROP with approximate violation inside the distance yield higher improvement than examples gen|X| erated considering standard log-likelihood; (v) the low computational complexity and high parallelizability of the proposed method allow for a fast data augmentation and retraining of HHAD. Figure 6 shows the results on the INTCATCH dataset [41].
Finally, we have recently addressed the problem of explainable anomaly detection, in order to provide useful information about the source of the anomaly for easier repair. To this aim, in [42] we showed that causal discovery based on Conditional Mutual Information (CMI) between time series can achieve higher performance than standard deep learning antomaly detectors, on a benchmark robotic dataset of the Pepper service robot3. Our methodology evaluates the variation of CMI between time series, thus providing a useful hint to the root cause of the anomaly. Moreover, it builds a nominal model of the real physical relations between variables of the system, thus resulting in higher robustness and more accurate anomaly detection, compared to DNN methods (95% vs 90 % F1-score and 100% precision).
Our methodologies aim at increasing transparency and safety at diferent development levels, from planning to execution and verification. Our current research direction includes the online integration of symbolic learning and formal verification approaches into RL, focusing on the current scalability issues.
Systems, AAMAS ’23, IFAAMAS, 2023, pp. 373–381. [31] D. Corsi, L. Marzari, A. Pore, A. Farinelli, A. Casals, [16] D. Meli, A. Castellini, A. Farinelli, Learning logic P. Fiorini, D. Dall’Alba, Constrained reinforcespecifications for policy guidance in POMDPs: an ment learning and formal verification for safe inductive logic programming approach, Journal of colonoscopy navigation, in: 2023 IEEE/RSJ InterArtificial Intelligence Research 79 (2024) 725–776. national Conference on Intelligent Robots and Sys[17] S. Muggleton, L. De Raedt, Inductive logic program- tems (IROS), IEEE, 2023, pp. 10289–10294. ming: Theory and methods, The Journal of Logic [32] E. Marchesini, L. Marzari, A. Farinelli, C. Amato, Programming 19 (1994) 629–679. Safe deep reinforcement learning by verifying task[18] S. C. Tran, E. Pontelli, M. Balduccini, T. Schaub, level properties, AAMAS ’23, International FounAnswer set planning: a survey, Theory and Practice dation for Autonomous Agents and Multiagent Sysof Logic Programming 23 (2023) 226–298. tems, 2023, p. 1466–1475. [19] D. Meli, H. Nakawala, P. Fiorini, Logic program- [33] L. Marzari, E. Marchesini, A. Farinelli, Online safety ming for deliberative robotic task planning, Artifi- property collection and refinement for safe deep cial Intelligence Review 56 (2023) 9011–9049. reinforcement learning in mapless navigation, in: [20] C. Veronese, D. Meli, F. Bistafa, M. Rodríguez-Soto, 2023 IEEE International Conference on Robotics A. Farinelli, J. A. Rodríguez-Aguilar, Inductive logic and Automation (ICRA), IEEE, 2023, pp. 7133–7139. programming for transparent alignment with mul- [34] A. Castellini, M. Bicego, F. Masillo, M. Zuccotto, tiple moral values, in: CEUR WORKSHOP PRO- A. Farinelli, Time series segmentation for stateCEEDINGS, 2023, pp. 84–88. model generation of autonomous aquatic drones: A [21] F. Trotti, A. Farinelli, R. Muradore, An online path systematic framework, Engineering Applications planner based on pomdp for uavs, in: 2023 Euro- of Artificial Intelligence 90 (2020).
pean Control Conference (ECC), IEEE, 2023. [35] A. Castellini, M. Bicego, D. Bloisi, J. Blum, F. Masillo, [22] A. Isidori, Nonlinear control systems II, Springer, S. Peignier, A. Farinelli, Subspace clustering for 2013. situation assessment in aquatic drones: a sensitivity [23] H. Khalil, Nonlinear Systems, Prentice Hall, 2002. analysis for state-model improvement, Cybernetics [24] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Er- and Systems 50 (2019) 658–671. han, I. Goodfellow, R. Fergus, Intriguing properties [36] A. Castellini, F. Masillo, M. Bicego, D. Bloisi, J. Blum, of neural networks, arXiv:1312.6199 (2013). A. Farinelli, Subspace clustering for situation assess[25] C. Liu, T. Arnon, C. Lazarus, C. Strong, C. Barrett, ment in aquatic drones, in: Proc. 33th ACM/SIGAPP M. J. Kochenderfer, et al., Algorithms for verifying Symposium on Applied Computing, SAC, 2019, pp. deep neural networks, Foundations and Trends® 930–937.
in Optimization 4 (2021) 244–404. [37] D. Azzalini, A. Castellini, M. Luperto, A. Farinelli, [26] G. Katz, C. Barrett, D. L. Dill, K. Julian, M. J. Kochen- F. Amigoni, et al., HMMs for anomaly detection derfer, Reluplex: An eficient smt solver for veri- in autonomous robots, in: Proc. AAMAS, 2020, p. fying deep neural networks, in: International con- 105–113. ference on computer aided verification, Springer, [38] A. Castellini, F. Masillo, D. Azzalini, F. Amigoni, 2017, pp. 97–117. A. Farinelli, Adversarial data augmentation for [27] L. Marzari, D. Corsi, F. Cicalese, A. Farinelli, The hmm-based anomaly detection, IEEE Transactions #DNN-Verification Problem: Counting Unsafe In- on Pattern Analysis and Machine Intelligence 45 puts for Deep Neural Networks, in: International (2023) 14131–14143.
Joint Conference on Artificial Intelligence (IJCAI), [39] E. Hellinger, Neue begründung der theorie 2023, pp. 217–224. quadratischer formen von unendlichvielen verän[28] L. Marzari, D. Corsi, E. Marchesini, A. Farinelli, derlichen, Journal für die reine und angewandte F. Cicalese, Enumerating safe regions in deep neural Mathematik 136 (1909) 210–271. networks with provable probabilistic guarantees, in: [40] F. Karim, S. Majumdar, H. Darabi, Adversarial atProceedings of the AAAI Conference on Artificial tacks on time series, IEEE Trans Pattern Anal Mach Intelligence, volume 38, 2024, pp. 21387–21394. Intell 43 (2020) 3309–3320. [29] S. S. Wilks, Statistical prediction with special refer- [41] A. Castellini, D. Bloisi, J. Blum, F. Masillo, ence to the problem of tolerance limits, The annals A. Farinelli, Multivariate sensor signals collected of mathematical statistics 13 (1942) 400–409. by aquatic drones involved in water monitoring: A [30] G. Amir, D. Corsi, R. Yerushalmi, L. Marzari, complete dataset, Data Brief 30 (2020) 105436.
D. Harel, A. Farinelli, G. Katz, Verifying learning- [42] D. Meli, Explainable online unsupervised anomaly based robotic navigation systems, in: 29th Inter- detection for cyber-physical systems via causal disnational Conference TACAS, Springer, 2023, pp. covery from time series, arXiv:2404.09871 (2024). 607–627.