The AAAI-21 Workshop on Artificial Intelligence Safety (SafeAI 2021) Huáscar Espinoza​1​, José Hernández-Orallo​2​, Xin Cynthia Chen​3​, Seán S. ÓhÉigeartaigh​4​, Xiaowei Huang​5​, Mauricio Castillo-Effen​6​, Richard Mallah​7​ and John McDermid​8 1​ CEA LIST, Gif-sur-Yvette, France huascar.espinoza@cea.fr 2​ Universitat Politècnica de València, Spain jorallo@upv.es 3​ University of Hong Kong, China cyn0531@hku.hk 4​ University of Cambridge, Cambridge, United Kingdom so348@cam.ac.uk 5​ University of Liverpool, Liverpool, United Kingdom xiaowei.huang@liverpool.ac.uk 6​ Lockheed Martin, Advanced Technology Laboratories, Arlington, VA, USA mauricio.castillo-effen@lmco.com 7​ ​Future of Life Institute, USA richard@futureoflife.org 8​ ​University of York,​ ​United Kingdom john.mcdermid@york.ac.uk Abstract trade-offs or alternative solutions. These choices can only We summarize the AAAI-21 Workshop on Artificial be analyzed holistically if we integrate technological and Intelligence Safety (SafeAI 2021)​1​, virtually held at the ethical perspectives into the engineering problem, and Thirty-Fifth AAAI Conference on Artificial Intelligence on consider both the theoretical and practical challenges for February 8. AI safety. This view must cover a wide range of AI paradigms, considering systems that are specific for a Introduction particular application, and also those that are more general, which may lead to unanticipated risks. We must bridge the Safety in Artificial Intelligence (AI) is increasingly short-term with the long-term perspectives, idealistic goals becoming a substantial part of AI research, deeply with pragmatic solutions, operational with policy issues, intertwined with the ethical, legal and societal issues and industry with academia, in order to build, evaluate, associated with AI systems. Even if AI safety is considered deploy, operate and maintain AI-based systems that are a design principle, there are varying levels of safety, truly safe. diverse sets of ethical standards and values, and varying The AAAI-21 Workshop on Artificial Intelligence degrees of liability, for which we need to deal with Safety (SafeAI 2021) seeks to explore new ideas in AI safety with a particular focus on addressing the following 1 Workshop series website: ​http://safeaiw.org/ questions: Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 ● What is the status of existing approaches for ensuring ● Session Discussants gave a critical review of the session AI and Machine Learning (ML) safety and what are the papers, and participated in the plenary debate. gaps? Papers were grouped by topic as follows: ● How can we engineer trustworthy AI software architectures? Session 1: Dynamic Safety and Anomaly Assessment ● How can we make AI-based systems more ethically aligned? ● Feature Space Singularity for Out-of-Distribution Detection, Haiwen Huang, Zhihan Li, Lulu Wang, ● What safety engineering considerations are required to Sishuo Chen, Xinyu Zhou and Bin Dong. develop safe human-machine interaction? ● What AI safety considerations and experiences are ● An Evaluation of “Crash Prediction Networks” (CPN) for Autonomous Driving Scenarios in CARLA relevant from industry? Simulator, Saasha Nair, Sina Shafaei, Daniel Auge and ● How can we characterize or evaluate AI systems Alois Knoll. according to their potential risks and vulnerabilities? ● From Black-box to White-box: Examining Confidence ● How can we develop solid technical visions and new Calibration under different Conditions, Franziska paradigms about AI safety? Schwaiger, Maximilian Henne, Fabian Küppers, Felippe ● How do metrics of capability and generality, and Schmoeller Roza, Karsten Roscher and Anselm trade-offs with performance, affect safety? Haselhoff. These are the main topics of the series of SafeAI Session 2: Safety Considerations for the Assurance of workshops. They aim to achieve a holistic view of AI and AI-based Systems safety engineering, taking ethical and legal issues into account, in order to build trustworthy intelligent ● The Utility of Neural Network Test Coverage Measures, autonomous machines. The first edition of SafeAI was held Rob Ashmore and Alec Banks. in January 27, 2019, in Honolulu, Hawaii (USA) as part of ● Safety Properties of Inductive Logic Programming, the Thirty-Third AAAI Conference on Artificial Gavin Leech, Nandi Schoots and Joar Skalse. Intelligence (AAAI-19), and the second edition was held in February 7, 2020 in New York City (USA) also as part of ● A Hybrid-AI Approach for Competence Assessment of AAAI. This third edition was held online (because of the Automated Driving functions, Jan-Pieter Paardekooper, COVID-19 situation) at the Thirty-Fifth AAAI Conference Mauro Comi, Corrado Grappiolo, Ron Snijders, Willeke on Artificial Intelligence on February 8, virtually. van Vught and Rutger Beekelaar. Session 3: Adversarial Machine Learning and Program Trustworthiness The Program Committee (PC) received 44 submissions. ● Adversarial Robustness for Face Recognition: How to Each paper was peer-reviewed by at least two PC Introduce Ensemble Diversity among Feature members, by following a single-blind reviewing process. Extractors?, Takuma Amada, Kazuya Kakizaki, Seng The committee decided to accept 13 full papers, 1 talks and Pei Liew, Toshinori Araki, Joseph Keshet and Jun 11 posters, resulting in a full-paper acceptance rate of Furukawa. 29.5% and an overall acceptance rate of 56.8%. ● Multi-Modal Generative Adversarial Networks Make The SafeAI 2021 program was organized in four Realistic and Diverse but Untrustworthy Predictions thematic sessions, one keynote and two (invited) talks. When Applied to Ill-posed Problems, John Hyatt and The thematic sessions followed a highly interactive Michael Lee. format. They were structured into short pitches and a group debate panel slot to discuss both individual paper ● DeepFakesON-Phys: DeepFakes Detection based on contributions and shared topic issues. Three specific roles Heart Rate Estimation, Javier Hernandez-Ortega, Ruben were part of this format: session chairs, presenters and Tolosana, Julian Fierrez and Aythami Morales. session discussants. Session 4: Safe Autonomous Agents ● Session Chairs introduced sessions and participants. The ● What Criminal and Civil Law Tells Us About Safe RL Chair moderated sessions and plenary discussions, Techniques to Generate Law-abiding Behaviour, Hal monitored time, and moderated questions and Ashton. discussions from the audience. ● Presenters gave a 10 minute paper talk and participated in the debate slot. 2 ● Performance of Bounded-Rational Agents With the Cartella, Orlando Anunciacao, Yuki Funabiki, Daisuke Ability to Self-Modify, Jakub Tětek, Marek Sklenka and Yamaguchi, Toru Akishita and Olivier Elshocht. Tomáš Gavenčiak. ● Correct-by-Construction Multi-Label Classification ● Deep CPT-RL: Imparting Human-Like Risk Sensitivity Networks, Eleonora Giunchiglia and Thomas to Artificial Agents, Jared Markowitz, Marie Chau and Lukasiewicz. I-Jeng Wang. ● Classification Confidence Scores with Point-wise Guarantees, Nivasini Ananthakrishnan, Shai Ben-David ● Challenges for Using Impact Regularizers to Avoid and Tosca Lechner. Negative Side Effects, David Lindner, Kyle Matoba and Alexander Meulemans. Acknowledgements SafeAI was pleased to have several additional inspirational researchers as invited speakers: We thank all researchers who submitted papers to SafeAI Keynote 2021 and congratulate the authors whose papers and posters were selected for inclusion into the workshop ● Not finalized at the date of publishing program and proceedings. We especially thank our distinguished PC members for Invited Talks reviewing the submissions and providing useful feedback to the authors: ● Juliette Mattioli (Thales, France) and Rodolphe Gelin (Renault, France). Methods and Tools for Trusted AI: an ● Stuart Russell, UC Berkeley, USA Urgent Challenge for Industry ● Francesca Rossi, IBM and University of Padova, Italy ● Sandhya Saisubramanian (University of Massachusetts ● Raja Chatila, Sorbonne University, France Amherst, USA), Challenges and Directions in Avoiding ● Roman V. Yampolskiy, University of Louisville, USA Negative Side Effects ● Gereon Weiss, Fraunhofer ESK, Germany Posters were presented with 2-minute pitches. Most posters ● Mark Nitzberg, Center for Human-Compatible AI, USA have also been included as short papers within this volume. ● Roman Nagy, Autonomous Intelligent Driving GmbH, Posters Germany ● François Terrier, CEA LIST, France ● Towards an Ontological Framework for Environmental ● Hélène Waeselynck, LAAS-CNRS, France Survey Hazard Analysis of Autonomous Systems, Christopher Harper and Praminda Caleb-Solly. ● Siddartha Khastgir, University of Warwick, UK ● Overestimation learning with guarantees, Adrien ● Orlando Avila-García, Atos, Spain Gauffriau, François Malgouyres and Mélanie Ducoffe. ● Nathalie Baracaldo, IBM Research, USA ● On the Use of Available Testing Methods for ● Peter Eckersley, Partnership on AI, USA Verification & Validation of AI-based Software and ● Andreas Theodorou, Umeå University, UK Systems, Franz Wotawa. ● Emmanuel Arbaretier, Apsys-Airbus, France ● Runtime Decision Making Under Uncertainty in ● Yang Liu, Webank, China Autonomous Vehicles, Vibhu Gautam, Youcef ● Philip Koopman, Carnegie Mellon University, USA Gheraibia, Rob Alexander and Richard Hawkins. ● Chokri Mraidha, CEA LIST, France ● Negative Side Effects and AI Agent Indicators: Experiments in SafeLife, John Burden, Jose ● Heather Roff, Johns Hopkins University, USA Hernandez-Orallo and Sean O'Heigeartaigh. ● Bernhard Kaiser, ANSYS, Germany ● Time for AI (Ethics) Maturity Model Is Now, Ville ● Brent Harrison, University of Kentucky, USA Vakkuri, Marianna Jantunen, Erika Halme, Kai-Kristian ● José M. Faria, Safe Perspective, UK Kemell, Anh Nguyen-Duc, Tommi Mikkonen and ● Toshihiro Nakae, DENSO Corporation, Japan Pekka Abrahamsson. ● John Favaro, Trust-IT, Italy ● AI-Blueprint for Deep Neural Networks, Ernest ● Rob Ashmore, Defence Science and Technology Wozniak, Henrik Putzer and Carmen Carlan. Laboratory, UK ● Neural Criticality: Validation of Convolutional Neural ● Jonas Nilsson, NVIDIA, USA Networks, Vaclav Divis and Marek Hruz. ● Michael Paulitsch, Intel, Germany ● Adversarial Attacks for Tabular Data: Application to Fraud Detection and Imbalanced Data, Francesco ● Philippa Ryan Conmy, Adelard, UK 3 ● Ramya Ramakrishnan, Massachusetts Institute of Technology, USA ● Stefan Kugele, Technical University of Munich, Germany ● Victoria Krakovna, Google DeepMind, UK ● Richard Cheng, California Institute of Technology, USA ● Javier Ibañez-Guzman, Renault, France ● Mehrdad Saadatmand, RISE SICS, Sweden ● Alessio R. Lomuscio, Imperial College London, UK ● Rick Salay, University of Waterloo, Canada ● Jérémie Guiochet, LAAS-CNRS, France ● Sandhya Saisubramanian, University of Massachusetts Amherst, USA ● Mario Gleirscher, University of York, UK ● Guy Katz, Hebrew University of Jerusalem, Israel ● Chris Allsopp, Frazer-Nash Consultancy, UK ● Daniela Cancila, CEA LIST, France ● Vahid Behzadan, University of New Haven, USA ● Simos Gerasimou, University of York, UK ● Brian Tse, Affiliate at University of Oxford, China ● Peter Flach, University of Bristol, UK ● Gopal Sarma, Broad Institute of MIT and Harvard, USA ● Huáscar Espinoza, CEA LIST, France ● Seán Ó hÉigeartaigh, University of Cambridge, UK ● Xiaowei Huang, University of Liverpool, UK ● José Hernández-Orallo, Universitat Politècnica de València, Spain ● Mauricio Castillo-Effen, Lockheed Martin, USA ● Xin Cynthia Chen, University of Hong Kong, China ● Richard Mallah, Future of Life Institute, USA ● John McDermid, University of York, United Kingdom As well as the additional reviewers: ● Fabio Arnez, ● Dashan Gao ● Anbu Huang We thank Juliette Mattioli, Rodolphe Gelin and Sandhya Saisubramanian for their inspiring talks. Finally we thank the AAAI-21 organization for providing an excellent framework for SafeAI 2021. 4