The IJCAI-19 Workshop on Artificial Intelligence Safety (AISafety 2019) Huáscar Espinoza1, Han Yu2, Xiaowei Huang3, Freddy Lecue4, Cynthia Chen5, José Hernández-Orallo6, Seán Ó hÉigeartaigh7, and Richard Mallah8 1 Commissariat à l´Energie Atomique, France 2 Nanyang Technological University, Singapore 3 University of Liverpool, UK 4 Thales, Canada 5 University of Hong Kong, China 6 Universitat Politècnica de València, Spain 7 University of Cambridge, UK 8 Future of Life Institute, USA Abstract • What safety engineering considerations are required to This preface introduces the IJCAI-19 Workshop on develop safe human-machine interaction in automated Artificial Intelligence Safety (AISafety 2019), held decision-making systems? at the 28th International Joint Conference on Arti- • What AI safety considerations and experiences are rele- ficial Intelligence (IJCAI) on August 11-12, 2019 vant from industry? in Macao, China. • How can we characterise or evaluate AI systems accord- 1 Introduction ing to their potential risks and vulnerabilities? In the last decade, there has been a growing concern on risks • How can we develop solid technical visions and para- of Artificial Intelligence (AI). Safety is becoming increas- digm shift articles about AI Safety? ingly relevant as humans are progressively side-lined from • How do metrics of capability and generality affect the the decision/control loop of intelligent and learning-enabled level of risk of a system and how trade-offs can be found machines. In particular, the technical foundations and as- with performance? sumptions on which traditional safety engineering principles are based, are inadequate for systems in which AI algo- • How do AI system feature for example ethics, explaina- rithms, and in particular Machine Learning (ML) algo- bility, transparency, and accountability relate to, or con- rithms, are interacting with people and/or the environment at tribute to, its safety? increasingly higher levels of autonomy. We must also con- • How to evaluate AI safety? sider the connection between the safety challenges posed by present-day AI systems, and more forward-looking research The main interest of AISafety 2019 is to look holistically at focused on more capable future AI systems, up to and in- AI and safety engineering, jointly with the ethical and legal cluding Artificial General Intelligence (AGI). issues, to build trustable intelligent autonomous machines. The IJCAI-19 Workshop on Artificial Intelligence Safety The first edition of AISafety was held in August 11-12, (AISafety 2019) seeks to explore new ideas on AI safety 2019, in Macao (China), as part of the 28th International with particular focus on addressing the following questions: Joint Conference on Artificial Intelligence (IJCAI-19). The • How can we engineer trustable AI software architec- AISafety workshop is organized as a “sister workshop” to tures? other two workshops: WAISE (https://www.waise.org/) and to SafeAI (http://www.safeai2019.org). • Do we need to specify and use bounded morality in sys- As part of this IJCAI workshop, we also started the AI tem engineering to make AI-based systems more ethical- Safety Landscape initiative. This initiative aims at defining ly aligned? an AI safety landscape providing a “view” of the current needs, challenges and state of the art and the practice of this • What is the status of existing approaches in ensuring AI field. Further information about this initiative can be found and ML safety and what are the gaps? at: https://www.ai-safety.org/ai-safety-landscape. 2 Programme • Detecting Spiky Corruption in Markov Decision Process- es. Alok Singh, Jason Mancuso, David Lindner and To- The Programme Committee (PC) received 36 submissions, masz Kisielewski. Detecting Spiky Corruption in Markov in the following categories: Decision Processes. • Short position papers – 9 submissions. • Modeling AGI Safety Frameworks with Causal Influence • Full scientific contributions – 23 submissions. Diagrams. Tom Everitt, Ramana Kumar, Victoria Kra- • Proposals of technical talks – 4 submissions. kovna and Shane Legg. Each of the papers was peer-reviewed by at least three PC Session 3: Safe Autonomous Vehicles members, by following a single-blind reviewing process. The committee decided to accept 13 papers (2 position pa- • On the Susceptibility of Deep Neural Networks to Natu- pers and 11 scientific papers) and 2 talks, resulting in an ral Perturbations. Mesut Ozdag, Sunny Raj, Steven L. overall acceptance rate of 42%. We additionally invited 1 Fernandes, Alvaro Velasquez, Laura Pullum and Sumit talk, which was not submitted to the call, and accepted 7 Kumar Jha. submissions as short papers for poster presentation. • Managing Uncertainty of AI-based Perception for Au- AISafety 2019 has been planned as a two-day workshop tonomous Systems. Maximilian Henne, Adrian with general AI Safety topics in the first day and AI Safety Schwaiger and Gereon Weiss. Landscape talks and discussions during the second day. • A Framework for Safety Violation Identification and Assessment in Autonomous Driving. Lukas Heinzmann, 2.1. First Workshop Day (Aug 11) Sina Shafaei, Mohd Hafeez Osman, Christoph Segler and The AISafety 2019 programme on Aug 11 was organized Alois Knoll. in four thematic sessions, one keynote and two invited talks. The thematic sessions followed a highly interactive for- Session 4: AI Value Alignment, Ethics and Bias mat. They were structured into short talks and a common panel slot to discuss both individual paper contributions and • The Glass Box Approach: Verifying Contextual Adher- shared topic issues. Three specific roles were part of this ence to Values. Andrea Aler Tubella and Virginia Dig- format: session chairs, presenters and session discussants. num. • Session Chairs introduced sessions and participants. The • Requisite Variety in Ethical Utility Functions for AI Val- Chair moderated session and plenary discussions, took ue Alignment. Nadisha-Marie Aliman and Leon Kester care of the time, and gave the word to speakers in the au- • Slam the Brakes: Perceptions of Moral Decisions in Driv- dience during discussions. ing Dilemmas. Holly Wilson, Andreas Theodorou and • Presenters gave a paper talk in 10 minutes and then par- Joanna Bryson. ticipated in the debate slot. • Understanding Bias in Datasets using Topological Data • Session Discussants prepared the discussion of individual Analysis. Ramya Srinivasan and Ajay Chander. papers and the plenary debate. The discussant gave a crit- ical review of the session papers. Additionally, AISafety was proud to bring great inspiration- al speakers: The mixture of topics has been carefully balanced, as fol- lows: Keynote Session 1: Safe Learning • Joel Lehman (Uber AI Labs, USA), AI Safety for Evolu- • Learning Modular Safe Policies in the Bandit Setting tionary Computation, Evolutionary Computation for AI with Application to Adaptive Clinical Trials. Hossein Safety. Aboutalebi, Doina Precup and Tibor Schuster. • Metric Learning for Value Alignment. Andrea Loreggia, Invited Talks Nicholas Mattei, Francesca Rossi and Kristen Brent Ve- nable. • Shlomo Zilberstein (University of Massachusetts Am- herst, USA), AI Safety Based on Competency Models. Session 2: Reinforcement Learning Safety • Yang Liu (WeBank, China), User Privacy, Data Confi- dentiality and AI Safety in Collaborative Learning. • Penalizing side effects using stepwise relative reachabil- ity. Victoria Krakovna, Laurent Orseau, Miljan Martic Posters were presented in 2-minutes pitches and are also and Shane Legg. part of this volume as poster papers. • Conservative Agency. Alexander Turner, Dylan Had- field-Menell and Prasad Tadepalli. Posters 3 Acknowledgements • Computational Strategies for the Trustworthy Pursuit and We thank all those who submitted papers to AISafety 2019 the Safe Modeling of Probabilistic Maintenance Com- and congratulate the authors whose papers and posters were mitments. Qi Zhang, Edmund Durfee and Satinder Singh selected for inclusion into the workshop program and pro- ceedings. • Categorizing Wireheading in Partially Embedded Agents. We specially thank our distinguished PC members, for Arushi Majha, Sayan Sarkar and Davide Zagami reviewing the submissions and providing useful feedback to • Adversarial Exploitation of Policy Imitation. Vahid the authors: Behzadan and William Hsu. • The Challenge of Imputation in Explainable Artificial • Stuart Russell, UC Berkeley, USA Intelligence Models. Muhammad Ahmad, Carly Eckert • Victoria Krakovna, Google DeepMind, UK and Ankur Teredesai • Peter Eckersley, Partnership on AI, USA • On the importance of system testing for assuring safety • Riccardo Mariani, Intel, Italy of AI systems. Franz Wotawa • Brent Harrison, University of Kentucky, USA • Towards Empathic Deep Q-Learning. Bart Bussmann, • Siddartha Khastgir, University of Warwick, UK Jacqueline Heinerman and Joel Lehman • Emmanuel Arbaretier, Apsys-Airbus, France • Watermarking of DRL Policies with Sequential Triggers. • Martin Vechev, ETH Zurich, Switzerland Vahid Behzadan and William Hsu. • Sandhya Saisubramanian, University of Massachusetts Amherst, USA • Alessio R. Lomuscio, Imperial College London, UK 2.2. Second Workshop Day (Aug 12): Landscape • Mauricio Castillo-Effen, Lockheed Martin, USA The second-day workshop (AI Safety Landscape) sessions • Yi Zeng, Chinese Academy of Sciences, China on August 12 were organized into by-invitation talks and • Brian Tse, Affiliate at University of Oxford, China panels with structured discussions. The by-invitation talks • Sandeep Neema, DARPA, USA focused on diverse topics contributing to understand the AI Safety Landscape, in terms of their scientific and technical • Michael Paulitsch, Intel, Germany challenges, industrial and academic opportunities, as well as • Elizabeth Bondi, University of Southern California, USA gaps and pitfalls. • Hélène Waeselynck, CNRS LAAS, France AISafety 2019 was proud to bring great industry, aca- • Rob Alexander, University of York, UK demic and research leaders as invited speakers: • Vahid Behzadan, Kansas State University, USA • Simon Fürst, BMW, Germany Invited Talks • Chokri Mraidha, CEA LIST, France • Fuxin Li, Oregon State University, USA • Richard Mallah (Future of Life Institute, USA): Creating • Francesca Rossi, IBM and University of Padova, Italy a Deep Model of AI Safety Research. • Ian Goodfellow, Google Brain, USA • John McDermid (University of York, UK): Towards a • Yang Liu, Webank, China Framework for Safety Assurance of Autonomous Sys- • Ramana Kumar, Google DeepMind, UK tems [published as part of this Proceedings volume]. • Javier Ibañez-Guzman, Renault, France • Gopal Sarma (Broad Institute of MIT and Harvard, • Dragos Margineantu, Boeing, USA USA): AI Safety and The Life Sciences. • Joanna Bryson, University of Bath, UK • Xiaowei Huang (University of Liverpool, UK): Formal • Heather Roff, Johns Hopkins University, USA Methods in Certifying Learning-Enabled Systems. • Raja Chatila, Sorbonne University, France • Virginia Dignum (University of Umeå, Sweden): AI • Hang Su, Tsinghua University, China Safety for Humans. • François Terrier, CEA LIST, France • Raja Chatila (Sorbonne University, France): Towards • Guy Katz, Hebrew University of Jerusalem, Israel Trustworthy Autonomous and Intelligent Systems. • Alec Banks, Defence Science and Technology Laborato- • Jeff Cao (Tencent Research Institute, China): AI Princi- ry, UK ples and Ethics by Design. • Gopal Sarma, Emory University, USA • Victoria Krakovna (Google DeepMind, UK): Specifica- • Lê Nguyên Hoang, EPFL, Switzerland tion, Robustness and Assurance Problems in AI Safety. • Roman Nagy, BMW, Germany One important ambition of this initiative is to align and syn- • Nathalie Baracaldo, IBM Research, USA chronize the proposed activities and outcomes with other • Toshihiro Nakae, DENSO Corporation, Japan related initiatives. This AI Safety Landscape work will fol- • Peter Flach, University of Bristol, UK low up with future meetings and workshops. • Richard Cheng, California Institute of Technology, USA • José M. Faria, Safe Perspective, UK • Ramya Ramakrishnan, Massachusetts Institute of Tech- nology, USA • Gereon Weiss, Fraunhofer ESK, Germany • Huáscar Espinoza, Commissariat à l´Energie Atomique, France • Han Yu, Nanyang Technological University, Singapore • Xiaowei Huang, University of Liverpool, UK • Freddy Lecue, Thales, Canada • Cynthia Chen, University of Hong Kong, China • José Hernández-Orallo, Universitat Politècnica de Valèn- cia, Spain • Seán Ó hÉigeartaigh, University of Cambridge, UK • Richard Mallah, Future of Life Institute, USA As well as the additional reviewers: • George Amariucai, Kansas State University, USA • Neale Ratzlaff, Oregon State University, USA We thank Joel Lehman, Shlomo Zilberstein, Yang Liu, Richard Mallah, John McDermid, Gopal Sarma, Xiaowei Huang, Virginia Dignum, Raja Chatila, Jeff Cao and Victo- ria Krakovna for their interesting talks on the current chal- lenges of AI safety. We would like to specially thank our sponsors, which funded the Best Paper Award and the video-recording of the AI Safety Landscape sessions: • Assuring Autonomy International Programme (AAIP). • Partnership on AI. • The Centre for the Study of Existential Risk (CSER). Finally yet importantly, we thank the IJCAI-19 organiza- tion for providing an excellent framework for AISafety 2019.