Explaining Possible Futures for Robust Autonomous Decision-Making Leilani H. Gilpin MIT CSAIL lgilpin@mit.edu Abstract The long-term goal for autonomous vehicles is to mini- mize the numbers of false positive and false negative detec- As humans, we consider multiple alternatives when mak- tion so that human fatalities become rare. In the meantime, ing decisions or choices. The way we decide between these explanations can be used to make the subsystems account- choices is to create stories and explain (to ourselves) why they are reasonable or not. But when machines make decisions, able and learn from their mistakes. For example, there are their processes are not interpretable (understandable by hu- two ways in which a previous working system can exhibit mans) nor explainable (able to recount the reasons and de- anomalous behavior. A local error is confined to a particu- pendencies leading up to a decision). In this paper, I present lar subsystem. An example of which is a subsystem that is a methodology to explain possible futures and utilize these calculating square roots, but the output squared is not “rea- explanations to make more robust and reasonable decisions sonably close” to the input. The second type of inconsis- moving forward. Internal explanations will be used dynami- tency is observed as a failed cooperation between subsys- cally by the parts of a complex machine to detect failure and tems; each subsystem is able to defend its observed behavior, intrusion. System-level explanations will provide a coherent but the larger neighborhood of subsystems is not executing and convincing story to humans for engineering, legal reason- its shared task as intended. For example, take a mechanism ing, and forensics. that is solving an optimization problem. Within this mecha- nism, there is a neighborhood of subsystems with the com- Introduction mon goal of calculating the next step gradient for each suc- cessive iteration. Within this neighborhood, there is a sub- Making decisions is a difficult human task. Often times, we system whose job is to calculate square roots. The square are left wondering if we made the correct decision (usually, root subsystem is only returning the positive square root. after the fact). These decisions may be due to overestimat- This output is not inconsistent local to the subsystem: it is ing, underestimating or miscalculating the impact; we may able to explain that its behavior is reasonable since the out- not have explained the alternatives accurately. Other times, put squared is close to input. However, when cooperating we may have had a time constraint, leading to an impulsive with other subsystems in its community, one or more subsys- decision without properly processing the outcomes and al- tems is expecting the conjugate root, and the community of ternatives. cooperating subsystems exhibits unexpected behavior. Al- Explaining possible futures is important as autonomous though each individual subsystem can explain that it is be- agents are increasingly deployed in real world-settings (e.g. having reasonably, this community of subsystems needs ad- driving), where there has been an increase in malfunc- ditional help to ameliorate the inconsistencies in neighbor- tions and errors leading to injuries1 and even deaths2 . Such hoods of interconnected subsystems and develop an expla- level of increased harm on human lives is undesirable and nation. completely untenable. My research addresses the uncer- tain, unstable, and error-prone decision-making of complex I have developed a proof-of-concept that can make these machine by imposing explanations: the symbolic reasons, kinds of deductions using explanations. This methodology premises, and support leading up to an intended decision. is focused on identifying, detecting, and explaining antic- ipatory subsystem decisions. In future work, the explana- Copyright c 2020 for this paper by its authors. Use permitted un- tions will be processed automatically to determine the “best” der Creative Commons License Attribution 4.0 International (CC next step. In this paper, I define the problem space, present a BY 4.0). proof-of-concept with initial results, and propose a method- 1 Mall robot injures a toddler: https://qz.com/730086/a-robot- ology for anomaly detection and monitoring with explana- mall-cop-did-more-harm-than-good/ 2 tions. Uber self-driving car pedestrian fatality: https: //www.nytimes.com/interactive/2018/03/20/us/self-driving- Not all successful systems will make decisions this way. uber-pedestrian-killed.html But, this methodology will serve as an inspiration for the development of machines that have to make safety-critical Given a set of premises whose relative validity may not be or mission-critical decisions. obvious, it can be difficult to decide which premises to ac- cept. There are of course situations in which premises can be Problem Statement identified as faulty – for example, if they lead to violations Complex systems have become a staple of daily life, pro- of reasonable constraints or contradictions of systematic ax- viding everything from smart thermostats to online bank- ioms. Often, though, we cannot reject any of the available ing to collision-avoidance systems. In a perfect world, these premises outright. In such cases we need a way to decide systems would have no errors, and false positives would be which system of premises, out of equally plausible-looking nearly impossible. Although these systems can be tested in possibilities, is to be accepted in practice – and, as a corol- simulation, simulation environments cannot contain all the lary, what future our system pursues. factors in the real world that can invoke errors and anoma- One can try to treat this problem as one of simulation- lies. At the same time, test suites of prone error modes can- based search, with a possible objective being to find the not possibly represent all possible error cases. Although sim- choice of premises resulting in the “least bad” set of con- ulation and test cases can represent and catch a multitude of sequences given the set of possible realities under consider- error conditions, we need better error detection and explana- ation. It is of course impossible to consider every potentially tion protocols in practice. available reality; for instance, one cannot, in this process, Tools and frameworks exist that permit the design and cre- reasonably account for events like a meteor strike during ation of some systems that are provably correct by construc- the interval under consideration, where no evidence exists tion. Unit testing is a well-accepted practice for verifying the to support the notion of incoming meteors. However, infor- behavior of a system once it is realized. However, these ap- mation reported by onboard perception supporting the ex- proaches are ultimately inadequate, as the specifications to istence of an “unusual” object, like a lawnmower, provides which systems are constructed are complex, subject to error, at least two possibilities – the existence of a lawnmover, or and constantly evolving in response to shifting requirements. a defect in the onboard perception – whose consequences As these systems become larger, containing more interacting must be examined. subsystems handling a wider range of tasks, the number of possible failure modes increases, so we can expect this al- Commonsense Reasoning ready challenging problem to grow worse over time. Humans are opaque systems. When something goes wrong, Instead of striving to produce a perfect design that never we cannot always say why. For example, when we are ill or fails I propose to build in mechanisms that robustly de- malfunctioning, we cannot always point to the exact subsys- tect failures and attempt to ameliorate their own anomalies tem causing the error. But we can form a coherent explana- through explanation. If part of a system produces undesir- tion of what we believe we are suffering from, by querying able behavior, either as an intrinsic error or as a result of previous data and commonsense. If we have a fever, we can external interference, the rest of the system should be able usually come up with a reason: we feel hot, then cold, and to dynamically limit the extent of the possible damage. that is similar to a previous time when we had a fever. We The goal of this methodology is three-fold: to correctly can also create explanations with commonsense. If we have identify (minimize false positives), detect (using constant in- a stomach ache, perhaps it was the spicy food that caused trospection and monitoring) and explain subsystem failures the pain. Or, it was the fact that we ate a heavy meal on a and alternatives. previously [starved] stomach. One way to mitigate perception errors is to supplement Imagining the Future with Explanations decisions with commonsense. One way to do this is to im- Sound decisions are not made based on some single instant pose a reasonableness monitor (Gilpin 2018). For example, in time; rather, they are made with careful consideration of in the stomach ache example, we can use commonsense to their consequences. Sometimes these consequences may not come up with multiple explanations. This requires the avail- be known beforehand. This was stated nicely by Donald ability of a commonsense knowledge base. We can formu- Rumsfeld: late explanations using “nearby” information: the stomach There are known knowns. These are things we know is close to the appendix, which may be ruptured. Or we that we know. There are known unknowns. That is to can create other causal explanations: stomach aches can be say, there are things that we know we don’t know. But caused by spicy food, or stomach aches can be caused by there are also unknown unknowns. There are things we eating too much on an empty stomach. It is difficult to de- don’t know we don’t know. termine which one of these explanations is “most” correct or plausible, which is left to future work. But, the ability for Thus, a system must be able to imagine each possible fu- intelligent machines to use commonsense to formulate these ture that may result from its choices, and evaluate whether explanations themselves is a promising area of research. that future might be reasonable. To do this well, the system must be able to simulate the behavioral and physical conse- quences of acting on any set of premises that may be chosen A Preliminary Demonstration by committee arbitration, particularly in the case where it To demonstrate how explanations could be used to imag- will have accepted premises that in fact represent the wrong ine possible futures, I constructed a small proof-of-concept situation. demonstration. Consider a toy model of a car, consisting of low-level actuation components, like the braking, steering, This vision perception is unreasonable. and power control systems, as well as driving tactics, Li- There is no commonsense data supporting DAR and the vision components, as seen in Figure 4 the similarity between a bike, vehicle There are monitors around each component, and a high- and unknown object except that they can level reasoner to reconcile component explanations for be located at the same location. higher-level decision making. The high-level reasoner takes This component should be ignored. in the input from the three underlying components, and pro- poses a few candidate high-level decisions. This high-level Figure 1: The output of a local reasonableness monitor on reasoner examines these proposed plans along with the ex- the input from the Uber self-driving car scenario, in which planations from the underlying parts to make a more in- the vision system was oscillating between 3 labels: a bike, a formed, explainable, and robust plan. The system also in- vehicle, and an unknown object. The perception is classified cludes a priority hierarchy to enforce individual needs when as unreasonable. there are conflicts. For example, the vehicle’s inhabitant(s) are prioritized, then other drivers and pedestrians, etc. This LiDAR perception is reasonable. An object moving of this size is a large Scenario Information moving object that should be avoided. The example for the proof-of-concept is the Uber self- driving vehicle accident. On March 18, 2018 at approx- Figure 2: The output of a local reasonableness monitor on imately 10pm, an Uber Advanced Technologies Group the LiDAR input from the Uber self-driving car scenario. (ATG) self-driving test vehicle (a modified 2017 Volvo Since a large object is detected, the monitor recommends it XC90) struck and killed a pedestrian in Tempe, Arizona. In to be avoided. the investigation findings: “The Uber ATG automated driv- ing system detected the pedestrian 5.6 seconds before im- pact. Although the system continued to track the pedestrian 1. Continue straight. until the crash, it never accurately identified the object cross- 2. Slow down to a stop. ing the road as a pedestrian – or predicted its path3 .” 3. Veer to the side of the road. Although the LiDAR system had correctly detected the pedestrian, since the vision system was unreliable, the plan- Note that these intended decisions are not necessarily output ning system was instructed to ignore4 the detected object in this human-understandable way. But using edge-detection as a false positive, and continued forward at a high speed. and interval analysis with explanations which was explored This error is due to an inconsistency between parts and an in previous work (Gilpin and Yuan ), I can directly generate inability to anticipate consequences. I can reconcile this in- these kinds of text explanations from symbolic descriptions. consistency using internal, subsystem explanations and a set Now, the high-level reasoner also requires explanations of future plans. These are the facts from the initial Uber re- from its underlying parts. In this scenario, the high-level port5 . reasoner receives input from the computer vision (percep- tion) system, the LiDAR/radar system, and the driving tac- 1. Radar and LiDAR detected the pedestrian about 6 seconds tics (consisting of the brakes, steering, gas, etc.) The system- before impact (vehicle speed was 43 mph). wide monitoring diagram for this example is shown in Fig- 2. The vision system classified the pedestrian as an unknown ure 4. The vision system output is a set of segmentations and object, as a vehicle, and then as a bicycle with varying their corresponding labels (e.g. person, tree, road, etc.) For expectations of future travel path. this Uber example case, I focus on the segmentation in the 3. 1.3 seconds before impact, the self-driving system en- upper left (from the point of view of the car). In the seconds gaged an emergency braking maneuver, to mitigate a col- before impact, the output of the reasonableness monitor for lision. the vision processing component is shown in Figure 1. But there is more sensory information: the LiDAR sensor Proof of Concept data log. The LiDAR reasonableness monitor first interprets the sensor log. Using edge detection and interval analysis, Consider the Uber scenario approximately 6 seconds before the raw sensor data is abstracted into a list of symbolic impact. The high-level reasoner (or a monitor around the descriptions that can be passed into the reasonableness mon- will generate 3 plans with some certainty. The high-level itor. The symbolic list produced for the LiDAR data in this reasoner will explain these high-level plans as follows: scenario is (’object, ’moving, ’5-ft-tall, 3 ’top-left-quadrant, ...). In the seconds before NTSB Accident Report Press Release-https://ntsb.gov/news/ press-releases/Pages/NR20191119c.aspx impact, the output of the reasonableness monitor for the 4 Uber data inconsistency: https://www.theinformation.com/ vision processing component is shown in Figure 2. articles/uber-finds-deadly-accident-likely-caused-by-software- Finally, the tactics system is similarly interpreted into a set-to-ignore-objects-on-road symbolic, qualitative description: (’moving-quickly, 5 NTSB Preliminary report-https://www.ntsb.gov/ ’straight, ’continued-straight, ...) investigations/AccidentReports/Reports/HWY18MH010- signifying that the vehicle has been proceeding straight, prelim.pdf quickly for the last 5-10 second horizon. The reasonableness The best option is to veer and slow down. The vehicle is traveling too fast to suddenly stop. The vision system is inconsistent, but the LiDAR system has provided a reasonable and strong claim to avoid the object moving across the street. Figure 3: The high-level reasoner output for the Uber self- driving vehicle example. monitor for the tactics system deduces that that system state is reasonable: The system state is reasonable given that the vehicle has been moving quickly and proceeding straight for the last 10 second history. With these three subsystem explanations, the high-level reasoner processes the explanations (which are also stored Figure 4: System diagram for the explanatory architecture as a list of symbolic triples). The reasoner examines and as- for a simplified self-driving car. Tactics provide communi- sesses at the strengths of each explanation, and compares cation to and from the brakes, gas and power subsystems. it to a hierarchy of needs to see which intended decision The tactics system reports its reasons and explanations to does not violate the the needs hierarchy. Several iterations of the reasoner, as well as the LiDAR and vision subsystems. this process may be necessary for more complex decisions (or a more complicated needs hierarchy). For this proof- of-concept, the high-level reasoner explains each of the in- tended plans against the component explanations and hierar- tection needs a novel scoring algorithm designed for stream- chy of needs: ing data, including a series of benchmarks (Lavin and Ahmad 2015). However, decreasing the number of false- 1. Continue forward (straight): this would result in injuring positives and false-negatives in anomaly detection is a dif- the object detected by the LiDAR system. The vision sys- ficult problem. Some tactics include smoothing the output tem cannot confirm this detection and is deemed unreli- (Grill, Pevnỳ, and Rehak 2017), or piece-wise approxima- able due to misjudgements. Therefore, the vehicle should tions (Vallis, Hochenbaum, and Kejariwal 2014). not continue forward. Another goal is to make autonomous systems naturally 2. Stop: It is unclear if stopping would guarantee limited resistant to intrusion through monitoring and continuous in- harm to the object detected by the LiDAR system. A sud- trospection. Intrusion detection research is a lively field with den stop at the speed of the vehicle may injure its occu- numerous proposed approaches in the literature. Some ap- pants. Therefore, the vehicle should not stop. (Although, proaches rely on a combination of topological vulnerability this intended decision will remain a possible choice since analysis and system alert data to detect attacks (Albanese et it does not produce as much damage as the first option). al. 2011). Other approaches are specifically for collections 3. Veer and slow down: this would result in avoiding the ob- of autonomous flying vehicles, directly examine deviations ject detected by the LiDAR system. The vision system from expected control algorithm behavior to detect faulty cannot confirm this detection and is deemed unreliable agents and route communication around them(Negash, Kim, due to misjudgements. This is consistent to safely avoid and Choi ). The primary deficiency of these approaches is the object. Veering and slowing down causes less damage that, while they provide fault detection, they do not attempt to the vehicle occupants. any explanation of how the faults might have arisen, a defi- ciency we hope to address through our work. And the final explanation produced by the high-level rea- The final goal of this work is to use provide interpretable soner is shown in Figure 3. explanations. Within the context of self-driving car, previ- ous work has used reasoning systems, propagation (Radul Previous Work and Sussman 2009), and models of expected vehicle physics One goal of this work is to decrease the number of false pos- and electromechanical behavior to create causal chains that itives in anomaly detection by using explanations. Anomaly explain the events leading up to and in an accident (Gilpin detection is a well-studied area in data science and machine and Yuan 2017). This work is also being extended to include learning (Chandola, Banerjee, and Kumar 2009), even as commonsense rules of vehicle actions, so that it could mon- a tactic to combat intrusion detection in networks (Garcia- itor planning systems for inconsistent tactics. Teodoro et al. 2009). In developing anomaly detection for Our approach and position is similar to that proposed in autonomous systems, it is also necessary to develop real- Explainable Agency (Langley et al. 2017). This refers to the time anomaly detection algorithms. Real-time anomaly de- ability of autonomous agents to explain their decisions and be questioned. Although I adhere to many of the principles Grill, M.; Pevnỳ, T.; and Rehak, M. 2017. Reducing false of explainable agency, my goal is to extend these principles positives of network anomaly detection by local adaptive to full system design. multivariate smoothing. Journal of Computer and System Sciences 83(1):43–57. Conclusion and Discussion Langley, P.; Meadows, B.; Sridharan, M.; and Choi, D. 2017. Explainable agency for intelligent autonomous systems. In Even if machines are robust, their failures are poorly de- AAAI, 4762–4764. tected and explained. Further, is nearly impossible to inspect if there were any plausible counterfactural decisions. I.e., in Lavin, A., and Ahmad, S. 2015. Evaluating real-time the Uber self-driving car case, were there any other planning anomaly detection algorithms–the numenta anomaly bench- decisions that could have avoided the trafic mark. In Machine Learning and Applications (ICMLA), As we add more components to this machines, either to 2015 IEEE 14th International Conference on, 38–44. IEEE. make function autonomously, or to add more capabilities Negash, L.; Kim, S.-H.; and Choi, H.-L. Distributed and features, we are also increasing the number of ways unknown-input-observers for cyber attack detection and iso- that they can fail. With more components and connections, lation in formation flying uavs. detecting the root-cause becomes difficult, and without a Radul, A., and Sussman, G. J. 2009. The art of the propaga- proper reason or detection of error, this can prevent the ma- tor. In Proceedings of the 2009 international lisp conference, chine or operator to learn from the failures. 1–10. At the current time, a machine can only justify their ac- Vallis, O.; Hochenbaum, J.; and Kejariwal, A. 2014. A novel tions with incomprehensible log trace, unconvincing to those technique for long-term anomaly detection in the cloud. In who demand a human-readable justification in order to trust HotCloud. this machines actions. Further, current testing protocols do not accurately mimic real life. Testing in simulation cannot cover all the test cases, how can we ensure that these ve- hicles are tested properly and how can we ensure that they perform to their best ability in real scenarios? In“state-of-the-art” diagnostic systems, root cause analy- sis and human experts are inadequate. Deep neural networks, our most powerful perceptual mechanisms, are opaque to even the most knowledgeable human experts. Even if ma- chines can somehow communicate their failures and anoma- lies, the appropriate next steps are rarely obvious. For exam- ple, a “check engine” light on a vehicle does not indicate a specific failure, but rather indicates a need for unspeci- fied maintenance. Our approach develops the capability for a complex machine to be aware of and report on its internal state, including multiple decisions and failures, supported by reasoning and history. References Albanese, M.; Jajodia, S.; Pugliese, A.; and Subrahmanian, V. S. 2011. Scalable Detection of Cyber Attacks. Berlin, Heidelberg: Springer Berlin Heidelberg. 9–18. Chandola, V.; Banerjee, A.; and Kumar, V. 2009. Anomaly detection: A survey. ACM computing surveys (CSUR) 41(3):15. Garcia-Teodoro, P.; Diaz-Verdejo, J.; Maciá-Fernández, G.; and Vázquez, E. 2009. Anomaly-based network intrusion detection: Techniques, systems and challenges. computers & security 28(1-2):18–28. Gilpin, L. H., and Yuan, B. Z. Getting up to speed on vehicle intelligence. 2017 AAAI Spring Symposium Series. Gilpin, L. H., and Yuan, B. Z. 2017. Getting up to speed on vehicle intelligence. In AAAI Spring Symposium Series. Gilpin, L. 2018. Reasonableness monitors. In The Twenty- Third AAAI/SIGAI Doctoral Consortium at AAAI-18. New Orleans, LA: AAAI Press.