Causal Knowledge Graph for Scene Understanding in Autonomous Driving Utkarshani Jaimini1 , Cory Henson2 and Amit Sheth1 1 Artificial Intelligence Institute, University of South Carolina, Columbia, SC, USA 2 Bosch Center for Artificial Intelligence, Pittsburgh, PA, USA Abstract The current approaches to autonomous driving focus on learning from observation or simulated data. These approaches are based on correlations rather than causation. For safety-critical applications, like autonomous driving, it’s important to represent causal dependencies among variables in addition to the domain knowledge expressed in a knowledge graph. This will allow for a better understanding of causation during scenarios that have not been observed, such as malfunctions or accidents. The causal knowledge graph, coupled with domain knowledge, demonstrates how autonomous driving scenes can be represented, learned, and explained using counterfactual and intervention reasoning to infer and understand the behavior of entities in the scene. Keywords Causality, causal knowledge graph, intervention, counterfactual, autonomous driving 1. Causality in Autonomous Driving Driving is a complex activity that requires careful planning and constant attention. Human drivers analyze their surroundings based on observations, past experiences, and anticipation of potential scenarios and necessary actions. While autonomous vehicles are trained on observa- tional data, they face challenges in unfamiliar, uncertain, and risky driving situations. These vehicles operate in environments with various elements, such as traffic signs, pedestrians, and other vehicles. Understanding the relationships and interactions among these elements is crucial for comprehending the behavior of autonomous vehicles in different contexts. To achieve level 5 full driving automation, which requires a system capable of handling all driving tasks under any condition without human intervention, artificial intelligence (AI) models need high-quality representation, discovery, and understanding of causal relationships among the elements in the driving scene1 . An understanding of causality, as expressed in a causal Bayesian network (CBN) [1], would benefit from explicit representation in a knowledge graph (KG). This idea brings forth many important research questions. Can CBN-based causal representation in the driving scene KG help in understanding AD scenes? Can using CBN-based causal representation in the KG perform intervention and counterfactual reasoning, such as determining how a specific Posters, Demos, and Industry Tracks at ISWC 2024, November 13–15, 2024, Baltimore, USA $ ujaimini@email.sc.edu (U. Jaimini); cory.henson@us.bosch.com (C. Henson); amit@sc.edu (A. Sheth)  0000-0002-1168-0684 (U. Jaimini); 0000-0003-3875-3705 (C. Henson); 0000-0002-0021-5293 (A. Sheth) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings CEUR Workshop Proceedings (CEUR-WS.org) http://ceur-ws.org ISSN 1613-0073 1 https://www.autonews.com/mobility-report/ai-lacks-causal-inference-needed-av-edge-cases CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings element, like a stop line marking, affects the behavior of the vehicle? Or predicting the vehicle’s response if a pedestrian is jaywalking? What would be the impact if the vehicle fails to identify the stop line marking on the behavior concerning a pedestrian? 2. Causal Knowledge Graph for Autonomous Driving Scene Understanding A Causal Knowledge Graph (CausalKG) incorporates causal knowledge into a KG, including causal domain knowledge encoded within a causal Bayesian network (CBN), and automates causal inference tasks [2]. It leverages the strengths of CBNs, causal ontology, and KGs to deliver robust and explainable insights. The primary benefit of building a CausalKG lies in integrating causal knowledge into reasoning and prediction processes, which is crucial for safety-critical applications2 . This integration not only boosts the accuracy and reliability of current AI algorithms but also provides improved explainability of outcomes, thereby enhancing trust and confidence in the system. In the context of scene understanding for autonomous driving, a real-world AD dataset, Pandaset3 , was used to build a causal knowledge graph4 . The CausalKG contains causal relations and causal effect weights estimated using the data from Pandaset and a derived CBN. The causal effect weights are quantitative analyses of interventions on one or more variables in the dataset. When queried, the CausalKG provided insights into intervention and counterfactual reasoning, demonstrating its relevance and applicability for scene understanding. It was observed that a stop line marking (STL) has a higher causal effect on a pedestrian walking with an object, such as a stroller, backpack, umbrella, etc. Pedestrians with objects seem to be more responsible citizens, following traffic rules while crossing the street. If a pedestrian with an object is jaywalking (walking in a scene with no STL), there is a positive causal effect on the stopping of a vehicle. Jaywalking pedestrians with an object have a higher causal effect on stopping a vehicle than jaywalking pedestrians without an object, as vehicles or drivers tend to be more alert of pedestrians walking with an object. Similarly, if a pedestrian is standing at an STL in a scene, but the vehicle fails to identify the STL, the vehicle will continue to move. The causal effect estimated using the CBN and AD dataset, incorporated into a KG, provides a better explanation and understanding of interactions between the entities in the driving scene, enlightening us about the complex dynamics of the driving scene. CausalKGs can be used in the future to predict new causal entities in the driving scene. Acknowledgments: NSF Awards #2335967 and #2119654. References [1] J. Pearl, Causality, Cambridge university press, 2009. [2] U. Jaimini, A. Sheth, Causalkg: Causal knowledge graph explainability using interventional and counterfactual reasoning, IEEE Internet Computing 26 (2022) 43–50. 2 https://tinyurl.com/m5ukmn8m 3 https://scale.com/open-av-datasets/pandaset 4 CausalKG for autonomous driving: https://github.com/utkarshani/CausalKG-Pandaset