=Paper=
{{Paper
|id=Vol-3828/ISWC2024_paper_43
|storemode=property
|title=Causal Knowledge Graph for Scene Understanding in Autnomous Driving
|pdfUrl=https://ceur-ws.org/Vol-3828/paper43.pdf
|volume=Vol-3828
|authors=Utkarshani Jaimini,Cory Henson,Amit Sheth
|dblpUrl=https://dblp.org/rec/conf/semweb/JaiminiHS24a
}}
==Causal Knowledge Graph for Scene Understanding in Autnomous Driving==
https://ceur-ws.org/Vol-3828/paper43.pdf
Causal Knowledge Graph for Scene Understanding in
Autonomous Driving
Utkarshani Jaimini1 , Cory Henson2 and Amit Sheth1
1
Artificial Intelligence Institute, University of South Carolina, Columbia, SC, USA
2
Bosch Center for Artificial Intelligence, Pittsburgh, PA, USA
Abstract
The current approaches to autonomous driving focus on learning from observation or simulated data.
These approaches are based on correlations rather than causation. For safety-critical applications, like
autonomous driving, it’s important to represent causal dependencies among variables in addition to
the domain knowledge expressed in a knowledge graph. This will allow for a better understanding of
causation during scenarios that have not been observed, such as malfunctions or accidents. The causal
knowledge graph, coupled with domain knowledge, demonstrates how autonomous driving scenes can
be represented, learned, and explained using counterfactual and intervention reasoning to infer and
understand the behavior of entities in the scene.
Keywords
Causality, causal knowledge graph, intervention, counterfactual, autonomous driving
1. Causality in Autonomous Driving
Driving is a complex activity that requires careful planning and constant attention. Human
drivers analyze their surroundings based on observations, past experiences, and anticipation of
potential scenarios and necessary actions. While autonomous vehicles are trained on observa-
tional data, they face challenges in unfamiliar, uncertain, and risky driving situations. These
vehicles operate in environments with various elements, such as traffic signs, pedestrians, and
other vehicles. Understanding the relationships and interactions among these elements is crucial
for comprehending the behavior of autonomous vehicles in different contexts. To achieve level
5 full driving automation, which requires a system capable of handling all driving tasks under
any condition without human intervention, artificial intelligence (AI) models need high-quality
representation, discovery, and understanding of causal relationships among the elements in the
driving scene1 . An understanding of causality, as expressed in a causal Bayesian network (CBN)
[1], would benefit from explicit representation in a knowledge graph (KG). This idea brings
forth many important research questions. Can CBN-based causal representation in the driving
scene KG help in understanding AD scenes? Can using CBN-based causal representation in
the KG perform intervention and counterfactual reasoning, such as determining how a specific
Posters, Demos, and Industry Tracks at ISWC 2024, November 13–15, 2024, Baltimore, USA
$ ujaimini@email.sc.edu (U. Jaimini); cory.henson@us.bosch.com (C. Henson); amit@sc.edu (A. Sheth)
0000-0002-1168-0684 (U. Jaimini); 0000-0003-3875-3705 (C. Henson); 0000-0002-0021-5293 (A. Sheth)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings CEUR Workshop Proceedings (CEUR-WS.org)
http://ceur-ws.org
ISSN 1613-0073
1
https://www.autonews.com/mobility-report/ai-lacks-causal-inference-needed-av-edge-cases
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
element, like a stop line marking, affects the behavior of the vehicle? Or predicting the vehicle’s
response if a pedestrian is jaywalking? What would be the impact if the vehicle fails to identify
the stop line marking on the behavior concerning a pedestrian?
2. Causal Knowledge Graph for Autonomous Driving Scene
Understanding
A Causal Knowledge Graph (CausalKG) incorporates causal knowledge into a KG, including
causal domain knowledge encoded within a causal Bayesian network (CBN), and automates
causal inference tasks [2]. It leverages the strengths of CBNs, causal ontology, and KGs to
deliver robust and explainable insights. The primary benefit of building a CausalKG lies in
integrating causal knowledge into reasoning and prediction processes, which is crucial for
safety-critical applications2 . This integration not only boosts the accuracy and reliability of
current AI algorithms but also provides improved explainability of outcomes, thereby enhancing
trust and confidence in the system. In the context of scene understanding for autonomous
driving, a real-world AD dataset, Pandaset3 , was used to build a causal knowledge graph4 . The
CausalKG contains causal relations and causal effect weights estimated using the data from
Pandaset and a derived CBN. The causal effect weights are quantitative analyses of interventions
on one or more variables in the dataset. When queried, the CausalKG provided insights into
intervention and counterfactual reasoning, demonstrating its relevance and applicability for
scene understanding. It was observed that a stop line marking (STL) has a higher causal effect on
a pedestrian walking with an object, such as a stroller, backpack, umbrella, etc. Pedestrians with
objects seem to be more responsible citizens, following traffic rules while crossing the street. If
a pedestrian with an object is jaywalking (walking in a scene with no STL), there is a positive
causal effect on the stopping of a vehicle. Jaywalking pedestrians with an object have a higher
causal effect on stopping a vehicle than jaywalking pedestrians without an object, as vehicles or
drivers tend to be more alert of pedestrians walking with an object. Similarly, if a pedestrian is
standing at an STL in a scene, but the vehicle fails to identify the STL, the vehicle will continue
to move. The causal effect estimated using the CBN and AD dataset, incorporated into a KG,
provides a better explanation and understanding of interactions between the entities in the
driving scene, enlightening us about the complex dynamics of the driving scene. CausalKGs can
be used in the future to predict new causal entities in the driving scene. Acknowledgments:
NSF Awards #2335967 and #2119654.
References
[1] J. Pearl, Causality, Cambridge university press, 2009.
[2] U. Jaimini, A. Sheth, Causalkg: Causal knowledge graph explainability using interventional
and counterfactual reasoning, IEEE Internet Computing 26 (2022) 43–50.
2
https://tinyurl.com/m5ukmn8m
3
https://scale.com/open-av-datasets/pandaset
4
CausalKG for autonomous driving: https://github.com/utkarshani/CausalKG-Pandaset