This paper extends previous work on rational agents and epistemic causation in the situation calculus to devise an explanatory framework. It incorporates agents' prioritized goals and intentions, utilizes a black-box goal recognition module, and accommodates causal analysis of observed efects involving knowledge and intentions, caused by knowledge-producing and intention-altering actions, respectively. Leveraging an action theory and mental state formalization, it then illustrates -through a theory of mind-grounded model of explanation- that, in contrast to purely machine learning-based systems, knowledge representation-based systems might indeed be more suitable for generating explanations of observed behaviour.
Logic.
In recent years, researchers have become increasingly interested in developing transparent AI systems whose behaviour can be easily understood. To this end, numerous studies have explored how decisions produced by otherwise opaque sub-symbolic approaches can be explained. This has also led to a renewed interest in the study of explainability in knowledge representation (KR), as advocates of KR argue that its declarative nature makes it cognitively more suited for explanation purpose.
Over the years, there has been some work on formalizing explanations in KR [
Our framework is based on the situation calculus (SC) [7, 8], a model of knowledge [9, 10] and intentions [11] in the SC, and a formalization of actual causation [12] and causal knowledge [13] therein. We extend the framework to include goal change due to request communication actions and to support causal analysis of conative efects (i.e. efects involving agent motivation). We also utilize a black-box module for recognizing agents’ intentions. Using these, we propose a definition of explanation of agent behaviour relative to observed efects, one that is grounded in theory of mind. Our proposal here is informal, and we mostly focus on an example to illustrate the idea; a full-fledged formal version is available in [ 14].
Our base framework for this is the Situation Calculus (SC), which is a second-order (SO) language for modeling and reasoning about dynamic systems where all changes are result of named actions. Here, a possible state of the domain is represented by a situation. The initial situation 0 denotes the empty sequence of actions and (, ) denotes the successor situation to resulting from performing the action . Thus the domain of situations can be viewed as a tree, where the root of the tree is the initial situation 0 and the arcs represent actions. Properties whose truth values vary from situation to situation, are called fluents. We will use the complex situation term ([ 1, ⋯ , ], 0) to represent the situation obtained by consecutively performing 1, ⋯ , starting from 0.
In the SC, a dynamic domain is formalized using an action theory that includes the following set of axioms: (1) first-order (FO) action precondition axioms (APA), one per action, (2) (FO) successor-state axioms (SSA), one per fluent, that succinctly encode both efect and frame axioms and specify exactly when the fluent changes, (3) (FO) initial state axioms describing what is true initially, (4) (FO) unique names axioms for actions, and (5) (SO) domain-independent foundational axioms describing the structure of situations [15].
Following [9, 10], we model knowledge using a possible worlds account adapted to the SC. There can now be multiple initial situations. Using an accessibility relation , the knowledge of an agent is defined as a necessity operator over . is constrained to be reflexive and Euclidean (and thus transitive) in the initial situations to capture the fact that the agent’s knowledge is true, and that it has positive and negative introspection.
In our framework, the dynamics of knowledge is specified using a SSA for that supports knowledge expansion as a result of sensing actions as well as “inform” communication actions. As shown in [10], the constraints on then continue to hold after any sequence of actions since they are preserved by the SSA for .
Thus to model knowledge, we will use a theory that is similar to above, but with modified foundational axioms to allow for multiple initial epistemic states. Also, action preconditions can now include knowledge preconditions and initial state axioms can now include axioms describing the epistemic states of the agents. Finally, the preconditions of inform and aforementioned axioms for are included. See [8] and [16] for details of these.
Following Khan and Lespérance (KL) [17], we will utilize the sort of paths in the SC, which are essentially infinite sequences of executable situations. KL [ 18] showed how one can interpret arbitrary computational tree logic (CTL∗) [19] formulae within SC with paths. Paths are useful for formalizing future-oriented concepts such as goals, intentions, and other motivational states. We assume that our theory includes the axiomatization for paths.
In [11], KL proposed a formalization of prioritized goals (p-goals), intentions, and their dynamics in the SC. The account supports a rich specification of the goals of an agent. In their agent theory, an agent can have multiple goals/desires at diferent priority levels, possibly inconsistent with each other. They assume that goals are totally ordered with respect to the priority ordering. Each goal is specified using its own goal-accessibility relation , parameterized by the priority level. KL defined intentions/chosen goals, i.e. the goals that the agent is actively pursuing, as the maximal set of highest priority goals that are consistent with each other and with the agent’s knowledge; semantically, this is handled by taking a prioritized intersection of goal-accessibility relations. Their model of goals supports the specification of general temporally extended goals (represented by CTL∗ formulae), not just achievement goals. They also specified how these goals evolve when actions/events occur, when the agent’s knowledge changes, or when the agent adopts or drops a goal. This is specified via the SSA for . Their formalization of prioritized goal dynamics ensures that the agent always optimizes their intentions. They will abandon a chosen goal if an opportunity to commit to a higher priority goal, that is inconsistent with , arises. As such their model displays an idealized form of rationality.
We propose to adopt and modify KL’s framework to accommodate multiple agents, by adding an agent argument to the hierarchy of goal-accessibility relations. We also modify goal dynamics by introducing a request action req(, ′, ) , that can be used by an agent to request another agent ′ to adopt a p-goal , simplifying the model to only include extremely cooperative agents that always adopts the requested goal as their intentions (even if it is inconsistent with their current intentions; note, the requestee’s intentions do remain consistent). For this we propose the APA for this req action and update the SSA for ; see [14] for the formal details.
Given a history of actions/events (often called a scenario) and an observed efect, actual causation involves figuring out which of these actions are responsible for bringing about this efect. When the efect is assumed to be false before the execution of the actions in the scenario and true afterwards, the notion is referred to as achievement (actual) causation. Based on Batusov and Soutchanski’s original proposal [12], KL recently ofered a definition of achievement cause in the SC [13]. Both of these frameworks assume that the scenario is a linear sequence of actions, i.e. no concurrent actions are allowed. KL’s proposal can deal with epistemic causes and efects; e.g., an agent may analyze the cause of some newly acquired knowledge, and the cause may include some knowledge-producing action, e.g. inform. They showed that an agent may or may not know all the causes of an efect, and can even know some causes while not being sure about others.
In this framework, causes are computed relative to a causal setting consisting of a domain theory , a scenario (which, again, is a linear sequence of actions), and an efect . Since all changes in the SC result from actions, they identified the potential causes with a set of ground action terms occurring in . However, since might include multiple occurrences of the same action, one also needs to identify the situations where these actions were executed; for brevity, we will ignore this component here. The underlying idea of computing causes is as follows. If some action of the action sequence in triggers the formula to change its truth value from false to true relative to , and if there are no actions in after that change the value of back to false, then is an actual cause of achieving in . Moreover, note that might have been non-executable initially; so other preceding actions that contributed to ensuring that its preconditions are brought about must also be considered as (indirect) cause of . Similarly, might have only brought about conditionally, and other preceding actions that achieved those conditions must be considered as (indirect) cause of . Using this reasoning, in addition to the single action that brings about the efect of interest, one can also capture the chain of actions that build up to it.
We propose to extend KL’s framework in [13] to include intentions in efect formulae; see [14] for how this can be done. With this extension, we can now analyze the causes of an agent having some intention in some scenario. In our framework, such efects are usually caused by request actions by others.
To propose a model of explanation on top of this framework, we need one last component, which for this work is considered to be a black-box module. Given an agent , an action , and a scenario , the component under consideration is a goal recognition module, which recognizes the intention of behind performing in . With this, we are now ready to give a definition of explanation. Just like causes, explanations in our framework are simply actions from the scenario. However, as we will see, they are not simply causes.
Explanation We say that the behaviour of a group of agents captured by a scenario relative to the observation that can be explained by the action if and only if is a cause of in ; or causes the intention behind some explanation of in , i.e. some other action ′ explains in , the agent of ′ is ′, ′ is recognized to have the intention that behind performing ′ in s, and was the cause of this intention in ′.
We consider a domain where we have two rescue drone agents, 1 and 2, navigating through four locations, , , 1, and ′1, and managed by a controller agent . We have a non-fluent relation, Route(, ′), that represents a flight path from location to ′. The routes are defined as: from to 1, from to ′1, from 1 to , and from ′1 to . In this domain, there are four self-explanatory actions (given and are drones and locations, resp.): takeOf (, ) , flyTo (, , ′), land(, ) , and the aforementioned communicative action inform. The fluents in this domain are At(, , ) , Flying(, ) , Vis(, , ) (i.e. has visited in ), and TStorm(, ) (i.e. there is a thunderstorm at in ).
The preconditions for the above actions are as follows. Agent can takeof at location in situation if it is at in and it is not flying in . can fly to ′ from in if it is at in , it is lfying in , there is a route from to ′, and does not know that there is a thunderstorm at ′ in . can land at in if it is at in and it is flying in . Finally, can inform ′ that Φ in if knows in that Φ and it does not know in that ′ knows that Φ.
The SSA for the above fluents are as follows. is at location after executing action in situation if refers to ’s action of flying from some location ′ to , or was already at in and is not its action of flying to a diferent location ′. is flying in (, ) if is ’s action of taking of at some location , or it was already flying in and is not its action of landing at some location. has visited in (, ) if refers to its action of flying to from some other location, or it has already visited in . Finally, there is a thunderstorm at in (, ) if this is the case in (for simplicity, we treat this as a non-fluent).
The Knowledge of agents initially are as follows. Drone 1 knows that it is at location , that it is not flying, and that it has only visited . Moreover, it does not know that there is a storm at location 1, but knows that there are no storms at ′1 and . There is indeed a thunderstorm at location 1 and the controller agent knows this. Finally, does not know however that the other agents know this fact.
Assume that, initially our drone agent 1 has the following two p-goals: 0 = ◇At( 1, ), i.e. that it is eventually at , at the highest priority level, and 1 = Vis( 1, 1) ℬ Vis( 1, ), i.e. that it visits 1 before it visits , at a lower priority level, respectively. Also, does not have any initial p-goals.
To see an example of intention dynamics, note that in our example, we can show that the agent 1 will have the intention that ◇Vis( 1, ′1) after 1 takes of from , informs 1 that there is a thunderstorm at 1, and requests 1 to eventually visit ′1, starting in the initial situation; thus we can show that 1 intends to eventually visit ′1 afterwards. But 1 will not have the intention that 1 afterwards as it has become impossible for 1 to visit 1 due to its knowledge of the thunderstorm at 1.
Next, let us consider an example of causation relative to conative efects. Assume that = do([takeOf ( 1, ), inform( , 1, TStorm( 1)), req( , 1, ◇Vis( 1, ′1)), inform( , 2, TStorm( 1)), req( , 2, ◇Vis( 1, ′1)), flyTo ( 1, , ′1), flyTo ( 1, ′1, )], 0). There are 7 actions in this scenario. For convenience, we will use ⃗ to denote the first actions in this trace, and so ([ ⃗5], 0) is the situation obtained from executing the first 5 actions starting in 0. Now assume that we want to reason about the causes of the efect 1 = Int( 1, ◇Vis( 1, ′1)) in scenario 1 = ([ ⃗5], 0). We can show that ’s request to 1 to eventually visit ′1 is the only cause of 1’s intention that ◇Vis( 1, ′1) in 1. Thus, e.g., req( , 2, ◇Vis( 1, ′1)) is not a cause.
Finally, we would like to explain the behaviour of drones as modeled by situation/scenario above relative to the efect that 2 = Vis( 1, ′1), i.e. we want to understand why 1 visited ′1 (rather than the usual path of 1). As expected, we can show that agent behaviour in w.r.t. visiting ′1 can be explained by 1’s action flyTo ( 1, , ′1). But perhaps more interestingly, assuming that the intention behind 1’s action of flyTo ( 1, , ′1) in was recognized to be ◇Vis( 1, ′1), we can further explain agent behaviour via the causes of having this intention. This will in turn reveal that 1 had this intention due to ’s request to 1 to eventually visit ′1, and thus agent behaviour w.r.t. 1 visiting ′1 can be explained by this request action as well.
In this paper, we sketched an account of causal reasoning about motivations. Using this, we ofered a novel take on explainable AI that is grounded in theory of mind: agent behaviour in our framework can be explained via the causal analysis of observed efects, which in turn can trigger the analysis of their mental states.
As mentioned, there has been some work on formalizing explanation in KR. For instance, in
his early work, Shanahan [
Our current formalization is limited in many ways. For instance, we only allow deterministic and fully observable actions. Scenarios in our framework are linear, i.e. we assume that the order of action occurrence is known. When dealing with causation and explanations, we computed achievement causes only. Incorporating other types of causes, e.g. maintenance causes [22], would have allowed us to explain efects further and in finer details. We leave these for future work.
We thank the anonymous reviewers of CAKR 2023, XLoKR 2023, and ECAI 2023 for directing us to the relevant literature. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), [funding reference number RGPIN-2022-03433]. MAS 2020, Auckland, New Zealand, May 9-13, 2020, Revised Selected Papers, volume 12175 of Lecture Notes in Computer Science, Springer, 2020, pp. 75–93. [5] T. Miller, Contrastive explanation: a structural-model approach, Knowl. Eng. Rev. 36 (2021) e14. [6] L. A. Dennis, N. Oren, Explaining BDI agent behaviour through dialogue, Auton. Agents
Multi Agent Syst. 36 (2022) 29. [7] J. McCarthy, P. J. Hayes, Some philosophical problems from the standpoint of artificial intelligence, Machine Intelligence 4 (1969) 463–502. [8] R. Reiter, Knowledge in Action. Logical Foundations for Specifying and Implementing
Dynamical Systems, MIT Press, Cambridge, MA, USA, 2001. [9] R. C. Moore, A formal theory of knowledge and action, in: Formal Theories of the
Commonsense World, Ablex, 1985, pp. 319–358. [10] R. B. Scherl, H. J. Levesque, Knowledge, action, and the frame problem, Artificial Intelligence 144 (2003) 1–39. [11] S. M. Khan, Y. Lespérance, A logical framework for prioritized goal change, in: W. van der Hoek, G. A. Kaminka, Y. Lespérance, M. Luck, S. Sen (Eds.), 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2010), Toronto, Canada, May 10-14, 2010, Volume 1-3, IFAAMAS, 2010, pp. 283–290. [12] V. Batusov, M. Soutchanski, Situation calculus semantics for actual causality, in: S. A.
McIlraith, K. Q. Weinberger (Eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, AAAI Press, 2018, pp. 1744–1752. [13] S. M. Khan, Y. Lespérance, Knowing why - on the dynamics of knowledge about actual causes in the situation calculus, in: F. Dignum, A. Lomuscio, U. Endriss, A. Nowé (Eds.), AAMAS ‘21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021, ACM, 2021, pp. 701–709. [14] S. M. Khan, M. Rostamigiv, On explaining agent behaviour via root cause analysis: A formal account grounded in theory of mind, in: Proceedings of the 26th European Conference on Artificial Intelligence ECAI, 30.09 - 5.10, 2023, Kraków, Poland, 2023. [15] H. J. Levesque, F. Pirri, R. Reiter, Foundations for the situation calculus, Electronic
Transactions on Artificial Intelligence (ETAI) 2 (1998) 159–178. [16] S. M. Khan, Y. Lespérance, ECASL: a model of rational agency for communicating agents, in: F. Dignum, V. Dignum, S. Koenig, S. Kraus, M. P. Singh, M. J. Wooldridge (Eds.), 4th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2005), July 25-29, 2005, Utrecht, The Netherlands, ACM, 2005, pp. 762–769. [17] S. M. Khan, Y. Lespérance, Infinite paths in the situation calculus: Axiomatization and properties, in: C. Baral, J. P. Delgrande, F. Wolter (Eds.), Principles of Knowledge Representation and Reasoning: Proceedings of the Fifteenth International Conference, KR 2016, Cape Town, South Africa, April 25-29, 2016, AAAI Press, 2016, pp. 565–568. [18] S. M. Khan, Y. Lespérance, Infinite Paths in the Situation Calculus, Technical Report EECS2015-05, Department of Electrical Engineering and Computer Science, York University, Toronto, Canada, 2015. [19] E. A. Emerson, J. Y. Halpern, “sometimes” and “not never” revisited: On branching versus linear time temporal logic, J. ACM 33 (1986) 151–178. [20] M. Sridharan, REBA-KRL: refinement-based architecture for knowledge representation, explainable reasoning and interactive learning in robotics, in: G. D. Giacomo, A. Catalá, B. Dilkina, M. Milano, S. Barro, A. Bugarín, J. Lang (Eds.), ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, IOS Press, 2020, pp. 2935–2936. [21] M. Sridharan, B. Meadows, Towards a theory of explanations for human-robot collaboration, Künstliche Intell. 33 (2019) 331–342. [22] V. Batusov, M. Soutchanski, Situation calculus semantics for actual causality, in: A. S.
Gordon, R. Miller, G. Turán (Eds.), Proceedings of the Thirteenth International Symposium on Commonsense Reasoning, COMMONSENSE 2017, London, UK, November 6-8, 2017, volume 2052 of CEUR Workshop Proceedings, CEUR-WS.org, 2017.