An assistive AI agent often has to collaborate with previously unseen humans. Methods considered state of the art for such ad hoc teamwork use a large labeled dataset of prior observations to model the behavior of other agents and to determine the ad hoc agent's behavior. These approaches are resource-hungry, and do not support rapid incremental revisions or transparency. Our previous work introduced an architecture that enabled an ad hoc agent to choose its actions in simple simulated domains based on non-monotonic logical reasoning with prior domain knowledge and models learned from limited examples to predict the behavior of other agents. Here, we extend this architecture to enable an ad hoc (AI) agent to collaborate with a human performing household tasks in a complex indoor environment, focusing on the ad hoc agent's ability to reason with relevant knowledge and provide relational descriptions as explanations of its behavior and that of the human. We evaluate our architecture's capabilities in VirtualHome, a realistic, physics-based, 3D simulation environment.
Humans and Embodied
CEUR Workshop
ISSN1613-0073
VirtualHome
State observations
Human action
Ad hoc agent (embodied AI)
ASP program & action predictions
Knowledge representation and
reasoning Declarative knowledge
Reasoning
Learning using decision heuristics Human behaviour model
Action predictions
Explanation generator text converter
analyser
State observations
Agent action
Query
Explanation 2. Automatically construct on-demand relational descriptions of its decisions and those of the human agent as explanations in response to diferent types of questions.
We evaluate these capabilities in the context of household tasks performed in VirtualHome, a 3D
simulation environment that allows us to explore an agent’s interaction with the environment and
other agents in a realistic and physically consistent manner [
Example Domain 1. [Example Embodied AI Agent Domain] Consider an embodied AI agent and a human agent collaborating to complete household tasks; Figure 1 (top left) shows a snapshot while preparing breakfast. They can interact with the environment through high-level actions, e.g., move to places, pick up or place objects, switch appliances on or of, and open or close appliances. Completing a task requires a sequence of such actions to be executed by the AI agent or the human who do not communicate directly with each other. Prior commonsense knowledge of the AI agent includes relational descriptions of some attributes of the domain, itself, and the human (Section 2.1); a learned (or encoded) graph of information about likely locations of objects in the domain; default statements that hold in all but a few exceptional circumstances; and some axioms governing actions and change, e.g., the agent can only pick one object at a time. The agent assumes that the human has access to the same state information and will make (what it considers) rational decisions. It determines its action choices through non-monotonic logical reasoning with the prior domain knowledge and an incrementally learned model of the human’s behavior, as described below.
In REACT, the transition diagram of any given domain is described using an extension of action
language ℒ [
causes _ℎ(, ) ℎ( ) if ( , ), ℎ impossible (, )
_(), ( ) if (, ), () (1a) (1b) (1c) Statement 1(a), a causal law, implies that grabbing an object causes it to be in the hand of the ad hoc agent; Statement 1(b), a state constraint, implies that a food item placed in an electrical appliance (e.g., microwave) that is switched on gets heated; and Statement 1(c), an executability condition, prevents the ad hoc agent from trying to grab an object from an appliance with a closed door. History ℋ is a record of observations of the form ( , , ) , and action executions of the form ℎ(, ) at specific time steps. It also includes default statements that are true in the initial state.
To reason with knowledge, we automatically construct program Π( , ℋ ) in CR-Prolog, an extension
to ASP that supports consistency restoring (CR) rules. Π( , ℋ ) includes statements from and
ℋ, inertia axioms, reality check axioms, closed world assumptions for defined fluents and actions,
helper relations, e.g., ℎ( , ) and (, ) to imply that a fluent is true and an
action is part of a plan at a time step, and helper axioms that define goals and guide planning and
diagnostics. ASP encodes default negation and epistemic disjunction, and supports non-monotonic logical
reasoning, an essential ability for agents reasoning and acting based on incomplete knowledge and
noisy observations. The CR rules allow the agent to make assumptions (e.g., that a default statement
does not hold) under exceptional circumstances to recover from inconsistencies. All reasoning tasks (i.e.,
planning, diagnostics, and inference) are then reduced to computing answer sets of Π. We use the SPARC
system [
Our example scenario is complex, with many objects and actions; the corresponding tasks require
multi-step plans. To enable the eficient use of logical reasoning in such scenarios, REACT represents
and reasons with knowledge at two (formally-coupled) levels of abstraction by building on our prior
work on a refinement-based architecture [
REACT enables the ad hoc agent to reason with models that predict the human agent’s action(s) in any
given state. Similar to our prior work, REACT uses the Ecological Rationality (ER) principle [
Specifically, REACT enables the ad hoc agent to learn an ensemble of Fast and Frugal (FF) trees that predict the human agent’s behavior; each FF tree provides a binary choice for a particular action and the number of leaves in the tree is limited by the number of attributes [19]. An example of one such FF tree in an ensemble is in Figure 2. The initial version of these trees were built using only 100 traces of human agent’s action choices and domain states from the VirtualHome domain, with the corresponding attributes in Table 1. Furthermore, consistent agreement (disagreement) between observations and
Description of the attribute Two previous actions of the human Two previous objects human interacted with Position and orientation of the human Distance from human to kitchen table Distance from human to kitchen counter Number of objects on kitchen table predictions of an existing model triggers model choice (revision); the ad hoc agent is thus able to quickly adapt to changes in the domain or the human’s behavior.
An automated decision-making system’s ability to justify its choices promotes acceptability [20, 21]. Unlike methods that seek to make a complex learned model interpretable, or to explain all the choices made by a reasoning system, REACT responds to any given question about specific decisions by quickly identifying the relevant information and constructing relational descriptions. It does so by building on the underlying logic-based representation of knowledge and simple predictive models, and on prior work in our group that identified the axioms and literals relevant to the questions posed to an AI system making decisions [22]. Specifically, REACT enables the ad hoc agent to respond to four types of questions identified as being important in work on explainable planning [ 21]. 1. (Causal questions) Why did you execute , i.e., action at step ? • If is not the last action of plan executed by the agent, extract actions {A } ∈ P that occurred after . Identify axioms that have negation of a + ∈ {A } in head, i.e., prevent a + from occurring. • Check if each such axiom’s body is satisfied by answer set at . If yes, identify fluent literals in body whose value changed from to + 1 . Collect all such literals { } over all identified axioms have been changed by the execution of . Use literals to construct answer.
• If is last action in , it contributed to achieving the goal. Use goal to construct answer. 2. (Contrastive questions) Why did you not execute , i.e., action at ? • Identify axioms with the negation of in head. If any such axiom’s body is satisfied by answer set at , collect fluent literals { } in body as they prevented consideration of . • If not, identify axioms in Π( , ℋ ) with in its body with other literals { ′}, i.e., causal laws.
Extract the precondition literals { ′} and identify axioms with ∈ { ′} in its head, i.e., state constraints. Check if each such axiom’s body is satisfied by answer set at step . If not, use literal ∈ { ′} and the body literals of axiom to construct answer. 3. (Justify beliefs) Why did you believe , i.e., l at step I? • Replace ground terms of with variables. Identify axioms with in head, and check whether body is satisfied by answer set at step . If yes, collect fluent literals { } in body as they support . • Create a tree with as its head and each selected axiom as a branch. With each axiom, store the supporting { } fluents. Repeat for each fluent literal in { } as target belief until no more axioms are identified. Use collected literals for answer. 4. (Counterfactual Questions) What will be the outcome of executing ′? • Retrieve most recent state of environment − with respect to step . Retrieve corresponding action for ad hoc agent ( − ) and human ({ − }, use predictive models). • Perform mental simulation of the future step − + 1 from − using existing knowledge and action choices of agents { − , − }. Repeat times, i.e., roll out the future and explore efects of actions of ad hoc agent and human until queried step . Collect state information at . • Retrieve agent’s action and predicted human action in . Traverse FF tree model of human to identify and collect active branches for to construct answer.
obtain the resultant state +1 to be used to construct answer. • Replace action ( ) with ′ for the human (ad hoc agent). Roll out environment one step to For each type of question, identified literals are processed using existing tools and templates to generate descriptions as responses (i.e., explanations) before, during, or after planning or execution.
We experimentally evaluated two hypotheses regarding REACT’s capabilities: • H1: The combination of reasoning and learning in REACT enables the ad hoc agent to adapt to changes and perform better than a logic-based baseline; and decisions and beliefs and those of the human.
• H2: REACT enables the ad hoc agent to generate relational descriptions as explanations of its The performance measures were precision and recall of explanations, number of steps (i.e., plan length), and total time taken (by human and AI agent) to complete the task.
In the VirtualHome domain, we modeled the human as a simulated entity that chooses its actions based on an ASP program. The human was assigned the same goal as the embodied AI (ad hoc) agent, e.g., prepare and eat breakfast. The baseline operation involved each of them receiving the same observations of the domain at each step; they then continued with their planned actions or computed a new plan as needed. There was no direct communication between them, and they did not have any prior knowledge or model of each other’s capabilities. With REACT, the key diference was that the ad hoc agent used the learned models to predict a couple of (future) actions of the human, and its ASP program included additional axioms for reasoning about these predicted actions. The agent’s plan thus expects the preconditions for some intermediate steps to be created by the actions executed by the human although the human may not always do so. The human, on the other hand, could not predict the ad hoc agent’s actions and the axioms to reason about these actions were not included in the corresponding ASP program. The human’s actions were primarily determined by the current state and goal. The human’s ASP program also includes actions such as eating and drinking that were not available to the ad hoc agent. Furthermore, the human’s ASP program encoded some priorities and preferences, e.g., when preparing breakfast, the human toasted the bread first before preparing the cereal.
As described in Section 2.2, each model predicting human behavior was an ensemble of FF trees based on just 100 prior traces of human actions and domain state. These observations also provided priors regarding likely location of objects, which were used to simplify planning (end of Section 2.1).
To evaluate H1, we designed Exp1 by first constructing 720 diferent configurations, each with a diferent arrangement of objects in the initial condition, e.g., bread on the kitchen table instead of the counter, microwave open instead of closed. We then randomly chose 100 of these configurations and measured the ability to achieve the shared goal (e.g., prepare and eat breakfast) with each of our options: REACT and baseline. To evaluate H2, we designed Exp2 in which we randomly selected 10 configurations (from the 100 in Exp1 and saved the corresponding answer sets (with REACT, baseline) to provide ground truth. Then, we posed 32 diferent questions (divided between four types of questions) about some chosen steps in each trial corresponding to one of these 10 configurations, with answers computed as described in Section 2.3. We recorded the precision and recall of retrieving literals to answer these questions. Furthermore, we considered execution traces as qualitative evaluation of H2.
based on the initial configuration, we computed the values of the performance measures for REACT as a fraction of those for the baseline in each trial, and reported the average of these ratios in Table 2. We observed that REACT significantly reduces the number of steps and the time taken to achieve the goal compared with the baseline, supporting H1. The prediction accuracy of the behavior models learned by the ad hoc agent for the human (Section 2.2) was 85.19%, i.e., the model does make errors, but it supports rapid revision and reasoning with it and prior knowledge significantly improves performance. In other experiments (not reported here), we noticed that the ad hoc agent was able to revise these models to track (or adapt to) changes in the human’s behavior.
Table 3 summarizes results of Exp2, with precision and recall calculated relative to all the literals expected to be included in the explanation. The observed high values of precision and recall indicate the ability to automatically extract the correct literals to provide relational descriptions as explanations in response to diferent types of questions, thus supporting H2. As an additional example, consider an exchange with the ad hoc agent after it collaborated with the human to prepare breakfast. • Questioner: “Why did you find bread slice in step 0 ?” • Ad hoc Agent: “Because I had not found the bread slice yet and wanted to grab it in step 1”.
This response identifies an action’s outcome being a requirement for a subsequent action. • Questioner: “Why did you not find the water glass in step 0?” • Ad hoc Agent: “Because I predicted that the human will find the water glass in step 0.”
The agent may be asked about the human’s actual or hypothetical action choices. • Questioner: “Why do you think human will grab water glass in step 1?” • Ad hoc Agent: “Because my prediction is that the human wants to bring the glass to the table.” • Questioner: “What if the human decided to grab the cutlets in step 1 ?” • Ad hoc Agent: “If the human grabs cutlets in step 1, they will be in human’s hands in step 2.” To answer questions about hypothetical situations, the ad hoc agent simulates the evolution of state, and the execution of actions by the human and the agent.
This paper described REACT, an AHT architecture for an embodied AI agent to collaborate with a human by reasoning with prior commonsense domain knowledge and incrementally learned models predicting the behavior of the human. REACT combines the principles of non-monotonic logical reasoning and ecological rationality, automatically identifying and reasoning eficiently with the relevant knowledge and observations. Also, the interplay between reasoning and learning enables the embodied AI agent to provide relational descriptions as on-demand explanations of its own decisions and those of the human. Experimental evaluation in a physics-based simulation environment demonstrates performance improvement compared with a logical reasoning baseline. Future work will explore: (a) humancontrolled avatars in our current experiments; (b) incremental learning of domain knowledge in more complex domains; (c) scalability to multiple ad hoc agents and humans; and (d) implementation and evaluation on physical robots in AHT settings.
This work was supported in part by the U.S. Ofice of Naval Research Award N00014-20-1-2390. All conclusions are those of the authors alone.
[19] K. Katsikopoulos, O. Simsek, M. Buckmann, G. Gigerenzer, Classification in the Wild: The Science and Art of Transparent Decision Making, MIT Press, 2021. [20] S. Anjomshoae, A. Najjar, D. Calvaresi, K. Framling, Explainable agents and robots: Results from a systematic literature review, in: International Conference on Autonomous Agents and Multiagent Systems, Montreal, Canada, 2019. [21] M. Fox, D. Long, D. Magazzeni, Explainable Planning, in: IJCAI Workshop on Explainable AI, 2017. [22] M. Sridharan, Integrated Knowledge-based Reasoning and Data-driven Learning for Explainable Agency in Robotics, in: Explainable Agency in Artificial Intelligence: Research and Practice, CRC Press, 2024.