In this work, we propose a novel architecture for agents to be employed in Human-AI Teaming in various, even critical, domains based upon afective computing, empathy, and Theory of Mind, and a description of the user profile and the operational, professional, and ethical requirements of the domain in which the agent operates. In this paper, we outline the architecture's building blocks and their interconnections. The architectural design agent encompasses a Knowledge Graph to enclose the above-mentioned kinds of knowledge and a Behavior Tree enhanced via a Neural component, where the latter elaborates sensor input from devices that monitor the user and input from the knowledge graph. The enhanced behavior tree actually interacts with the user, making actions or providing suggestions, and returns feedback to feed to the knowledge graph as a novelty in the literature. We briefly present a case study, on which to experiment once the implementation, which is presently at an initial stage, will be completed. We discuss in some detail the Prolog program implementing the behavior tree, and discuss why we chose Prolog.
One recent focus in Artificial Intelligence (AI) is building intelligent systems where humans and AI systems form teams. This to exploit the potentially synergistic relationships between human and automation, thus devising “hybrid” systems where the partners should cooperate to perform complex tasks, possibly involving a high degree of risk. As a simple example, in an AI-supported self-driving or assisted-driving vehicle, the AI component can be expected to evaluate and co-manage situations and risks, where the driver can provide the AI component with useful information on practical driving in all conditions and can self-manage the risks in the case this should be required by the circumstances. Human-automation interaction is, in fact, one of the main themes of Human-centered AI. This issue also falls in the realm of Trustworthy AI, whose requirements include respect for human autonomy, prevention of harm, fairness, and explainability, and of Responsible AI, whose goal is to employ AI in a safe, trustworthy and ethical fashion.
AI and humans, if working together in Human-AI Teaming (HAIT), can produce results exceeding what either can achieve alone, whereas they can control and improve each other. For instance, a human driver might train to cope with previously unseen situations through co-driving automation via a cooperative task shared between the human driver and the AI-based system installed on the vehicle. At the same time, AI helps drivers in case of dificulties and immediate risks. In this synergistic relationship, humans may improve automation eficacy and capabilities. At the same time, automation may enhance human performance in a task and compensate for human inadequacies, catching and correcting possible misbehaviors, possibly also due to physically or emotionally impaired states, and providing valuable suggestions.
For the tasks of adopting AI agents in crucial tasks such as, e.g., improving care-giving in medicine and teaching and constructing efective human-AI teams, agents should be endowed with an emotion recognition and management module, capable of empathy and modelling aspects of the Theory of Mind (ToM), in the sense of being able to reconstruct what someone is thinking or feeling. Modelling a Theory of Mind is often based on forms of “Afective Computing”, which is a set of techniques aimed at eliciting a human’s emotional condition from physical signs, to enable the system to respond intelligently to human emotional feedback.
In this work, we devise the architectural design of an agent to be employed in HAIT, based upon afective computing, empathy, and Theory of Mind, and a description of the user profile and of the operational, professional and ethical requirements of the domain in which the agent operates. Our main contribution is the design of the architecture, which includes a Knowledge Graph (KG), a Neural component, and a Behavior Tree (BT) as its main components. As such, we term the proposed framework NEMO (“Neural EMOtional”).
The KG will encompass definitions concerning all the required kinds of knowledge, plus a history of past interactions of the agent with the user. The BT enhanced via a Neural component, where the latter elaborates sensor input from devices that monitor the user and input from the KG, will actually interact with the user, making actions or providing suggestions, and return feedback to feed to the KG. This two-way relationship between KG and BT constitutes a novelty in the literature.
We provide a Prolog implementation of the BT and its relationships with the other components of the proposed NEMO framework. We have chosen Prolog for our implementation for reasons of flexibility and readability apt for the task.
The paper is organized as follows. Section 2 provides some background on behavior trees and knowledge graphs. Section 3 provides the overview of the envisaged framework. The Prolog implementation of the BT component is illustrated in Section 4, and a relevant case study for the future implemented system is outlined in Section 4.1. After discussing related work and some open issues in Section 5, we conclude the paper in Section 6.
Behavior trees were introduced as a tool to enable modular AI in computer games. A BT is essentially a
mathematical model of plan execution, where each element (task and action) of a plan is associated
with a node in the tree. Their strength comes from their ability to create complex tasks composed of
simple tasks without worrying about how the simple functions are implemented. For a comprehensive
survey of BTs in Artificial Intelligence and Robotic applications, see [
The critical nodes in a BT include leaf nodes and inner nodes. An action is a leaf node representing a behavior that the character can perform. The action returns success or failure when it completes its execution, depending on the outcome. An action is depicted as a white circle. A condition is a leaf node that checks an internal or external state. It returns either success or failure. A condition is represented as a grey rounded rectangle. A sequence selector is an inner node with several child nodes executed sequentially. Once a child node completes its execution successfully, the sequence selector continues executing the next child node. If every child node returns success, then the sequence selector returns success. If one of the child nodes returns failure, the sequence selector immediately returns failure. A sequence selector is depicted as a grey square with an arrow across the links to its child nodes. A priority selector is an inner node. It has a list of child nodes that it tries to execute one at a time until one of the child nodes returns success. If none of the child nodes executes successfully, the priority selector returns failure. A priority selector is represented with a grey circle with a question mark.
To consider empathy and mimic human decision-making, in [
The emotional selector is a node that orders its child nodes based on the agent’s current afective state. The agent elaborates on the afective state during repeated interactions with the user and then tunes its reaction accordingly. Once the ordering has been established, the emotional selector behaves as a priority selector. A white circle with the character E represents an emotional selector. In contrast, an empathy node provides an emotional evaluation of its single child node. An empathy node can only be a child of an emotional selector. Its child can be a leaf or an inner node. A dashed circle line with the name of the empathy emotion represents an empathy node.
To enable the integration of deep learning models for emotion recognition and symbolic models for planning and decision-making within emotional behavior trees, we introduced neural nodes. A neural node takes the current state of the environment and agent as input and, using a deep learning model, makes inferences about the emotional state. It contains a model, such as an emotion recognition system, that estimates the emotional state. These estimates are then mapped into the agent’s afective state variables that parameterize the emotional selector. The neural node continually updates the agent’s internal emotional state, allowing the dynamic adaptation of behavior trees to the emotional context.
At present, machine learning algorithms can help classify individuals’ emotions depending on the
input data. Emotions can be recognized from a wide spectrum of input data, encompassing physiological
data, speech [
Knowledge Graphs are a particular type of knowledge base [
A noteworthy technique that is commonly exploited for tasks on KGs is Knowledge graph embeddings
(KGEs) [
In this work, we use KGs to represent (various kinds of) knowledge in the proposed NEMO framework. The main reasons why we employ KGs and prefer it over other types of knowledge base are as follows. • First, the graph structure underlying KGs allows for capturing well the interrelations between the various entities of interest, and having such interrelations always and easily available, without performing possibly expensive operations (e.g., join operations in relational databases). This is eagerly needed in our context, as the proposed NEMO framework needs to continuously select specific portions of knowledge to be passed to the NEABT (see Section 3). Thus, this operation needs to be done eficiently. • Second, KGs allow for the representation of heterogeneous entities and relationships among them. • Third, KGs are highly flexible and versatile, as the knowledge therein need not to be organized according to any a-priori fixed schema. As such, KGs can easily integrate knowledge from diferent sources and perform updates on the acquired knowledge. Support for source heterogeneity and lfexibility/versatility are particularly required for the proposed NEMO framework. In fact, as detailed in Section 3, the KG in NEMO encompasses diferent types of knowledge, possibly coming from diferent sources (i.e., domain knowledge of various forms and user profiles). Also, our NEMO framework has a mechanism for continuously updating the KG. • Lastly, KGs can easily, yet efectively be represented as numerical vectors. This is required in our context in order to make KGs amenable to be processed by components of the proposed NEMO framework which do require numerical representations (i.e., the NEABT component, see Section 3). KGE techniques (see above) can represent KGs numerically. These are well-established techniques for which one can exploit the results of the corresponding research area, which has been very active and fruitful in the last few years.
The architecture of the proposed NEMO framework is illustrated in Figure 1. The main components of the architecture are the User, the Environment, a Knowledge Graph, and a Behavior Tree, specifically a NEABT. In this framework, the agent is identified by the NEABT. The overall interaction between the components of the framework is described next.
The NEABT is fed with signals from Environment, User, and KG. Such signals are exploited by the NEABT to perform its computation and to output () an action to be suggested to the User, () an action actually performed by the agent (e.g., an empathetic action), and () User’s emotion detected by its neural node (‘N’, see below). Threefold NEABT’s output passes through an “Aggregator”, responsible for suitably aggregating and presenting NEABT’s outputs to the User. The Aggregator may perform something either very simple (e.g., derive a textual representation of the three outputs and concatenate them) or more sophisticated (e.g., exploit a large language model (LLM)). NEABT’s outputs and User’s feedback are used to update back the KG. This way, we have a two-way, loop-back mechanism in which the NEABT exploits the KG for its internals, and the NEABT is exploited to update the KG properly.
In the following, we describe the User, Environment, KG, and NEABT components in more detail. User. The User performs reactions and actions based on the signals provided by the NEABT. Sensory data from users flow into the NEABT through a sensor, which represents them in a proper numerical format. Also, the User’s feedback – e.g., whether (or to which extent) the User has adopted the Agent’s suggested action – is sent back to the KG. User’s reactions/actions are assumed to be determined by all three NEABT output types. In particular, the User’s emotion detected by the NEABT at the previous iteration is essential for establishing the emotional conditions that most influence the User. In fact, the User likely performs certain actions only under certain emotional circumstances. Thus, the emotional conditions that determine a certain User’s behavior are an important signal to consider for both the presentation of NEABT’s output to the User and the subsequent KG update step.
Environment. A sensor detects signals from the surrounding environment and represents them in some numerical format. Thus, the NEABT (along with the KG representation) is ready to process them. KG. The Knowledge Graph (KG) contains information about domain knowledge and user profiles. KG’s information is provided to the NEABT in a twofold form. It is first encoded in some proper numerical format and passed to NEABT’s neural node (see below). The encoding is performed by a KG encoder component, which can be implemented, e.g., with a KGE (see Section 2.3). KG’s encoded information is then decoded into a format suitable for processing by the internal nodes of the NEABT. A KG-to-BT decoder performs KG’s information decoding. This can be implemented, e.g., as a neural network component whose training can be performed on a ground truth defined through manual annotation or the agent’s historical data. The KG is fed with NEABT’s output and user feedback. Such data in input to the KG need to be translated in a format suitable for updating the KG; for example, a set of KG triples to be added and a set of KG triples to be removed. A further encoder-decoder component performs this translation. Again, such an encoder-decoder can be implemented as a neural network and trained with a ground truth defined manually or through historical data.
Note that more sophisticated implementations and training strategies of the two encoder-decoder components may be possible. For instance, one could consider training the two encoder-decoder components simultaneously in an end-to-end fashion. This is expected to increase the efectiveness of those components. However, a downside of this solution is that it is technically challenging. A major hardness in this regard consists in making the NEABT a “diferentiable” component so that it can be safely involved in the envisaged end-to-end neural training setting. For this reason, this could be an interesting direction of research that deserves dedicated efort, and we defer to future work.
Another interesting technical challenge is how to self-supervise the training of the two encoderdecoder components to overcome the burden of building a ground truth. An idea in this regard could be to mask a subset of the triples of the KG and use them in substitution for the ground truth. However, this leaves non-trivial technical questions open, such as how to select the triples of the KG to be masked and, more importantly, how to map the output of the NEABT into KG triples. For this reason, we leave this problem for future work as well.
Finally, a further intriguing idea – still deferred to future work – consists in avoiding the encoderdecoder components at all, and letting the interaction between the KG and the NEABT be carried out in an unsupervised way.
NEABT. The NEMO framework deploys a NEABT as a behavior tree. NEABT’s neural node receives the KG’s information and the user’s sensory data and makes inferences about the user’s emotional state. These estimates are mapped into the user afective state variables that parametrize the neural node child, the emotional selector. In turn, the emotional state selector passes the values of the afective state variables to its child nodes, empathy nodes. Each child empathy node provides an empathic evaluation of its subtree. In Figure 1, every subtree has a root node, a sequence selector with a condition node as a child and several action nodes. The condition child node returns success/failure by performing a test condition upon the input pair (KG’, Env). The corresponding action child nodes are executed if the condition node returns success. By doing so, the NEABT can execute actions over the environment. Some of these action nodes define the NEABT threefold output.
Specifically, in the envisaged class of applications, to Human-AI Teaming, the Knowledge Graph will include the user profile along with the history of past interactions, allowing the KG to interact with the NEABT in two ways: by passing data from the user profile to the neural node indicating the user’s “classification” among several possibilities (in Section 4 some basic examples will be provided), and then passing to the leaves the type of action most suitable for that user (here too, we will provide some minimal example). Additionally, the KG should encompass an ontology on driving to assess, for instance, whether the user’s action is allowed and/or ethical.
The implementation of the envisaged system is in an embryonic phase. However, we have first chosen to complete the implementation of our NEABT, making its connections with the other architecture components explicit. To do so, we chose to employ Prolog for the following reasons: • The Prolog representation of a BT is easily readable and modifiable; • Prolog rules provide the modularity and flexibility to represent the various components of the BT; • Due to Prolog’s fast prototyping capabilities, the implementation is ready to use and to be connected with any other component when ready.
In Figure 2, we show the first implementation of our NEABT. This implementation adheres to the workflow described in Figure 1: the neural network component processes knowledge graph embeddings and sensor data from the user to estimate the user’s emotional state and related probability. The emotion selector component uses the predicted emotional state and context from the knowledge graph to select relevant sub-trees or nodes representing potential empathetic responses or actions. The context and action evaluation component tests the conditions. It executes the appropriate actions from the selected empathetic nodes, considering the user’s emotional state and the context from the knowledge graph. If an action fails or the context changes, the system can select and execute alternative empathetic responses or actions based on the updated information.
Accordingly, the main predicates of our program are: • _/2, it extracts all the structural knowledge about the behavior trees described utilizing the ℎ/3 facts. • _/4, (node N in Figure 1) this predicate encodes a neural network (indicated as nn, and accessed via a dedicated plugin). It takes as input the user sensor data, the KG embeddings, and the associated probability values from which it estimates the recognized emotional state. • _/6, (node E in Figure 1) this predicate is responsible for selecting relevant subtrees or nodes based on the user’s emotional state, the associated probability values, the context provided by the knowledge graph decoder and the environment state. It uses knowledge about the NEABT structure to discover nodes that succeed given the current situation using the ℎ__/2 predicate. • ℎ__/2 (nodes [e1, .., en] in Figure 1), this predicate attempts to execute the sequence of actions within a sub-tree while testing the context provided by the knowledge graph decoder () and environment context (), recursively descending the tree; if a condition is met, it then launches the respective sequence of actions within the sub-tree or descends to sub-nodes. If an action fails, either because the user does not “comply" or impeding obstacles show up, it returns the failure to the upper selector node _/6, so that the next best empathy node can be selected according to the probability ranking. • _/1, it generates commands to the agent to execute the action as an external plugin. Each atomic action can fail independently, even if the context is appropriate. If a failure occurs, it returns the failure signal to the main _/6 predicate, which then selects the next best emotional sub-tree. • /4 is the main entry point, combining the neural network node, emotion selector, KG, environment and execution of appropriate empathy nodes. The predicate _ encodes the KG into a suitable format for the neural network (), and generates a decode representation () that provide the context.
Here, we envision a case study that involves developing an intelligent agent that actively functions as a "companion" (co-driver) and support system for drivers. The agent will assist drivers by providing interventions in risky situations that may arise due to external circumstances and/or the driver’s health condition and emotional state, taking into account emotional aspects that could impact driving performance.
The intelligent agent will also be trained through interaction with the human user following the recent "Human-AI teaming" paradigm. A human driver could cooperatively train the agent by collaboratively performing various driving-related tasks, even under challenging scenarios. In this synergistic relationship during the training phase, humans enhance the efectiveness of automation (capabilities and performance). At the same time, the agent installed in each vehicle improves human eficiency and compensates for human inadequacies by intercepting and correcting potential erroneous behaviors, possibly resulting from compromised physical or emotional states.
Potential intervention modes for the agent to assist a struggling driver could include automatically activating (semi-)autonomous driving mode (if available) so the driver can momentarily divert their attention. Alternatively, the agent could more actively engage with the driver to regain attentiveness, for instance, by recommending stimulating music on a dedicated radio station. In case of health issues, the agent could recommend pulling over to rest or take medication (e.g., for hypertension) or, in critical cases, seek emergency assistance by contacting emergency services.
Consider, for instance, the case of truck drivers. Trucks come equipped with tachographs, devices that capture various data, including the operational hours of the vehicle and the driver’s behavior concerning, e.g., speed, stops, etc. Regulations govern drivers’ activities for safety and compliance purposes. The data necessary for checking compliance are stored within the tachograph to identify any breaches and generate a report detailing any violations that can lead to fines (also from the truck company) or even constitute a criminal ofence. The KG can be interfaced with such devices to gather the data to feed the BT so as to issue suitable actions to help the driver avoid or alleviate violations, possibly explaining their reasons (e.g., too much stop because of some driver’s discomfort). Importantly, our system might detect severe physical or psychological pain and alert human operators and emergency services.
Accordingly, in our use case, the system considers the following values for its decision-making process: • The neural network can detect and categorize the driver’s emotional state into one of the following values: [sad, angry, happy, neutral, tired] • Based on the assessed emotional state and environmental factors, the system can select from a set of predefined actions to assist the driver: 1. advisory, provide advisory messages or recommendations to the driver. 2. explicative, ofer explanatory information or guidance to the driver. 3. stop, recommend or initiate the process of stopping the vehicle in critical situations. 4. reassurance, deliver reassuring messages to alleviate the driver’s stress or frustration. • The system continuously monitors the driving environment through various sensors, collecting data such as: 1. _, information about the current road conditions, trafic, and potential hazards. 2. _ℎ, the duration of the current driving session. 3. _, the vehicle’s speed data, including also current speed and historical speed patterns.
4. _, the frequency and duration of stops taken during the driving session.
To provide an example, consider the following scenario: a user is driving, and the system receives environment sensor data [road_status(busy), driving_hours(5), speed_data(120, 150), stop_duration(1, 30)], indicating prolonged driving at high speeds with few stops. The emotional state detected is also identified as tired. Based on this input, our framework reasoning flow is as follows: 1. The neural network node processes the user sensor data and detects the emotional state tired. 2. The emotion selector identifies relevant nodes based on the tired emotional state. 3. The empathy node execution evaluates the user’s state and selects an appropriate action, such as an advisory:
AgentAction = advisory_message("Slow down and take a break soon.").
Neural architecture and the role of emotions. According to Damasio [
The NEMO architecture follows Damasio’s definition of emotions. Emotions are elicited in the neural nodes of NEABTs from both the environment and the agent’s input. The recognized emotions form the afective state of the agent. The agent is conscious of the emotions in its afective state. The emotional values are then passed down in the behavior tree to the emotional and empathy nodes. Doing so triggers changes in the agent’s behavior.
Knowledge graphs. Knowledge Graphs [
KGs have been extensively used in a plethora of application scenarios, including knowledge
completion [
A well-established technique that is commonly exploited for tasks on KGs is Knowledge graph
embeddings (KGEs) [
In this work, we use KGs as a building block of the proposed NEMO framework and KGEs as a tool to derive numerical representations of the KG. The latter is needed to make the knowledge in the KG processable by other components of our NEMO framework (which require knowledge represented in a numerical form).
Knowledge graphs and behavior trees. A significant use of the KG component in our NEMO framework is to have it interact with the NEABT component so that () the KG can adequately guide the actions to be performed by the agent and to be suggested to the user and () the output of NEABT can be employed to update the knowledge in the KG suitably. This gives a “two-way” interaction between KGs and BTs, where the KG influences the processing logic of the BT, and, the other way around, the BT contributes to the knowledge stored in the KG. To the best of our knowledge, this is the first work where a two-way interaction of this kind between KGs and BTs has been employed. A few works exist in the literature about the simultaneous use of KGs and BTs. For instance, Axelsson and Skantze [39] devise a BT-based model that exploits a KG to generate the presentation of information by an agent to an audience. However, Axelsson and Skantze’s model utilizes the KG solely for presenting information to the user, and not for guiding the choices made by the BT, like in our NEMO framework. Zhou et al. [40] propose a methodology that exploits a KG to generate a BT for robot task planning. However, in Zhou et al.’s methodology, the KG is used to generate a BT (to be in turn used to help a robot perform its task planning), and not for interacting with a pre-existing BT, and contributing together with it to perform agent’s actions and suggest actions to the user, like in our NEMO framework. Venkata et al. [41] devise a framework for knowledge transfer through BTs in a multi-agent system. Specifically, Venkata et al.’s framework encompasses a mechanism where the knowledge acquired by single agents is shared, through BTs, with the other agents of the multi-agent system. Thus, the use of BTs and knowledge representation Venkata et al.’s work is conceptually far from the use we do in our NEMO framework. Moreover, Venkata et al.’s framework does not use KGs to represent agents’ knowledge.
From the above discussion, it is apparent that the literature about the simultaneous use of KGs and BTs is only marginally related to what we propose in this work. This makes our proposal of a two-way interaction between KGs and BTs entirely novel.
We have proposed an architecture for a system to be adopted in human-computer interaction and human-AI teaming. The architecture features relevant elements of novelty, encompassing a Knowledge Graph, a Neural Network, and a Behavior Tree, that moreover, as never done before, interact in two ways. We have presented a partial implementation of a part of the architecture, particularly the behavior tree and the related neural nodes, developed in Prolog, and we have discussed the advantages of the implementation. We have outlined a significant case study for testing our future system. However, once implemented, we intend to apply our system to other applications we are developing, firstly, assistive robotics. Future work includes further development of the implementation – including investigation of alternative paradigms, e.g., answer set programming (ASP) – and a suitable experimentation phase. [37] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, G. Bouchard, Complex embeddings for simple link prediction, in: ICML, 2016, pp. 2071–2080. [38] T. Dettmers, P. Minervini, P. Stenetorp, S. Riedel, Convolutional 2d knowledge graph embeddings, in: AAAI, 2018. [39] N. Axelsson, G. Skantze, Using knowledge graphs and behaviour trees for feedback-aware presentation agents, in: Proc. of Int. Conf. on Intelligent Virtual Agents (IVA), 2020, pp. 4:1–4:8. [40] Y. Zhou, S. Zhu, W. Song, J. Gu, J. Ren, X. Xi, T. Jin, Z. Mu, Robot planning based on behavior tree and knowledge graph, in: Proc. of Int. Conf. on Robotics and Biomimetics ROBIO, 2022, pp. 827–832. [41] S. S. O. Venkata, R. Parasuraman, R. M. Pidaparti, KT-BT: A framework for knowledge transfer through behavior trees in multirobot systems, IEEE Trans. Robotics 39 (2023) 4114–4130.