The NEMO co-pilot

Abstract

In this work, we describe an agent to be employed in Human-AI Teaming in various, even critical, domains, based upon afective computing, empathy, and Theory of Mind, and a description of the user profile and of the operational, professional, and ethical requirements of the domain in which the agent operates. The architecture of the proposed agent encompasses a Knowledge Graph, a Neural component and a Behaviour Tree. We briefly discuss a case study.

Keywords

• eol>Human-AI Interaction
• Human-AI Teaming
• Trustworthy AI
• Responsible AI

1. Introduction

AI and humans, if working together in Human-AI

Teaming (HAIT), can produce results exceeding what One recent focus in Artificial Intelligence (AI) is building either can achieve alone, whereas they can control and intelligent systems where humans and AI systems form improve each other. For instance, a human driver might teams. This with the aim of exploiting the potentially train to cope with previously unseen situations through synergistic relationships between human and automa- co-driving automation via a cooperative task shared betion, thus devising “hybrid” systems where the partners tween the human driver and the AI-based system inshould cooperate to perform complex tasks, possibly in- stalled on the vehicle. At the same time, AI helps drivers volving a high degree of risk. As a simple example, in an in case of dificulties and immediate risks. In this synAI-supported self-driving or assisted-driving vehicle, the ergistic relationship, humans may improve automation AI component can be expected to evaluate and co-manage eficacy and capabilities. At the same time, automation situations and risks, where the driver can provide the AI may enhance human performance in a task and compencomponent with useful information on practical driving sate for human inadequacies, catching and correcting in all conditions and can self-manage the risks in the case possible misbehaviors, possibly also due to physically this should be required by the circumstances. Human- or emotionally impaired states, and providing valuable automation interaction is, in fact, one of the main themes suggestions. of Human-centered AI. This issue also falls in the realm For the tasks of adopting AI agents in crucial tasks of Trustworthy AI, whose requirements include respect such as, e.g., improving caregiving in medicine and teachfor human autonomy, prevention of harm, fairness, and ing and constructing efective human-AI teams, agents explainability, and of Responsible AI, whose goal is to should be endowed with an emotion recognition and employ AI in a safe, trustworthy and ethical fashion. management module, capable of empathy, and modelling Ital-IA 2023: 3rd National Conference on Artificial Intelligence, orga- aspects of the Theory of Mind (ToM), in the sense of being nized by CINI, May 29–31, 2023, Pisa, Italy able to reconstruct what someone is thinking or feeling. * Corresponding author. Modelling a Theory of Mind is often based on forms of † These authors contributed equally. “Afective Computing”, which is a set of techniques aimed $ [e-mail] (S. Costantini); at eliciting a human’s emotional condition from physical [e-mail] (P. Dell’Acqua); signs, to enable the system to respond intelligently to fgriaonvcaensncio.d.geuglalosp@[e-mail](Fq..iGtu(Gll.oD);.aGnadsrpeae.rrias)f;[e-mail] human emotional feedback. (A. Rafanelli) In this work, we describe an agent to be employed in € http://www.di.univaq.it/stefcost (S. Costantini); HAIT, based upon afective computing, empathy, and https://dellacqua.se/ (P. Dell’Acqua); https://fgullo.github.io/ Theory of Mind, and a description of the user profile and (F. Gullo) of the operational, professional and ethical requirements (P. 0D0e0l0l’-A00cq02u-a5)6;8060-0601-20400(1S-.9C5o21st-a4n71tin( Gi).; D00.0G0a-0sp00e3ri-s3)7;80-0389 of the domain in which the agent operates. The archi0000-0002-7052-1114 (F. Gullo); 0000-0001-8626-2121 (A. Rafanelli) tecture of the proposed agent encompasses a Knowledge © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Graph, a Neural component and a Behavior Tree. CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) 2. Background 2.1. Behavior Trees nodes based on the agent’s current afective state. The agent elaborates on the afective state during repeated interactions with the user and then tune its reaction accordingly. Once the ordering has been established, the emotional selector behaves as a priority selector. A white circle with the character E represents an emotional selector. In contrast, an empathy node provides an emotional evaluation of its single child node. An empathy node can only be a child of an emotional selector. Its child can be a leaf or an inner node. A dashed circle line with the name of the empathy emotion represents an empathy node.

To enable the integration of deep learning models for emotion recognition and symbolic models for planning and decision-making within emotional behavior trees, we introduced neural nodes. A neural node takes the current state of the environment and agent as input and, using a deep learning model, makes inferences about the emotional state. It contains a model, such as an emotion recognition system, that estimates the emotional state. These estimates are then mapped into the agent’s afective state variables that parameterize the emotional selector. The neural node continually updates the agent’s internal emotional state, allowing the dynamic adaptation of behavior trees to the emotional context.

Behaviour Trees (BTs) were introduced as a tool to enable modular AI in computer games. A behavior tree is essentially a mathematical model of plan execution, where each element (task and action) of a plan is associated with a node in the tree. Their strength comes from their ability to create complex tasks composed of simple tasks without worrying about how the simple functions are implemented. For a comprehensive survey of BTs in Artificial Intelligence and Robotic applications, see [ 1, 2 ].

A BT is a directed acyclic graph consisting of diferent types of nodes, each one associated with executable code (where such code enacts an element composing a plan).

In most cases, a BT is tree-shaped, hence the name. However, unlike a traditional tree, a node in a BT can have multiple parents, allowing the reuse of that part of the tree. The traversal of a behavior tree starts at the top node. When a node is traversed, the associated code is executed, returning one of the three states: success, failure, or running.

The critical nodes in a BT include leaf nodes and inner nodes. An action is a leaf node representing a behavior that the character can perform. The action returns success or failure when it completes its execution, de- 2.3. Knowledge Graphs pending on the outcome. An action is depicted as a white Knowledge Graphs (KGs) [ 4, 5 ] is a particular type of circle. A condition is a leaf node that checks an internal knowledge base [ 6 ] where knowledge is organized in a or external state. It returns either success or failure. A graph-like structure, i.e., with triples that define relationcondition is represented as a grey rounded rectangle. A ships (edges) among entities (nodes) of interest. KGs are sequence selector is an inner node that typically has sev- also known as information graphs [ 7 ], or heterogeneous eral child nodes that are executed sequentially. Once a information networks [ 8 ]. child node completes its execution successfully, the se- KGs have been extensively used in a plethora of apquence selector continues executing the next child node. plication scenarios, including knowledge completion [ 9 ], If every child node returns success, then the sequence head/tail prediction [ 10 ], rule mining [ 11 ], query answerselector returns success. If one of the child nodes return ing [ 12 ], and entity alignment [ 13, 14, 15 ]. KGs have failure, the sequence selector immediately returns failure. also recently recently emerged as supporting tools for A sequence selector is depicted as a grey square with Retrieval-Augmented Generation (RAG) for Large Lanan arrow across the links to its child nodes. A priority guage Models (LLMs) [ 16, 17, 18, 19 ]. selector is an inner node. It has a list of child nodes that it A well-established technique that is commonly extries to execute one at a time until one of the child nodes ploited for tasks on KGs is Knowledge graph embeddings returns success. If none of the child nodes executes suc- (KGEs) [ 20, 10 ]. KGEs generate numerical vector reprecessfully, the priority selector returns failure. A priority sentations for entities and relationships of a KG, thus selector is represented with a grey circle with a question making them amenable to be processed in downstream mark. tasks where a numerical representation is required (e.g., neural network-based machine-learning tasks). Although 2.2. Neural Empathy-Aware Behavior KGEs can difer (significantly) from one another in their Trees definition, a shared key aspect of all KGEs is that they are typically defined based on a so-called embedding scoring To consider empathy and mimic human decision-making, function or simply embedding score. This function quanin [ 3 ] we introduced neural empathy-aware behavior trees tifies how likely a triple exists in the KG based on the (NEABTs) by introducing a selector node called emotional embeddings of the entities and the relationship of that selector, an empathy node, and a neural node. triple. Several KGEs have appeared in the last few years.

The emotional selector is a node that orders its child The distinctive features among embeddings are the score

ENVIRONMENT

KNOWLEDGE GRAPH

Domain Knowledge User Profile KG KG encoder KG ' (KG', Env)

KG decoder KGNEABT-to-KG encoder

User Feedback KG-toNEABT decoder Sensor

USER

Aggregator

N E ... e1 e2 (KG', Env)

A1

A2 ... An (KG', Env)

A1

A2 ... An Agent's suggested action to User

Agent Action

Detected User

Emotion function and the optimization loss. Translational embed- to the User. The Aggregator may perform something dings in the TransE [ 21 ] family and the recent PairRE [ 22 ] either very simple (e.g., just derive a textual representaassumes that the relationship of a triple performs a trans- tion of the three outputs and concatenate them) or more lation between the entities of that triple. Semantic em- sophisticated (e.g., exploit a large language model (LLM)). beddings, such as DistMult [ 11 ] or HolE [ 23 ], interpret BT’s outputs and User’s feedback are used to update back the relationship as a multiplicative operator. Complex the KG. This way, we have a loop-back mechanism in embeddings, such as RotatE [ 24 ] and ComplEx [ 25 ], use which the KG is exploited by the BT for its internals, and complex-valued vectors and operations in the complex the BT is exploited to update the KG properly. plane. Neural-network embeddings, such as ConvE [ 26 ], Next, we describe the User, Environment, KG, and perform sequences of nonlinear operations. NEABT components in more detail.

3. Framework

The architecture of the proposed agent is illustrated in

Figure 1. The main components of the architecture are the User, the Environment, a Knowledge Graph (KG), and a Behavior Tree (BT). The overall interaction between such components is described next.

The BT is fed with signals from Environment, User, and KG. Such signals are exploited by the BT to perform its computation and to output () an action to be suggested to the User, () an action actually performed by the agent Environment. Signals from the surrounding environ(e.g., an empathetic action), and () User’s emotion de- ment are detected by a sensor, representing them in some tected by its neural node (‘N’, see below). Threefold BT’s numerical format, and are thus ready to be processed by output passes through an “Aggregator”, responsible for the NEABT (along with the KG representation). suitably aggregating and presenting three BT’s outputs User. The User performs reactions and actions based on the signals provided by the NEABT. The user’s sensory data flow into the NEABT through a sensor, which represents them in some proper numerical format. Also, the User’s feedback—e.g., whether (or to which extent) the User has adopted the Agent’s suggested action—is sent back to the KG. User’s reactions/actions are assumed to be determined by all three types of BT’s output. In particular, the User’s emotion detected by the BT at the previous iteration is important for establishing the emotional conditions that most influence the user.

KG. The KG contains information about domain knowledge and user profile. KG’s information is provided to ous driving-related tasks, even under challenging scenarthe NEABT in a twofold form. It is first encoded in some ios. In this synergistic relationship during the training proper numerical format and passed to BT’s neural node phase, humans enhance the efectiveness of automation (see below). The encoding is performed by a KG encoder (capabilities and performance). At the same time, the component, which can be implemented, e.g., with a KGE agent installed in each vehicle improves human eficiency (see Section 2.3). KG’s encoded information is then de- and compensates for human inadequacies by intercepting coded into a format suitable for processing by the internal and correcting potential erroneous behaviours, possibly nodes of the BT. A KG-to-BT decoder performs KG’s in- resulting from compromised physical or emotional states. formation decoding. This can be implemented, e.g., as a Potential intervention modes for the agent to assist neural network component whose training can be per- a struggling driver could include automatically activatformed on a ground truth defined through either manual ing (semi-)autonomous driving mode (if available) so the annotation or the agent’s historical data. The KG is fed driver can momentarily divert their attention. AlternaBT’s output and user feedback. Such data in input to tively, the agent could more actively engage with the the KG are represented in a format suitable for updating driver to regain attentiveness, such as by recommending the KG, e.g., a set of KG triples should be added and a stimulating music on a dedicated radio station. In case of set of KG triples removed. Such a translation from BT’s health issues, the agent could recommend pulling over and User’s signals to KG updating signal is performed to rest or take medication (e.g., for hypertension) or, in by a further encoder-decoder component. Again, such critical cases, seek emergency assistance by contacting an encoder-decoder can be implemented as a neural net- emergency services. work and trained with a ground truth defined manually or through historical data.

NEABT. The NEMO framework deploys a NEABT as a

behavior tree. The BT’s neural node receives the KG’s information and the user’s sensory data and makes inferences about the user’s emotional state. These estimates are mapped into the user afective state variables that parametrize the neural node child, the emotional selector. In turn, the emotional state selector passes the values of the afective state variables to its child nodes, empathy nodes. Each child empathy node provides an empathic evaluation of its subtree. In Figure 1, every subtree has a root node that is a sequence selector with a condition node as a child and several action nodes. The condition child node returns success/failure by performing a test condition upon the input pair (KG’, Env). The corresponding action child nodes are executed if the condition node returns success. By doing so, the NEABT can execute actions over the environment. Some of these action nodes define the BT threefold output.

4. Case Study: Driver Co-Pilot

Here, we envision a case study that involves developing an intelligent agent that actively functions as a "companion" (co-driver) and support system for drivers. The agent will assist drivers by providing interventions in risky situations that may arise due to external circumstances and/or the driver’s health condition and emotional state, taking into account emotional aspects that could impact driving performance.

The intelligent agent will also be trained through interaction with the human user following the recent "HumanAI teaming" paradigm. A human driver could cooperatively train the agent by collaboratively performing vari

References

[1] M. Colledanchise , P. Ögren , Behavior trees in robotics and AI: An introduction , CRC Press, 2018 .

[2] M. Iovino , E. Scukins , J. Styrud , P. Ögren , C. Smith , A survey of behavior trees in robotics and ai , Robotics and Autonomous Systems 154 ( 2022 ) 104096 .

[3] S. Costantini , P. Dell'Acqua , G. De Gasperis , A. Rafanelli , Empowering emotional behavior trees with neural computation for digital forensic , 15th European Symposium on Computational Intelligence and Mathematics (ESCIM 2024 ) (in press).

[4] A. Hogan , E. Blomqvist, M. Cochez , C. d'Amato, G. de Melo, C. Gutierrez , S. Kirrane , J. E. L. Gayo , R. Navigli , S. Neumaier , A. N. Ngomo , A. Polleres , S. M. Rashid , A. Rula , L. Schmelzeisen , J. F. Sequeda , S. Staab , A. Zimmermann , Knowledge graphs , ACM CSUR 54 ( 2022 ) 71 : 1 - 71 : 37 .

[5] G. Weikum , Knowledge graphs 2021 : A data odyssey , PVLDB 14 ( 2021 ) 3233 - 3238 .

[6] O. Deshpande , D. S. Lamba , M. Tourn , S. Das , S. Subramaniam , A. Rajaraman , V. Harinarayan , A. Doan , Building, maintaining, and using knowledge bases: a report from the trenches , in: SIGMOD , 2013 , pp. 1209 - 1220 .

[7] M. Lissandrini , D. Mottin , T. Palpanas , D. Papadimitriou , Y. Velegrakis , Unleashing the power of information graphs , ACM SIGMOD Record 43 ( 2015 ) 21 - 26 .

[8] C. Shi , Y. Li , J. Zhang , Y. Sun , S. Y. Philip , A survey of heterogeneous information network analysis , TKDE 29 ( 2016 ) 17 - 37 .

[9] X. Wang , L. Chen , T. Ban , M. Usman , Y. Guan , S. Liu, T. Wu, H. Chen , Knowledge graph quality control: A survey , Fundamental Research 1 ( 2021 ) 607 - 626 .

[10] S. Ji , S. Pan , E. Cambria, P. Marttinen , S. Y. Philip , A survey on knowledge graphs: Representation, acquisition, and applications , Trans. Neural Netw. Learn. Syst . 33 ( 2021 ) 494 - 514 .

[11] B. Yang , S. W.-t. Yih, X. He , J. Gao , L. Deng , Embedding entities and relations for learning and inference in knowledge bases , in: ICLR , 2015 .

[12] Y. Wu , Y. Xu , X. Lin , W. Zhang, A holistic approach for answering logical queries on knowledge graphs , in: ICDE , 2023 , pp. 2345 - 2357 .

[13] S. S. Bhowmick , E. C. Dragut , W. Meng, Globally aware contextual embeddings for named entity recognition in social media streams , in: ICDE , 2023 , pp. 1544 - 1557 .

[14] J. Huang , Z. Sun , Q. Chen , X. Xu , W. Ren , W. Hu, Deep active alignment of knowledge graph entities and schemata , PACMMOD 1 ( 2023 ) 159 : 1 - 159 : 26 .

[15] A. Zeakis , G. Papadakis , D. Skoutas , M. Koubarakis , Pre-trained embeddings for entity resolution: An experimental analysis , PVLDB 16 ( 2023 ) 2225 - 2238 .

[16] Y. Gao , Y. Xiong , X. Gao , K. Jia , J. Pan , Y. Bi , Y. Dai , J. Sun , Q. Guo , M. Wang , H. Wang , Retrievalaugmented generation for large language models: A survey , CoRR abs/2312 .10997 ( 2023 ).

[17] S. Hao , T. Liu , Z. Wang , Z. Hu, ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings , in: NeurIPS, 2023 .

[18] X. Wang , Q. Yang , Y. Qiu , J. Liang , Q. He , Z. Gu , Y. Xiao , W. Wang, KnowledGPT: Enhancing large language models with retrieval and storage access on knowledge bases , CoRR abs/2308 .11761 ( 2023 ).

[19] J. Zhang , Graph-toolformer: To empower LLMs with graph reasoning ability via prompt augmented by ChatGPT , CoRR abs/2304 .11116 ( 2023 ).

[20] Q. Wang , Z. Mao , B. Wang , L. Guo , Knowledge graph embedding: A survey of approaches and applications , TKDE 29 ( 2017 ) 2724 - 2743 .

[21] A. Bordes , N. Usunier , A. Garcia-Duran , J. Weston , O. Yakhnenko , Translating embeddings for modeling multi-relational data , NeurIPS 26 ( 2013 ).

[22] L. Chao , J. He , T. Wang , W. Chu, PairRE: Knowledge graph embeddings via paired relation vectors , in: ACL , 2021 , pp. 4360 - 4369 .

[23] M. Nickel , V. Tresp , H.-P. Kriegel , et al., A three-way model for collective learning on multi-relational data , in: ICML, 2011 , pp. 3104482 - 3104584 .

[24] Z. Sun , Z. Deng , J. Nie , J. Tang, RotatE: Knowledge graph embedding by relational rotation in complex space , in: ICLR , 2019 .

[25] T. Trouillon , J. Welbl , S. Riedel, É. Gaussier, G. Bouchard, Complex embeddings for simple link prediction , in: ICML, 2016 , pp. 2071 - 2080 .

[26] T. Dettmers , P. Minervini , P. Stenetorp , S. Riedel , Convolutional 2d knowledge graph embeddings , in: AAAI , 2018 .