When humans and software agents collaborate on taking decisions together in hybrid teams, they typically share knowledge and goals based on their individual intentions. Goals can be modelled as the efects that are caused by events or actions taken. In order to decide and plan which actions to take, it is necessary to understand which actions or events cause the intended efects. In other words, we consider causal inferencing in a reverse way: instead of asking whether certain actions or events indeed cause corresponding efects, we consider establishing and using a causal model for determining the appropriate cause or causes, such that the causal chain results in the desired and intended outcomes. For example, your goal can be to arrive at a destination at a given time. By reasoning back which actions are required to get you there, piece by piece, a causal path can be constructed to determine the departure time and modes of trafic along the route. Due to shared intentions and causal models, humans and agents can mutually trust each other regarding their actions and outcomes.
The aim to empower humans by using systems with artificial intelligence, where the AI systems
can be trusted [
= { , } = { } = { ∶ = 2 + 3 } (a) Structural Causal Model (SCM) (b) Structural Equation Model (SEM)
humans and agents can trust each other, even when they disagree in their goals and preferred
When humans and agents share goals to collaborate in hybrid teams [
Structural Causal Models (SCM) or Structural Equation Models (SEM) [8] represent such understanding of casual relationships as graphs (SCM) and sets of equations (SEM), as shown in figure 1.
Both representations are useful for diferent purposes. SCMs support the human understanding and explanations, while SEMs are more suitable for representing causality in combination with logical expressions. The latter enables the possibility to communicate about causal models and intentions formally and concisely among agents.
Shared causal models increase trust among team members, because they help to explain to each other why certain actions are to be taken. They can explain the relationships between causes and efects. Delegation without reason or motivation is not trustworthy (unless the authority or reputation of the delegator is very high). This enables users to better understand the rationale and have greater confidence in others making a fair and unbiased decision.
There are several important aspects by which causality can improve the trustworthiness of AI systems (Causality for Trust, C4T). Besides precision and accuracy, which are fundamental to trustworthiness in AI, they are [9, 10, 11, 12]: • Transparency & Interpretability. The reasoning behind decisions is explainable and easily understood by humans. Causal models provide the reasons for predictions and causal explanations help to build a correct mental model of the problem. • Reproducibility. The ability to repeat experiments and get the same results increases the trustworthiness and accuracy of scientific output. • Fairness. Causal AI can remove bias because it understands how variables are interconnected and dependent on each other. Understanding causal relationships between sensitive input variables (such as gender or race) and predicted outcomes is important for assessing biased behaviour. Counterfactual fairness is achieved when the output is identical for each sensitive input variable. • Robustness. Causal models can avoid the brittleness of most machine learning systems, due to spurious correlations. They can handle data that is not independent and identically distributed (IID) or out of distribution (OOD), because they can discern between relevant and irrelevant data and variables [13, 14]. • Privacy. The robustness of causal models helps in preventing privacy attacks, because weaknesses of trained models cannot easily be exploited, for example in federated learning. • Safety & Accountability (Auditing). Regulations for safe-guarding AI systems for use in critical applications and domains demand impact assessment (IA) to prevent from algorithmic and data-driven harm by finding potential negative efects before (large-scale) deployment. Causal models that represent dependencies between system design and impact can be used to assess and mitigate corresponding risks by identifying which system elements are responsible for undesired efects.
Causal inference [15, 16] is typically concerned with the resulting efect when a corresponding event (cause) occurs, according to a given causal model, such that the respective dependency can be verified. Causal inference asks whether an event indeed causes a certain efect by determining the likelihood that one event was the cause of another. In contrast to statistical correlations, causal relationships are asymmetrical [17, 18, 19], i.e. that there is a directed relationship from a cause to an efect, rather than a spurious co-occurrence of events.
Causal discovery, on the other hand, is concerned with determining whether a change in one variable (representing a state, action or event) indeed causes a change in another, in order to distinguish between correlated and causal relationships. Approaches to make the distinction are interventions, random control trials and counterfactual reasoning.
Counterfactuals refer to alternative choices that could have been made in the past and the corresponding efects that they might have caused.
On the other hand, we want to find the causes (events or actions) that achieve a given efect. Therefore, we are concerned with counterfactual exploration for the future. Starting from intended efects (such as individual or collaborative goals) we search for appropriate causes from which these efects will follow. If we know the causes for our intended efects, we can plan the actions that will lead to them. Thus, we are interested in establishing the causal chain that results in the desired and intended outcomes. Therefore, we are concerned with counterfactual exploration to answer the following questions. Which efects will result from diferent alternative choices for actions that we are going to take now? And which of those efects match with our goals and intentions?
For collaborative decision-making (CDM), it is essential that each human and agent is aware of each others’ points of view and understands that others possess mental states that might difer from one’s own - which is known as a Theory of Mind (ToM). ToM is defined as the human cognitive ability to perceive and interpret others in terms of their mental states, such as beliefs, desires, goals, intentions and emotions, and it is considered an indispensable requirement of human social life [20, 21, 22, 23, 24].
We distinguish three diferent ways in which shared causal models involve a Theory of Mind. The intentions and causal models are either (1) shared explicitly, (2) observed and anticipated from others’ behaviours or (3) based on expectations of average behaviour patterns.
Using option (1), each agent knows several causal models and other facts about its context
(beliefs). These models can be shared among agents by negotiating about group goals and
individual intentions [
In causal inferencing, deliberate and controlled actions are called interventions. Interventional distributions are typically written as probabilities ( |( = 0)), where is the desired efect and the -operator represents the intervention of deliberately adjusting to the value of 0. Consequently, we need to find 0 in a process that we call counterfactual exploration.
Counterfactual exploration searches backwards along the paths in a causal model to find the most probable outcome, as close as possible to , by setting to likely values and estimating the resulting efects along the causal chain. Eficient causal paths can be found by searching to satisfy anticipated intentions and group goals. This process makes use of the backdoor criteria, as described in [32]. Hence, counterfactual exploration is a complementary method to either causal inference or causal discovery.
By using counterfactual exploration, the desired efects that were established in a hybrid human-agent team can be achieved with high probability and reliability. The use of a causal model guarantees that the outcomes are indeed causally related to the interventions and interpretable. Therefore, humans and agents can trust each other regarding the right actions to take.
The above-mentioned concepts are explored in a scenario of optimising the energy consumption of users of a multi-tenant building. Besides finding sustainable ways of producing more energy (out of scope of this project), we must reduce our energy consumption and use the available energy as eficiently as possible. Therefore, it is crucial to create awareness and means of control at the points of storage and consumption.
The Talking Buildings project is an applied AI project that involves hybrid interaction in a social urban context. The project aims to find new forms of living, working and urban rhythms related to energy sustainability and flexibility [ 33, 34, 35, 36].
Our activities partly focus on operational climate management using real-life data from actual buildings. This involves activities related to situations inside the building, such as learning to control the climate system in an eficient way, while maintaining the level of comfort perceived by its users.
The other part of our work is the study of energy consumption patterns and interactions among buildings. This part involves the development of a computational model of interactions in urban energy regulation based on epistemic logic and Theory of Mind [27, 37, 38]. The setting requires agents to communicate not purely with facts, but also based on their beliefs and knowledge. The goal is to create an agent-based computational model of eficient knowledge transfer in a human-AI ecosystem.
As outlined above, we use our counterfactual exploration process to find eficient causal paths that achieve the shared intentions in hybrid teams. The causal relationships are explainable and transparent to each of the actors involved (both humans and agents), which leads to the emergence of mutual trust.
This research was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. [6] A. S. Rao, M. P. George, BDI agents: From theory to practice, in: Proceedings of the First International Conference on Multi-Agent Systems (ICMAS-95), 1995, pp. 312–319. URL: http://www.agent.ai/doc/upload/200302/rao95.pdf. [7] D. C. Dennett, The intentional stance, The intentional stance, The MIT Press, Cambridge,
MA, US, 1987. Pages: xi, 388. [8] J. Pearl, An Introduction to Causal Inference, The International Journal of Biostatistics 6 (2010). URL: https://www.degruyter.com/document/doi/10.2202/1557-4679.1203/html. doi:10.2202/1557-4679.1203, publisher: De Gruyter. [9] B. Greifeneder, Three Ways A Causal Approach Can Improve Trust In AI, 2021. URL: https://www.forbes.com/sites/forbestechcouncil/2021/11/01/ three-ways-a-causal-approach-can-improve-trust-in-ai/, section: Innovation. [10] N. Ganguly, D. Fazlija, M. Badar, M. Fisichella, S. Sikdar, J. Schrader, J. Wallat, K. Rudra, M. Koubarakis, G. K. Patro, W. Z. E. Amri, W. Nejdl, A Review of the Role of Causality in Developing Trustworthy AI Systems, 2023. URL: http://arxiv.org/abs/2302.06975. doi:10. 48550/arXiv.2302.06975, arXiv:2302.06975 [cs]. [11] B. Bartling, E. Fehr, D. Hufman, N. Netzer, The Causal Efect of Trust, 2018. URL: https: //papers.ssrn.com/abstract=3286177. doi:10.2139/ssrn.3286177. [12] J. Y. Yap, A. Tomlinson, A Causality-Based Model for Describing the Trustworthiness of a Computing Device, in: M. Yung, J. Zhang, Z. Yang (Eds.), Trusted Systems, Lecture Notes in Computer Science, Springer International Publishing, Cham, 2016, pp. 130–149. doi:10.1007/978-3-319-31550-8_9. [13] E. Sherman, I. Shpitser, Identification and Estimation of Causal Efects from Dependent Data, 2019. URL: http://arxiv.org/abs/1902.01443. doi:10.48550/arXiv.1902.01443, arXiv:1902.01443 [stat]. [14] C. Zhang, K. Mohan, J. Pearl, Causal Inference under Interference and Model Uncertainty, 2023. URL: https://openreview.net/forum?id=TYKk9SWhke0. [15] J. Pearl, Causality: Models, Reasoning and Inference, 2nd edition ed., Cambridge University
Press, Cambridge, U.K. ; New York, 2009. [16] J. Pearl, D. Mackenzie, The Book of Why: The New Science of Cause and Efect, 1st edition ed., Basic Books, New York, 2018. [17] H. Price, Agency and Causal Asymmetry, Mind 101 (1992) 501–520. URL: https://www.
jstor.org/stable/2253900, publisher: [Oxford University Press, Mind Association]. [18] D. Kutach, Causal Asymmetry, in: D. Kutach (Ed.), Causation and its Basis in Fundamental Physics, Oxford University Press, 2013, p. 0. URL: https://doi.org/10.1093/acprof: oso/9780199936205.003.0007. doi:10.1093/acprof:oso/9780199936205.003.0007. [19] J. Ismael, Reflections on the asymmetry of causation, Interface Focus 13 (2023) 20220081.
URL: https://royalsocietypublishing.org/doi/10.1098/rsfs.2022.0081. doi:10.1098/rsfs. 2022.0081, publisher: Royal Society. [20] D. Premack, G. Woodruf, Does the chimpanzee have a theory of mind?, Behavioral and Brain Sciences 1 (1978) 515–526. URL: https://www.cambridge.org/core/journals/ behavioral-and-brain-sciences/article/does-the-chimpanzee-have-a-theory-of-mind/ 1E96B02CD9850016B7C93BC6D2FEF1D0. doi:10.1017/S0140525X00076512, publisher: Cambridge University Press. [21] S. Baron-Cohen, A. M. Leslie, U. Frith, Does the autistic child have a “theory of mind” City, Frontiers in Sustainable Cities 2 (2020). URL: https://www.frontiersin.org/articles/10. 3389/frsc.2020.00038. [35] A. Luusua, J. Ylipulli, M. Foth, A. Aurigi, Urban AI: understanding the emerging role of artificial intelligence in smart cities, AI & SOCIETY (2022). URL: https://doi.org/10.1007/ s00146-022-01537-5. doi:10.1007/s00146-022-01537-5. [36] C. Nevejan, P. Sefkatli, S. Cunningham, City Rhythm, 2018. [37] R. Verbrugge, Testing and Training Theory of Mind for Hybrid Human-agent Environments, in: A. P. Rocha, L. Steels, H. J. v. d. Herik (Eds.), Proceedings of the 12th International Conference on Agents and Artificial Intelligence, ICAART 2020, Volume 1, Valletta, Malta, February 22-24, 2020, SCITEPRESS, 2020, p. 11. URL: https://vimeo.com/396473042. [38] I. Tsoukalas, Theory of Mind: Towards an Evolutionary Theory, Evolutionary Psychological Science 4 (2018) 38–66. URL: https://doi.org/10.1007/s40806-017-0112-x. doi:10.1007/s40806-017-0112-x.