The concept of trust has long been studied, initially in the context of human interactions and, more recently, in human-machine or human-agent interactions. Despite extensive studies, defining trust remains challenging due to its inherent complexities and the diverse factors that influence its dynamics in multi-agent environments. This paper focuses on a specific formalization of a trust factor: predictive reliability, defined as the ability of agents to accurately forecast the actions of their peers in a shared environment. By realizing this trust factor within the framework of multi-agent reinforcement learning (MARL), we integrate it as a criterion for agents to assess and select collaborators. This approach enhances the functionality of MARL systems, promoting improved cooperation and overall efectiveness.
CEUR ceur-ws.org
With the advent of artificial intelligence, the number of applications that require co-existence
and the interaction between intelligent agents and humans is increasing over time. Such
applications include autonomous vehicles [
Trust is a concept that has been defined numerous times in the literature [
A MARL solution for a DEC-POMDP is a set of functions, called policies ∶ Ω → , which map the local observations of each agent to its actions, in order to maximize the expected joint sum of discounted rewards: ∑= ( , ), where 0 ≤ < 1 .
Trust factors We refer to Frattolillo et al. [
TrustFactor( | , Γ) = ( , , , [ → ], [ → ], [ → ]) (1) In this template, denotes the observation of the trustor, is the action of the trustee, is the immediate reward, and → is the current belief that the trustor currently maintains with respect to the trustee. Finally, → and → represent known facts that result from communication from trustor to trustee and vice versa. The brackets denote that these are optional components.
In this work, we propose a specific instantiation of the function in eq. (1) to capture the
dependency identified above between the actions of the trustee and the trustor’s expectations.
Specifically, we model one of these trust factors as the ability of one agent, acting as a trustor,
to predict the actions of another agent, acting as trustee. Therefore, among all agents, we select
one to be considered as the primary agent who, in addition to learning its own policy, learns
how to predict the actions computed by other agents. Specifically, for each trustee , the primary
agent estimates the Trust Score defined as the number of correctly predicted actions over the
(a)
(b)
(c)
number of true actions. For predicting the others’ actions, we adopt a simple neural network
that we define PredNet, which is trained in a supervised fashion and that takes as input the
observations of other agents and returns a prediction about their actions. The action predicted
by the PredNet is concatenated to the state of the primary agent; this allows us to influence its
decisions based on other agents’ intentions. All agents used in the experiment are trained in a
decentralized way through a MARL algorithm called Independent PPO [
The environment used is a customized version of the Level-based Foraging (LBF) [13]. This is a grid-world multi-agent environment in which agents should navigate and cooperate to collect food, which can be collected only if the sum of the levels of agents is equal to or higher than the level of the food. In our experiments, the primary agent selects which agent to communicate with among the agents in its field of view based on their trust score learned during the training. We did some experiments in a three agents environment, where the other two agents are defined respectively as trustable and unreliable. The trustable agent executes its actions according to its learned policy, and its goal is to cooperate with the primary agent. On the other side, the unreliable agent performs actions according to a bad policy that, with a certain probability, leads to incorrect action performed. The results of the experiment are shown in Figure 1. Here, the trust score with respect to the trustable agent (b) is much higher than the one referred to the unreliable agent (c), and additionally, the average return of the primary agents is drastically better when relying on the former (a). In conclusion, we showed that using the trust score as a mechanism to select which agent to communicate with improves the performance in the case where an agent is not reliable.
This work is supported by the Air Force Ofice of Scientific Research under award number FA8655-23-1-7257. abs/2011.09533 (2020). URL: https://arxiv.org/abs/2011.09533. arXiv:2011.09533. [13] F. Christianos, L. Schäfer, S. V. Albrecht, Shared experience actor-critic for multi-agent reinforcement learning, in: Advances in Neural Information Processing Systems (NeurIPS), 2020.