=Paper=
{{Paper
|id=Vol-3894/paper18
|storemode=property
|title=Sentimental Agents: Exploring Deliberation, Cognitive Biases, and Decision-making in LLM-based Multiagent Systems
|pdfUrl=https://ceur-ws.org/Vol-3894/paper18.pdf
|volume=Vol-3894
|authors=Elizabeth A. Ondula,Daniele Orner,Nick Mumero Mwangi,Casandra Rusti
|dblpUrl=https://dblp.org/rec/conf/kil/OndulaOMR24
}}
==Sentimental Agents: Exploring Deliberation, Cognitive Biases, and Decision-making in LLM-based Multiagent Systems==
https://ceur-ws.org/Vol-3894/paper18.pdf
Sentimental Agents: Exploring Deliberation, Cognitive
Biases, and Decision-making in LLM-based Multiagent
Systems
Elizabeth A. Ondula2,∗,† , Daniele Orner1,† , Nick Mumero Mwangi1 and Casandra Rusti2
1
Brave Venture Labs
2
University of Southern California, Los Angeles, USA
Abstract
How does sentiment affect deliberative opinion dynamics in multi-agent systems using Large Language Models (LLMs)?
In this paper, we introduce Sentimental Agents, a framework designed to study collaborative decision-making in a society
of agents, each equipped with a distinct Mental Model of Self. We propose a method to integrate sentiment analysis and a
non-Bayesian update mechanism, to analyze and interpret agents’ beliefs and interactions systematically. This method allows
us to observe the volatility of the sentiment associated with different agent statements, as well as the change in opinion
throughout the agents’ conversation. We further use it to model and compare collaborative decision-making approaches.
We situate these agents in a simulated Human Resource recruiting environment as a case study to evaluate a candidate’s fit
for a role. We present a set of metrics to assess the quality of the agents’ output. Finally, we explore cognitive biases in the
agents’ individual and collective opinion formation, a fundamental step to enhance decision-making capabilities and mitigate
distortions in the system and the agents’ collective reasoning.
Keywords
Multi-Agent Systems, Large Language Models, Sentiment Analysis, Cognitive Biases, Decision-Making, Opinion Dynamics,
1. Introduction not adequately account for the influence of behavioral
states like sentiment and cognitive biases on the decision-
Multi-agent systems (MAS), composed of interactive making process. Our work adopts a nuanced approach to
agents have been pivotal in modeling social phenomena, understanding how the output of LLM agents influences
decision-making processes and collaborative tasks. Large one another within these frameworks.
Language Models (LLMs) such as GPT-4 [1] have opened We introduce Sentimental Agents, a framework de-
new possibilities for exploring complex social dynamics signed to study and analyze collaborative decision pro-
through the simulation of linguistic interactions among cesses. These agents are not only equipped with language
agents. These models can provide the necessary capabili- capabilities but also possess a unique Mental Model of
ties for simulating communication scenarios. Integrating Self. This allows them to process and exhibit behaviors
LLMs into MAS facilitates the study of conversations and that can offer a comprehensive view of how opinions are
interaction patterns in a more detailed manner. formed and evolve in a multi-agent setting.
LLMs have demonstrated exceptional performance in Our system is designed primarily to observe and de-
generating text that embodies sentiment and in execut- scribe agents’ behavior, rather than to design or direct
ing sentiment analysis tasks [2]. However, the effect it. We do not currently include objectives, reward func-
of sentiment on deliberative opinion dynamics within tions, utility metrics or payoffs in our model. The focus
an artificial society of agents is a domain that has not is on the natural evolution of interactions among agents
yet been fully explored. Traditional agent models may without imposing external incentives or goals. Our study
concentrates on non-strategic interactions. Unlike strate-
KiL’24: Workshop on Knowledge-infused Learning co-located with
gic agents, which model the behavior of others and act
30th ACM KDD Conference, August 26, 2024, Barcelona, Spain
∗
Corresponding author. based on these predictions, our non-strategic agents do
†
These authors contributed equally. not possess such models. This distinction is crucial as it
Envelope-Open ondula@usc.edu (E. A. Ondula); daniele@braveventurelabs.com means our agents are not engaging in behaviors such as
(D. Orner); nick@braveventurelabs.com (N. M. Mwangi); scheming or deceiving to achieve a specific objective. If
rusti@usc.edu (C. Rusti) LLM-based multi-agent systems are ultimately to be used
GLOBE https://eondula.github.io/ (E. A. Ondula); https://bravelabs.ai/
(D. Orner); https://bravelabs.ai/ (N. M. Mwangi);
to support decision-making, it is critical to understand
https://www.linkedin.com/in/casandrarusti/ (C. Rusti) and explain how their decisions are made. This is espe-
Orcid 0000-0003-0403-0306 (E. A. Ondula); 0009-0005-1264-1985 cially true in the hypothetical case of such systems being
(D. Orner); 0009-0004-6654-2635 (N. M. Mwangi); designed to evaluate, rank or recommend humans. At
0009-0007-5668-1991 (C. Rusti) present, there are no unified solutions that can system-
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Figure 1: The Sentimental Agents framework consists of 7 modules: The Brief, Agent Initialization, Opinion Dynamics, Conversation,
Decision, Cognitive Bias and Evaluation Modules.
atically analyze the opinions and interactions of these the use of LLMs in multi-agent settings with a focus on
agents, and the potential correlation between the two. To ”Theory of Mind” (TOM), which is the ability of an agent
remediate this, we make the following key contributions: to understand and predict the mental states and inten-
• We develop a framework, Sentimental Agents [], tions of others. Although crucial for collaboration, our
to explore and study collective decision-making focus looks more at how agents make decisions rather
processes in a society of agents. than understanding others’ mental states. Other stud-
• We propose using sentiment analysis as a method ies, like those of [4] look at how agents can debate and
to quantify content generated by LLM-based make collective decisions using a method known as grad-
agents for evaluation and recommendation tasks. ual semantics, where agents exchange arguments and
• We propose a method to apply a non-Bayesian progressively update their opinions to reach a shared
model for opinion dynamics within a multi-agent decision. Our approach is different in that it explains the
system. This offers a perspective on how opin- agent interactions and decision processes leveraging a
ions are formed and altered in a sentiment-driven mental model of self and sentiment tracking. Further,
environment. our agents don’t have access to other agents’ memories.
[5] explores how agents coordinate in complex tasks that
Additionally, this work introduces metrics for assessing
necessitate both working together on the same task (co-
the quality of conversation and decision-making in lan-
operation) and dividing the task into smaller parts to be
guage model-based multi-agent systems. These metrics,
done individually (divide-and-conquer). This study high-
namely nuance, platitudinal score, drift, and defensibility,
lights the need for flexible strategies to manage tasks that
offer a toolkit for evaluating the effectiveness of such
require both joint and individual efforts, differing from
systems in diverse scenarios. Furthermore, we evalu-
our work which doesn’t focus on specific task coordina-
ate cognitive biases including negativity, positivity, and
tion but rather on general deliberations on various topics.
saliency biases. This assessment offers valuable insights
Similarly, [6] demonstrate the potential of collaborative
into the cognitive influences and tendencies within multi-
mechanisms with LLMs in enhancing social interactions
agent decision-making processes. Finally, the framework
among agents, providing valuable insights into how these
is applied in a simulated Human Resource recruiting
technologies can foster collaborative intelligence within
environment, serving as a practical case study. This ap-
multi-agent settings.
plication not only validates the theoretical model but
also highlights the practical potential of the approach in
real-world settings. 2.2. LLM-based Multi-Agent Frameworks
An LLM-based agent is defined as an AI system compris-
2. Related Works ing three core components: the brain, perception, and
action modules [7]. The brain module stores knowledge
2.1. Multi-Agent Collaboration and memories, facilitating information processing and
decision-making, essential for reasoning and handling
In the study of multi-agent systems, understanding how new tasks. The perception module extends the agent’s
agents collaborate to achieve collective objectives is es- sensory capabilities to include textual, auditory, and vi-
sential. One interesting approach, explored [3] examines
sual modalities. This enhances its understanding of the 2.4. Evaluating LLM-based Systems
environment. The perception module extends the agent’s
Evaluation for LLMs is emerging as a discipline to as-
sensory capabilities to include textual, auditory, and vi-
sess the performance of different of AI systems. Cur-
sual modalities. This enhances its understanding of the
rently, for LLMs, there is no single benchmark or proto-
environment. The action module enables the agent to
col that emerges as universally superior. This reflects the
perform physical tasks and interact with its environment.
diversity of tasks and model capabilities. [2] provides
In terms of operating mechanism, the agent use natural
an exhaustive summary and discussion based on exist-
language for communication, with the brain processing
ing works. This work covers evaluation tasks, methods
information from the perception module to form strate-
and benchmarks that are crucial for assessing the per-
gies and make decisions. In our work, we introduce the
formance of LLMs. In our work, we adopt a nuanced
concept of a Mental Model of Self (MMS). This concept has
approach to evaluation. We define specific metrics to
been discussed in social psychology [8]. It refers to an
assess the conversation quality. These metrics include
integrated theory and understanding that an agent forms
nuance, platitudinal score, drift and defensibility scores,
to organize and make sense of one’s self-knowledge, ex-
which are detailed in Section 5.6.
periences and memories into broader principles that can
guide anticipation of future behaviors and consequences.
In our implementation, it serves as an important organi- 3. Preliminaries
zational function in making sense of self-knowledge.We
summarize and show differences between the Sentimental 3.1. Conversation protocols
Agents framework and prior works in Table 1.
Consider a conversational simulation system with a set
of agents denoted as ℳ = {𝑚1 , 𝑚2 , … , 𝑚𝑛 }. Each agent
2.3. Non-strategic Multi-Agent Systems 𝑚𝑖 ∈ ℳ is initialized with a Mental Model of Self (MMS)
Opinion dynamics has been extensively explored for and a memory component for storing an opinion log. In
over six decades, predominantly in the fields of sociol- this system, the engagement among agents in each round
ogy and psychology. It delves into the mechanisms and 𝑡 is ordered with equal participation.
principles that dictate the formation and alteration of
Definition 1. Argument (𝐴) is a component of an opin-
individual opinions under the influence of others. This
ion that contributes to its overall sentiment.
involves examining a range of models and frameworks
to comprehend collective behaviors and the process of For each argument 𝐴 a sentiment value 𝑆𝐴 is assigned,
consensus formation [9]. Our work focuses on a non- mapping the argument to a spectrum of sentiment values
strategic model within opinion dynamics, meaning the (positive, negative, neutral, and their intensities):
model does not incorporate game theory principles, nor
does it involve agents optimizing specific utilities. 𝑆𝐴 = 𝑓 (𝐴) (1)
Non-Bayesian updating, in this context, signifies a where 𝑓 ∶ 𝐴 ↦ 𝑆 is the sentiment mapping function
𝐴
process wherein opinions are modified not based on a for arguments.
factual or probabilistic framework that converts prior
probabilities into posterior probabilities. Instead, this Definition 2. 𝑂(𝑚𝑖 , 𝑡) is the opinion 𝑚𝑖 in a given round
approach entails agents updating their opinions influ- 𝑡 is a set of arguments 𝐴.
enced by the views of others, without basing these on an
The sentiment of an opinion 𝑆𝑂 is the average of the
unknown state of nature. The updating mechanism in
sentiment values 𝑆𝐴 of all its arguments:
such models can be either synchronous, where all agents
update their opinions simultaneously, or asynchronous, 1
𝑆𝑂 (𝑚𝑖 , 𝑡) = ∑ 𝑆 (2)
where updates occur at different times. A recent survey |𝑂(𝑚𝑖 , 𝑡)| 𝐴∈𝑂(𝑚 ,𝑡) 𝐴
𝑖
categorizes and discusses various models prevalent in
existing literature [10]. The Ordered Engagement in the system is repre-
We further use Sentiment Analysis to investigate sented by a function 𝐸 ∶ ℳ × 𝑡 → 𝑚𝑖 , which establishes
opinions which manifest as either positive or negative the speaking order of agents in each round 𝑡. Under this
[11]. Studies have shown that generative models, such as model, each agent 𝑚𝑖 contributes exactly one opinion
Large Language Models (LLMs), are capable of producing per round. The collective state of opinions at any given
text, which can include opinions with specific sentiments, round 𝑡 is represented as a vector:
depending on their application [12].
𝑋𝐸 (𝑡) = [𝑂(𝑚1 , 𝑡), 𝑂(𝑚2 , 𝑡), … , 𝑂(𝑚𝑛 , 𝑡)] (3)
In each conversation round 𝑡, the sentiment value 𝑆𝑂 𝑚𝑖 , 𝑡
for each agent 𝑚𝑖 is updated to reflect the sentiment of the
Related Work Sentiment Analysis Engagement type Memory Decision module Bias Evaluation
[13] No Ordered Belief No Confirmation bias
[14] No Ordered Store/Retrieve Yes No
[15] No Ordered Internal critic Yes Fact-checking
[16] No Varies Chat history Yes No
[17] No Ordered Specialized roles Yes No
[18] No Ordered User-driven Yes User-preference
[19] No Ordered Rationale analysis Yes Credibility check
[20] No Ordered Dynamic Memory No Opinion classifier
Sentimental Agents Yes Ordered Opinion logs Yes Cognitive bias
Table 1
A comparison of different language model-based multi-agent frameworks.
newly formed opinion. This process considers the senti- The Valence 𝑉𝑗 for each item 𝑗 is determined based on
ment values 𝑆𝐴 of the arguments within the opinion 𝑂. the sentiment of opinion 𝑆𝑂 . For 𝑆𝑂 < −0.5, 𝑉𝑗 = −1; for
The sentiment update is executed using a Non-Bayesian −0.5 ≤ 𝑆𝑂 ≤ 0.5, 𝑉𝑗 = 0; and for 𝑆𝑂 > 0.5, 𝑉𝑗 = 1. Items
method, mathematically represented by: are classified into three tiers according to 𝑉𝑗 : Tier 1 for
𝑉𝑗 = 1, Tier 2 for 𝑉𝑗 = 0, and Tier 3 for 𝑉𝑗 = −1.
1 Definition 5. Gut-feeling List Protocol: A method to
𝑆𝑖𝑂 (𝑡) = 𝛼 ⋅ ( ∑ 𝑆 ) + (1 − 𝛼) ⋅ 𝑆𝑖𝑂 (𝑡 − 1) (4)
|𝐴| 𝐴∈𝑂(𝑚 ,𝑡) 𝐴 collectively rank a list of items based on the confidence of
𝑖
individuals’ feeling toward each item.
Here, 𝑆𝑖𝑂 (𝑡) represents the average sentiment of all the
arguments expressed by agent 𝑚𝑖 at round 𝑡, with each The volatility 𝜈𝑚𝑖 ,𝑗 of agent 𝑚𝑖 ’s sentiment towards item
argument 𝐴 having its sentiment value 𝑆𝐴 = 𝑓 (𝐴). The 𝑗 over several rounds is calculated. Conviction 𝐼𝑚𝑖 ,𝑗 is
parameter 𝛼 is a weighting factor that determines the derived as a function of both volatility 𝜈𝑚𝑖 ,𝑗 and the final
influence of the new opinion’s average sentiment on the sentiment score 𝑆𝑚𝑖 ,𝑗 for item 𝑗. The Gut-feeling list is
agent’s updated sentiment. then generated using a Borda count based on 𝐼𝑚𝑖 ,𝑗 for each
The change in sentiment Δ𝑆𝑖𝑂 (𝑡) for agent 𝑚𝑖 is then item across all agents, and items are ranked based on the
calculated as the absolute difference between the updated |ℳ|
total Conviction points 𝑇𝑗𝐼 = ∑𝑖=1 𝐼𝑚𝑖 ,𝑗 in descending
sentiment value 𝑆𝑖𝑂 (𝑡) at round 𝑡 and the agent’s previous order.
sentiment value 𝑆𝑖𝑂 (𝑡 − 1) at round 𝑡 − 1:
Δ𝑆𝑖𝑂 (𝑡) = |𝑆𝑖𝑂 (𝑡) − 𝑆𝑖𝑂 (𝑡 − 1)| (5) 4. Applying the Framework
Our framework is applied to a simulated environment
3.2. Collective decision protocols inspired by Human Resource recruiting to evaluate the
When the conversation ends we take the total sentiment. effectiveness of Sentimental Agents. These agents are de-
We have the final sentiment score and we have the aver- signed to generate opinions reflecting their unique exper-
age of the 𝑆0 for the gut feeling protocol tise, contributing to collective decision-making. The sim-
ulation explores opinion formation and decision-making
Definition 3. Borda Count Protocol: A method to col- processes within an LLM-based multi-agent setting, mir-
lectively rank a list of items, given each individual’s order roring real-world HR recruitment where employers as-
of preference. sess candidates through discussions with various experts.
Given 𝑛, the number of items, each agent 𝑚𝑖 ranks these In this context, LLM-based agents are expected to engage
items. The point assignment for an item 𝑗 by agent 𝑚𝑖 in conversation and form diverse opinions that influence
is 𝑃𝑚𝑖 ,𝑗 , with the top-ranked item receiving 𝑛 points and their decision-making in a simulated recruiting scenario.
the last receiving 1 point. The total points for each item
|ℳ|
𝑗 is calculated as 𝑇𝑗 = ∑𝑖=1 𝑃𝑚𝑖 ,𝑗 , and items are ranked in 4.1. Configuration
descending order of their total points 𝑇𝑗 .
In the HR recruiting simulation, advisor agents analyze
Definition 4. Tiered List Protocol: A method to collec- candidates’ CVs and engage in discussions to provide
tively classify a list of items in 3 tiers, given the items that opinions about each candidate. These agents, with ex-
each individual can’t accept, and the items they like the pertise in roles like Chief Financial Officer (CFO), Vice
most. President of Engineering, and Recycling Plant Manager,
evaluate profiles and generate text reports. They also and transparent data. Figure 2 shows agent initialization
score candidates and, through collective decision-making prompt.
protocols like the Borda Count, rank candidates or select
the top performers.
4.1.1. Dataset
We sourced our dataset from the study conducted by
[21]. This dataset is a collection of resumes represented
in a multi-label format. To facilitate easy access and in-
tegration of this dataset into our framework, we have
developed a script that automates the process of down-
loading and parsing the data.
5. Sentimental Agents Framework
The system design as shown in Fig 1, consists of 7 mod-
ules. We describe each of them here.
Figure 2: Series of prompts used to create a group of agents
Mental Model of Self, for one instance of the system
5.1. Brief Module
The module provides a configuration interface for sys-
tem initialization with four components: input type ,
output type , task type , and context . It handles sin-
5.3. Opinion Dynamics Module
gle and multiple item formats for input and output and This module coordinates agent conversations and
requires user-defined context specifying task object decision-making, consisting of conversation and decision-
and subject , with optional Knowledge base integration. making protocols. It focuses on: defining the number of
Predefined rules in the module automatically associate agents, engagement type and stopping mechanism. The
Input, Output, and Task Types. The logic enforces spe- current implementation employs ordered engagement
cific task types Evaluate, Score, Classify for single-item with equal participation. For the stop mechanism, the
inputs and broader tasks for multi-item inputs. For rank module uses a non-strategic approach, differentiating
tasks, the output is structured as a list to match task from strategic interactions. In this non-strategic context,
requirements. This design ensures alignment between conversations conclude based on non-Bayesian updating
input/output formats and system functionality. as shown in Algorithm 1 , where they end once agents’
opinions reach stability. This contrasts with strategic
5.2. Agent Initialization interactions, which involve different mechanisms like
rewards or objectives.
The Agent Initialization module includes two main ele-
ments: Mental Model of Self (MMS) and Memory with
qualitative and quantitative opinion logs. It configures 5.4. Conversation Module
agent interaction types for opinion formation as dynamic This module analyzes agents’ statements in conversa-
or independent. The module requires user input to set the tions, comprising four components: Argumentation,
number of agents and their expertise, which informs the which breaks down statements into arguments for quali-
creation of detailed agent profiles, including priorities, tative logging; Sentiment Analysis, which evaluates and
objectives, and evaluation criteria. Figure 3 shows an in- quantitatively logs the sentiment of each argument; Opin-
stance of an MMS. Key parameters include the tolerance ion Change, using non-Bayesian updating to monitor sen-
level, affecting opinion change propensity, and the drift timent shifts; and Conversation Trends, gauging signifi-
metric, which tracks MMS variability. Strategies for main- cant changes across rounds to infer opinion stabilization
taining agent consistency involve controlled character and conversation conclusion.
prompts and setting MMS prompt temperature. The opin-
ion formation, generated via boolean input, influences
5.5. Decision Module
the nature of agents’ decision-making processes. Each
agent’s opinion log is stored in a central memory system, The Decision Making Protocol module is designed to ac-
ensuring decision-making is based on comprehensive commodate various decision-making protocols, including
Borda Count, Tiered List, and Gut Feeling List, as detailed
opinion, potentially overshadowing other relevant infor-
mation [26].
Algorithm 1 Non-Bayesian Updating
1: for each round 𝑡 do
2: for each agent 𝑚𝑖 ∈ ℳ do
3: if 𝑡 > 0 then
4: Equation 5 Δ𝑆𝑖𝑂 (𝑡) = |𝑆𝑖𝑂 (𝑡) − 𝑆𝑖𝑂 (𝑡 − 1)|
5: Equation 4 𝑆𝑖𝑂 (𝑡) = 𝛼 ⋅𝑆𝑂 (𝑚𝑖 , 𝑡)+(1−𝛼)⋅𝑆𝑖𝑂 (𝑡 −
1)
6: end if
7: end for
8: if all Δ𝑆𝑖𝑂 (𝑡) < threshold for each 𝑚𝑖 ∈ ℳ or 𝑡 =
Figure 3: An instance of an agent’s Mental Model of Self in a max_rounds then
simulated HR environment. In this case, the agent took a Job Title 9: Set 𝑐𝑜𝑛𝑣𝑒𝑟𝑠𝑎𝑡𝑖𝑜𝑛_𝑎𝑐𝑡𝑖𝑣𝑒 to False
as input, and generated a Description, Priorities, and Evaluation
10: end if
Criteria for a given Job Description.
11: Increment 𝑡
12: end for
in the preliminaries (Section 3). It operates by capturing
the final sentiment of each agent and the average sen-
timent throughout the conversation. The functionality 6. Experiments
and outcomes of these different decision-making pro-
cesses are further explored and discussed in the results In this study, we aim to investigate the dynamics of sen-
(Section 6). timent and opinion formation in an LLM-based multi-
agent system. We focus on understanding how agents’
opinions evolve through deliberation, and how sentiment
5.6. Evaluation and Cognitive Bias influences their decision-making processes. Our research
Modules questions are as follows:
This module evaluates the quality of conversations 1. How do agents’ opinions change as a result of
through various metrics. deliberating with each other, and can we quantify
these changes?
• Nuance: Examines the diversity of themes and
perspectives, quantified by the number of top- 2. Do agents adopt each other’s arguments during
ics identified within individual statements or the the deliberation process, and can we observe this
entire conversation. in qualitative results?
• Platitudinal Score: Calculated using cosine sim- 3. Does the sentiment of an argument (valence,
ilarity, it measures the uniqueness of outcomes arousal) affect its adoption by other agents?
in the conversation rounds, with higher scores 4. Do agents exhibit cognitive biases in their opinion
indicating less similarity between different runs. formation, and how can we identify and mitigate
• Drift: Assesses the stability of each agent’s Men- these biases?
tal Model of Self, monitoring the relevance of
results to the advisors’ profiles and checking for 6.1. Experimental Setting
consistency throughout the conversation.
In our experiments, we conducted the simulation with 3
• Defensibility: Evaluates the strength and evi-
agents and 10 candidates. We used the data set within the
dence backing of the agents’ arguments, ensuring
simulation environment described in Section 4. For the
they are well-supported and referenceable.
LLM, we used the gpt-3.5-turbo-0613 version of ChatGPT
In this research, we examine three cognitive biases: [27]. For the result shown, the language model param-
negativity, positivity, and saliency. Negativity bias eters were set as alpha 𝛼 = 0.5, tolerance = 0.00001, and
might lead agents to give undue weight to adverse opin- temperature = 1.5.
ions [22] [23], while positivity bias could result in an
overemphasis on favorable views [24] [25]. Saliency bias,
on the other hand, might cause agents to focus on the
most prominent or emotionally striking aspects of an
(a) (b) (c) (d) (e)
(f) (g) (h) (i) (j)
Figure 4: Sentiment change, and corresponding Opinion Change in conversations for two different candidates. Each conversation
stops after five rounds.
(a) (b) (c) (d) (e)
Figure 5: Platitudinal Score: The uniqueness of outcomes in the conversation rounds among agents (a lower Score indicates a more
original contribution)
6.2. Results Drift scores. In Table 3 it is observed that the CFO
agent generally exhibits moderate drift, while the VP
6.2.1. Evaluation Metrics
of Engineering (VPE) and the Recycling Plant Manager
The non-Bayesian updating data from the simulation, (RPM) show higher drift values, suggesting a more dy-
shown in Figure 4, reveals sentiment fluctuations among namic adaptation of their MMS in response to the con-
agents. For instance, Figure 5a shows the VP of Engineer- versation. This variability in drift signifies the agents’
ing exhibiting the most dramatic change, especially in the differing levels of adaptability and potential reevaluation
final round. This volatility, captured by sentiment and of their initial stances
change metrics, highlights the dynamic nature of opinion
Candidate CFO VPE RPM
formation in multi-agent conversations and suggests that
Kimberly Carr 0.4504 0.7556 0.7138
agents’ opinions evolve and respond to the unfolding dis-
Melissa Morgan 0.6224 0.6308 0.5810
course, emphasizing the effectiveness of non-Bayesian Mikayla Garrison 0.3254 0.5878 0.5720
updating in capturing real-time perspective shifts. 1
Emily Marshall 0.4678 0.7998 0.7390
Platitudinal score. The inter-agent similarities Justin Davis 0.3458 0.3638 0.3940
heatmap shown in Figure 5 reveals a contrast in sen- Tamara Brown 0.3842 0.6030 0.6574
timent alignment among the agents. This divergence Taylor Mahoney 0.3814 0.4794 0.4154
contributes to an overall lower platitudinal score for this Joshua Alvarado 0.3756 0.5238 0.5788
specific run for the given candidate. Such diversity in Melissa Baldwin 0.4228 0.7988 0.5714
sentiment, as captured by the platitudinal metric, un- James Wallace 0.4240 0.6342 0.6926
derscores the variation in decision-making approaches Table 2
within the agent group, emphasizing the balance between Agent Drift Values for hypothetical candidates
consensus and individual thought in the simulation out-
comes.
Nuance Scores We use Latent Dirichlet Allocation
1
For brevity, we only show results for 5 candidates, but the experi-
ment was conducted with 10 candidates for the platitudinal scores, Sentiment and Opinion change
(LDA) to extract topics from text statements. The data is well as the data point distribution, provides insights into
preprocessed by tokenization and removal of stop words the cognitive tendencies of the agents. Additionally, by
and unwanted words. A dictionary and corpus are con- adjusting our three parameters, alpha, tolerance, and tem-
structed using the Gensim library. The LDA model iden- perature, we aim to better understand how these factors
tifies 5 topics, with the top 10 words per topic being most affect agents’ cognitive biases. This study offers impor-
significant. Figure 6 and ?? show the number of unique tant insights into the decision-making processes in multi-
words per topic and word clouds for each candidate, re- agent systems, particularly in sentiment-influenced con-
spectively. texts.
In our sensitivity analysis, we varied key parameters:
setting alpha to 0.3, 0.5, and 0.7; tolerance to 0.001, 0.005,
and 0.0001; and temperature to 0.7, 1, and 1.5, to eval-
uate their impact on sentiment changes. The outcome,
depicted in Figure 7 for ten random candidates, provides
insight into negativity and positivity biases through the
slopes of the OLS regressions. Our findings on this varia-
tion of model parameters show a modest positivity bias,
evidenced by the positive slope being approximately 29%
steeper than its negative counterpart. A slight positivity
or negativity bias trend persisted across varied param-
eter settings, with some scenarios, notably alpha = 0.3,
Figure 6: The nuance score for each candidate by showing the tolerance = 0.005, and temperature = 1.5, showing a more
number of unique words used across all topics. pronounced positivity bias with a slope more than twice
as steep on the positive side than on the negative side.
Defensibility Scores Candidate resumes are pro- The absence of saliency bias was noted in all experi-
cessed through the Langchain embedding2 and trans- ments, as indicated by slopes remaining below 1. Linear
formed into a format suitable for detailed analy- regression was determined as the most suitable model
sis. The llama index libraries, VectorStoreIndex and based on our evaluation of the 𝑅2 values. Notably in the
ServiceContext , are used to create an indexed reposi- shown experiment, agents displayed a tendency towards
tory of the vectorized documents. This index serves as a expressing stronger negative sentiments, with the most
searchable database, allowing efficient retrieval of text negative reaching -0.76, compared to a maximum posi-
segments that are contextually similar to a given input. tive sentiment of 0.62. This inclination towards stronger
When evaluating agents’ arguments, the indexed space negative expressions was marked in most scenarios. Ad-
is searched to find text segments from the resumes that ditionally, the alpha parameter was observed to signifi-
closely match the argument. The similarity between an cantly influence sentiment ranges, with lower alpha val-
agent’s argument and the retrieved text is quantified as ues yielding more constrained ranges.
a score, with higher scores indicating stronger support For future studies, we aim to extend our examination of
for the argument. If no relevant text is found, a score of the cognitive bias to larger candidate sample sizes. This
zero is assigned, suggesting an unsupported argument. expansion will enable us to deepen our understanding
of how parameter tuning influences cognitive biases and
6.2.2. Cognitive Bias Testing decision-making processes within our framework.
We hypothesize that agents’ updates in sentiment during 6.2.3. Collective decision-making
conversational rounds might be influenced by their peers’
positive, negative, or prominent opinions. To investigate The decision-making data reveals diverse agent prefer-
this, we chart each agent’s sentiment change from the ences, as evidenced by the variation in candidate ranks
second round onwards, against the recent sentiments across Borda Count, Tier, and Conviction. We use the av-
of other agents. This analysis reveals the correlation erage sentiment score, 𝑆𝑖𝑂 (𝑡), from equation 4, where 𝑡 is
between an agent’s changing sentiment and the influence the last round, as the basis for collective decision-making.
of peer opinions. While some candidates consistently rank higher or lower,
We apply Ordinary Least Squares (OLS) regression to suggesting a consensus on their suitability, discrepancies
analyze nega tive and positive sentiments separately, set- in ranks among agents could reflect unique valuations of
ting the y-intercept at zero to indicate that neutral peer candidate qualities.
statements might not impact an agent’s sentiment. An- Table 4 shows the overall sentiment scores. The
alyzing the regression’s strength (𝑅2 ) and the slope, as CFO shows the highest sentiment score, of 0.46 towards
2 Melissa Baldwin, indicating a strong positive inclination.
https://github.com/langchain-ai/langchain
Final Sentiment Scores
Candidate CFO VPE RPM
Kimberly Carr -0.51 -0.13 0.26
Melissa Morgan -0.74 -0.21 -0.46
Mikayla Garrison 0.69 0.38 0.59
Emily Marshall 0.29 -0.29 0.05
Justin Davis -0.01 -0.12 0.37
Tamara Brown 0.11 0.36 0.20
Taylor Mahoney -0.49 -0.55 -0.41
Joshua Alvarado 0.28 0.58 0.25
Melissa Baldwin 0.46 0.34 0.17
James Wallace -0.30 0.11 -0.30
Table 4
Final Sentiment Scores for each candidate
Figure 7: The results of the cognitive bias testing using a sample
of ten candidates and model parameters of alpha = 0.5, tolerance
= 0.00001, and temperature = 1.5. Candidates Rank
Candidate CFO VPE RPM Borda Count Rank
Kimberly Carr 9 5 3 4
Candidate Borda Count Tiered Gut-Feeling Melissa Morgan 10 6 10 2
Kimberly 9 3 10 Mikayla Garrison 2 9 4 7
Melissa 10 3 8 Emily Marshall 3 8 6 4
Mikayla 2 2 1 Justin Davis 5 4 2 8
Emily 3 2 3 Tamara Brown 6 7 7 3
Justin 5 2 5 Taylor Mahoney 8 10 9 1
Tamara 6 2 6 Joshua Alvarado 4 3 1 9
Taylor 8 3 7 Melissa Baldwin 1 1 5 10
Joshua 4 2 4 James Wallace 7 2 8 4
Melissa 1 2 2
James 7 2 9 Table 5
Rank of each candidate, including the final Rank taking into
Table 3 account each agent’s individual rankings (calculated through
Candidate Ranking Metrics Borda count)
In contrast, the CFO’s lowest sentiment score is -0.74 to one candidate. Consequently, no candidate was clas-
towards Melissa Morgan, signaling a significant negative sified as Tier 1, with most classified as Tier 2, except for
view. Similarly, the VPE aligns with the CFO in favor- the three candidates with negative valence, who were
ing Melissa Baldwin with the highest score of 0.34, but classified as Tier 3.
diverges in its lowest sentiment, which is directed to-
wards Taylor Mahoney with a score of -0.55. The RPM, Valence
on the other hand, exhibits the most positive sentiment Candidate CFO VPE RPM
towards Mikayla Garrison with a score of 0.59, while Kimberly Carr -1 0 0
sharing the CFO’s negative sentiment towards Melissa Melissa Morgan -1 0 0
Morgan, albeit at a less intense level of -0.46. The sen- Mikayla Garrison 0 0 0
timent scores from the conversations directly influence Emily Marshall 0 0 0
Justin Davis 0 0 0
the ranking of candidates as shown in Table 5. Applying
Tamara Brown 0 0 0
the Borda Count method to the combined rankings yields Taylor Mahoney 0 -1 0
a collective decision. Although individual agents might Joshua Alvarado 0 0 0
rank candidates differently based on their interactions, Melissa Baldwin 0 0 0
the aggregated results provide a more comprehensive James Wallace 0 0 0
assessment. This approach demonstrates how sentiment
Table 6
analysis combined with a voting system could inform
Valence for each candidate
hiring decisions in a multi-agent setting.
The intensity of an agent’s final sentiment score deter-
mines the valence score. In Table 6, this occurred only The sentiment volatility of the agents, as shown in Ta-
three times: the CFO attributed a negative valence to two ble 7, was mostly moderate, indicating strong conviction
candidates, while the VPE attributed a negative valence in their opinions. However, there were instances of high
volatility, such as the CFO’s sentiment towards Kimberly Gut Feeling
Carr and Mikayla Garrison, and the VPE’s sentiment Candidate CFO VPE RPM Borda Count Rank
towards Kimberly, Joshua, and James. The RPM’s senti- Kimberly Carr 2 6 2 5
ment was volatile towards Mikayla and Melissa Baldwin. Melissa Morgan 8 5 10 2
Mikayla Garrison 2 9 4 7
The agents’ conviction in their opinions is calculated by
Emily Marshall 3 8 6 6
dividing the final sentiment by the volatility, with higher
Justin Davis 5 4 1 8
values indicating stronger intuition about a candidate’s Tamara Brown 6 7 7 3
suitability for the role. Taylor Mahoney 7 9 8 1
Joshua Alvarado 4 3 1 9
Sentiment Volatility Melissa Baldwin 2 1 4 10
Candidate CFO VPE RPM James Wallace 9 2 9 3
Kimberly Carr 0.58 0.64 0.40
Table 9
Melissa Morgan 0.16 0.32 0.47
Gut feeling for each candidate (Ranking of candidates that com-
Mikayla Garrison 0.31 -0.47 0.17
bines both the Sentiment Score, and the Conviction an agent has
Emily Marshall 0.17 0.42 0.21
in this sentiment
Justin Davis 0.49 0.16 0.34
Tamara Brown -0.06 -0.27 -0.16
Taylor Mahoney 0.13 0.26 0.17
Joshua Alvarado 0.09 0.09 0.42 posed framework integrates a non-Bayesian updating
Melissa Baldwin 0.14 0.46 0.61 mechanism to track sentiment volatility and opinion evo-
James Wallace 0.39 0.54 0.39 lution. In a simulated HR recruiting scenario, we assess
Table 7
these agents’ decision-making abilities, noting their di-
Sentiment Volatility for each candidate verse opinions and preference shifts over multiple rounds.
The findings suggest model parameters, such as alpha
Conviction and tolerance, significantly influence sentiment expres-
Candidate CFO VPE RPM
sion and thus cognitive bias within the system. This re-
Kimberly Carr -0.30 -.0.08 0.11
search offers a foundation for advanced tool development
Melissa Morgan -0.12 -0.07 -0.22
Mikayla Garrison 0.21 -0.18 0.10 applicable to domains such as HR recruiting, medical di-
Emily Marshall 0.05 -0.12 0.01 agnostics, or educational domains.
Justin Davis 0.00 -0.02 0.13
Tamara Brown -0.01 -0.10 -0.03
Taylor Mahoney -0.06 -0.14 -0.07 References
Joshua Alvarado 0.02 0.05 0.10
Melissa Baldwin 0.06 0.16 0.10 [1] R. OpenAI, Gpt-4 technical report. arxiv 2303.08774,
James Wallace -0.12 0.06 -0.12 View in Article 2 (2023) 13.
[2] Y. Chang, X. Wang, J. Wang, Y. Wu, K. Zhu, H. Chen,
Table 8 L. Yang, X. Yi, C. Wang, Y. Wang, A survey on
Conviction for each candidate (the Sentiment Score of given by evaluation of large language models, arXiv preprint
agents to a candidate, taking into account the Sentiment Volatility
arXiv:2307.03109 (2023). URL: https://arxiv.org/abs/
during a conversation about this candidate)
2307.03109.
[3] H. Li, Y. Q. Chong, S. Stepputtis, J. Campbell,
Gut Feeling Rank for each candidate is a revised rank- D. Hughes, M. Lewis, K. Sycara, Theory of mind
ing that takes into account an agent’s conviction in its for multi-agent collaboration via large language
own sentiment. In Table 9, the Gut Feeling of the RPM models, arXiv preprint arXiv:2310.10701 (2023).
toward Joshua is still to rank him in the first place. But [4] L. D. de Tarlé, E. Bonzon, N. Maudet, Multia-
the CFO revises its ranking of Kimberley, from the 9th gent dynamics of gradual argumentation semantics,
place to the 8th place. The more generous ranking can in: 21st International Conference on Autonomous
be interpreted as a result of the ”acknowledgement” of Agents and Multiagent Systems (AAMAS 2022),
the RPM agent that it is not sure of its opinion toward 2022.
Kimberly. [5] S. A. Wu, R. E. Wang, J. A. Evans, J. B. Tenenbaum,
D. C. Parkes, M. Kleiman-Weiner, Too many cooks:
Bayesian inference for coordinating multi-agent
7. Conclusion collaboration, Topics in Cognitive Science 13 (2021)
In this paper, we introduce Sentimental Agents, 414–432.
LLM-based agents that generate opinions for collective [6] J. Zhang, X. Xu, S. Deng, Exploring Collaboration
decision-making within conversational settings. Our pro- Mechanisms for LLM Agents: A Social Psychology
View, 2023. URL: http://arxiv.org/abs/2310.02124, [20] Y.-S. Chuang, A. Goyal, N. Harlalka, S. Suresh,
arXiv:2310.02124 [cs]. R. Hawkins, S. Yang, D. Shah, J. Hu, T. T. Rogers,
[7] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, Simulating Opinion Dynamics with Networks of
M. Zhang, J. Wang, S. Jin, E. Zhou, et al., The rise LLM-based Agents, 2023. URL: http://arxiv.org/abs/
and potential of large language model based agents: 2311.09618, arXiv:2311.09618 [physics].
A survey, arXiv preprint arXiv:2309.07864 (2023). [21] K. Jiechieu, N. Tsopze, Skills prediction based
[8] D. Hart, S. Fegley, Social imitation and the emer- on multi-label resume classification using cnn
gence of a mental model of self. (1994). with model predictions explanation, Neural
[9] A. Sîrbu, V. Loreto, V. D. P. Servedio, F. Tria, Computing and Applications (2020). URL: https://
Opinion Dynamics: Models, Extensions and Exter- doi.org/10.1007/s00521-020-05302-x. doi:10.1007/
nal Effects, in: V. Loreto, M. Haklay, A. Hotho, s00521- 020- 05302- x .
V. D. Servedio, G. Stumme, J. Theunis, F. Tria [22] T. A. Ito, J. T. Larsen, N. K. Smith, J. T. Cacioppo,
(Eds.), Participatory Sensing, Opinions and Collec- Negative information weighs more heavily on the
tive Awareness, Springer International Publishing, brain: the negativity bias in evaluative categoriza-
Cham, 2017, pp. 363–401. URL: http://link.springer. tions., Journal of personality and social psychology
com/10.1007/978-3-319-25658-0_17. doi:10.1007/ 75 (1998) 887.
978- 3- 319- 25658- 0_17 , series Title: Understand- [23] P. Rozin, E. B. Royzman, Negativity bias, negativity
ing Complex Systems. dominance, and contagion, Personality and social
[10] M. Grabisch, A. Rusinowska, A survey on non- psychology review 5 (2001) 296–320.
strategic models of opinion dynamics, Games 11 [24] M. W. Matlin, D. J. Stang, The Pollyanna princi-
(2020) 65. ple: Selectivity in language, memory, and thought,
[11] E. Cambria, D. Das, S. Bandyopadhyay, A. Feraco, Schenkman Publishing Company, 1978.
et al., A practical guide to sentiment analysis, vol- [25] P. S. Dodds, E. M. Clark, S. Desu, M. R. Frank, A. J.
ume 5, Springer, 2017. Reagan, J. R. Williams, L. Mitchell, K. D. Harris, I. M.
[12] U. Maqsud, Synthetic text generation for sentiment Kloumann, J. P. Bagrow, et al., Human language re-
analysis, in: Proceedings of the 6th Workshop on veals a universal positivity bias, Proceedings of the
Computational Approaches to Subjectivity, Senti- national academy of sciences 112 (2015) 2389–2394.
ment and Social Media Analysis, 2015, pp. 156–161. [26] M. P. Inderbitzin, A. Betella, A. Lanatá, E. P. Scilingo,
[13] G. Betz, Natural-language multi-agent simulations U. Bernardet, P. F. Verschure, The social perceptual
of argumentative opinion dynamics, arXiv preprint salience effect., Journal of experimental psychology:
arXiv:2104.06737 (2021). human perception and performance 39 (2013) 62.
[14] Y. Li, Y. Zhang, L. Sun, Metaagents: Simulating [27] https://platform.openai.com/docs/models/gpt-3-5,
interactions of human behaviors for llm-based task- 2024. Accessed: 2024-10-9.
oriented coordination via collaborative generative
agents, arXiv preprint arXiv:2310.06500 (2023).
[15] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, I. Mor-
datch, Improving factuality and reasoning in lan-
guage models through multiagent debate, arXiv
preprint arXiv:2305.14325 (2023).
[16] C.-M. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang,
J. Fu, Z. Liu, Chateval: Towards better llm-based
evaluators through multi-agent debate, arXiv
preprint arXiv:2308.07201 (2023).
[17] G. Chen, S. Dong, Y. Shu, G. Zhang, J. Sesay, B. F.
Karlsson, J. Fu, Y. Shi, Autoagents: A framework
for automatic agent generation, arXiv preprint
arXiv:2309.17288 (2023).
[18] J. Park, B. Min, X. Ma, J. Kim, ChoiceMates: Support-
ing Unfamiliar Online Decision-Making with Multi-
Agent Conversational Interactions, 2023. URL: http:
//arxiv.org/abs/2310.01331, arXiv:2310.01331 [cs].
[19] X. Sun, X. Li, S. Zhang, S. Wang, F. Wu, J. Li,
T. Zhang, G. Wang, Sentiment Analysis through
LLM Negotiations, 2023. URL: http://arxiv.org/abs/
2311.01876, arXiv:2311.01876 [cs].