=Paper=
{{Paper
|id=Vol-2978/casa-paper5
|storemode=property
|title=A Probabilistic Model for Personality Trait Focused Explainability
|pdfUrl=https://ceur-ws.org/Vol-2978/casa-paper5.pdf
|volume=Vol-2978
|authors=Mohammed N. Alharbi,Shihong Huang,David Garlan
|dblpUrl=https://dblp.org/rec/conf/ecsa/AlharbiHG21
}}
==A Probabilistic Model for Personality Trait Focused Explainability==
https://ceur-ws.org/Vol-2978/casa-paper5.pdf
A Probabilistic Model for
Personality Trait Focused Explainability
Mohammed N. Alharbi Shihong Huang David Garlan
Department of Computer & Electrical Department of Computer & Electrical Institute for Software Research
Engineering and Engineering and School of Computer Science
Computer Science Computer Science Carnegie Mellon University
Florida Atlantic University Florida Atlantic University Pittsburgh PA USA
Boca Raton FL USA Boca Raton FL USA garlan@cs.cmu.edu
malharbi2016@fau.edu shihong@fau.edu
Abstract— Explainability refers to the degree to
which a software system’s actions or solutions can be
understood by humans. Giving humans the right
amount of explanation at the right time is an
important factor in maximizing the effective
collaboration between an adaptive system and
humans during interaction. However, explanations
come with costs, such as the required time of
explanation and humans’ response time. Hence it is
not always clear whether explanations will improve
overall system utility and, if so, how the system should
effectively provide explanation to humans,
particularly given that different humans may benefit
from different amounts and frequency of explanation.
To provide a partial basis for making such decisions,
this paper defines a formal framework that
incorporates human personality traits as one of the
important elements in guiding automated decision-
making about the proper amount of explanation that
should be given to the human to improve the overall
system utility. Specifically, we use probabilistic model
analysis to determine how to utilize explanations in an Fig. 1 A probabilistic model for personality trait focused explainability
effective way. To illustrate our approach, Grid – a framework: this framework incorporates two basic personality traits (Openness and
virtual human and system interaction game -- is Need for Cognition) as important elements in a human model that can be used to guide
developed to represent scenarios for human-systems a system in deciding the appropriate amount of explanation that should be given to the
collaboration and to demonstrate how a human’s human
personality traits can be used as a factor to consider for systems While explanation is an increasingly desirable – even,
in providing appropriate explanations.
essential – capability of a system, it is not at all obvious when
Keywords— explainability, human system co-adaptation, and how explanation should be given, particularly since
human computer interaction (HCI), personality traits, self- explanation comes with a cost on human attention and delays
adaptive systems, human-in-the-loop, model checking, in system-human interaction and the fact that different humans
probabilistic model. may need different kinds of explanation. To partially address
this problem this paper defines a formal framework, as
I. INTRODUCTION illustrated in Figure 1, for reasoning about the proper amount
As systems become more autonomous and intelligent of explanation that a system should provide to the human
through the incorporation of AI techniques and self-adaptive based on their personality traits. Specifically, leveraging
research in the psychology of human personality, this
approaches, it becomes increasingly important for those
framework incorporates two basic personality traits
systems to be able to “explain” themselves to their human
(Openness and Need for Cognition) as important elements in
users and collaborators [1][2]. In particular, there are four a human model that can be used to guide a system in deciding
main purposes of explainability: (1) explain to justify: use the appropriate amount of explanation that should be given to
explanations to justify some results to the human, particularly the human in order to improve overall system utility. The
when decisions are made suddenly; (2) explain to control: effects of given explanations (which are determined based on
explanations can help not only to justify, but also to prevent personality traits of the human) affect human-system co-
systems from going wrong; (3) explain to improve: improving adaptation, represented through the Opportunity-Willingness-
the systems continuously through human involvement; (4) Capability (OWC) model, a commonly used model for
explain to discover: discovering and gathering new facts that adaptive systems’ reasoning about human-in-the-loop
help us to learn and to gain knowledge. In the context of this behavior [3]. We incorporate our approach into the MAPE-K
paper, explainability refers to the degree to which a software architecture [4] to formally model and analyze human
system’s actions or solutions can be understood by humans, involvement at different stages of system management and
and explainability is used to improve a system’s overall utility. adaptation. To illustrate our approach, Grid – a virtual human
and system interaction game – is developed to represent
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
scenarios for human-systems collaboration and to demonstrate • An evaluation system based on a collaborative game,
how a human’s personality traits can be used as a factor to to simulate the effects of decision making under
consider for systems in providing appropriate explanations. various scenarios.
The organization of the paper is as follows: Section II III. BACKGROUND AND RELATED WORK
describes the research problem and goals, Section III
represents background information and related work, Section This section introduces some background on personality
IV shows methodology, Section V shows the Stochastic traits, the OWC (Opportunity-Willingness-Capability) model,
Multi-player Games (SMG) model while Section VI shows model checking of stochastic multi-player games (SMG), and
results and analysis, Section VII represents discussion and some of state-of-the-art studies that focus on explainability
future work and the last section focuses on the conclusion. and human-system co-adaptation. Section IV will then
illustrate how this background and related work are related to
II. PROBLEM STATEMENT AND what we do in this research.
RESEARCH GOALS A. Personality Traits
A. Problem Statement Psychological studies have demonstrated that human
A co-adaptation system is symbiotic human-in-the-loop personality traits play a strong role in determining human
system where human-system cooperation is required in behavior [8]. Personalities can be characterized in terms of
achieving shared goals, and system and human actions traits that are relatively stable characteristics of a human that
mutually impact each other’s behavior in accomplishing influence our behavior across many situations. An individual's
coordinated tasks [5]. In this context, providing effective personality is the combination of traits and patterns that
explanations to humans is an important factor in maximizing influence his/her behavior, thought, motivation, and emotion.
the co-adaptation outcomes between the system and the It drives individuals to consistently think, feel, and behave in
human [6]. Maximizing co-adaptation outcomes implies that specific ways.
the relationship between system and humans has become a There are, of course, many differences between
partnership, or collaborative relationship, in which humans individuals; however, personality traits are one of the more
and systems act semi-autonomously – in contrast to traditional important measurable characteristics that can be used to
systems that wait for the human's inputs and commands to take distinguish one person from another. In the psychological
action [6]. literature the Big Five (also called the Five Factor) model of
Given that different humans may benefit from different personality is one of the most widely accepted personality
amounts and frequency of explanation, in this paper we argue taxonomies. In the Big Five model, the five dimensions of
that adapting the explanation to the particular human through personality include extraversion, neuroticism, openness to
knowledge of their personality traits can help the system in experience, agreeableness, and conscientiousness [9].
determining what are appropriate explanations and, therefore, Openness to experience is one of the personality traits that
maximize the benefits of co-adaptation. In particular, given is used to describe individual personality in the Five Factor
that there are tradeoffs in determining what kind of Model. Open people tend to be intellectually curious, creative
explanations to give, it is important to be able to tailor the and imaginative. Open people have a high openness to
explanations to the user [7]. Providing longer and more- embrace new things, fresh ideas, and novel experiences [10].
frequent explanations may increase the effectiveness of
collaboration between the system and the human; however, In addition to the Five Factor Model, the psychological
this comes at the cost of taking more time for humans to literature also identifies Need for Cognition is an important
understand the explanations and respond accordingly. Thus, distinguishing characteristics of human personality trait
key questions that must be answered by a system are: What [9][11].
should the contents of an explanation be, and how frequently
Need for Cognition (NFC) is defined as the "individual’s
should they be given? Further, how can we formalize and
tendency to engage in and enjoy effortful cognitive tasks.”
mechanize the decision process that a system uses in
People with higher NFC levels typically prefer more detail,
determining the answers to these questions?
while those with low levels of NFC want to quickly
B. Research Goals understand the big picture and avoid engaging through more
In this paper we attempt to answer these questions by detail. Based on the NFC 10-item testing instrument [9][11],
defining a formal framework for reasoning about how a self- a score above 80 is generally considered to be High NFC (or
adaptive system should provide explanations based on its high personality trait), and below 50 is Low NFC.
knowledge of a person’s personality traits. This framework As we elaborate later, we adopt these two basic personality
uses probabilistic analysis to decide how explanations should traits (Openness to Experience and Need for Cognition) as
be given, based on a formal human model that includes important elements in a human model that can be used to guide
psychologically relevant aspects of personality. Specifically, a system in deciding the proper amount of explanation that
we focus on answering the following research question: How should be given to the human to improve overall system
to use knowledge about an individual’s personality traits to utility.
improve the overall system utility?
B. OWC (Opportunity-Willingness-Capability) Model
The main contributions of this paper are:
Prior research in adaptive systems has investigated various
• A formal framework that incorporates human models of humans that can be used at run time to effectively
personality traits and guides adaptive human-in-the- characterize humans when deciding how best to incorporate
loop systems to decide how much explanations should them into a co-adaptive system. One of the more prominent
be given in order to improve system utility.
models is the OWC (Opportunity-Willingness-Capability) the user's inputs and commands to take an action. Self-
model [3]. adaptation refers to a process in which an interactive system
co-adapts its behavior to a human based on its internal model
OWC categorizes human attributes into: (1) Opportunity: of the human, dynamic information acquired about the human,
indicates whether a human is available to participate in a the context of use and its surrounding environment [4][5][6].
cooperative task with the system (such as whether the human
is physically present). (2) Willingness: identifies the human’s Several related works have studied explainability focused
inclination to perform the task (affected by cognitive load, on a human-system co-adaptation perspective. In [20] the
human attention, stress level, and motivation). (3) Capability: authors propose a method that generates verbal explanations
defines the human’s abilities and skills that are necessary to of multi-objective probabilistic planning. This method
execute the task successfully (affected by level of experience explains why a particular behavior is chosen on the basis of
or training, knowledge of the task, and cognitive or physical the optimization objectives. Their explainability method relies
skills) [3]. on describing the values of the objective of a generated
behavior and, therefore, explaining tradeoffs that were made
This model has been used effectively in a number of to reconcile competing objectives.
papers to determine, for example, whether to involve the user
in a task or to carry it out automatically [12][5], whether to In [21], the authors define a formal framework to reason
proactively gain the user’s attention [13], and when to provide about explainability of co-adaptive system behaviors and the
an explanation [14]. As we detail later in this paper we use situations under which they are warranted. Specifically, they
OWC to capture the co-adaptation attributes of the human (see characterized explainability in terms of explainability cost,
Section IV. B). effect, and content. They propose a dynamic adaptation
approach that uses a probabilistic reasoning technique, similar
C. Model Checking Stochastic Multiplayer Games (SMG) to ours, in order to determine when the explanations should be
and PRISM used for the purpose of improving system utility.
Probabilistic model checking is used as a technique to
analyze the systems that exhibit stochastic behavior. In another related work [14], the authors use a similar
Stochastic Multi-player Games (SMG) is a form of framework of [21] to reason about explainability of adaptive
probabilistic modelling that allows us to reason quantitatively system behaviors and the conditions under which they are
about reward-based properties and probability such as time, warranted. They characterize explainability in terms of the
usage, and resources in a multi-agent system [15][16][17]. effects on a human operator’s ability to engage in co-adaptive
Our approach is to use SMG models to reason about the actions effectively. They present a decision-making
appropriate amount of explanation that should be given to the mechanism to plan in self-adaptation that provides a
humans based on their personality traits where we model the probabilistic reasoning tool to determine when explanations
system and humans as (cooperating) players in a game. should be used in an adaptation.
PRISM is “a probabilistic model checker, a tool for formal While this prior work shares with our research the goal of
modelling and analysis of systems that exhibit random or reasoning about explanation in the context of human-system
probabilistic behavior” [18]. PRISM-games is an extension of co-adaptation, and also use probabilistic reasoning to account
PRISM that is used to analyze probabilistic systems where for inherent uncertainties in our human models, none of these
players can incorporate competitive or collaborative behavior, studies take into consideration specific personality traits of
modelled as stochastic multiplayer games SMG [19]. humans – the main focus of our work.
Analyzing systems using PRISM has been carried out in
IV. METHODOLOGY
variety of application domains, including: security protocols,
communication and multimedia protocols, randomized In this section, we illustrate how we use explanation as a
distributed algorithms, biological systems and many others. tactic (or action) that systems can use to improve the
PRISM can analyze a wide range of quantitative properties of efficiently and effectiveness of human-system co-adaptation
stochastic models automatically (e.g., "what is the probability based on human personality traits. We describe also how we
of a failure causing the system to shut down within 4 hours?”). utilize a probabilistic planner [19] to determine the optimal
PRISM further supports the specification and analysis of amount of explanation according to those personality traits.
properties based on costs and rewards. These allow it to A. Selection of Personality Traits
reason, not only about the probability that a model behaves in
a certain way, but about a wide range of quantitative measures An important question is which personality traits to
related to the behavior of the model (e.g., "expected number consider with respect to explanation? As noted earlier, the
of lost messages", "expected time", or "expected power psychological literature has classified a variety of important
consumption"). distinguishing characteristics for human personality.
However, not all traits are relevant to explainability. In this
In this paper we use PRISM to dynamically determine work we have adopted two personality traits: Need for
appropriate levels of explanation to maximize expected utility Cognition (NFC) and Openness to experience, since there is a
(expressed as a reward).
direct relationship between NFC and explainability and
D. Human-in-the-Loop Self-Adaptation and Explainability between Openness and capability in OWC [9][10] (see
Human-system integration or human-system co- Section IV. B).
adaptation is advancing the fields of human-system We use the “Openness to experience” trait as one factor
interaction. Integration here means that the relationship that affects the human’s capability to continue and complete a
between system and humans has become a partnership or task, since open people tend to be intellectually curious and
symbiotic relationship in which humans (i.e., users) and have a high level of capability to do creative tasks [10]. We
systems act with autonomy instead of the system waiting for consider the Openness level as an important human factor
since an individual’s Openness level reflects their capability
to engage in cognitive tasks.
In our work we assume that the human’s personality traits
are known (for example, by using the NFC 10-item testing
instrument in [9][11]) and do not change over the time horizon
of a particular set of interactions with the system. While the
traits are assumed to be known, there does, however, remain
some uncertainty about the impact of the amount of
explanation that should be provided to the human, which we
incorporate into our reasoning framework. We will further
assume for concreteness that both selected personality traits
are relevant, and that their weights are equally important
(although the relative importance can be adjusted in the
model).
B. Incorporating the OWC (Opportunity-Willingness-
Capability) Model
We use the OWC model (described in Section III.B) to Fig. 2 The Grid Game we defined that embodies a representative
scenario for human-system ao-adaptation
capture the co-adaptation attributes of the human. In this
paper, the following indicators show the connection between
our model and the OWC model and how the OWC is D. Grid Game
incorporated in the context of the collaborative Grid game: (1) To illustrate our approach, we defined the Grid game– a
time and location represent the set of variables of the virtual game -- as shown in Figure 2, as a game that embodies
Opportunity category. Is the player located at the correct a representative scenario for human-system co-adaptation.
location? Has the timer expired? (2) Human satisfaction In the Grid game the system S instructs a player P verbally
represents the Willingness category. Is the human satisfied to move on a 5×5 grid from the top right corner (start) to the
with the given explanation? That category is applied through bottom left corner (end). The game is designed to rely on
the playerFeedback (pF) tactic. (3) Human performance explanations, at various levels of detail, to instruct the user on
represents the Capability category. The Capability category what tasks to perform and how to perform them.
identifies the ability of the human to complete Grid task.
Giving an explanation increases the capability of the human Game objectives:
to successfully carry out that particular task [12].
• Follow the system instruction through a certain path
C. Utlizing Model Checking Stochastic Multiplayer Games within a certain maximum amount of time (60
(SMG) seconds).
The probabilistic model checker (PRISM-games) is • Minimize the time t to complete the task.
utilized to formally model our approach. PRISM-games is
particularly suitable for our study because it helps us to reason • Traverse an optimal number of blocks to complete the
quantitatively under unpredictability and uncertainty about end-to-end task, avoiding obstacles.
“how much” explanations should be given. The uncertainty
(or stochasticity) that is relevant in this context is about the Game rules:
proper amount of explanations and the impact of different • The player can move either horizontally or vertically.
amounts of explanations that should be given to the human.
• Game score (100 points): points are deducted for
We model the system (the Grid game described in Section traversing extra blocks or moving into or through
IV.D below) as a turn-based SMG, which means exactly one obstacle squares (e.g., in Figure 2 there are four
player in each state of the modeled system can choose an obstacles: the house, a traffic light, a mountain, and a
action, where the outcome of that state will be probabilistic. tree).
Players in a SMG may cooperate to achieve a common goal,
or compete to accomplish different goals. In our examples, we The Grid game can involve the use of five tactics for
model two players1, the human and the system, and we assume interacting with the player, as shown in Table 1. The system
that they share a common goal. provides two levels of explanation to command the human to
move from one point to another. The choice of level of
We use rPATL, a probabilistic temporal logic, to express explanation is based on the run-time calculation and
properties of stochastic multi-player games quantitatively. explanation generation based on the probabilistic model.
rPATL helps us to reason about the collective ability of a
group of players to achieve a goal relating to the probability In this case “less explanation (LExp)” provides an
of an occurring event [22]. abbreviated command (e.g., “Go 2 blocks left”), while “more
1
Note that a multiplayer here (i.e., two players) does not mean that the Grid game is a multiuser game. The concept
“multiplayers” in PRISM refers to multiple agents, such as system, human, or environment. In our model, the system and the
human are the only two players and they are working cooperatively (taking turns) to achieve the best possible outcome.
TABLE I: GRID GAME TACTICS FOR INTERACTING WITH THE PLAYER
Model Categories Tactics Role Example
Less Commands the human to carry out an action. “Go 2 blocks left”
lessExplain (lExp)
Explanation “Move south 4 blocks”
The system further explains information when the “You will go between a
System human is confused and loses track.
More moreExplain house and traffic light”
Explanation (mExp) “You go straight, and you
see a car on your left side”
The human requests the system to confirm information “North?”
Clarification Check (Chk) that they not entirely sure about. “Should I continue above
Request the tree?”
Human playerFeedback Human feedback is collected about his satisfaction for Helpful, Not helpful,
Feedback (pF) each given explanation Neutral
The human confirms information and follows the “Yeah”, “Thanks”
Acknowledgement confirm (conf)
instructions. “Okay”
explanation (mExp)” provides an abbreviated command (e.g., b) Not helpful (NH) reflects utility decrements (↓),
“You will go between a house and traffic light”) contains
additional details. The human may request clarification about c) Neutral (N) reflects neither utility increments nor
a given explanation if they are not entirely sure about it (Chk) decrements (-).
(e.g., “Should I continue above the tree?”). Or the user can 3) Utility Functions
confirm the information and follow the instructions (conf).
The human also gives feedback (pF) about the given To compare different explainability tactics (i.e., lengths of
explanation as to whether it was (a) helpful, (b) not helpful, or explanation), we use probabilistic temporal logic with
(c) neutral. (This supports explanation assessment in the rewards, rPATL, which enables us to analyze the utilities of
framework - Figure 1). the system that explainability can influence. rPATL
(described in Section IV.C) is used to reason about the ability
1) Utility Attributes of a group of players (system and human) to collectively
The four utility attributes of the game are: RequiredTime achieve a specific goal [18].
(t), Blocks (B), LengthOfExplanations (xL), and In the formal model we define formulas that represent the
ExplainEfficiency (xE). B and t are used for calculating the accrued utility (The Scores function ∪ 𝑠 and the
game score, and xL and xE are used as explainability ExplainEfficiency Function ∪ 𝑥𝐸) as the maximum real
attributes. Game score(s) depend on the time elapsed for
immediate utility that the human can achieve along the whole
completing the game (t), associated with the optimal number
task.
of the blocks (B) that the player is supposed to end the task
with: TABLE II: COST/BENEFIT ’ IMPACTS ON UTILITY DIMENSIONS
• RequiredTime (t): the total elapsed time for Time
completing the game. ∆ ExplainEfficiency
∆ RequiredTime (t) (xE) a
Tactics ∆ LengthOfExplanations
• Blocks (B): the number of the blocks traversed to (xL)
complete the task. ↑ ↓ -
↑
• LengthOfExplanations (xL): the amount of delay (or lessExplain (lExp) H NH N
+3
time) required to explain.
moreExplain (mExp) +6 +1 -1 0
• ExplainEfficiency (xE): a measurement that
determines how happy the player is with the given Check (Chk)
explanations. xE is associated with the playerFeedback +3
confirm (conf)
(pF) tactic which can be one of the following values:
Helpful, Not helpful, or Neutral. playerFeedback (pF)
a.
2) Tactics Cost/Benefit and Utility Dimensions H: Helpful, NH: Not helpful, N: Neutral
The Scores function ∪𝑠, as shown in function (1), maps
Table 2 lists the tactics in the Grid game, and their impacts
high scores to high utility derived by dividing the number of
on utility dimensions. Different tactics cause an increase in
blocks 𝐵 by the maximum level of RequiredTime 𝑡 𝑚𝑎𝑥 (tmax=
Time (three seconds for lExp, Chk, and conf; six seconds, for
60), where B must be greater than or equal to 𝑜𝑝𝑡𝑖𝑚𝑎𝑙𝐵 (the
mExp). The upward ↑ or downward arrow ↓ reflects utility
optimal number of blocks that the player is supposed to
increments and decrements, respectively. For example, the complete the task with):
lExp tactic increases both t and xL by three seconds, which is
associated with a smaller amount of costs. Human feedback is 𝐵
∪ 𝑠(𝐵) = (1 − 𝑚𝑎𝑥) 𝑥100 where 𝐵≥𝑜𝑝𝑡𝑖𝑚𝑎𝑙𝐵 ()
collected about the user’s satisfaction for the given 𝑡
explanation (lExp) which can be: The ExplainEfficiency Function ∪ 𝑥𝐸, as shown in
a) Helpful (H) reflects utility increments (↑), function (2), maps higher levels of ExplainEfficiency (xE)
derived by dividing the accumulated player Feedback (∑ 𝑝𝐹)
by the total number of feedbacks (𝑝𝐹 𝑚𝑎𝑥 ), where
𝑝𝐹𝜖[1,0, −1] represent Helpful, Neutral, Not helpful,
respectively:
∑ 𝒑𝑭
∪ 𝑥𝐸(𝑝𝐹) ≈ ( ) 𝑥100 ()
𝑝𝐹𝒎𝒂𝒙
Both personality trait variables (Openness and
NFC) are initialized with some constants (as inputs)
that represent the human traits. Personality traits are
directly mapped to the level of explanation (the
amount), and are used to calculate the probability of
getting explanations in that amount. Function (3)
shows combined personality traits, which will be 0 in
case both traits are 0, or 1 in case the human has the
highest personality traits levels (i.e., 0 → 1).
Opennessmax and NFCmax are 100, which represent the
highest personality trait levels. The values of
personality traits are determined based on the NFC 10-
item testing instrument in [9][11] that produces scores
between 0-100. Fig. 3 The strategy we use to model the SMG: the proper amount of explanation
For example, if a human has 75 Openness and 90 is determined based on the three personality levels of the human (represented in light
blue). The two explanation amounts (less or more) are determined by using function
NFC. The combined human traits are 0.82 which 3. Human feedback is collected that will can be helpful, neutral, or not helpful
means he has high personality traits (by using function (represented in yellow). The human confirms information that means he moved
(3)). That means the system will explain less 18% of successfully to the next point (conf), or checks/requests the system to clarify
the time (i.e., lExp) and explain more 82% of the time information that they not entirely sure about (chk).
(i.e., mExp) during the task. As another example,
suppose a human with low personality traits has 43
system took 27 seconds for explanations (xL). At the end of
openness and 49 NFC. The combined human traits are 0.46
the task, the score of the player is 75 (by using the function
(using function (3)). That means the system will explain less
(1), where B=15 and tmax= 60), and the ExplainEfficiency
54% of the time (i.e., lExp) and explain more 46% of the
(xE) is 43 (by using the function (2), where ∑ 𝑝𝐹 is three and
time (i.e., mExp) while playing the Grid game.
𝑝𝐹 𝑚𝑎𝑥 is seven (which means seven feedbacks are
𝐻𝑢𝑚𝑎𝑛 𝑇𝑟𝑎𝑖𝑡𝑠 =
𝑂𝑝𝑒𝑛𝑛𝑒𝑠𝑠+𝑁𝐹𝐶
() collected)).
𝑂𝑝𝑒𝑛𝑛𝑒𝑠𝑠 𝑚𝑎𝑥 +𝑁𝐹𝐶 𝑚𝑎𝑥
TABLE III: AN EXAMPLE DIALOGUE OF A SCENARIO V. THE STOCHASTIC MULTI-PLAYER GAMES
BETWEEN THE SYSTEM (S) AND A HUMAN (H) (SMG) MODEL
Scenario Tactics Time pF We model the Stochastic Multi-player Games (SMG)
model as two players, where the players try to collaboratively
S: Can you go 2 blocks down? (lExp) 3s H
maximize accumulated reward(s): (1) Player SYS specifies
H: Yeah (conf) 3s - the actions that are controlled by the system (i.e., it represents
S: Then go 2 blocks left. (lExp) 3s NH the Grid game). (2) Player HUMAN specifies the actions
belonging to the human (i.e., it represents the game player).
H: Could you repeat that? (Chk) 3s - The models represent the behavior of a set of agents (or
S: Go west. You will go between a house (mExp) 6s H “players”) that take turns making moves, where the choice of
and traffic light. move is specified probabilistically or non-deterministically. A
H: Okay (conf) 3s - game solver for such a system (such as PRISM-games [19])
determines an optimal strategy for the players by resolving the
S: Go after that 2 blocks up. (lExp) 3s N non-deterministic transitions in such a way that the expected
H: The human is on the wrong track - reward for each player is maximized (assuming rational play
S: No, not south. You go north (mExp) 6s H
by each). Figure 3 shows the strategy we use to model the
SMG. The proper amount of explanation is determined based
H: Okay (conf) 3s - on the three personality levels of the human (i.e., Low,
S: Go 2 blocks left (lExp) 3s N Average, or High personality levels (represented in light
blue)). The two explanation amounts (less or more) are
……… determined by using Function 3 (describes in the previous
S: Go south 4 blocks. (lExp) 3s H section). Human feedback is collected that is of the form
H: Okay, thanks a lot. (conf) 3s -
Helpful, Neutral, or Not Helpful (represented in yellow). The
human confirms information that means he moved
4) Example Scenario successfully to the next point (conf), or checks/requests the
system to clarify information that they not entirely sure about
Figure 2 and Table III show an example dialogue of a (Chk).
scenario between the system (S) and a human (H).
The Stochastic Multi-player Games (SMG) model consists
The human spent 42 seconds (t) and used 15 blocks (B) to of the following four parts:
finish the task. However, the number of blocks B that the
player is supposed to end the task with is 12 (𝑜𝑝𝑡𝑖𝑚𝑎𝑙𝐵). The
A. Player Definition Similarly, the system instructs the human with average
Player definition includes the declaration of the two personality traits through executing the command labeled as
players in the SMG and different modules that each player has lExpAvg (line 8-10), or the system instructs the human with
control of. The two players in our game are shown in Listing high personality traits through executing the command labeled
1. Player SYS (lines 1-2) specifies the actions that are as lExpHigh (line 11-13).
controlled by the system (i.e., it represents the Grid game). The human wins by executing the command labeled as win
Player HUMAN (lines 3-4) specifies the actions belonging to (line 25). That means the human (turn=SYS) has arrived at the
the human (i.e., it represents the game player). Our Grid game bottom left corner ((x1= 1)&(y1=1)) within the time limit
is played in turns by the two players SYS and HUMAN. Turn (t<60). However, the human loses the game by executing the
(line 5) is a global variable used as a controller to take turns command labeled as lose (line 26) when the end time of the
between different players, ensuring that only one player can task has been reached (t=60).
take an action at each state of the model execution. Tactics are
executed sequentially in our model.
1. global lExp_state: bool init false;
2. global mExp_state: bool init false;
1. player SYS 3. …
Game, [lExpLow],[ lExpAvg],[ lExpHigh], 4. module Game
[mExpLow],[mExpAvg],[mExpHigh] 5. [lExpLow] (turn=SYS)&(human_traits<.5)&(x1= 5)
2. endplayer &(y1=5)&(t<60)
3. player HUMAN 6. ->human_traits:(mExp_state'= true)
Play, [conf], [Chk] 7. + 1-human_traits:(x2'=5) & (y2'=3)&(xL'=xL+3)
4. endplayer & (t'=t+3)&( lExp_state'= true)&(turn'=HUMAN);
5. global turn:[SYS..HUMAN] init SYS; 8. [lExpAvg] (turn=SYS)&(x1= 5)&(y1=5)&(t<60)
6. const SYS=1; const HUMAN=2; 9. ->0.5:(x2'=5)&(y2'=3)&(xL'=xL+3)&(t'=t+3)
&(lExp_state'= true)&(turn'=HUMAN)
Listing. 1 Player definition includes the declaration of the two players in 10. +0.5:(mExp_state'= true);
the SMG and different modules that each player has control of 11. [lExpHigh] (turn=SYS)&(human_traits>.8)
&(x1= 5)&(y1=5)&(t<60)
B. Game Model 12. ->human_traits:(mExp_state'= true)
13. + 1-human_traits:(x2'=5) & (y2'=3)&(xL'=xL+3)
Player SYS has control of the Game model, illustrated in & (t'=t+3)&( lExp_state'= true)&(turn'=HUMAN);
Listing 2. Opportunity elements are used as execution 14. …
conditions of different tactics such as: the human is at the 15. [mExpHigh] (turn=SYS)&(human_traits>.8)&
correct location ((x= 1)&(y=1)) and is not involved in a crash, (conf_state= false)&(Chk_state= true)&(t<60)
and the time has not expired (t<60). The Game module is 16. ->human_traits:(mExp_state'=true)&(xL'=xL+6)
parameterized by the variables (lines 1-2), which indicate the &(t'=t+6)&(Chk_state'= false)&(turn'=HUMAN)
17. + 1-human_traits: (lExp_state'= true);
state of tactic execution, where false means this tactic is not 18. [mExpAvg] (turn=SYS)&(conf_state= false)
in use (i.e., lExp_state, and mExp_state). &(Chk_state= true)&(t<60)
19. ->0.5:(mExp_state'= true)&(xL'=xL+6)&(t'=t+6)
During the system’s turn, the system executes these &(Chk_state'= false)&(turn'=HUMAN)
tactics sequentially: lExp (lines 5-13), and mExp (lines 15- 20. +0.5:( lExp_state'= true);
23). For the sake of clarity, we will describe only the 21. [mExpLow] (turn=SYS)&(human_traits<.5)
lExpLow tactic to illustrate how tactic execution is modeled. &(conf_state= false)&(Chk_state= true)&(t<60)
The other explainability tactics follow the same structure. The 22. ->human_traits:( lExp_state'= true)
system instructs the human with low personality traits 23. + 1-human_traits:(mExp_state'=true)&(xL'=xL+6)
&(t'=t+6)&(Chk_state'= false)&(turn'=HUMAN);
through executing the command labeled as lExpLow (line 5). 24. …
This tactic executes only if: 25. [win] (turn=SYS)&(x1= 1)&(y1=1)&(t<60)
-> (win'=true)&(turn'=0);
• It is the turn of the SYS. 26. [lose] ((turn=SYS)|(turn=HUMAN))&(t=60)
-> (win'=false)&(loser'= true)&(turn'=0);
• The human traits are low (<0.50).
27. endmodule
• The player position is on a certain block (x1,y1). 28. …
• The end time of the task has not been reached yet
(t<60). Listing. 2 Game Model
If the guard is satisfied, the system will explain more by C. Play Model
flagging mExp_state tactic true with probability
human_traits (line 6). Otherwise, the system will explain less Player HUMAN has control of the Play model, illustrated
by flagging lExp_state tactic true with probability 1- in Listing 3. The encodings of the HUMAN module are similar
human_traits (line 7) and the system will: to those of the SYS module. The Play module is
parameterized by variables (lines 1-2), which indicate the state
• Commands the player to move to the position (x2,y2). of tactic execution, where false means this tactic is not in use
(e.g., Chk_state, and conf_state). Personality Traits are
• Increases the time 3 seconds (xL'=xL+3)&(t'=t+3).
initialized with values that represent the human’s personality
• Flags the lExp tactic as true (lExp_state'= true). (lines 3-5).
• Updates the value of the variable turn, changing During the human’s turn, the human can execute one of
control to the human player (turn’=HUMAN). these tactics: conf (line 8), and Chk (line 10). We explain only
the conf tactic to illustrate how tactic execution is modeled.
The human confirms (conf) and follows the system the encoded Function (2) described in Section IV.D. Line 5
instructions (i.e., the human moves successfully from the 1st shows the encoded combined traits function (3).
point to the second) by executing the command labeled as
conf. This tactic executes only if: 1. rewards "Scores"
[win] true:(1-(B/tMax))*100;
• It is the turn of the HUMAN. [lose] true:0;
• The system instructs the player to move to the position [crash] true: -5;
2. endrewards
(x2,y2).
3. rewards "ExplainEfficiency"
• The end time of the task has not been reached yet [win] true:(pF/pfMAX)*100;
(t<60). [lose] true:(pF/pfMAX)*100;
4. endrewards
1. global Chk_state: bool init false; 5. formula human_traits =
2. global conf_state: bool init false; (human_Open+ human_NFC)/(Max_Open+Max_NFC);
3. const int INIT_OPN; const int INIT_NFC;
4. global human_Open: [1..100] init INIT_OPN; Listing 4. Utility profile and reward structure: formulas and reward
5. global human_NFC:[1..100] init INIT_NFC; structures are used to encode the utility functions that allow us to quantify the
6. … utilities of different task states. Formulas calculate system utility of the
7. module Play different states.
8. [conf] (turn=HUMAN)&(x2= 5)&(y2=3)&(t<60)
->(x1'=5) & (y1'=3)& (t'=t+3)&(B'=B+2)
&(pF'=pF+1)&(pfMAX'=pfMAX+1)&(conf_state'= true) VI. RESULTS AND ANALYSIS
&(lExp_state'= false)&(turn'=SYS); In this section, we illustrate how our modeling framework
9. …
can produce optimal decisions with respect to how adaptive
10. [Chk] (turn=HUMAN)&(conf_state= false)&(x1= 5)
& (y1=3)&(t<60) systems should explain to the human based on their
->(Chk_state'= true)&(t'=t+3)&(pF'=pF1) personality traits. Specifically, we use SMG models of
&(pfMAX'=pfMAX+1)&(turn'=SYS); explainability to determine the expected outcome utilities of
11. [wrong] (turn=HUMAN)&(x2= 3)&(y2=5)&(loser=false) using different explainability tactics (i.e., explanation
&(t<60)-> (x1'=3) & (y1'=1) &(t'=t+3) amounts) based on the personality traits of the human. Our
& (B'=B+2)&(pF'=pF-1)&(pfMAX'=pfMAX+1)
modeling is done as a simulation (or set of “experiments” in
&(conf_state'= false) &(turn'=SYS);
12. [crash] (turn=HUMAN)& ((x1=obj1x & y1=obj1y) PRISM terms). We use rPATL to ask PRISM a variety of
| (x1=obj2x & y1=obj2y)| (x1=obj3x & y1=obj3y) questions such as “what is the maximum/minimum
| (x1=obj4x & y1=obj4y))->(turn'=SYS); probability a human with high/low personality traits can
13. endmodule guarantee to win with high/low utilities?” [22].
14. …
Table IV and Figure 4 show the analysis results of 44
Listing. 3 Play Model rounds run on PRISM. All possible combinations of
personality traits are taken into consideration, where high
If the guard is satisfied, the player: traits are (>80) (represented by orange color), average traits
are (≥50 and ≤80) (represented by blue-gray color), and low
• Moves to the position (x1,y1). traits are (<50) (represented by gray color). Plot (a) shows the
• Increases the time three seconds (t'=t+3). 44 simulations of different personality traits and the given
amounts of explanations (LengthOfExplanations (xL)) to
• Increases the number of Blocks by two (B'=B+2). complete the task. The average of different personality traits
and the amounts of explanations (xL) is shown in Plot (b).
• Gets the player feedback (pF= 1 means the explanation
39% of the iterations (17 rounds) of humans with high
was helpful).
personality traits (>80) needed more explanations to finish the
• Increases the player feedback counter pfMAX by 1. task with an average of 21 seconds. 32% of the iterations (14
rounds) of humans with low personality traits (<50) needed
• Flagging the conf tactic true (conf_state'= true). less amount of explanations with an average of 20 seconds.
Moreover, the tactic wrong (line 11) will be executed The remaining 30% of the iterations (13 rounds) belongs to a
when the human moves in the wrong direction, and the tactic human with average personality traits (≥50 and ≤80), where
crash (line 12) will be executed when the human moves to they use average amounts of explanation with an average of
one of the obstacle squares (the house, a traffic light, a 19 seconds to complete the task. Table 5 shows the average
mountain, or a tree in Figure 2). of different utilities based on the three personality trait levels.
D. Utility Profile and Reward Structure We can conclude from the results that a human with high
personality traits needs more detailed information (i.e.,
Utility functions are described in Section IV.D and
explanations), while a human with low personality traits needs
illustrated in Listing 4. Formulas and reward structures are
less detailed explanation. These conclusions are all consistent
used to encode the utility functions that allow us to quantify
with psychology studies (discussed in Section III. A) that
the utilities of different task states.
human with higher personality trait levels typically prefer
The Scores function, ∪𝑠, as in lines (1-2), represents the more explanations, while those with low levels of personality
encoded Function (1) as described in Section IV.D. trait want to quickly understand the big picture and avoid
ExplainEfficiency function ∪𝑥𝐸, as in lines (3-4) , represents engaging through more explanations [9][11].
TABLE IV: RESULTS OF 44 ROUNDS RUN ON PRISM
Human Traits Utilities
# Combined Traits LengthOfExplanations ExplainEfficiency
Openness NFC Scores
(xL) (xE)
1 75 90 82.5 27 28.5 93.4
2 100 100 100 21 50 96.7
3 50 90 70 15 80 100
4 95 30 62.5 21 50 96.7
5 95 85 90 15 80 100
6 45 88 66.5 15 80 100
31 47 47 47 33 12.5 90
32 83 83 83 21 50 96.7
33 22 19 20.5 15 80 100
34 96 77 86.5 15 80 100
35 69 55 62 27 28.5 93.4
36 39 11 25 21 50 96.7
37 33 19 26 15 80 100
38 17 15 16 27 28.5 93.4
39 9 30 19.5 21 50 96.7
40 49 29 39 15 80 100
41 51 71 61 15 80 100
42 93 100 96.5 21 50 96.7
43 100 90 95 15 80 100
44 81 80 80.5 15 80 100
Minimum 9 11 16 15 -12.5 90
Maximum 100 100 100 36 80 100
Average 66.02 61.98 64 20.39 57.07 97.23
TABLE V: AVERAGE UTILITIES OF THE EXPERIMENTS BASED
ON THE THREE PERSONALITY TRAITS LEVEL
Personality Traits Average Utilities
Level
xL xE Scores
High Traits 21.18 53.50 91.388
Average Traits 19.15 62.27 97.708
Low Traits 20.57 56.57 96.921
Plot (a): The 44 simulations of different personality traits and
the given amounts of explanations (LengthOfExplanations (xL))
VII. DISCUSSION AND FUTURE WORK to complete the task.
In this research we presented an approach based on
probabilistic model checking of SMGs to determine how
much explanation should be given to the human based on their
personality traits. Providing the right amount of explanation
to the right human is an important factor to maximize co-
adaptation between the system and the human during their
interaction.
There is a number of limitations of this research that future
research can address based on the foundations that we have
described in this paper. These explanation decisions are ideal Plot (b): The average of different personality traits and the
scenarios without having actual proof of that in reality. To amounts of explanations (xL)
address this, the most important next step is to conduct an
empirical study to validate these models on actual real-world Fig. 4 Results of 44 rounds run on PRISM show that human with
higher personality trait levels typically prefer more explanations, while
systems with humans in the loop. those with low levels of personality trait prefer less explanations
As we explained earlier, there are many reasons to use
explanations, and examine in more detail questions such as
explainability and improving a system’s overall utility is one
how explanations should be presented: graphically, textually,
of the main reasons (see Section I). Explainability can help not
verbally? A further extension of this research is to have more
only to improve the systems continuously through human
detailed models that allow the system to determine in a more
involvement, but also to justify some information given to the
nuanced way the ideal contents of the explanations that should
human, particularly when decisions are made suddenly.
be considered.
Gaining more information improves the capability of the
human to perform a task. Our results in this paper suggest one
of the next steps of research is to go beyond the length of
VIII. CONCLUSION Systems,” In Proceedings of the 2020 IEEE Conference on Autonomic
Computing and Self-organizing Systems (ACSOS), Washington, D.C.,
In this research we presented a formal framework that 19-23 August 2020.
incorporates human personality traits as one of the important [15] P. D. Kwiatkowska M., Norman G., “Probabilistic Model Checking:
elements in guiding automated decision-making about the Advances and Applications,” Form. Syst. Verif. Springer, Cham., pp.
proper amount of explanation that should be given to the 73–121, 2018.
human to improve overall system utility. To accomplish our [16] C. Baier, “Probabilistic model checking,” Dependable Softw. Syst.
goal of this paper, we use probabilistic model analysis to Eng., vol. 45, no. August, pp. 1–23, 2016.
determine how to utilize explanations in an effective way [17] S. W. Cheng and D. Garlan, “Stitch: A language for architecture-based
self-adaptation,” J. Syst. Softw., 2012.
based on the difference of human’s personality traits. Grid – a
[18] M. Kwiatkowska, G. Norman, and D. Parker, “PRISM 4.0: Verification
virtual human and system interaction game – was developed of probabilistic real-time systems,” in Lecture Notes in Computer
to illustrate our approach, to represent scenarios for human- Science (including subseries Lecture Notes in Artificial Intelligence
system co-adaptation, and to demonstrate through simulation and Lecture Notes in Bioinformatics), 2011.
how a human’s personality traits can be used as a factor to [19] M. Kwiatkowska, G. Norman, D. Parker, and G. Santos, “PRISM-
consider for systems in providing appropriate explanations. games 3.0: Stochastic Game Verification with Concurrency, Equilibria
and Time,” in Lecture Notes in Computer Science (including subseries
ACKNOWLEDGMENT Lecture Notes in Artificial Intelligence and Lecture Notes in
Bioinformatics), 2020.
This research was supported in part by the NSA under [20] R. Sukkerd, R. Simmons, and D. Garlan, “Towards explainable multi-
Award No. H9823018D0008 and Award No. N00014172899 objective probabilistic planning,” Proc. - Int. Conf. Softw. Eng., pp.
from the Office of Naval Research. Any views, opinions, 19–25, 2018.
findings and conclusions or recommendations expressed in [21] N. Li, S. Adepu, E. Kang, and D. Garlan, “Explanations for human-on-
this material are those of the author(s) and do not necessarily the-loop: A probabilistic model checking approach,” Proc. - 2020
reflect the views of the NSA or the Office of Naval Research. IEEE/ACM 15th Int. Symp. Softw. Eng. Adapt. Self-Managing Syst.
SEAMS 2020, pp. 181–187, 2020.
REFERENCES [22] T. Chen, V. Forejt, M. Kwiatkowska, D. Parker, and A. Simaitis,
“Automatic verification of competitive stochastic systems,” Form.
[1] G. Vilone and L. Longo, “Explainable Artificial Intelligence: a Methods Syst. Des., vol. 43, no. 1, pp. 61–92, 2013.
Systematic Review.” arXiv preprint arXiv:2006.00093, 2020.
[2] A. Adadi and M. Berrada, “Peeking Inside the Black-Box: A Survey
on Explainable Artificial Intelligence (XAI),” IEEE Access, vol. 6. pp.
52138–52160, 2018.
[3] D. Eskins and W. H. Sanders, “The multiple-asymmetric-utility system
model: A framework for modeling cyber-human systems,” Proc. 2011
8th Int. Conf. Quant. Eval. Syst. QEST 2011, pp. 233–242, 2011.
[4] J. O. Kephart and D. M. Chess, “The vision of autonomic computing,”
Computer (Long. Beach. Calif)., 2003.
[5] E. Lloyd, S. Huang, and E. Tognoli, “Improving Human-in-the-Loop
Adaptive Systems Using Brain-Computer Interaction,” Proceedings -
2017 IEEE/ACM 12th International Symposium on Software
Engineering for Adaptive and Self-Managing Systems, SEAMS 2017.
pp. 163–174, 2017.
[6] M. Alharbi and S. Huang, “A Survey of Incorporating Affective
Computing for Human-System Co-adaptation,” in Proceedings of the
2020 The 2nd World Symposium on Software Engineering, 2020, pp.
72–79.
[7] B. Mittelstadt, C. Russell, and S. Wachter, “Explaining explanations in
AI,” FAT* 2019 - Proc. 2019 Conf. Fairness, Accountability,
Transpar., pp. 279–288, 2019.
[8] C. G. H. Jung, “Psychological Factors Determining Human
Behaviour,” Collect. Work. C.G. Jung, Vol. 8 Struct. Dyn. Psyche, pp.
114–126, 2015.
[9] C. J. Sadowski and H. E. Cogburn, “Need for cognition in the big-five
factor structure,” Journal of Psychology: Interdisciplinary and Applied,
vol. 131, no. 3. pp. 307–312, 1997.
[10] R. R. McCrae, “Openness to Experience as a Basic Dimension of
Personality,” Imagin. Cogn. Pers., 1993.
[11] R. E. Petty, J. T. Cacioppo, R. E. Petty, J. A. Feinstein, and W. B. G.
Jarvis, “Dispositional Differences in Cognitive Motivation : The Life
and Times of Individuals Varying in Need for Cognition Dispositional
Differences in Cognitive Motivation : The Life and Times of
Individuals Varying in Need for Cognition,” Psychol. Bull., vol. 119,
no. August, pp. 197–253, 2015.
[12] J. Cámara, G. Moreno, and D. Garlan, “Reasoning about Human
Participation in Self-Adaptive Systems,” Proc. - 10th Int. Symp. Softw.
Eng. Adapt. Self-Managing Syst. SEAMS 2015, no. i, pp. 146–156,
2015.
[13] N. Li, C. Javier, D. Garlan, and B. Schmerl, “Hey ! Preparing Humans
to do Tasks in Self-adaptive Systems .” In Proceedings of the 16th
Symposium on Software Engineering for Adaptive and Self-Managing
Systems, Virtual, 18-21 May 2021.
[14] N. Li, J. Cámara, D. Garlan, and B. Schmerl, “Reasoning about When
to Provide Explanation for Human-in-the-loop Self-Adaptive