A Probabilistic Model for Personality Trait Focused Explainability Mohammed N. Alharbi Shihong Huang David Garlan Department of Computer & Electrical Department of Computer & Electrical Institute for Software Research Engineering and Engineering and School of Computer Science Computer Science Computer Science Carnegie Mellon University Florida Atlantic University Florida Atlantic University Pittsburgh PA USA Boca Raton FL USA Boca Raton FL USA garlan@cs.cmu.edu malharbi2016@fau.edu shihong@fau.edu Abstract— Explainability refers to the degree to which a software system’s actions or solutions can be understood by humans. Giving humans the right amount of explanation at the right time is an important factor in maximizing the effective collaboration between an adaptive system and humans during interaction. However, explanations come with costs, such as the required time of explanation and humans’ response time. Hence it is not always clear whether explanations will improve overall system utility and, if so, how the system should effectively provide explanation to humans, particularly given that different humans may benefit from different amounts and frequency of explanation. To provide a partial basis for making such decisions, this paper defines a formal framework that incorporates human personality traits as one of the important elements in guiding automated decision- making about the proper amount of explanation that should be given to the human to improve the overall system utility. Specifically, we use probabilistic model analysis to determine how to utilize explanations in an Fig. 1 A probabilistic model for personality trait focused explainability effective way. To illustrate our approach, Grid – a framework: this framework incorporates two basic personality traits (Openness and virtual human and system interaction game -- is Need for Cognition) as important elements in a human model that can be used to guide developed to represent scenarios for human-systems a system in deciding the appropriate amount of explanation that should be given to the collaboration and to demonstrate how a human’s human personality traits can be used as a factor to consider for systems While explanation is an increasingly desirable – even, in providing appropriate explanations. essential – capability of a system, it is not at all obvious when Keywords— explainability, human system co-adaptation, and how explanation should be given, particularly since human computer interaction (HCI), personality traits, self- explanation comes with a cost on human attention and delays adaptive systems, human-in-the-loop, model checking, in system-human interaction and the fact that different humans probabilistic model. may need different kinds of explanation. To partially address this problem this paper defines a formal framework, as I. INTRODUCTION illustrated in Figure 1, for reasoning about the proper amount As systems become more autonomous and intelligent of explanation that a system should provide to the human through the incorporation of AI techniques and self-adaptive based on their personality traits. Specifically, leveraging research in the psychology of human personality, this approaches, it becomes increasingly important for those framework incorporates two basic personality traits systems to be able to “explain” themselves to their human (Openness and Need for Cognition) as important elements in users and collaborators [1][2]. In particular, there are four a human model that can be used to guide a system in deciding main purposes of explainability: (1) explain to justify: use the appropriate amount of explanation that should be given to explanations to justify some results to the human, particularly the human in order to improve overall system utility. The when decisions are made suddenly; (2) explain to control: effects of given explanations (which are determined based on explanations can help not only to justify, but also to prevent personality traits of the human) affect human-system co- systems from going wrong; (3) explain to improve: improving adaptation, represented through the Opportunity-Willingness- the systems continuously through human involvement; (4) Capability (OWC) model, a commonly used model for explain to discover: discovering and gathering new facts that adaptive systems’ reasoning about human-in-the-loop help us to learn and to gain knowledge. In the context of this behavior [3]. We incorporate our approach into the MAPE-K paper, explainability refers to the degree to which a software architecture [4] to formally model and analyze human system’s actions or solutions can be understood by humans, involvement at different stages of system management and and explainability is used to improve a system’s overall utility. adaptation. To illustrate our approach, Grid – a virtual human and system interaction game – is developed to represent Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). scenarios for human-systems collaboration and to demonstrate • An evaluation system based on a collaborative game, how a human’s personality traits can be used as a factor to to simulate the effects of decision making under consider for systems in providing appropriate explanations. various scenarios. The organization of the paper is as follows: Section II III. BACKGROUND AND RELATED WORK describes the research problem and goals, Section III represents background information and related work, Section This section introduces some background on personality IV shows methodology, Section V shows the Stochastic traits, the OWC (Opportunity-Willingness-Capability) model, Multi-player Games (SMG) model while Section VI shows model checking of stochastic multi-player games (SMG), and results and analysis, Section VII represents discussion and some of state-of-the-art studies that focus on explainability future work and the last section focuses on the conclusion. and human-system co-adaptation. Section IV will then illustrate how this background and related work are related to II. PROBLEM STATEMENT AND what we do in this research. RESEARCH GOALS A. Personality Traits A. Problem Statement Psychological studies have demonstrated that human A co-adaptation system is symbiotic human-in-the-loop personality traits play a strong role in determining human system where human-system cooperation is required in behavior [8]. Personalities can be characterized in terms of achieving shared goals, and system and human actions traits that are relatively stable characteristics of a human that mutually impact each other’s behavior in accomplishing influence our behavior across many situations. An individual's coordinated tasks [5]. In this context, providing effective personality is the combination of traits and patterns that explanations to humans is an important factor in maximizing influence his/her behavior, thought, motivation, and emotion. the co-adaptation outcomes between the system and the It drives individuals to consistently think, feel, and behave in human [6]. Maximizing co-adaptation outcomes implies that specific ways. the relationship between system and humans has become a There are, of course, many differences between partnership, or collaborative relationship, in which humans individuals; however, personality traits are one of the more and systems act semi-autonomously – in contrast to traditional important measurable characteristics that can be used to systems that wait for the human's inputs and commands to take distinguish one person from another. In the psychological action [6]. literature the Big Five (also called the Five Factor) model of Given that different humans may benefit from different personality is one of the most widely accepted personality amounts and frequency of explanation, in this paper we argue taxonomies. In the Big Five model, the five dimensions of that adapting the explanation to the particular human through personality include extraversion, neuroticism, openness to knowledge of their personality traits can help the system in experience, agreeableness, and conscientiousness [9]. determining what are appropriate explanations and, therefore, Openness to experience is one of the personality traits that maximize the benefits of co-adaptation. In particular, given is used to describe individual personality in the Five Factor that there are tradeoffs in determining what kind of Model. Open people tend to be intellectually curious, creative explanations to give, it is important to be able to tailor the and imaginative. Open people have a high openness to explanations to the user [7]. Providing longer and more- embrace new things, fresh ideas, and novel experiences [10]. frequent explanations may increase the effectiveness of collaboration between the system and the human; however, In addition to the Five Factor Model, the psychological this comes at the cost of taking more time for humans to literature also identifies Need for Cognition is an important understand the explanations and respond accordingly. Thus, distinguishing characteristics of human personality trait key questions that must be answered by a system are: What [9][11]. should the contents of an explanation be, and how frequently Need for Cognition (NFC) is defined as the "individual’s should they be given? Further, how can we formalize and tendency to engage in and enjoy effortful cognitive tasks.” mechanize the decision process that a system uses in People with higher NFC levels typically prefer more detail, determining the answers to these questions? while those with low levels of NFC want to quickly B. Research Goals understand the big picture and avoid engaging through more In this paper we attempt to answer these questions by detail. Based on the NFC 10-item testing instrument [9][11], defining a formal framework for reasoning about how a self- a score above 80 is generally considered to be High NFC (or adaptive system should provide explanations based on its high personality trait), and below 50 is Low NFC. knowledge of a person’s personality traits. This framework As we elaborate later, we adopt these two basic personality uses probabilistic analysis to decide how explanations should traits (Openness to Experience and Need for Cognition) as be given, based on a formal human model that includes important elements in a human model that can be used to guide psychologically relevant aspects of personality. Specifically, a system in deciding the proper amount of explanation that we focus on answering the following research question: How should be given to the human to improve overall system to use knowledge about an individual’s personality traits to utility. improve the overall system utility? B. OWC (Opportunity-Willingness-Capability) Model The main contributions of this paper are: Prior research in adaptive systems has investigated various • A formal framework that incorporates human models of humans that can be used at run time to effectively personality traits and guides adaptive human-in-the- characterize humans when deciding how best to incorporate loop systems to decide how much explanations should them into a co-adaptive system. One of the more prominent be given in order to improve system utility. models is the OWC (Opportunity-Willingness-Capability) the user's inputs and commands to take an action. Self- model [3]. adaptation refers to a process in which an interactive system co-adapts its behavior to a human based on its internal model OWC categorizes human attributes into: (1) Opportunity: of the human, dynamic information acquired about the human, indicates whether a human is available to participate in a the context of use and its surrounding environment [4][5][6]. cooperative task with the system (such as whether the human is physically present). (2) Willingness: identifies the human’s Several related works have studied explainability focused inclination to perform the task (affected by cognitive load, on a human-system co-adaptation perspective. In [20] the human attention, stress level, and motivation). (3) Capability: authors propose a method that generates verbal explanations defines the human’s abilities and skills that are necessary to of multi-objective probabilistic planning. This method execute the task successfully (affected by level of experience explains why a particular behavior is chosen on the basis of or training, knowledge of the task, and cognitive or physical the optimization objectives. Their explainability method relies skills) [3]. on describing the values of the objective of a generated behavior and, therefore, explaining tradeoffs that were made This model has been used effectively in a number of to reconcile competing objectives. papers to determine, for example, whether to involve the user in a task or to carry it out automatically [12][5], whether to In [21], the authors define a formal framework to reason proactively gain the user’s attention [13], and when to provide about explainability of co-adaptive system behaviors and the an explanation [14]. As we detail later in this paper we use situations under which they are warranted. Specifically, they OWC to capture the co-adaptation attributes of the human (see characterized explainability in terms of explainability cost, Section IV. B). effect, and content. They propose a dynamic adaptation approach that uses a probabilistic reasoning technique, similar C. Model Checking Stochastic Multiplayer Games (SMG) to ours, in order to determine when the explanations should be and PRISM used for the purpose of improving system utility. Probabilistic model checking is used as a technique to analyze the systems that exhibit stochastic behavior. In another related work [14], the authors use a similar Stochastic Multi-player Games (SMG) is a form of framework of [21] to reason about explainability of adaptive probabilistic modelling that allows us to reason quantitatively system behaviors and the conditions under which they are about reward-based properties and probability such as time, warranted. They characterize explainability in terms of the usage, and resources in a multi-agent system [15][16][17]. effects on a human operator’s ability to engage in co-adaptive Our approach is to use SMG models to reason about the actions effectively. They present a decision-making appropriate amount of explanation that should be given to the mechanism to plan in self-adaptation that provides a humans based on their personality traits where we model the probabilistic reasoning tool to determine when explanations system and humans as (cooperating) players in a game. should be used in an adaptation. PRISM is “a probabilistic model checker, a tool for formal While this prior work shares with our research the goal of modelling and analysis of systems that exhibit random or reasoning about explanation in the context of human-system probabilistic behavior” [18]. PRISM-games is an extension of co-adaptation, and also use probabilistic reasoning to account PRISM that is used to analyze probabilistic systems where for inherent uncertainties in our human models, none of these players can incorporate competitive or collaborative behavior, studies take into consideration specific personality traits of modelled as stochastic multiplayer games SMG [19]. humans – the main focus of our work. Analyzing systems using PRISM has been carried out in IV. METHODOLOGY variety of application domains, including: security protocols, communication and multimedia protocols, randomized In this section, we illustrate how we use explanation as a distributed algorithms, biological systems and many others. tactic (or action) that systems can use to improve the PRISM can analyze a wide range of quantitative properties of efficiently and effectiveness of human-system co-adaptation stochastic models automatically (e.g., "what is the probability based on human personality traits. We describe also how we of a failure causing the system to shut down within 4 hours?”). utilize a probabilistic planner [19] to determine the optimal PRISM further supports the specification and analysis of amount of explanation according to those personality traits. properties based on costs and rewards. These allow it to A. Selection of Personality Traits reason, not only about the probability that a model behaves in a certain way, but about a wide range of quantitative measures An important question is which personality traits to related to the behavior of the model (e.g., "expected number consider with respect to explanation? As noted earlier, the of lost messages", "expected time", or "expected power psychological literature has classified a variety of important consumption"). distinguishing characteristics for human personality. However, not all traits are relevant to explainability. In this In this paper we use PRISM to dynamically determine work we have adopted two personality traits: Need for appropriate levels of explanation to maximize expected utility Cognition (NFC) and Openness to experience, since there is a (expressed as a reward). direct relationship between NFC and explainability and D. Human-in-the-Loop Self-Adaptation and Explainability between Openness and capability in OWC [9][10] (see Human-system integration or human-system co- Section IV. B). adaptation is advancing the fields of human-system We use the “Openness to experience” trait as one factor interaction. Integration here means that the relationship that affects the human’s capability to continue and complete a between system and humans has become a partnership or task, since open people tend to be intellectually curious and symbiotic relationship in which humans (i.e., users) and have a high level of capability to do creative tasks [10]. We systems act with autonomy instead of the system waiting for consider the Openness level as an important human factor since an individual’s Openness level reflects their capability to engage in cognitive tasks. In our work we assume that the human’s personality traits are known (for example, by using the NFC 10-item testing instrument in [9][11]) and do not change over the time horizon of a particular set of interactions with the system. While the traits are assumed to be known, there does, however, remain some uncertainty about the impact of the amount of explanation that should be provided to the human, which we incorporate into our reasoning framework. We will further assume for concreteness that both selected personality traits are relevant, and that their weights are equally important (although the relative importance can be adjusted in the model). B. Incorporating the OWC (Opportunity-Willingness- Capability) Model We use the OWC model (described in Section III.B) to Fig. 2 The Grid Game we defined that embodies a representative scenario for human-system ao-adaptation capture the co-adaptation attributes of the human. In this paper, the following indicators show the connection between our model and the OWC model and how the OWC is D. Grid Game incorporated in the context of the collaborative Grid game: (1) To illustrate our approach, we defined the Grid game– a time and location represent the set of variables of the virtual game -- as shown in Figure 2, as a game that embodies Opportunity category. Is the player located at the correct a representative scenario for human-system co-adaptation. location? Has the timer expired? (2) Human satisfaction In the Grid game the system S instructs a player P verbally represents the Willingness category. Is the human satisfied to move on a 5×5 grid from the top right corner (start) to the with the given explanation? That category is applied through bottom left corner (end). The game is designed to rely on the playerFeedback (pF) tactic. (3) Human performance explanations, at various levels of detail, to instruct the user on represents the Capability category. The Capability category what tasks to perform and how to perform them. identifies the ability of the human to complete Grid task. Giving an explanation increases the capability of the human Game objectives: to successfully carry out that particular task [12]. • Follow the system instruction through a certain path C. Utlizing Model Checking Stochastic Multiplayer Games within a certain maximum amount of time (60 (SMG) seconds). The probabilistic model checker (PRISM-games) is • Minimize the time t to complete the task. utilized to formally model our approach. PRISM-games is particularly suitable for our study because it helps us to reason • Traverse an optimal number of blocks to complete the quantitatively under unpredictability and uncertainty about end-to-end task, avoiding obstacles. “how much” explanations should be given. The uncertainty (or stochasticity) that is relevant in this context is about the Game rules: proper amount of explanations and the impact of different • The player can move either horizontally or vertically. amounts of explanations that should be given to the human. • Game score (100 points): points are deducted for We model the system (the Grid game described in Section traversing extra blocks or moving into or through IV.D below) as a turn-based SMG, which means exactly one obstacle squares (e.g., in Figure 2 there are four player in each state of the modeled system can choose an obstacles: the house, a traffic light, a mountain, and a action, where the outcome of that state will be probabilistic. tree). Players in a SMG may cooperate to achieve a common goal, or compete to accomplish different goals. In our examples, we The Grid game can involve the use of five tactics for model two players1, the human and the system, and we assume interacting with the player, as shown in Table 1. The system that they share a common goal. provides two levels of explanation to command the human to move from one point to another. The choice of level of We use rPATL, a probabilistic temporal logic, to express explanation is based on the run-time calculation and properties of stochastic multi-player games quantitatively. explanation generation based on the probabilistic model. rPATL helps us to reason about the collective ability of a group of players to achieve a goal relating to the probability In this case “less explanation (LExp)” provides an of an occurring event [22]. abbreviated command (e.g., “Go 2 blocks left”), while “more 1 Note that a multiplayer here (i.e., two players) does not mean that the Grid game is a multiuser game. The concept “multiplayers” in PRISM refers to multiple agents, such as system, human, or environment. In our model, the system and the human are the only two players and they are working cooperatively (taking turns) to achieve the best possible outcome. TABLE I: GRID GAME TACTICS FOR INTERACTING WITH THE PLAYER Model Categories Tactics Role Example Less Commands the human to carry out an action. “Go 2 blocks left” lessExplain (lExp) Explanation “Move south 4 blocks” The system further explains information when the “You will go between a System human is confused and loses track. More moreExplain house and traffic light” Explanation (mExp) “You go straight, and you see a car on your left side” The human requests the system to confirm information “North?” Clarification Check (Chk) that they not entirely sure about. “Should I continue above Request the tree?” Human playerFeedback Human feedback is collected about his satisfaction for Helpful, Not helpful, Feedback (pF) each given explanation Neutral The human confirms information and follows the “Yeah”, “Thanks” Acknowledgement confirm (conf) instructions. “Okay” explanation (mExp)” provides an abbreviated command (e.g., b) Not helpful (NH) reflects utility decrements (↓), “You will go between a house and traffic light”) contains additional details. The human may request clarification about c) Neutral (N) reflects neither utility increments nor a given explanation if they are not entirely sure about it (Chk) decrements (-). (e.g., “Should I continue above the tree?”). Or the user can 3) Utility Functions confirm the information and follow the instructions (conf). The human also gives feedback (pF) about the given To compare different explainability tactics (i.e., lengths of explanation as to whether it was (a) helpful, (b) not helpful, or explanation), we use probabilistic temporal logic with (c) neutral. (This supports explanation assessment in the rewards, rPATL, which enables us to analyze the utilities of framework - Figure 1). the system that explainability can influence. rPATL (described in Section IV.C) is used to reason about the ability 1) Utility Attributes of a group of players (system and human) to collectively The four utility attributes of the game are: RequiredTime achieve a specific goal [18]. (t), Blocks (B), LengthOfExplanations (xL), and In the formal model we define formulas that represent the ExplainEfficiency (xE). B and t are used for calculating the accrued utility (The Scores function ∪ 𝑠 and the game score, and xL and xE are used as explainability ExplainEfficiency Function ∪ 𝑥𝐸) as the maximum real attributes. Game score(s) depend on the time elapsed for immediate utility that the human can achieve along the whole completing the game (t), associated with the optimal number task. of the blocks (B) that the player is supposed to end the task with: TABLE II: COST/BENEFIT ’ IMPACTS ON UTILITY DIMENSIONS • RequiredTime (t): the total elapsed time for Time completing the game. ∆ ExplainEfficiency ∆ RequiredTime (t) (xE) a Tactics ∆ LengthOfExplanations • Blocks (B): the number of the blocks traversed to (xL) complete the task. ↑ ↓ - ↑ • LengthOfExplanations (xL): the amount of delay (or lessExplain (lExp) H NH N +3 time) required to explain. moreExplain (mExp) +6 +1 -1 0 • ExplainEfficiency (xE): a measurement that determines how happy the player is with the given Check (Chk) explanations. xE is associated with the playerFeedback +3 confirm (conf) (pF) tactic which can be one of the following values: Helpful, Not helpful, or Neutral. playerFeedback (pF) a. 2) Tactics Cost/Benefit and Utility Dimensions H: Helpful, NH: Not helpful, N: Neutral The Scores function ∪𝑠, as shown in function (1), maps Table 2 lists the tactics in the Grid game, and their impacts high scores to high utility derived by dividing the number of on utility dimensions. Different tactics cause an increase in blocks 𝐵 by the maximum level of RequiredTime 𝑡 𝑚𝑎𝑥 (tmax= Time (three seconds for lExp, Chk, and conf; six seconds, for 60), where B must be greater than or equal to 𝑜𝑝𝑡𝑖𝑚𝑎𝑙𝐵 (the mExp). The upward ↑ or downward arrow ↓ reflects utility optimal number of blocks that the player is supposed to increments and decrements, respectively. For example, the complete the task with): lExp tactic increases both t and xL by three seconds, which is associated with a smaller amount of costs. Human feedback is 𝐵 ∪ 𝑠(𝐵) = (1 − 𝑚𝑎𝑥) 𝑥100 where 𝐵≥𝑜𝑝𝑡𝑖𝑚𝑎𝑙𝐵 () collected about the user’s satisfaction for the given 𝑡 explanation (lExp) which can be: The ExplainEfficiency Function ∪ 𝑥𝐸, as shown in a) Helpful (H) reflects utility increments (↑), function (2), maps higher levels of ExplainEfficiency (xE) derived by dividing the accumulated player Feedback (∑ 𝑝𝐹) by the total number of feedbacks (𝑝𝐹 𝑚𝑎𝑥 ), where 𝑝𝐹𝜖[1,0, −1] represent Helpful, Neutral, Not helpful, respectively: ∑ 𝒑𝑭 ∪ 𝑥𝐸(𝑝𝐹) ≈ ( ) 𝑥100 () 𝑝𝐹𝒎𝒂𝒙 Both personality trait variables (Openness and NFC) are initialized with some constants (as inputs) that represent the human traits. Personality traits are directly mapped to the level of explanation (the amount), and are used to calculate the probability of getting explanations in that amount. Function (3) shows combined personality traits, which will be 0 in case both traits are 0, or 1 in case the human has the highest personality traits levels (i.e., 0 → 1). Opennessmax and NFCmax are 100, which represent the highest personality trait levels. The values of personality traits are determined based on the NFC 10- item testing instrument in [9][11] that produces scores between 0-100. Fig. 3 The strategy we use to model the SMG: the proper amount of explanation For example, if a human has 75 Openness and 90 is determined based on the three personality levels of the human (represented in light blue). The two explanation amounts (less or more) are determined by using function NFC. The combined human traits are 0.82 which 3. Human feedback is collected that will can be helpful, neutral, or not helpful means he has high personality traits (by using function (represented in yellow). The human confirms information that means he moved (3)). That means the system will explain less 18% of successfully to the next point (conf), or checks/requests the system to clarify the time (i.e., lExp) and explain more 82% of the time information that they not entirely sure about (chk). (i.e., mExp) during the task. As another example, suppose a human with low personality traits has 43 system took 27 seconds for explanations (xL). At the end of openness and 49 NFC. The combined human traits are 0.46 the task, the score of the player is 75 (by using the function (using function (3)). That means the system will explain less (1), where B=15 and tmax= 60), and the ExplainEfficiency 54% of the time (i.e., lExp) and explain more 46% of the (xE) is 43 (by using the function (2), where ∑ 𝑝𝐹 is three and time (i.e., mExp) while playing the Grid game. 𝑝𝐹 𝑚𝑎𝑥 is seven (which means seven feedbacks are 𝐻𝑢𝑚𝑎𝑛 𝑇𝑟𝑎𝑖𝑡𝑠 = 𝑂𝑝𝑒𝑛𝑛𝑒𝑠𝑠+𝑁𝐹𝐶 () collected)). 𝑂𝑝𝑒𝑛𝑛𝑒𝑠𝑠 𝑚𝑎𝑥 +𝑁𝐹𝐶 𝑚𝑎𝑥 TABLE III: AN EXAMPLE DIALOGUE OF A SCENARIO V. THE STOCHASTIC MULTI-PLAYER GAMES BETWEEN THE SYSTEM (S) AND A HUMAN (H) (SMG) MODEL Scenario Tactics Time pF We model the Stochastic Multi-player Games (SMG) model as two players, where the players try to collaboratively S: Can you go 2 blocks down? (lExp) 3s H maximize accumulated reward(s): (1) Player SYS specifies H: Yeah (conf) 3s - the actions that are controlled by the system (i.e., it represents S: Then go 2 blocks left. (lExp) 3s NH the Grid game). (2) Player HUMAN specifies the actions belonging to the human (i.e., it represents the game player). H: Could you repeat that? (Chk) 3s - The models represent the behavior of a set of agents (or S: Go west. You will go between a house (mExp) 6s H “players”) that take turns making moves, where the choice of and traffic light. move is specified probabilistically or non-deterministically. A H: Okay (conf) 3s - game solver for such a system (such as PRISM-games [19]) determines an optimal strategy for the players by resolving the S: Go after that 2 blocks up. (lExp) 3s N non-deterministic transitions in such a way that the expected H: The human is on the wrong track - reward for each player is maximized (assuming rational play S: No, not south. You go north (mExp) 6s H by each). Figure 3 shows the strategy we use to model the SMG. The proper amount of explanation is determined based H: Okay (conf) 3s - on the three personality levels of the human (i.e., Low, S: Go 2 blocks left (lExp) 3s N Average, or High personality levels (represented in light blue)). The two explanation amounts (less or more) are ……… determined by using Function 3 (describes in the previous S: Go south 4 blocks. (lExp) 3s H section). Human feedback is collected that is of the form H: Okay, thanks a lot. (conf) 3s - Helpful, Neutral, or Not Helpful (represented in yellow). The human confirms information that means he moved 4) Example Scenario successfully to the next point (conf), or checks/requests the system to clarify information that they not entirely sure about Figure 2 and Table III show an example dialogue of a (Chk). scenario between the system (S) and a human (H). The Stochastic Multi-player Games (SMG) model consists The human spent 42 seconds (t) and used 15 blocks (B) to of the following four parts: finish the task. However, the number of blocks B that the player is supposed to end the task with is 12 (𝑜𝑝𝑡𝑖𝑚𝑎𝑙𝐵). The A. Player Definition Similarly, the system instructs the human with average Player definition includes the declaration of the two personality traits through executing the command labeled as players in the SMG and different modules that each player has lExpAvg (line 8-10), or the system instructs the human with control of. The two players in our game are shown in Listing high personality traits through executing the command labeled 1. Player SYS (lines 1-2) specifies the actions that are as lExpHigh (line 11-13). controlled by the system (i.e., it represents the Grid game). The human wins by executing the command labeled as win Player HUMAN (lines 3-4) specifies the actions belonging to (line 25). That means the human (turn=SYS) has arrived at the the human (i.e., it represents the game player). Our Grid game bottom left corner ((x1= 1)&(y1=1)) within the time limit is played in turns by the two players SYS and HUMAN. Turn (t<60). However, the human loses the game by executing the (line 5) is a global variable used as a controller to take turns command labeled as lose (line 26) when the end time of the between different players, ensuring that only one player can task has been reached (t=60). take an action at each state of the model execution. Tactics are executed sequentially in our model. 1. global lExp_state: bool init false; 2. global mExp_state: bool init false; 1. player SYS 3. … Game, [lExpLow],[ lExpAvg],[ lExpHigh], 4. module Game [mExpLow],[mExpAvg],[mExpHigh] 5. [lExpLow] (turn=SYS)&(human_traits<.5)&(x1= 5) 2. endplayer &(y1=5)&(t<60) 3. player HUMAN 6. ->human_traits:(mExp_state'= true) Play, [conf], [Chk] 7. + 1-human_traits:(x2'=5) & (y2'=3)&(xL'=xL+3) 4. endplayer & (t'=t+3)&( lExp_state'= true)&(turn'=HUMAN); 5. global turn:[SYS..HUMAN] init SYS; 8. [lExpAvg] (turn=SYS)&(x1= 5)&(y1=5)&(t<60) 6. const SYS=1; const HUMAN=2; 9. ->0.5:(x2'=5)&(y2'=3)&(xL'=xL+3)&(t'=t+3) &(lExp_state'= true)&(turn'=HUMAN) Listing. 1 Player definition includes the declaration of the two players in 10. +0.5:(mExp_state'= true); the SMG and different modules that each player has control of 11. [lExpHigh] (turn=SYS)&(human_traits>.8) &(x1= 5)&(y1=5)&(t<60) B. Game Model 12. ->human_traits:(mExp_state'= true) 13. + 1-human_traits:(x2'=5) & (y2'=3)&(xL'=xL+3) Player SYS has control of the Game model, illustrated in & (t'=t+3)&( lExp_state'= true)&(turn'=HUMAN); Listing 2. Opportunity elements are used as execution 14. … conditions of different tactics such as: the human is at the 15. [mExpHigh] (turn=SYS)&(human_traits>.8)& correct location ((x= 1)&(y=1)) and is not involved in a crash, (conf_state= false)&(Chk_state= true)&(t<60) and the time has not expired (t<60). The Game module is 16. ->human_traits:(mExp_state'=true)&(xL'=xL+6) parameterized by the variables (lines 1-2), which indicate the &(t'=t+6)&(Chk_state'= false)&(turn'=HUMAN) 17. + 1-human_traits: (lExp_state'= true); state of tactic execution, where false means this tactic is not 18. [mExpAvg] (turn=SYS)&(conf_state= false) in use (i.e., lExp_state, and mExp_state). &(Chk_state= true)&(t<60) 19. ->0.5:(mExp_state'= true)&(xL'=xL+6)&(t'=t+6) During the system’s turn, the system executes these &(Chk_state'= false)&(turn'=HUMAN) tactics sequentially: lExp (lines 5-13), and mExp (lines 15- 20. +0.5:( lExp_state'= true); 23). For the sake of clarity, we will describe only the 21. [mExpLow] (turn=SYS)&(human_traits<.5) lExpLow tactic to illustrate how tactic execution is modeled. &(conf_state= false)&(Chk_state= true)&(t<60) The other explainability tactics follow the same structure. The 22. ->human_traits:( lExp_state'= true) system instructs the human with low personality traits 23. + 1-human_traits:(mExp_state'=true)&(xL'=xL+6) &(t'=t+6)&(Chk_state'= false)&(turn'=HUMAN); through executing the command labeled as lExpLow (line 5). 24. … This tactic executes only if: 25. [win] (turn=SYS)&(x1= 1)&(y1=1)&(t<60) -> (win'=true)&(turn'=0); • It is the turn of the SYS. 26. [lose] ((turn=SYS)|(turn=HUMAN))&(t=60) -> (win'=false)&(loser'= true)&(turn'=0); • The human traits are low (<0.50). 27. endmodule • The player position is on a certain block (x1,y1). 28. … • The end time of the task has not been reached yet (t<60). Listing. 2 Game Model If the guard is satisfied, the system will explain more by C. Play Model flagging mExp_state tactic true with probability human_traits (line 6). Otherwise, the system will explain less Player HUMAN has control of the Play model, illustrated by flagging lExp_state tactic true with probability 1- in Listing 3. The encodings of the HUMAN module are similar human_traits (line 7) and the system will: to those of the SYS module. The Play module is parameterized by variables (lines 1-2), which indicate the state • Commands the player to move to the position (x2,y2). of tactic execution, where false means this tactic is not in use (e.g., Chk_state, and conf_state). Personality Traits are • Increases the time 3 seconds (xL'=xL+3)&(t'=t+3). initialized with values that represent the human’s personality • Flags the lExp tactic as true (lExp_state'= true). (lines 3-5). • Updates the value of the variable turn, changing During the human’s turn, the human can execute one of control to the human player (turn’=HUMAN). these tactics: conf (line 8), and Chk (line 10). We explain only the conf tactic to illustrate how tactic execution is modeled. The human confirms (conf) and follows the system the encoded Function (2) described in Section IV.D. Line 5 instructions (i.e., the human moves successfully from the 1st shows the encoded combined traits function (3). point to the second) by executing the command labeled as conf. This tactic executes only if: 1. rewards "Scores" [win] true:(1-(B/tMax))*100; • It is the turn of the HUMAN. [lose] true:0; • The system instructs the player to move to the position [crash] true: -5; 2. endrewards (x2,y2). 3. rewards "ExplainEfficiency" • The end time of the task has not been reached yet [win] true:(pF/pfMAX)*100; (t<60). [lose] true:(pF/pfMAX)*100; 4. endrewards 1. global Chk_state: bool init false; 5. formula human_traits = 2. global conf_state: bool init false; (human_Open+ human_NFC)/(Max_Open+Max_NFC); 3. const int INIT_OPN; const int INIT_NFC; 4. global human_Open: [1..100] init INIT_OPN; Listing 4. Utility profile and reward structure: formulas and reward 5. global human_NFC:[1..100] init INIT_NFC; structures are used to encode the utility functions that allow us to quantify the 6. … utilities of different task states. Formulas calculate system utility of the 7. module Play different states. 8. [conf] (turn=HUMAN)&(x2= 5)&(y2=3)&(t<60) ->(x1'=5) & (y1'=3)& (t'=t+3)&(B'=B+2) &(pF'=pF+1)&(pfMAX'=pfMAX+1)&(conf_state'= true) VI. RESULTS AND ANALYSIS &(lExp_state'= false)&(turn'=SYS); In this section, we illustrate how our modeling framework 9. … can produce optimal decisions with respect to how adaptive 10. [Chk] (turn=HUMAN)&(conf_state= false)&(x1= 5) & (y1=3)&(t<60) systems should explain to the human based on their ->(Chk_state'= true)&(t'=t+3)&(pF'=pF1) personality traits. Specifically, we use SMG models of &(pfMAX'=pfMAX+1)&(turn'=SYS); explainability to determine the expected outcome utilities of 11. [wrong] (turn=HUMAN)&(x2= 3)&(y2=5)&(loser=false) using different explainability tactics (i.e., explanation &(t<60)-> (x1'=3) & (y1'=1) &(t'=t+3) amounts) based on the personality traits of the human. Our & (B'=B+2)&(pF'=pF-1)&(pfMAX'=pfMAX+1) modeling is done as a simulation (or set of “experiments” in &(conf_state'= false) &(turn'=SYS); 12. [crash] (turn=HUMAN)& ((x1=obj1x & y1=obj1y) PRISM terms). We use rPATL to ask PRISM a variety of | (x1=obj2x & y1=obj2y)| (x1=obj3x & y1=obj3y) questions such as “what is the maximum/minimum | (x1=obj4x & y1=obj4y))->(turn'=SYS); probability a human with high/low personality traits can 13. endmodule guarantee to win with high/low utilities?” [22]. 14. … Table IV and Figure 4 show the analysis results of 44 Listing. 3 Play Model rounds run on PRISM. All possible combinations of personality traits are taken into consideration, where high If the guard is satisfied, the player: traits are (>80) (represented by orange color), average traits are (≥50 and ≤80) (represented by blue-gray color), and low • Moves to the position (x1,y1). traits are (<50) (represented by gray color). Plot (a) shows the • Increases the time three seconds (t'=t+3). 44 simulations of different personality traits and the given amounts of explanations (LengthOfExplanations (xL)) to • Increases the number of Blocks by two (B'=B+2). complete the task. The average of different personality traits and the amounts of explanations (xL) is shown in Plot (b). • Gets the player feedback (pF= 1 means the explanation 39% of the iterations (17 rounds) of humans with high was helpful). personality traits (>80) needed more explanations to finish the • Increases the player feedback counter pfMAX by 1. task with an average of 21 seconds. 32% of the iterations (14 rounds) of humans with low personality traits (<50) needed • Flagging the conf tactic true (conf_state'= true). less amount of explanations with an average of 20 seconds. Moreover, the tactic wrong (line 11) will be executed The remaining 30% of the iterations (13 rounds) belongs to a when the human moves in the wrong direction, and the tactic human with average personality traits (≥50 and ≤80), where crash (line 12) will be executed when the human moves to they use average amounts of explanation with an average of one of the obstacle squares (the house, a traffic light, a 19 seconds to complete the task. Table 5 shows the average mountain, or a tree in Figure 2). of different utilities based on the three personality trait levels. D. Utility Profile and Reward Structure We can conclude from the results that a human with high personality traits needs more detailed information (i.e., Utility functions are described in Section IV.D and explanations), while a human with low personality traits needs illustrated in Listing 4. Formulas and reward structures are less detailed explanation. These conclusions are all consistent used to encode the utility functions that allow us to quantify with psychology studies (discussed in Section III. A) that the utilities of different task states. human with higher personality trait levels typically prefer The Scores function, ∪𝑠, as in lines (1-2), represents the more explanations, while those with low levels of personality encoded Function (1) as described in Section IV.D. trait want to quickly understand the big picture and avoid ExplainEfficiency function ∪𝑥𝐸, as in lines (3-4) , represents engaging through more explanations [9][11]. TABLE IV: RESULTS OF 44 ROUNDS RUN ON PRISM Human Traits Utilities # Combined Traits LengthOfExplanations ExplainEfficiency Openness NFC Scores (xL) (xE) 1 75 90 82.5 27 28.5 93.4 2 100 100 100 21 50 96.7 3 50 90 70 15 80 100 4 95 30 62.5 21 50 96.7 5 95 85 90 15 80 100 6 45 88 66.5 15 80 100 31 47 47 47 33 12.5 90 32 83 83 83 21 50 96.7 33 22 19 20.5 15 80 100 34 96 77 86.5 15 80 100 35 69 55 62 27 28.5 93.4 36 39 11 25 21 50 96.7 37 33 19 26 15 80 100 38 17 15 16 27 28.5 93.4 39 9 30 19.5 21 50 96.7 40 49 29 39 15 80 100 41 51 71 61 15 80 100 42 93 100 96.5 21 50 96.7 43 100 90 95 15 80 100 44 81 80 80.5 15 80 100 Minimum 9 11 16 15 -12.5 90 Maximum 100 100 100 36 80 100 Average 66.02 61.98 64 20.39 57.07 97.23 TABLE V: AVERAGE UTILITIES OF THE EXPERIMENTS BASED ON THE THREE PERSONALITY TRAITS LEVEL Personality Traits Average Utilities Level xL xE Scores High Traits 21.18 53.50 91.388 Average Traits 19.15 62.27 97.708 Low Traits 20.57 56.57 96.921 Plot (a): The 44 simulations of different personality traits and the given amounts of explanations (LengthOfExplanations (xL)) VII. DISCUSSION AND FUTURE WORK to complete the task. In this research we presented an approach based on probabilistic model checking of SMGs to determine how much explanation should be given to the human based on their personality traits. Providing the right amount of explanation to the right human is an important factor to maximize co- adaptation between the system and the human during their interaction. There is a number of limitations of this research that future research can address based on the foundations that we have described in this paper. These explanation decisions are ideal Plot (b): The average of different personality traits and the scenarios without having actual proof of that in reality. To amounts of explanations (xL) address this, the most important next step is to conduct an empirical study to validate these models on actual real-world Fig. 4 Results of 44 rounds run on PRISM show that human with higher personality trait levels typically prefer more explanations, while systems with humans in the loop. those with low levels of personality trait prefer less explanations As we explained earlier, there are many reasons to use explanations, and examine in more detail questions such as explainability and improving a system’s overall utility is one how explanations should be presented: graphically, textually, of the main reasons (see Section I). Explainability can help not verbally? A further extension of this research is to have more only to improve the systems continuously through human detailed models that allow the system to determine in a more involvement, but also to justify some information given to the nuanced way the ideal contents of the explanations that should human, particularly when decisions are made suddenly. be considered. Gaining more information improves the capability of the human to perform a task. Our results in this paper suggest one of the next steps of research is to go beyond the length of VIII. CONCLUSION Systems,” In Proceedings of the 2020 IEEE Conference on Autonomic Computing and Self-organizing Systems (ACSOS), Washington, D.C., In this research we presented a formal framework that 19-23 August 2020. incorporates human personality traits as one of the important [15] P. D. Kwiatkowska M., Norman G., “Probabilistic Model Checking: elements in guiding automated decision-making about the Advances and Applications,” Form. Syst. Verif. Springer, Cham., pp. proper amount of explanation that should be given to the 73–121, 2018. human to improve overall system utility. To accomplish our [16] C. Baier, “Probabilistic model checking,” Dependable Softw. Syst. goal of this paper, we use probabilistic model analysis to Eng., vol. 45, no. August, pp. 1–23, 2016. determine how to utilize explanations in an effective way [17] S. W. Cheng and D. Garlan, “Stitch: A language for architecture-based self-adaptation,” J. Syst. Softw., 2012. based on the difference of human’s personality traits. Grid – a [18] M. Kwiatkowska, G. Norman, and D. Parker, “PRISM 4.0: Verification virtual human and system interaction game – was developed of probabilistic real-time systems,” in Lecture Notes in Computer to illustrate our approach, to represent scenarios for human- Science (including subseries Lecture Notes in Artificial Intelligence system co-adaptation, and to demonstrate through simulation and Lecture Notes in Bioinformatics), 2011. how a human’s personality traits can be used as a factor to [19] M. Kwiatkowska, G. Norman, D. Parker, and G. Santos, “PRISM- consider for systems in providing appropriate explanations. games 3.0: Stochastic Game Verification with Concurrency, Equilibria and Time,” in Lecture Notes in Computer Science (including subseries ACKNOWLEDGMENT Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020. This research was supported in part by the NSA under [20] R. Sukkerd, R. Simmons, and D. Garlan, “Towards explainable multi- Award No. H9823018D0008 and Award No. N00014172899 objective probabilistic planning,” Proc. - Int. Conf. Softw. Eng., pp. from the Office of Naval Research. Any views, opinions, 19–25, 2018. findings and conclusions or recommendations expressed in [21] N. Li, S. Adepu, E. Kang, and D. Garlan, “Explanations for human-on- this material are those of the author(s) and do not necessarily the-loop: A probabilistic model checking approach,” Proc. - 2020 reflect the views of the NSA or the Office of Naval Research. IEEE/ACM 15th Int. Symp. Softw. Eng. Adapt. Self-Managing Syst. SEAMS 2020, pp. 181–187, 2020. REFERENCES [22] T. Chen, V. Forejt, M. Kwiatkowska, D. Parker, and A. Simaitis, “Automatic verification of competitive stochastic systems,” Form. [1] G. Vilone and L. Longo, “Explainable Artificial Intelligence: a Methods Syst. Des., vol. 43, no. 1, pp. 61–92, 2013. Systematic Review.” arXiv preprint arXiv:2006.00093, 2020. [2] A. Adadi and M. Berrada, “Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI),” IEEE Access, vol. 6. pp. 52138–52160, 2018. [3] D. Eskins and W. H. Sanders, “The multiple-asymmetric-utility system model: A framework for modeling cyber-human systems,” Proc. 2011 8th Int. Conf. Quant. Eval. Syst. QEST 2011, pp. 233–242, 2011. [4] J. O. Kephart and D. M. Chess, “The vision of autonomic computing,” Computer (Long. Beach. Calif)., 2003. [5] E. Lloyd, S. Huang, and E. Tognoli, “Improving Human-in-the-Loop Adaptive Systems Using Brain-Computer Interaction,” Proceedings - 2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2017. pp. 163–174, 2017. [6] M. Alharbi and S. Huang, “A Survey of Incorporating Affective Computing for Human-System Co-adaptation,” in Proceedings of the 2020 The 2nd World Symposium on Software Engineering, 2020, pp. 72–79. [7] B. Mittelstadt, C. Russell, and S. Wachter, “Explaining explanations in AI,” FAT* 2019 - Proc. 2019 Conf. Fairness, Accountability, Transpar., pp. 279–288, 2019. [8] C. G. H. Jung, “Psychological Factors Determining Human Behaviour,” Collect. Work. C.G. Jung, Vol. 8 Struct. Dyn. Psyche, pp. 114–126, 2015. [9] C. J. Sadowski and H. E. Cogburn, “Need for cognition in the big-five factor structure,” Journal of Psychology: Interdisciplinary and Applied, vol. 131, no. 3. pp. 307–312, 1997. [10] R. R. McCrae, “Openness to Experience as a Basic Dimension of Personality,” Imagin. Cogn. Pers., 1993. [11] R. E. Petty, J. T. Cacioppo, R. E. Petty, J. A. Feinstein, and W. B. G. Jarvis, “Dispositional Differences in Cognitive Motivation : The Life and Times of Individuals Varying in Need for Cognition Dispositional Differences in Cognitive Motivation : The Life and Times of Individuals Varying in Need for Cognition,” Psychol. Bull., vol. 119, no. August, pp. 197–253, 2015. [12] J. Cámara, G. Moreno, and D. Garlan, “Reasoning about Human Participation in Self-Adaptive Systems,” Proc. - 10th Int. Symp. Softw. Eng. Adapt. Self-Managing Syst. SEAMS 2015, no. i, pp. 146–156, 2015. [13] N. Li, C. Javier, D. Garlan, and B. Schmerl, “Hey ! Preparing Humans to do Tasks in Self-adaptive Systems .” In Proceedings of the 16th Symposium on Software Engineering for Adaptive and Self-Managing Systems, Virtual, 18-21 May 2021. [14] N. Li, J. Cámara, D. Garlan, and B. Schmerl, “Reasoning about When to Provide Explanation for Human-in-the-loop Self-Adaptive