We propose a framework toward more explainable reinforcement learning (RL) agents. The framework uses introspective analysis of an agent's history of interaction with its environment to extract several interestingness elements regarding its behavior. Introspection operates at three distinct levels, first analyzing characteristics of the task that the agent has to solve, then the behavior of the agent while interacting with the environment, and finally by performing a meta-analysis combining information gathered at the lower levels. The analyses rely on data that is already collected by standard RL algorithms. We propose that additional statistical data can easily be collected by a RL agent while learning that helps extract more meaningful aspects. We provide insights on how an explanation framework can leverage the elements generated through introspection. Namely, they can help convey learned strategies to a human user, justify the agent's decisions in relevant situations, denote its learned preferences and goals, and identify circumstances in which advice from the user might be needed.
• Computing methodologies → Artificial intelligence ; Intelligent agents; Sequential decision making; • Human-centered computing → Human computer interaction (HCI); Collaborative interaction.
Reinforcement learning (RL) is a very popular computational
approach to address problems of autonomous agents facing a
sequential decision problem in a dynamic and often times uncertain
environment [
RL focus on developing memoryless (reactive) agents, i.e., that
select their actions based solely on their current observation [
On one hand, RL agents do not need to plan or reason about their
future in order to select actions. On the other hand, this means that
it is hard for them to come up with explanations for their behavior—
all they know is that they should perform one action in any given
situation, in the case of deterministic policies, or select an action
according to a probability distribution over actions, in the case of
stochastic policies. Importantly, the “why” behind decision-making
is lost during the learning process as the policy converges to an
optimal action-selection mechanism. At most, agents know that
choosing some action is preferable over others, or that some actions
have a higher value associated with them. Another key insight of
RL that complicates explainability is that it allows an agent to learn
from delayed rewards [
Ultimately, RL agents lack the ability of knowing why some actions are preferable over others, of identifying the goals that they are currently pursuing, of making sense of the dynamics of the environment or recognizing what elements are more desirable, of identifying situations that are “hard to learn”, or of summarizing the strategy learned to solve the task. Without such an explanation mechanism, autonomous RL agents may sufer from lack of trust by end users of the system or human collaborators that wish to delegate on them critical tasks. 1.1
In this paper, we present a framework towards making autonomous RL agents more explainable. The framework is illustrated in Fig. 1 and relies on introspective analysis. In particular, the agent examines its history of interaction with the environment to extract interestingness elements denoting meaningful situations, i.e., potentially “interesting” characteristics of the interaction. Each element is generated by applying statistical analysis and machine learning methods to data collected during the interaction. In that regard, we argue that there is a great amount of information that is disregarded by typical RL algorithms that can be of most importance to understand the behavior of the agent and address some the aforementioned problems. Some of such information is already collected by standard algorithms but used only for updates of the policy or history of interaction
RL algorithm
Observation human user value function, while other data can easily be collected and updated by the agent during learning. As depicted in Fig. 1, the introspection framework can be the basis of an explanation system operating on top of the interestingness elements, selecting the ones that better help explain the agent’s behavior.
There are a few characteristics of the proposed introspection framework that are worth noting. First, the framework is domainindependent, meaning that the data that is collected and analyzed is agnostic to the specific learning scenario. Second, the framework is also algorithm-independent in the sense that it can be used in conjunction with standard RL tabular methods without having to modify the learning mechanism itself. Furthermore, the framework makes no assumptions with regards to optimality of the observed behavior—it captures important aspects specific to some history of interaction, independently of whether the agent was exploring the environment, exploiting its knowledge after learning, or simply acting randomly. Moreover, the framework is suitable to be used at diferent times and for diferent explanation modes, namely: during learning, by tracking the agent’s learning progress and acquired preferences; after learning, by summarizing the most relevant aspects of the interaction; passively, where a user can query the agent regarding its current goals or ask it to justify its behavior at any given situation; proactively, where the agent requests input from the user in situations where its decision-making is more uncertain or unpredictable. 1.2
RL agents can be modeled using the partially observable Markov
decision process (POMDP) framework [
In this paper, we deal with the problem of explainability in
situations where the agent may have limited sensing capabilities, i.e., the
environment may be partially-observable. Notwithstanding, as in
typical RL scenarios, we assume that the agent’s observations are
suficient for it to solve the intended task, in which case Z is treated
as if it were S, and O is discarded. The simplified model thus
obtained, represented as a tuple M = (S, A, P, R, γ ), is referred to as
a Markov decision process (MDP) [
The goal of the agent can be formalized as that of gathering as much reward as possible throughout its lifespan discounted by γ . This corresponds to maximizing the value v = E Ít γ t r . To that end, the agent must learn a policy, denoted by π : Z → A, that maps each observation z ∈ Z directly to an action π (z) ∈ A. In the case of MDPs, this corresponds to learning a policy π ∗ : S → A referred to as the optimal policy maximizing the value v.
We focus on value-based RL, where a function Q∗ : S × A → R
associated with π ∗ verifies the recursive relation Q∗(s, a) = r +
γ Ís′ ∈S P(s ′ | s, a) maxb ∈A Q∗(s ′, b). Q∗(s, a) represents the value
of executing action a in state s and henceforth following the optimal
policy. Standard RL algorithms like Q-learning [
As depicted in Fig. 1, the introspection framework relies on interaction data that is collected by the agent during its interaction with the environment. Namely, the analyses rely on the following data: • n(z), n(z, a) and n(z, a, z′), which respectively denote the number of times that some observation z, observation-action pair (z, a) and transition (z, a, z′) were observed by the agent while interacting with the environment; • Pˆ(z′ | z, a) corresponds to the estimated probability of observing z′ when executing action a after observing z. This can be estimated by using n(z, a, z′)/n(z, a); • Rˆ(z, a) is the agent’s estimate of the reward received for performing action a after observing z. This can be estimated by maintaining a running average of the rewards received; • Q(z, a) is the agent’s estimate of the Q function. It corresponds to the expected value of executing a having observed z and henceforth following the current policy. It can be estimated by using any value-based RL algorithm; • ∆ cQ(z, a) is the expected prediction (Bellman) error associated with Q(z, a). For a transition (z, a, r, z′), the prediction error corresponds to ∆ Q(z, a) = r + γ maxb ∈A Q(z′, b) − Q(z, a). As such, the agent can maintain a running average of the prediction errors after each visit to (z, a). As we can see, some of the data is already collected by value-based RL methods, namely the Q function, and by model-based algorithms, namely Pˆ and Rˆ . The remaining data can easily be collected by the agent during its interaction with the environment by updating counters and running averages.
In addition, as is the case with many RL scenarios, let us assume that, at each time-step t , the agent observes its environment through a finite set of features Zti = zi , i = 1, . . . , N , each taking values in some feature space Zi . The observation-space thus corresponds to the cartesian product Z = Z1 × . . . × ZN . When this is the case, the structure exhibited by such factored MDPs can also be exploited to derive interesting aspects related to specific observation elements.
All these data are used by the agent as sources of information for the several introspection analyses. Table 1 lists the analyses implemented so far, how they are group at diferent levels, and the elements that each generates.1 The idea is to take the most out of the data collected during the interaction to highlight information that helps explain the agent’s behavior. 2.2
In this level we analyze characteristics of the task or problem that
the agent has to solve.
2.2.1 Transition Analysis. The estimated transition probability
function Pˆ(z′ | z, a) can be used to expose the environment’s dynamics.
Namely, it allows the identification of (un)certain transitions. Given
an observation z and an action a, the transition certainty
associated with (z, a) is measured according to how concentrated the
observations z′ ∈ Z following (z, a) are. In particular, we use the
evenness of the distribution over observations following (z, a) as the
normalized true diversity (information entropy) [
next state is hard/easy to be predicted by the agent.
tion features that are active, on average, whenever certain/uncertain observation-action pairs occur.
Transitions leading to many diferent states—according to a given
threshold—have a high evenness and are considered uncertain.
Likewise, transitions leading only to a few states have a low
evenness and are considered certain. This analysis thus highlights the
(un)certain elements of the agent’s transitions, actions and of its
observation features. Uncertain elements are especially important
as people tend to resort to “abnormal” situations for the explanation
of behavior [
average among all other states and actions, the agent received 1Due to space restrictions, only some elements are detailed here. significantly more/less reward. 2 This information may be used to identify situations in which the agent is likely to receive relatively low or high rewards.
Feature-action reward outliers: correspond to feature-action pairs that are, on average among all observations taken, significantly more or less rewarding than other pairs. The rationale is that, in typical RL scenarios, the agent designer defines rewards to be provided to the agent when it interacts with elements of the environment in a (in)correct manner, e.g., the agent may have to interact with the appropriate object to achieve some subgoal. However, the reward from executing some action after making some observation is “diluted” among all the features that were active at that time. Therefore, this element may be used to denote significant individual contributions of features to the agent’s reward. 2.3
The purpose of this level is to help characterize the environment’s dynamics and extract important aspects of the agent’s behavior and history of interaction with it. 2.3.1 Observation Frequency Analysis. This analysis identifies interestingness elements that can be found given information stored in the counters n, namely: Observation coverage: corresponds to how much of the observationspace—regarding all possible combinations between the observation features—were actually observed by the agent. This information may provide an indication of how much of the state-space was covered by the agent’s behavior, which is an important quality of its exploration strategy.
Observation evenness: corresponds to how even the distribution of visits to the observation space was. In particular, it analyzes the histogram of observations using the aforementioned distribution evenness metric. It can be used to infer how unbalanced the visits to states were, which in turn may denote how interesting the dynamics of the environment are, e.g., denoting situations that are physically impossible to occur, and how exploratory the agent was during the interaction with it.
Frequent/infrequent observations: these correspond to observations that appeared less/more frequently than others during the interaction. This element can assess the agent’s (in)experience with its environment, denoting not only common situations it encounters but also rare interactions. Importantly, the latter may indicate states that were not suficiently explored by the agent, e.g., locations that are hard to reach in a maze or encounters with elements that are scarce, or situations that had such a negative impact on the agent’s performance that its action-selection and learning mechanisms made sure they were rarely visited, such as a death situation in a game.
tion features (feature-sets) that frequently/rarely co-occur. To that
end, we resorted to frequent pattern-mining (FPM) [
In this case, each observation corresponds to a transaction containing the features that are active in that observation. We then 2In our analyses, we detect outliers based on whether the distance to the mean is higher than a certain number of standard deviations.
(Un)Certain transitions; (Un)Certain actions; (Un)certain features of the task (POMDP) analyze agent’s history of interaction with the environment combine elements generated by the diferent analyses
Obs.-Action Freq. Obs.-action coverage; Obs.-action evenness; (Un)Certain obs. and features Reward Obs. Freq. and Recency Value Transition Value Sequence Contradiction
Mean reward; Obs.-action reward outliers; Actions mean reward;
Feature-action reward outliers
Obs. coverage; Obs. Dispersion; (In)Freq. obs.; Strongly/weakly-associated
feature-sets; Associative feature-rules; Earlier obs. and actions
Mean value; Obs.-action value outliers; Feature-action value outliers; Mean
pred. error; Obs.-action pred. outliers; Actions mean value and pred. error
Local minima/maxima; Absolute minima/maxima; Maximal
strict-diference obs.; Obs. variance outliers
Uncertain-future obs.; Certain sequences to subgoal
Contradictory-value obs.; Contradictory-count obs.; Contradictory-goal
obs.; Contradictory feature-actions
created a frequent-pattern tree (FP-tree) using the algorithm in [
Typical FPM techniques rely on the relative frequency of
cooccurrences of items to judge whether a certain item-set is
considered a pattern or not. However, other metrics exist that allow
the discovery of more meaningful patterns—for example, if two
features are always observed together by the agent, the pair should
be consider interesting even if their relative frequency (i.e.,
compared to all other observations) is low. Therefore, we use the Jaccard
index [
This element may be used to denote both patterns in the agent’s perceptions (or regularities in its environment), and also rare on inexistent combinations of features. In turn, these aspects may be important to explain the agent’s physical interaction with the environment and expose its perceptual limitations to an external observer.
Associative feature-rules: these are rules generated in the form
antecedent ⇒ consequent . The idea is to determine sets of features–
the antecedent—that frequently appear conditioned on the
appearance of another set of features—the consequent. We used the lift
statistical measure [
actions were executed in the observations made. Similarly to the observation coverage, this element reveals how exploratory the interaction with the environment was.
ness of action executions per observation. It can be used to determine how (un)balanced the selection of actions in certain observations were. In turn, this may denote either how exploratory the agent was during the interaction, or how stochastic the agent’s policy is.
Certain/uncertain observations and features: besides transition certainty, we can calculate how certain or uncertain each observation is with regards to action execution. Observations where many diferent actions have a high count (high evenness) are considered uncertain, while those in which only a few actions were selected are considered certain. This may denote situations in which the agent is (un)certain of what to do, therefore providing good opportunities to ask for a human user for intervention. Similarly, we can identify features denoting situations in which, on average, action selection is very even/uneven. Uneven features may be particularly useful to abstract action-execution rules, i.e., actions that are very likely to be executed whenever some feature is active. 2.3.3 Value Analysis. This analysis uses information stored in Q, V and ∆ cQ to generate the following interestingness elements:
that are significantly more or less valued. This element denotes desirable situations with regards to the agent’s goals—high-value pairs indicate situations conducive for the agent to attain its goals while low-valued situations might prevent the agent of fulfilling its task.
Mean prediction error: the mean prediction error among all states and actions. This evaluates the accuracy of the agent’s world model, i.e., how well can the agent predict the consequences and future value of its actions in most situations.
pairs that have associated a significantly higher/lower mean
prediction error. These are situations that are very hard (or easy) for
the agent to learn. Together with transition uncertainty, this is an
important element for explanation as people use social attribution
to determine causes and judge others’ behavior [
This level refers to analyses combining information from the diferent interaction data and the previous levels. 2.4.1 Transition Value Analysis. This analysis combines information from the agent’s estimated V function and the transition function Pˆ (z′ | z, a). The goal is to analyze how the value attributed to some observation changes with regards to possible observations taken at the next time-step. The following elements are produced: Local minima/maxima: refers to observations whose values are greater/lower than or equal to the values of all possible next observations. These help explain the desirability attributed by the agent to a given situation. Specifically, local maxima denote subgoals or acquired preferences—e.g., this may help explain situations in which the agent prefers to remain in the same state rather than explore the surrounding environment. In contrast, local minima denote highly-undesirable situations that the agent will want to avoid and in which typically any action leading to a diferent state is preferable.
Observation variance outliers: correspond to observations where
the variance of the diference in value to possible next observations
is significantly higher or lower. This element is important to identify
highly-unpredictable and especially risky situations, i.e., in which
executing actions might lead to either lower- or higher-valued next
states.
2.4.2 Sequence Analysis. This analysis combines information from
the analyses of observation frequencies, transitions and values. The
goal is to extract common and relevant sequences of actions from
to the agent’s interaction with its environment. In particular,
interesting sequences involve starting from important observations
identified by the other analyses, then executing the most likely
action, and henceforth performing actions until reaching a local
maximum. To discover sequences between observations we first
used the information stored in Pˆ to create a graph where nodes are
observations and edges are the actions denoting the observed
transitions. We then implemented a variant of Dijkstra’s algorithm [
Certain sequences to subgoal: these denote likely sequences
starting from an observation in the same set of sources used for
the previous element, and then performing actions until reaching a
subgoal. This element determines the agent’s typical or most likely
actions when in relevant situations. Thus, it can be used to
summarize its behavior in the environment, e.g., by distilling a visual
representation from the graph or visualizing a sequence of
observations. The sequence-finding procedure can also be used by the
user to query the agent about its future goals and behavior in any
possible situation. Notably, this can be used to provide contrastive
explanations which help reasoning about why the alternatives to
some actions—the foils—are not as desirable as those chosen by the
agent [
observations in which the actions’ selection distribution diverges from that of their values. We use the same technique as above to calculate the count-value JSD by using the data stored in n and Q. This element can identify situations where the agent’s actionselection mechanism contradicts what it has learned, e.g., by selecting more often actions in situations in which they have lower values. In turn, this could indicate one of several things: an inadequate action-selection mechanism; an unstable and hard-to-learn situation, where the value of an action changed throughout learning; that the agent has acquired a preference for an action but selected other actions with the same frequency during learning, which means that the agent has not started exploiting its learned knowledge.
Contradictory-goal observations: to identify these elements, we assume that the system is provided with domain-knowledge regarding goal states. These correspond to situations that would normally be considered as highly-desirable for the agent to perform the task by an external observer, e.g., collecting relevant items from the environment, reaching a new level in a game, etc. Based on this information, we determine which observations that were found to be subgoals for the agent (local maxima) are not in the known list of goals. The idea is to identify surprising situations in which the agent can justify its behavior by resorting to other interestingness elements. 3
Recent works within eXplainable RL (XRL) have recently started to
addressed some of the problems identified in this paper. Some
frameworks simply develop language templates to translate elements of
the problem into human-understandable explanations. Namely, the
work in [
We note that, overall, these works usually require a great deal of manual adjustments for specific domains while our framework relies on generic data that is already collected by standard RL algorithms and on a factored-state structure. Moreover, while some can summarize the agent’s behavior, they do not perform an analysis of the reasons of behavior and thus cannot provide insights about the agent’s decision-making. In addition, they lack the capability of automatically detecting situations requiring external human intervention. Finally, current XRL systems operate only after learning, making it hard to recover key aspects of the interaction. 4
We introduced a framework capable of analyzing a RL agent’s history of interaction with its environment. The framework operates at three distinct levels, first analyzing characteristics of the task that the agent has to solve, then the behavior of the agent while interacting with the environment, and finally by performing a metaanalysis combining information gathered at the lower levels. In general, the proposed analyses generate meaningful information from data that is already collected by standard RL algorithms, such as the Q and V functions generated by value-based methods, and state frequencies and expected rewards collected by model-based techniques. In addition, we proposed statistical data that can be easily collected by an RL agent while performing the task that helps further summarizing its history of interaction. We then detailed how, based on this interaction data, several interestingness elements can be generated by the analyses framework. 4.1
Explanation for autonomous agents requires determining not just
what to explain but also how and when. The interestingness
elements described here provide the content for explanation.
Throughout the paper, we provided insights on how each element can be
used to expose the agent’s behavior to a human user, justifications
regarding its decisions, situations in which input from the user
might be needed, etc. For most of the elements, explanations can
easily be converted from the data by coupling them with natural
language templates, similarly to what has been done in [
In an autonomy setting—particularly for learned autonomy— explanation should be capable of operating with diferent modes. As mentioned earlier, in a passive mode the agent constructs explanations in response to explicit queries from the user. For example, after training the agent in some task, the user may wish to validate the learning by asking the agent to analyze particular situations, its motivations, its foreseeable plans, etc. To that end, we can use the analyses proposed in this paper to summarize the learned policy, i.e., abstract a strategy, or identify the most important situations.
The explanation framework should also be able to operate in a
proactive mode, where the agent initiates explanation, whether to
avert surprise [
Being able to operate at diferent times is another desirable feature of an explanation framework. In that regard, we can use the proposed framework to perform a diferential analysis identifying how the interestingness elements change given two histories of interaction with an environment. By analyzing these changes, we can explain transformations of the agent’s behavior, changes in the environment, identify novel situations, acquired knowledge, etc. This analysis can be used during training to assess the agent’s learning progress. Another possibility is to compare between the agent’s behavior during and after training to identify which challenges were overcome after learning, and which situations remained confusing, uncertain or unpredictable. Finally, one can apply the diferential analysis to data captured by a novice/learning agent and an expert in the task. This can be useful to “debug” the agent’s behavior and identify its learning dificulties, assess how its acquired goals difer from those of the expert, etc.