Deep reinforcement learning (DRL) models have shown great promise in various applications, but their practical adoption in critical domains is limited due to their opaque decision-making processes. To address this challenge, explainable AI (XAI) techniques aim to enhance transparency and interpretability of black-box models. However, most current interpretable systems focus on supervised learning problems, leaving reinforcement learning relatively unexplored. This paper extends the work of PW-Net, an interpretable wrapper model for DRL agents inspired by image classification methodologies. We introduce Shared-PW-Net, an interpretable deep learning model that features a fully trainable prototype layer. Unlike PW-Net, Shared-PW-Net does not rely on pre-existing prototypes. Instead, it leverages the concept of ProtoPool to automatically learn general prototypes assigned to actions during training. Additionally, we propose a novel prototype initialization method that significantly improves the model's performance. Through extensive experimentation, we demonstrate that our Shared-PW-Net achieves the same reward performance as existing methods without requiring human intervention. Our model's fully trainable prototype layer, coupled with the innovative prototype initialization approach, contributes to a clearer and more interpretable decision-making process. The code for this work is publicly available for further exploration and applications.
Despite the successful and promising results of deep reinforcement learning (DRL) models over
the last few decades, their practical use in critical domains remains constrained due to their
unclear and enigmatic decision-making processes. To address this problem, explainable AI
(XAI) techniques aim to enhance the transparency and understandability of black-box models
[
Kenny et al. [
In this paper, we expand the work of Kenny et al. [
Overall, the contributions of this work are the following1: • we propose a fully trainable prototype layer that automatically learns prototypes without human supervision, achieving PW-Net reward results; • we apply the diferentiable prototype assignment mechanism to each class, previously introduced in ProtoPool, in order to enhance prototype sharing and network eficiency; • we propose a new prototype initialization based on the relevance of certain actions to certain states.
The remainder of this work is organized as follows: Section 2 presents the most related literature to our work; in Section 3 we provide a general background about reinforcement learning and prototype-based models’ architectures; Section 4 introduces our proposed approach in detail; in Section 5 we carry out the experimental setup and the analysis of the obtained results; Section 6 wraps up the results of this paper by analyzing its limitations and setting the base for future work.
XAI is a rapidly evolving field that aims to develop artificial intelligence systems that can provide
human-understandable explanations for their decisions and actions. One approach within XAI
is the development of self-interpretable models [
Inspired by ProtoPNet, Rymarczyk et al. [
In the field of XRL, some works propose post-hoc methods involving attention weights
[
In the context of this study, which combines RL and self-interpretable XAI models, highly
significant work in the field of RL is the research conducted by Kenny et al. [
Incorporating the concepts and methodologies proposed in the aforementioned works [
Reinforcement Learning (RL) is a prominent subfield of machine learning that focuses on teaching autonomous agents how to perceive and interpret their environment in order to take actions. Unlike supervised learning, in fact, where labeled examples are provided, or unsupervised learning, which seeks patterns in unlabeled data, RL focuses on the concept of an agent which interacts with an environment, receives feedback in the form of rewards or penalties, and learns to behave trying to maximize cumulative rewards.
RL relies on the concept of Markov Decision Process (MDP), a stochastic decision-making
process that uses a mathematical framework to model the decision-making of a dynamic
system. In an MDP, defined by the (, , , , ) tuple [
that maximizes the expected discounted reward and serves as a strategic mapping from states to actions. By continuously exploring and exploiting the environment, the agent aims to refine its understanding of the state-action relationships, seeking the most advantageous course of action at any given state.
In this study, we deal with the so-called “black-box policies”, pre-trained neural networks’ models that operate without providing clear and transparent explanations for their decisionmaking process.
equal to 1.
a specific patch of the image:
considering the distribution :
ProtoPool [
where the values in indicate the probabilities of assigning successive prototypes to the -th slot, with the sum of these probabilities ‖ ‖ that is Given an input image , () is a set of ×
vectors of dimension , each corresponding to = { ∈ () ∶ ∈ ℝ = 1, … , ⋅ }.
Then, for each slot, calculates the aggregated similarity between and all prototypes where is a novel focal similarity function: =1 = ∑
⋅ , () = log(1 +
1 ‖ − ‖ 2 ), ∈ = max () − mean∈ ().
Finally, in ℎ the similarity scores per class are multiplied by the weight matrix ℎ to obtain the output logits that then undergo a softmax transformation to produce the prediction: =̂ softmax ( ℎ ⋅ ).
The soft prototypes’ assignment has two main constraints: first, they assign one prototype
per slot, and second, they assign successive slots of a class to diferent prototypes. The first
constraint is satisfied by the use of the diferentiable Gumbel-Softmax estimator [
Gumbel-softmax (, ) = ( 1, … , ) ∈ , (1) (2) (3) (4) (5) (6) where = distribution. function orth: 3.2.2. PW-Net exp( + )
∑=1 exp( + )
representation , : , and the prototype , : .
Regarding the second constraint, the slots’ orthogonality is enforced by the orthogonal loss orth = ∑
⟨ , ⟩ < ‖ ‖2 ⋅ ‖ ‖2 .
PW-Net [
) and two human-defined layers, the prototype layer and the weight matrix ′. Initially, a state is mapped to a latent representation through the encoder network: = enc().
Then, for each action and prototype a separate linear layer ℎ, transforms z to a specific
and for ∈ 1, … ,
are samples drawn from the Gumbel
(7)
(8)
(9)
(10)
(11)
Successively, the similarity function from Chen et al. [
In the end, output actions are generated by combining the similarity scores with ′: =1 ′ = ∑ ,′ sim( , , , ).
Inspired by the work of Kenny et al. [
In Figure 1, we report a scheme of Shared-PW-Net in the Car Racing environment. The network
consists of a enc stage, a projection network , a prototype layer , and a fully connected layer
. As in Kenny et al. [
The network takes the RL agent’s current state , already fed in enc, and passes it through to , which projects it in the latent space: (12) (13) (14) = enc(), ′ = () sim( ′, , ) = log ( ( ′ − , )2 + 1 ( ′ − , )2 + ) .
Successively, the network performs the similarity function from Chen et al. [
After this, the similarity score is fed into and then to the output activation function corresponding to that particular action in order to obtain the predicted action ′. = ⋅
sim( ′, , ), ′ = (), (15) (16) (17) where is the general activation function. In the specific case of the Car Racing domain (see Section 5.1.1), for the steering action we use the Tanh activation function because we suppose that the network, through the fully connected layer, learns that the value +1 is associated to ”Turn right” action and the value -1 is associated to ”Turn left” action. For braking and accelerate actions, instead, we employ the ReLU activation function because we want a unique value for expressing the intensity of the actions.
Apart from attempting the conventional initialization of prototypes with random numbers drawn from a standard distribution, we also investigate a novel prototype initialization technique. This novel approach involves a clustering operation on all states from which the pre-trained RL agent executes a specific action. In detail, given the list of states and actions of the dataset , for each action, we perform a clustering on the states where the black-box model performs that action. In particular, we use the number of slots as the number of clusters in order to have at least prototypes for each action. This procedure allows us to extract ⋅ centroids of the clusters. We select the first centroids, and we use them as prototypes.
In the specific case of the continuous action space for the Car Racing environment, at each step the agent performs the action ∈ 3 where each of the three components is the continuous action: ⎨ ⎩braking ⎧steering ∈ [−1, +1] accelerate ∈ [0, maxAcc)
∈ [0, maxBrake)
This indicates that the agent, for example, can steer to the right and brake simultaneously. For this reason, for the prototype initialization, we simply find the mean value of each action performed by the RL agent in . Then, for each action, we collect the set of states where that action is executed with an absolute value that is greater than . In the end, we use the K-means clustering to identify the centroids and we use them as prototypes.
As already proposed and studied by Rymarczyk et al. [
Equation 6 shows the computations performed to ensure diferentiability for each slot. At training time, we initialize the Gumbel-Softmax distribution with = 1 and gradually reduce it to 0.001 over 30 epochs. Moreover, we extend the loss function to enforce orthogonality between prototypes assigned to the same action. This orthogonality constraint (previously shown in Equation 7) prevents multiple slots from being assigned to the same prototype, maximizing the potential of the prototype pool layer.
Prototype projection is an essential stage in the training process, as it aids in visualizing the learned prototypes and helps in better understanding the features they represent. In this step, abstract prototypes, which the model learns during training, are replaced with the actual representations of the nearest training data. By using the representations of actual training data, the model’s prototypes become more interpretable, allowing one to gain insights into the learned features and their relevance to the task. We perform prototype projection every two epochs, only after the first 10 epochs.
In this study, we investigate how Shared-PW-Net performs compared to its counterpart, PW-Net, and the black-box agents in diferent DRL settings. We conduct tests in two diferent types of environments: first, we assess Shared-PW-Net performance in a DRL domain with multiple continuous output actions, and then, we conduct tests in a diferent domain with discrete output actions. These two experiments use a state space solely represented by raw pixel data; however, PW-Net could also be applied to other state-action representations, such as tabular environments. Finally, we study the influence of the proposed initialization procedure and the role of the number of prototypes and slots of the model.
In this section we present the setup we use to carry out the experiments. In particular, we
evaluate each model on 30 episodes for 15 seeds. We compare our proposed method with
PW-Net [
We test our model in Car Racing continuous action domain from OpenAI’s gym environment
[
For the discrete action domain, we use Atari Pong (from OpenAI’s gym environment). In this
environment, the agent has to control a paddle and try to hit a ball back and forth, with the
goal of preventing the ball from going past its paddle. The agent is rewarded for successfully
scoring against the opponent, while it incurs penalties for being scored against. The agent’s
discrete actions are 6, each allowing the agent to move up or down. We use the pre-trained
model from Şentürk [
Table 1 illustrates the results achieved for the Car Racing and Atari Pong environments. We report the mean and standard deviation values over the diferent seeds for statistical significance. In the following paragraphs, we comment on the results obtained on the two environments.
Concerning the Car Racing environment, we evaluate the performance of each model using the average reward of the RL agent and the mean squared error (MSE) between the interpretable model’s actions and the black-box’ ones. In Table 1 we observe that Shared-PW-Net succeeds in reaching the same reward levels of PW-Net (and the black-box agent) even if it is less accurate. In light of this, Shared-PW-Net represents a valid alternative and a workable solution for setting up a self-interpretable model that does not rely on human intervention. The two straightforward modifications of PW-Net with trainable prototypes, PW-Net* and PW-Net**, on the other hand, do not achieve remarkable results. This leads to demonstrating that the application of soft assignment of prototypes to classes and the utilization of the new prototype initialization (adopted in Shared-PW-Net) are fundamental for achieving significant results.
In this domain, we use the reward of the RL agent and the accuracy of the model with respect to the black-box to evaluate the agent’s performance in the environment. Also in the Atari Pong environment, the Shared-PW-Net outperforms PW-Net* and PW-Net** and successfully attains the same rewards values of PW-Net and the black-box model (Table 1).
To study the efect of the prototype number and initialization, we test various hyperparameter combinations and report the result in reward and distillation performances (Table 2).
For Car Racing, we test the following combinations: (i) 4 prototypes and two slots; (ii) 6 prototypes and two slots; (iii) 6 prototypes and three slots; (iv) 8 prototypes and three slots. We ifnd the model (ii) to be the best performing. The choice in the number of prototypes and slots is guided by the number of classes (actions) within the Car Racing environment. Moreover, for each model, we test both the random initialization and the one proposed in Section 4.1.2. We see that all the models trained with the novel prototype initialization perform better than the randomly initialized ones. Moreover, while we find consistent diferences in performances for the randomly initialized models, all the models using the novel initialization reach comparable performances.
The model exhibits consistently high performance in Atari Pong across diferent hyperparameter variations, notably concerning the number of prototypes and slots. While the novel initialization technique shows success in some instances, it is the Shared-PW-Net configuration, featuring eight prototypes and two slots per action, that attains the highest reward value. In this particular setup, the novel initialization proves to be the key factor in achieving the best model performance.
In this section, we show how to obtain insights on the reasoning process of Shared-PWNet. In particular, in Figure 2, we report an illustration of the prototype assignment matrix learned by the model with 6 prototypes and 3 slots per action. We see that for the Steering action, the Shared-PW-Net successfully learns to diferentiate between right ( 4 , 5 ) and left (6 ) directions. Moreover, the model associates prototypes 3 , 1 , 6 to the Accelerate action, identifying with a straightforward road (or a road without pronounced curves) those situations in which acceleration is possible. The same logical reasoning can be observed for the third class, the Braking action, to which the model associates prototypes 4 , 2 , 5 , revealing that the time to decelerate is when the car is approaching a curve. Such observations contribute to a more comprehensive understanding of the model’s decision-making process. Examining the prototypes and their assignment gives us valuable insights into how the model perceives diferent actions and scenarios within the car racing environment. This analysis of the Shared-PW-Net’s interpretability sheds light on its strengths and limitations, providing crucial information to enhance the model’s performance and adaptability in varying real-world scenarios.
This paper introduces a novel prototype-based neural network that builds upon the foundational work of PW-Net. Our proposed model is an interpretable wrapper for any pre-trained reinforcement learning agent, enabling a deeper understanding of its decision-making process. By leveraging prototypes assigned to actions, our network achieves comparable reward performances with respect to the state-of-the-art in two environments, without the need for human intervention. Indeed, the use of trainable prototypes eliminates the need for humandefined prototypes, enhancing the model’s autonomy and adaptability. Although this study demonstrates the eficacy of behavior cloning on a pre-trained RL agent, showcasing promising results in various scenarios, it is essential to acknowledge the inherent limitation of relying on a pre-existing agent for this approach. While leveraging a pre-trained agent ofers certain advantages, such as reduced training time and access to valuable learned knowledge, it can also introduce constraints that limit the full exploration of the model’s capabilities. With respect to the introduction of the novel prototype initialization method, it has shown its efectiveness as the results indicate an improvement in the reward values when it is utilized, compared to the cases where a simple random prototypes initialization is employed.
As a natural progression of this research, a future direction could be to avoid using behavior cloning or distillation as the training method for our prototype-based neural network in order to allow the model to learn from data directly, potentially reducing the introduction of biased behaviors and increasingly improving performance from an interpretable perspective. Furthermore, the exploration of additional environments (which is ongoing) could provide an additional validation to this work.
Overall, our contribution expands the frontiers of interpretable reinforcement learning by presenting a powerful and versatile framework that enhances the interpretability of pre-trained RL agents. By empowering researchers and practitioners to gain deeper insights into agent behavior, our prototype-based neural network opens new avenues for improving the transparency and trustworthiness of RL applications. Furthermore, this framework could hold significant promise in real-world applications such as autonomous driving because it could help passengers trust the vehicle’s decision-making processes by providing clear explanations for the car’s driving choices and by enhancing safety and accountability in autonomous driving systems. We believe our work provides a strong foundation for interpretable DRL in real-world applications, where ethics, as seen in contexts like autonomous driving, plays a crucial role in responsible decision-making and trustworthiness.