Robots in social environments raise the requirement for explainability of robot behavior [1]. As the tendency of robots' presence in society grows, this requirement becomes more pronounced. The introduction of the "Right to explanation" [2] in the European Union as a part of the General Data Protection Regulation (GDPR) [3] underlines the human right to explanation in the face of machines making decisions that affect humans. Current decision-making methods in robotics largely lack explainability and thus limit the faster adoption of robots in important tasks. A lack of explainability can also become a safety issue when robots behave unexpectedly, putting humans in highly sensitive environments at risk.
We address explainability in robotics by focusing on explainable robot navigation in social environments: Imagine a robot navigating in a known environment with the possibility of encountering humans and obstacles. Local path planners allow robots to follow a global path plan while dynamically reacting to unexpected occurrences. Some of the robot's decisions may require abrupt stops or changes of direction and path deviations, thus surprising people in the neighborhood or even scaring them. This can lead to trust loss, which needs to be mitigated [4]. One mitigation strategy is explanation. We want to mitigate trust loss by enabling robots to explain their navigational choices. Using Local Interpretable Model-agnostic Explanations (LIME) [5], an established method from Explainable Artificial Intelligence (XAI) [6], we demonstrate how a robot can generate visual explanations of its local decision-making in path planning and obstacle avoidance. To approach explanation generation in real-time, we train a Generative Adversarial Network (GAN) [7] model on a dataset produced by LIME. We demonstrate how the trained GAN model generates visual explanations of local path plans.
LIME [5] is a model-agnostic local XAI technique that explains predictions of a black-box model by learning an interpretable model around the instance of interest. The instance of interest can be anything that is an input to an AI model, be it text, numerical data, or images. We focus on visual explanations and use an image (viz., the local costmap, see below) as the instance of interest. LIME for images 1 takes the input image and partitions it into segments -superpixels, thereby creating interpretable features. Then, it perturbs interpretable features, turning them off to generate perturbed samples (perturbations) in the neighborhood of the instance of interest. For every perturbation, LIME queries the black-box model and thereby generates a local data set of (perturbed) neighbor images and the respective black-box model's predictions. On this new dataset, LIME trains an interpretable model, viz., a weighted linear regression model. The explanation is obtained by interpreting the coefficients of the trained linear model: The importance of each segment in the image for the behavior of the black-box model is represented by one coefficient in the linear model. Depending on the sign of the coefficient, the interpretable feature (viz., the segment in the image) positively or negatively affects the black-box model's prediction. That said, applying LIME to explain local navigation visually, one needs to provide a suitable method for computing a segmentation of a local costmap (viz., the interpretable features). Moreover, the output of the local planner has to be interpreted as the prediction of some black-box model.
GANs were introduced by Ian Goodfellow et al. [7] as a deep learning framework for the estimation of generative models. Estimation is done by an adversarial process where a generative model, Generator (G), and a discriminative model, Discriminator (D), are trained concurrently. G generates new samples by learning the training data distribution, while D estimates whether the provided sample is from the training data or is produced by G. Mehdi and Osindero [8] introduced conditional GAN (cGAN), where G and D are conditioned on some information. Isola and colleagues [9] show how cGAN can be used for image-to-image translation by conditioning on images. G is trained to learn translation between input and output images and fool D, while D learns to classify output images as real (coming from the training dataset) or fake (generated by G). In our work, we employ their pix2pix cGAN architecture 2 to achieve a fast explanation generation of local navigation decisions. D is trained by minimizing the negative log-likelihood of identifying real and fake images conditioned on input images, while G is trained using the adversarial loss of D (whether it fools the discriminator or not) and L1 loss (mean absolute per pixel difference between real and fake images) which are combined into a composite loss function. We condition G and D on local costmaps (see Fig. 1b,1f,1j) as inputs and explanation images (LIME outputs) (see Fig. 1c,1g,1k) as outputs. Both input and output images include (besides obstacle information) the robot's location, the local plan, and the global plan.
Our set-up is situated in the context of the ROS navigation stack [10]. A global path planner has generated a global path plan for the robot to navigate to a specified goal position. For path following and obstacle avoidance, a local path planner takes the local costmap and the global path as input and outputs a local path (in terms of a velocity vector) for the robot to execute. For LIME to be applicable, the black-box behavior needs to be deterministic. Therefore, we do not employ sampling-based planners, such as DWA or RRT, but instead, employ the TEB planner [11]. To use LIME for generating visual explanations of local path plans in terms of obstacles in the local neighborhood of the robot, we use the local costmap as an instance of interest and the TEB planner as the black box that takes that costmap as input and outputs some local path. LIME first segments the local costmap into obstacles as interpretable features. The SLIC [12] segmentation algorithm is used to get obstacle segments. As the second step, LIME obtains perturbations of the segmented costmap by turning off segments by replacing them with free space. The perturbed local costmap, together with the global plan, the robot's footprint, and its current velocities, form the input to the TEB, which then outputs a local plan for the perturbation at hand. The deviation of the so-calculated local plan from the global plan is taken as a target for the interpretable model and is calculated as a sum of the minimal point-to-point L2 differences between the local and the global plan.
We get an explanation image for each local navigation decision by coloring segments based on their LIME coefficients. The sign of the coefficient dictates the color: Positive-weighted segments are colored green, and negative-weighted segments are colored red. A green-colored segment contributed positively to the deviation; that is, green indicates "without that segment, the local plan would deviate less from the global plan". Conversely, a red-colored segment indicates "without that segment, the local plan would deviate more from the global plan". Color intensity is set proportional to the coefficient with intensities in the range [0, 255] in RGB color space.
Figures 1a, 1e, and 1i show three characteristic local navigation cases (C1, C2, and C3) in our lab. The robot (a TIAGo from PAL robotics) tries to follow the global plan that leads it through the doorway. Figure 1b, 1f, and 1j show the local costmaps for three local navigation cases with black obstacles, white robot's location, and grey free space. In C1, the local plan (yellow dots) mostly coincides with the global plan (blue dots), while in C2, the starting and ending points of the local plan could not be connected into a joint trajectory. In C3, the local plan deviates from the global plan. The LIME explanation explains how obstacles and/or parts of obstacles contribute to the deviation. From the explanation images 1c, 1g, and 1k we have:
• C1: "The right (green) wall segment increases deviation, while the left (red) wall segment decreases it, squeezing it to the doorway. " • C2: "The (green) obstacle increases the deviation because if it were not there, the robot would follow the global plan. If the wall (red) were not there, the local path planner could create the connected local plan and deviate from the global plan. " • C3: "Both obstacles increase the deviation, but the round one does so more significantly.
If it were not there, the robot would follow the global plan. If the rectangular obstacle were not there, the robot would still deviate, but less. "
We analyze explanation runtime. LIME's runtime is generally high and increases linearly in the number of perturbations as shown in Fig. 2a, where the runtimes of the most important parts of the LIME are plotted. Planner total time takes the biggest part of the total explanation runtime and includes the preparation of input data for the planner (TEB), the planner calculation time of all the paths for each perturbation, and the collection of the planner's outputs. The planner calculation time takes the biggest part of the planner total time. Both runtimes increase relative to the increase in the number of perturbations. As segmentation only needs to be done once for each explanation, its runtime is unaffected by the number of perturbations. LIME has clear limitations in that this method alone cannot be used for real-time explanations. Fast-changing and socially complex environments like streets or places with people might require real-time explanations. Even when using a small number of perturbations (which affects the explanation quality), not every TEB call (every 200ms) can be explained in real time.
The first experiment showed that LIME can be used to generate meaningful visual explanations but that the generation procedure is too slow for online usage. To approach explanation generation in real time, we utilize GAN as an explanation method. Our main idea is to use LIME only offline to generate a dataset of pairs of local costmaps and respective explanations. With this dataset, we train GAN for image-to-image translation. This way, explanation generation becomes independent of the number of samples.
We have trained an image-to-image GAN for 200 epochs with 240 training image pairs, 60 validation image pairs, and 60 test image pairs. The image-pairs dataset was generated using LIME with the configuration as outlined in the description of our first experiment; see Section 3.1. Of the important GAN settings, resnet_9blocks is used as Generator architecture. Other settings are kept as in the pix2pix standard implementation. The trained GAN model generates an explanation image by taking a local costmap with a plotted robot's position and local and global plans as input. In the following, we refer to the trained GAN model simply as GAN.
We assess the quality of the visual explanations generated by the GAN by human visual examination. This is a recommended practice [13,14]. Figures 1d, 1h, and 1l show GAN explanations for the use-cases C1, C2, and C3, respectively. One can see distortions in the GAN explanations, which, however, do not do any harm to the conveyed meaning. In C1, one can see that the colors of colored segments are not as sharp as in LIME, but the explanation does not suffer qualitatively. GAN's explanation for C2 has similar properties, with even somewhat different coloring of less important segments on the right wall. Still, this does not hamper the explanation very much, as the main contributors are still clearly distinguished. In the GAN explanation for C3, the green color is somewhat duller and blurred compared to LIME, but the contributions of the segments are still visible. We report a mean GAN calculation runtime of 0.25 seconds and a mean GAN model loading runtime of 0.36 seconds. Hence, once the GAN model is loaded, it can output four explanations per second.
We demonstrate how GAN explanations can be visualized in Rviz in real-time in Fig. 2b. The GAN output is published as PointCloud2 and overlayed over the map view in RViz as a local explanation layer. The GAN model is loaded once at the beginning of navigation and called periodically with every new local plan produced by TEB, allowing for the local explanation layer refresh frequency of 4Hz. This tool thus enables humans to observe which parts of the environment the robot considers important for its navigational decisions. We envision the tool to be used for inspection and debugging, teaching path planning, and demonstrating the robot's internal reasoning processes to interested laymen.
GAN achieves huge runtime savings compared to LIME and approaches the upper real-time performance limit of 200 ms. Most importantly, explanations generated by GAN do not depend on any image segmentation preprocessing, and the performance-hungry process of replanning the local path for every input image perturbation is no longer needed. This translates to the possibility of achieving explanations in near real-time even when many obstacles are considered potential explanations. This allows for explanation generation in highly dynamic environments.
One drawback of the GAN model is some distortions in the visual explanation. However, these are not too harmful as they are local and do not significantly affect the color and shade of color. A limitation of our work is that we have not systematically analyzed how well the GAN explanations generalize to very complex environments. The GAN explanation procedure does not make assumptions about the robot platform and its kino-dynamic constraints. It also does not assume a specific underlying local planner. It generates the explanation only based on an image containing the local obstacles along with the local plan and the global plan. Thus, it
may turn out that the GAN has to be retrained for every robotic platform. Another limitation is that our explanation approach relies on the underlying local planner to be deterministic. This is necessary because the procedure must be certain that a variation in the local path is due to the obstacles in the surroundings rather than random fluctuations. In the future, we will also investigate how non-deterministic path planners could be explained.