We describe Explainable Artificial Intelligence (XAI) methods for image segmentation for autonomous driving applications. The analysis is conducted using metrics such as efficiency, robustness, localization, and complexity. Four XAI methods, namely Gradient-weighted Class Activation Mapping (GradCAM), Local Interpretable Model Agnostic Explanations (LIME), Feature Ablation, and Saliency are applied and assessed on a dataset of street images.
truth masks is Cityscapes [
To evaluate the segmentation of street elements, the pre-trained models DeeplabV3 and the
Fully Convolutional Network (FCN) both with ResNet101 architecture from PyTorch were
analyzed. These models were trained using part of the COCO val2017 dataset and the 20
segmentation classes from Pascal VOC. The decision to use pre-trained models was because
these 20 classes include relevant street elements such as bicycles, buses, cars, motorbikes,
people, and trains. The models were selected due to their high average Intersection over
Union (IoU) scores, which are listed on the PyTorch website [
For the evaluation of the segmentation models, two metrics were employed: the IoU value and the Matching Area. The evaluation, shown in Table 1, involved calculating the average values of 122 images from Zurich for both metrics.
Various XAI methods can be utilized. These methods are classified into two categories:
model-specific and model-agnostic. Model-specific methods analyze how changes in the
input features change the output, whereas model-agnostic methods work by manipulating
input data and analyzing the respective model predictions. Within the subclasses of specific
and agnostic, a further differentiation is made based on whether the method is local or
global. Additionally, local methods explain the individual predictions of models, while global
methods explain the behavior of the model [
Gradient-weighted Class Activation Mapping (GradCAM) [
Local Interpretable Model Agnostic Explanations (LIME) [
Feature Ablation [
Saliency [
In the implementation of XAI methods, an image representing a street scene was selected. The following two segmentation classes were examined: cars and persons. The calculated attribution values from the methods were normalized and a heatmap was generated and overlaid on the original image, as shown in Figure 1. While the original image had dimensions of 2048 x 1024 pixels, it was resized to 512 x 256 pixels to enhance computational speed.
Hedström et al. [
To measure the efficiency of a method, the runtime was recorded. The time taken to compute the attributions for the "Car" and "Person" classes was measured.
To calculate robustness, the average sensitivity [
= 1 1=1| − | (1) levels.
To obtain comparable values, the average sensitivity was calculated using the same method for the "Car" and "Person" classes. A low value of average sensitivity indicates good robustness, where 0 represents the lowest and 1 the highest possible value. Figure 2 illustrates an example of how the average attribution may appear at different brightness ensured when the average attribution remains equal to or only slightly deviates from the original.
Next, we consider the metric Relevance Rank Accuracy (RRA) [
The sparseness [
∑ =1 ( ) − (2)
Let be the indexing for the position of an attribution in an array, ( ) be the value of the attribution at position , and be the number of pixels in the image. The higher the Gini index, the better. A lower value indicates that the importance of all pixels is equal. Our images consist of different objects and only the pixels representing the object should be marked as important, which favors a higher G value.
In the evaluation, the metrics of average sensitivity, RRA and sparseness were calculated for the targets Car and Person. The target is a parameter that can be selected in every method. The first number in Table 2 represents the Car class, while the second number represents the Person class.
In terms of efficiency, GradCAM clearly outperforms all other XAI methods, followed by Saliency, then Feature Ablation and LIME. In terms of robustness, measured by average sensitivity, both GradCAM and Saliency show remarkable performance. In particular, Saliency shows almost no change in attribution values despite brightness variations. In the RRA and FP RRA category, Feature Ablation and LIME using Max(0,RRA - FP RRA) show identical top performance. The performance of Saliency in this category is considered significantly off target. The low, almost zero RRA means that Saliency only finds very few highly attributed pixels, present in the ground truth mask. In terms of the sparseness metric, the Saliency method emerges as the top performer. The other three methods show similar values that are significantly different from that of Saliency. Furthermore, it is observed that LIME and Feature Ablation exhibit similar or even identical values. The similarity may arise from the fact that both LIME and Feature Ablation are perturbation-based methods.
Additionally, it is notable that in most cases, the values from the category "Car" are better than those from the category "Person." One possible reason for the lower performance values in the "Person" class could be the downsizing of the image, as described in Chapter 3. Table 3 shows how the values of the Matching Area, which indicates how well the segmentation aligns with the ground truth mask, get worse when the image is resized. Another reason could be the general size of the objects in the image. A car, in comparison, has a significantly larger area than a person and is therefore easier to detect. In Figure 4, the measured metrics are visualized in a radar chart with best scores on the outer circle and worse scores in the inner area.
To evaluate the methods by the used metrics, we need to distinguish between their importance. Two of the most relevant metrics are shown in the lower part of the spider plot, namely the metrics RRA and FP RRA. They are a central aspect in the human interpretability, as they indicate in the heatmap (Figure 1) what is considered important. Due to the bad interpretability, we decide not to pursue Saliency further. Setting Saliency aside as a method, the three remaining methods can be ordered according to their performance: GradCAM demonstrates superiority, followed by Feature Ablation and LIME.
In summary, GradCAM emerges in this study as the first choice for evaluating XAI methods for image segmentation, not only because of its favorable metrics but also because of its interpretability in the heatmap. Figure 1 illustrates how GradCAM considers not only the objects themselves but also contextual features such as the road. The heatmap shows a gradient from orange (important) to red (very important), which is missing in other methods. The situation for Saliency was different: While it performed well in the metrics, its heatmap provided almost no information, making interpretation difficult. This highlights the importance of understanding and selecting a comprehensive set of metrics that can provide a clear understanding of the reliability of the methods. We acknowledge that the results are specifically tailored to this case and cannot be directly applied to other fields, such as medicine. Nevertheless, they represent a crucial first step toward future autonomous driving applications.
The current study is to the best of our knowledge the first one to evaluate a subset of XAI methods for image segmentation, an application area of computer vision that has not received as much attention as, e.g., image classification. Future research should involve a variety of data sets and segmentation models and prioritize the evaluation of a broader range of XAI methods for image segmentation. This research should include the development and application of various evaluation metrics, with a particular focus on interpretability. By optimizing these evaluation criteria, an understanding of the specific strengths and weaknesses of different XAI methods can be achieved, which eventually leads to the identification of the most suitable methods for specific applications.