Computer vision through machine learning is a relevant topic in the field of autonomous driving. Within the field of computer vision, there are various tasks such as image classification, object detection, semantic segmentation, and instance segmentation. The focus of this paper lies in the evaluation of Explainable Artificial Intelligence (XAI) methods on segmentation models. Because we want to use the segmentation models for autonomous driving in the future, we focused on images showing street scenes. The current model quality indicates that there is a lot of further work necessary before their implementation in a real-world application.
The methods refer to algorithms that are included in the Captum package [1]. For the evaluation of XAI methods, four metrics were used, three of which originate from the categories of complexity, robustness, and localization as listed in Quantus [2]. Complexity is based on sparseness, robustness on average sensitivity, and localization on Relevance Rank Accuracy (RRA) and accordingly the False Positive Rate of RRA. The fourth metric, efficiency, was measured based on runtime.
To apply and evaluate XAI methods, it is necessary to have images and a model capable of segmenting images. A publicly available dataset that provides street images with ground truth masks is Cityscapes [3]. Images from Zurich, Switzerland, were selected from this dataset. For the segmentation, pre-trained models from PyTorch were used [4].
To evaluate the segmentation of street elements, the pre-trained models DeeplabV3 and the Fully Convolutional Network (FCN) both with ResNet101 architecture from PyTorch were analyzed. These models were trained using part of the COCO val2017 dataset and the 20 segmentation classes from Pascal VOC. The decision to use pre-trained models was because these 20 classes include relevant street elements such as bicycles, buses, cars, motorbikes, people, and trains. The models were selected due to their high average Intersection over Union (IoU) scores, which are listed on the PyTorch website [4].
For the evaluation of the segmentation models, two metrics were employed: the IoU value and the Matching Area. The evaluation, shown in Table 1, involved calculating the average values of 122 images from Zurich for both metrics.
A conclusion drawn from the results of this table is that both the Deeplabv3 and FCN models demonstrate similar performances in segmenting road-related classes. Specifically, the "Car" and "Person" classes tend to exhibit the best results in segmentation for both models. Therefore, only these two classes will be used for further evaluations.
Various XAI methods can be utilized. methods explain the behavior of the model [5]. In this evaluation, a total of four XAI methods are applied. The selection of methods was partially based on Munn and Pitman [6]. The authors dedicated a chapter in their book to the topic of explainability for image data. In this chapter, the methods, GradCAM and LIME, were introduced, which was one reason for choosing these two methods. Two further methods were selected: Feature Ablation and Saliency.
Gradient-weighted Class Activation Mapping (GradCAM) [7] is a technique that analyzes gradient information for any convolutional layer of a model and generates a heatmap that highlights important regions in the image.
Local Interpretable Model Agnostic Explanations (LIME) [8] involves multiple iterations of removing specific regions of an image to determine which specific areas are more or less important. It is a model-agnostic, perturbation-based method.
Feature Ablation [9] is also a perturbation-based method. It calculates the difference of the attribution in the model output of each feature when it is active and when it is replaced. Multiple features can be turned off together instead of one at a time.
Saliency [10] is a method that follows the gradient of a class through the model using backpropagation. During this process, each pixel is minimally changed, and the resulting changes are observed in the prediction of the class score.
In the implementation of XAI methods, an image representing a street scene was selected. The following two segmentation classes were examined: cars and persons. The calculated attribution values from the methods were normalized and a heatmap was generated and overlaid on the original image, as shown in Figure 1. While the original image had dimensions of 2048 x 1024 pixels, it was resized to 512 x 256 pixels to enhance computational speed.
HedstrΓΆm et al. [2] compare various XAI libraries of evaluation metrics for XAI methods. Metrics were divided into six categories: Faithfulness, Robustness, Localisation, Complexity, Axiomatic and Randomisation. Three metrics based on these categories were used to evaluate XAI methods and an additional efficiency metric was employed. The selection of the following metrics was guided by the idea to test and understand metrics from different categories but else chosen somewhat arbitrary.
To measure the efficiency of a method, the runtime was recorded. The time taken to compute the attributions for the "Car" and "Person" classes was measured.
To calculate robustness, the average sensitivity [11] was applied. Zhou et al. [12] describe that models usually do not adapt well to new environments when new factors such as weather or illumination conditions are introduced. For this reason, the original images were modified to various brightness levels while preserving the objects in the image. For calculating the average sensitivity, the following formula (1) was used:
To obtain comparable values, the average sensitivity was calculated using the same method for the "Car" and "Person" classes. A low value of average sensitivity indicates good robustness, where 0 represents the lowest and 1 the highest possible value. Figure 2 illustrates an example of how the average attribution may appear at different brightness levels. Next, we consider the metric Relevance Rank Accuracy (RRA) [13], which falls in the category of Localization. This metric measures how many of the highly attributed pixels are located within the ground truth mask. In our evaluation, highly attributed pixels are defined as those falling within the top 20% range. This means when attributions range from 0 to 1, values from 0.8 to 1 are marked as highly attributed. An example of calculating the RRA is illustrated in Figure 3. Not only is the RRA an important indicator, but also the False Positive (FP) Rate of the RRA. A method might represent all pixels in the image as highly important, resulting in an RRA of 1, which is the highest possible value. Therefore, to achieve more comparable results, the FP RRA was additionally examined. The sparseness [14] was measured as the last metric, falling in the category of Complexity. Here, the Gini index (G) is used for calculation. In our case, G indicates how scattered or concentrated the attributions are distributed in an image. A value of 0 (the smallest value) indicates a large dispersion, where each pixel is crucial for segmentation. A value of 1 (the largest value) indicates that the important attributions are concentrated. For calculating the Gini index, the following formula (2) was used [15]:
Let π be the indexing for the position of an attribution in an array, π₯(π) be the value of the attribution at position π, and π be the number of pixels in the image. The higher the Gini index, the better. A lower value indicates that the importance of all pixels is equal. Our images consist of different objects and only the pixels representing the object should be marked as important, which favors a higher G value.
In the evaluation, the metrics of average sensitivity, RRA and sparseness were calculated for the targets Car and Person. The target is a parameter that can be selected in every method. The first number in Table 2 represents the Car class, while the second number represents the Person class.
Evaluation of the methods GradCAM, Feature Ablation, Saliency and LIME on the metrics. The arrow direction indicates whether higher or lower results are considered better.
In terms of efficiency, GradCAM clearly outperforms all other XAI methods, followed by Saliency, then Feature Ablation and LIME. In terms of robustness, measured by average sensitivity, both GradCAM and Saliency show performance. In particular, Saliency shows almost no change in attribution values despite brightness variations. In the RRA and FP RRA category, Feature Ablation and LIME using Max(0,RRA -FP RRA) show identical top performance. The performance of Saliency in this category is considered significantly off target. The low, almost zero RRA means that Saliency only finds very few highly attributed pixels, present in the ground truth mask. In terms of the sparseness metric, the Saliency method emerges as the top performer. The other three methods show similar values that are significantly different from that of Saliency. Furthermore, it is observed that LIME and Feature Ablation exhibit similar or even identical values. The similarity may arise from the fact that both LIME and Feature Ablation are perturbation-based methods.
Additionally, it is notable that in most cases, the values from the category "Car" are better than those from the category "Person." One possible reason for the lower performance values in the "Person" class could be the downsizing of the image, as described in Chapter 3. Table 3 shows how the values of the Matching Area, which indicates how well the segmentation aligns with the ground truth mask, get worse when the image is resized. Another reason could be the general size of the objects in the image. A car, in comparison, has a significantly larger area than a person and is therefore easier to detect.
In Figure 4, the measured metrics are visualized in a radar chart with best scores on the outer circle and worse scores in the inner area. To evaluate the methods by the used metrics, we need to distinguish between their importance. Two of the most relevant metrics are shown in the lower part of the spider plot, namely the metrics RRA and FP RRA. They are a central aspect in the human interpretability, as they indicate in the heatmap (Figure 1) what is considered important. Due to the bad interpretability, we decide not to pursue Saliency further. Setting Saliency aside as a method, the three remaining methods can be ordered according to their performance: GradCAM demonstrates superiority, followed by Feature Ablation and LIME.
In summary, GradCAM emerges in this study as the first choice for evaluating XAI methods for image segmentation, not only because of its favorable metrics but also because of its interpretability in the heatmap. Figure 1 illustrates how GradCAM considers not only the objects themselves but also contextual features such as the road. The heatmap shows a gradient from orange (important) to red (very important), which is missing in other methods. The situation for Saliency was different: While it performed well in the metrics, its heatmap provided almost no information, making interpretation difficult. This highlights the importance of understanding and selecting a comprehensive set of metrics that can provide a clear understanding of the reliability of the methods. We acknowledge that the results are specifically tailored to this case and cannot be directly applied to other fields, such as medicine. Nevertheless, they represent a crucial first step toward future autonomous driving applications.
The current study is to the best of our knowledge the first one to evaluate a subset of XAI methods for image segmentation, an application area of computer vision that has not received as much attention as, e.g., image classification. Future research should involve a variety of data sets and segmentation models and prioritize the evaluation of a broader range of XAI methods for image segmentation. This research should include the development and application of various evaluation metrics, with a particular focus on interpretability. By optimizing these evaluation criteria, an understanding of the specific strengths and weaknesses of different XAI methods can be achieved, which eventually leads to the identification of the most suitable methods for specific applications.