Deep Neural Networks (DNNs) are successful but work as black-boxes. Elucidating their inner workings is crucial as DNNs are prone to reproducing data biases and potentially harm underrepresented or historically discriminated demographic groups. In this work, we demonstrate an approach for visualizing DNN activations that facilitates to visually detect biases in learned representations. This approach displays activations as topographic maps, similar to common visualization of brain activity. In addition to visual inspection of activations, we evaluate diferent measures to quantify the quality of the topographic maps. With visualization and measurement of quality, we provide qualitative and quantitative means for investigating bias in representations and demonstrate this for activations of a pre-trained image recognition model when processing images of peoples' faces. We find biases for diferent sensitive variables, particularly in deeper layers of the investigated DNN, and support the subjective evaluation with a quantitative measure of visual quality.
Deep Neural Networks (DNNs) are highly successful but it is dificult to interpret how they
perform their learned task [
Feature visualization explains learned patterns by creating inputs that
maximally activate particular filters [
In this section, we introduce our approach of visualizing DNN activity as topographic maps.
An implementation is available at https://github.com/valeriekrug/ANN-topomaps.
Group-Specific Activations We use an averaging approach to characterize DNN activity
for groups of examples [
Topographic Activation Map Inspired by how brain activity is displayed as topographic
maps, we map DNN neurons to allow for a similar activation visualization [
) (, ) = 15 ⋅ −((,)/2) (1) where = Euclidean distance of particle coordinates and apply it for 1000 iterations.
Finally, we visualize the NAP in the computed layout, mapping the values to a 0-symmetric continuous color scale from blue over white to red. This way, equal colors represent the same value in each group. Then, we linearly interpolate the colors. In Convolutional Neural Networks (CNNs), we compute the layout such that each feature map is assigned a position in a 2D space and that similar feature maps are close to each other. To assign each position a color for a group, we use the respective feature map’s mean NAP value.
Quality Measures We investigate approaches to measure the visual quality of topographic activation maps in terms of ease of visual interpretability. We consider the quality as high if there are few distinguishable regions which jointly cover a large area. This does not imply quality of the activations themselves but only the visual quality. Figure 1 shows relevant steps of computing the diferent measures.
To test whether each position in a topographic map is similar to its neighborhood, we measure robustness against image perturbations. We perturb either by a Gaussian blur or by downscaling and then upscaling to the original size (with bicubic interpolation). Then, we compute the Mean Squared Error (MSE) between the perturbed and the original image. We use Gaussian blur with radii 2 px to 20 px in steps of 2 px and investigate downscaling sizes to 55 × 55 px to 10 × 10 px in steps of 5 px (see Figure 1A). Finally, we aggregate the MSEs for the diferent parameters with an estimated area under the curve (AUC) value (trapezoidal rule). We call the measures “blur MSE AUC” and “resize MSE AUC”.
We further quantify topographic map quality based on connected components (compare Figure 1B). First, we separate the image into the red and blue channel. For both channels, we apply a binary threshold at pixel value of 240 to separate regions from the background. Note that small values in the red channel indicate a blue region and vice versa. In the binarized images, we detect connected components using OpenCV1. We compute the number of components larger than 10 px area (“count”) and the average component size relative to the circle area.
Generally, few large components are considered as high quality. However, this does not
account for whether the components are large but interwoven with others. Therefore, we
further compute the convexity of each connected component as ℎ/ , where ℎ is the length of
the convex hull and the perimeter of the connected component (Figure 1B on the right depicts
perimeter and hull). Finally, we aggregate convexity values of the components and reward
larger components. To this end, we compute the fraction of total circle area occupied by each
component and use these as weights for a weighted sum of convexity values. This results in a
value in the range of [
For evaluation of the measures, we use a simple model and data set. As data, we use MNIST [
We manually created the topographic activation maps shown in Figure 2 as representative cases to see which quality measure aligns best with our expectations. Maps A and B represent ideal topographic activation maps for which we expect the highest quality. C represents a poor activation map that is dificult to visually interpret, so its quality should be low. D has few large components, however, they are interwoven with each other and should have lower visual quality. E1 and E2 are more realistic and visually qualitative examples which difer in that the blue region is split by a small gap. Both should obtain good visual quality and the gap should not afect the value too strongly. F 1 and F2 are examples to test whether nested components are separated correctly. For our last example G, which represents sparse regions, we expect a low quality that should be higher than that of C as it contains more information.
Comparing the measures, size-weighted convexity is most consistent with our expectations as it gives the highest quality to A and B and the lowest to C and G. Only the low convexity of D in comparison to E1,2 or F1,2 does not reflect well. Therefore, we will continue using the size-weighted convexity quality measure in the following. Note that average relative component size and size-weighted convexity are not 1 or 0.5 for the ideal examples A and B due to the pixel grid and approximations when obtaining connected components.
We perform the bias experiments for representations of VGG16 [32], a pre-trained CNN model
that can be used as feature extractor for downstream applications like image recognition DNNs.
As test data, we use FairFace [
Topographic maps for sensitive variables in layers 6 and 18 of VGG16 are shown in Figure 3. Appendix Figure 4 shows results for several more layers of in VGG16.
“race”: In layer 6, each category has a specific activation pattern. However, in layer 18, class activations become more similar between particular groups. Specifically, Black and Indian categories are highly similar, as well as East Asian, Southeast Asian and Latino Hispanic. The observations indicate that there is a racial bias in deeper layers. Surprisingly, the Middle Eastern and White categories do not show clear activation patterns, potentially because the model learns more individual representations for these categories. 2https://github.com/keras-team/keras
“gender”: Topographic activation maps of Female and Male category almost are the inverse of each other, which is expected for a binary grouping. Clearly, in layer 18, the Female and Male groups show stronger over-/under-activation than the random groups, indicating that the representation in deeper layers distinguishes between these categories.
“age”: Groups of similar age are similarly activated, which is reasonable considering the categories’ fuzzy boundaries. In layer 6, we observe clusters of high similarity: age groups 0-2, 10-49 and >50. Layer 18 shows more continuous changes and strongest activation deviation from the mean in the lowest and highest age groups. There seems to be no systematic disadvantage for any individual group but the groups would be distinguishable in a downstream application. Significance To evaluate the significance of the results, we contrast sensitive variables and random groups regarding color intensity of topographic maps and visual diference of groups.
We observe stronger color intensity for sensitive variables than random groups. Further, activation diferences are more pronounced between the sensitive variables than between the random groups. Both indicate that the observed similarities and patters are not only a random efect but really related to the sensitive variables.
We notice that visual quality is generally lower in layer 18. This might be related to the higher number of feature maps in layer 18 (512) compared to layer 6 (128). Visualizing more feature maps means that each influences a smaller part of the map. Therefore, the regions are likely to become less convex. We further observe that there is a large white region common to all groups, which decreases the visual quality. It represents feature maps that are inactive or unspecific to the groups but potentially sensitive to features that are not present in face images.
In general, the quality of visualizations for sensitive variables is higher than for the corresponding random groups which supports the significance of the results.
Diversity of Groups We appreciate the eforts of the FairFace data set to provide balanced evaluation data. However, the sensitive variables still have potential to be further diversified. For example, “gender” is only considered as a binary or age group “>70” includes a larger range of ages than other groups. We still consider the data suitable to demonstrate our technique but encourage the community to conduct studies with more diverse data sets upon availability.
Topographic activation maps are a promising tool to visually inspect bias in representation of DNNs and our visual quality measure supports the otherwise only subjective evaluation.
Our approach does not provide an explanation of the patterns responsible for the bias, neither do we mitigate biases. We consider our method to be a visual overview to spot likely biases to look for. Moreover, we expect that topographic activations maps are useful for people without expert knowledge in Machine Learning to get a simplified insight into DNN internals.
In this work, we only considered the sensitive variables independently. As this does not consider intersectionality, we will investigate combinations of sensitive variables in future work.
This research has been funded by the Federal Ministry of Education and Research of Germany (BMBF) as part of the project “CogXAI – Cognitive neuroscience inspired techniques for eXplainable AI”. We also thank Raihan Kabir Ratul for implementing FairFace dataset pre-processing. arXiv:1412.6980 (2014). [32] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014).