Quantitatively evaluating explainability methods is a notoriously hard endeavor. One reason for this is the lack of real-world benchmark datasets that contain local feature importance annotations done by domain experts. We present SancScreen, a dataset from the domain of financial sanction screening. It allows for both evaluating explainability methods and uncovering errors made during model training. We showcase two possible ways to use the dataset for evaluating and debugging a Random Forest and a Neural Network model. For evaluation, we compare a total of 8 configurations of state-ofthe-art explainability methods to the expert annotations. The dataset and code is available under
https://github.com/MatthiasJakobs/sancscreen.
class label. In our evaluation, we compare diferent state-of-the-art explainability methods on two popular model families (Neural Networks and Random Forests) by measuring the distance between the generated annotations and the expert annotations. Also, we outline a potential use case for debugging. We measure the influence of a feature that is deemed to be irrelevant by the experts to the explainability methods evaluated in the previous experiment.
Our contributions can be summarized in the following points: • We provide a novel real-world dataset, which contains local feature importance annotations by experts in the field. • We show the usefulness of our dataset by conducting two experiments. In the first experiment, we evaluate state-of-the-art explainability methods by comparing them to the expert annotations. The second experiment outlines a possible use case for model debugging.
The rest of the paper is structured as follows. Section 2 discusses synthetic datasets for evaluating explainability methods, as well as state-of-the-art explainability methods. Section 3 presents the dataset and annotation process. In Section 4 we show how the dataset could be used to evaluate state-of-the-art explainability methods for Random Forests and Neural Networks. Lastly, Section 5 summarizes the findings and presents possible future work directions.
Local explainability methods With the rise of complex machine learning models like
Random Forests and Deep Neural Networks, concerns about their black-box nature rose as well.
This is especially true for safety-critical application domains. To make models more transparent
after training, multiple so called post-hoc explainability methods have been devised over the
years. LIME [
LIME generates samples by perturbing around the data point to explain. Then, it trains a linear classifier on the samples and their predictions of the original model. The proximity to the original data point is taken into account during training of the linear classifier, where closer samples get a higher weight. After training, the coeficients of the linear model can be used as local feature importance measures in the neighborhood around the data point.
SHAP is a framework based on Shapley values [
In computer vision, one popular approach for generating explanations is Integrated Gradients
[
Recent work experimented with generating synthetic
datasets with known local explanations [
[
)and ( ) = | |. Since all these approaches are additive and the functions are one-dimensional, this means that feature interactions are not taken into account. While the individual features might be highly correlated, the functions only take one feature into account at the same time. This is unrealistic for many real-world datasets. Consider the trivial example = 1 ⋅ 2, where the features are interacting and influencing the output together, regardless of their correlation. We argue that these datasets might not be suficient to evaluate explainability methods in a way that reflects their applicability on complex, real-world datasets, where we cannot assume the absence of feature interactions in general.
The basis of our dataset is a real-world dataset, consisting of international transactions that entered a sanction screening pipeline. All data points contain 19 features. From these 19 features, 14 are binary indicators. For example, one feature called Name hit encodes whether the name of the person involved in the transaction can be found in internal reference data. The remaining features are numeric. For example, Number Countries aggregates the number of diferent countries involved in the transaction. We want to emphasize that, due to the features being indicators and aggregators of internal reference data, no personal information about the people and companies involved in the transactions are exposed in or can be inferred from our dataset. For an overview of all features see Tab. 1. Notice that the meaning of three features
(hidden_feature_1,hidden_feature_2 and hidden_feature_3) cannot be disclosed in respect to trade secrets. The task is to classify whether a transaction needs to be halted. We will further refer to the possible outcomes of the sanction screening process as STOP and GO, respectively.
We sampled a representative set of transactions to provide local feature importance annotations for each data point and devised an annotation guide to rate each features relevance. Our guide is based on a numeric rating scale, ranging from −3 to +3 (see Fig. 1). We chose the range to be large enough to allow for some nuance in the annotation process but not be too overwhelming at the same time. The lower the value, the more the feature indicates the label STOP, while positive values indicate GO. It is also possible to rate the features influence as neither, meaning that the value of the feature is uninformative to the decision.
This results in a quantification of local feature importance, encoding expert knowledge in a way that allows us to compare it with attribution-based explanations. See Tab. 2 for the amount of data labeled with class labels and expert annotations.
In total, our dataset contains 1, 002, 860 data points for which only the class labels are available.
We split the dataset into training, validation and test parts to train the models. 2, 288 data points with equal amount of class labels were used for the training split. Additionally, 572 data points with equal amount of class labels were used for the validation split. We made sure to use equal amounts of both classes for training and validation in order to not prefer one class in production. This is due to the fact that unsuspicious transactions vastly outnumber those that are. We randomly picked a day in the logs of the real-world sanction screening pipeline and noticed that only 3 transactions were suspicious. This is reflected in the test split, where 3 transactions were suspicious and the remaining 999, 997 were not.
To gain more insight into the dataset, we calculated diferent statistics. First, we plotted the distribution of the binary features, shown in Fig. 2. The distributions show that many feature values are highly imbalanced. As an example, consider the features Other hit, Address hit and Account hit, which are zero for most data points, meaning that these indicators are probably highly relevant if they are one. The opposite case as with Name hit indicates that the presence of a name being recognized in the reference data is not unusual and is thus likely less informative. We conducted a similar analysis of the numerical features, as shown in Fig. 3. As can be seen, Unique hits, Diferent hits as well as Number of countries feature a small amount of variation, while the Indicator value is most spread out since it is a percentage value.
Lastly, we investigated the distribution of the expert annotations per feature. The distributions are shown in Fig. 4. As can be seen, many features contain a median value of zero, meaning that they can either be salient for a GO or a STOP, depending on the context of the other features. Notice that the median value of Name hit is positive, with most of distribution of annotations being positive as well. Additionally, Account hit is mostly negative or zero.
Unique values in numerical features Indicator value
Number abnormalities
Unique hits
Different hits
Number countries
3 g itn 2 a r e can 1 t r o ipm 0 e r tau 1 e ftr ep 2 x E 3 ilItrcvoaaenud ilitrroaaeeunbbNmm iftraeeehnudd3 iittyhC ittcconuhA itaehNm ittrehhO iftraeeenuhdd1 ittryonuhC itrssehddA iittcoannhS iftraeeenuhdd2 itskanhB ittrccoeunnhAOw ittsxTeh iiifttrccyoeunhSpC iitsenuhqU iiffttrseenhD itrrscoeeuunbNm
s
To show the potential of the dataset, we conducted two experiments. First, we compare explanations generated by multiple state-of-the-art explainability methods on Random Forests and Neural Networks to the expert annotations. Second, we show a potential use case for debugging the models by focusing on which method better identifies unimportant features. Metrics Since to our knowledge there appears to be no consensus in the literature on one specific metric, we propose using the following three metrics, since they are scale independent and vary in what aspect they measure. We will define () to be an explanation or annotation of data point and () being its corresponding ranking.
First, we compute a ranking of both generated explanations and expert annotations to see the relative importance of each feature. Since during the rating process multiple features can be equally (un)informative, we compute fractional rankings, meaning that features get assigned the mean of all possible, equally valid rankings. As an example, consider an expert annotation () = (+3, 0, 0, −1). Clearly, both rankings () = (4, 3, 2, 1)and () = (4, 2, 3, 1)are equally valid.
Using fractional ranking, the computed rank of () is equal to () = (4, 2.5, 2.5, 1), where the
ranking of each value is given by the mean of all possible equal rankings. This procedure is also
necessary if an explanation generated by an explainability method contains multiple identical
values. From now on, we will refer to fractional rankings () simply as () for ease of notation.
Then, we compute the distance between both rankings using Spearman’s Footrule:
( () , () ) = (⌊ 1 2⌋)
As shown by [
Second, we use cosine distance between two explanations to get a sense of alignment between them without taking scale into account. We argue for the use of cosine distance since diferent explanation methods provide varying scales, which also vary from the rating scale. 2 1 2 ( () , () ) = (1 −
() ⋅ () || () ||2 || () ||2 )
Third, we propose a distance measure based on Hamming distance. We average three Hamming distances ℎ between indicators of positive values + , negative values − , and those values whose absolute value is smaller than or equal to a fixed value = 10 −8 indicated by . ℎ ( () , () ) =
ℎ( () , () ) 1
∑ 3 ∈{+,−,}
We train both a Random Forest and a fully-connected Neural Network on a split of the dataset for which only class labels are present. See Tab. 3 for a breakdown of the model performance on the training and test split used. We used the implementation of scikit-learn for the Random Forest and PyTorch for the implementation of our Neural Network.
For our Neural Network model architecture, we chose a simple fully-connected Neural Networks with two hidden layers, of which each layer contains 50 neurons. Since we achieved good results using this model architecture, we did not investigate the choice of these hyperparameters further. We trained the model with a learning rate of 0.0003 using the Adam optimizer and a binary cross-entropy loss. To avoid overfitting, we monitored the loss on the validation split and utilized early stopping if the loss did not reach a new minimum for 50 consecutive epochs. We saved a model checkpoint on every new minimum validation loss value and reset the model to the latest checkpoint if early stopping came into efect.
After training the network, we tested it on the test split and found that no mistakes in predicting the suspicious transactions were made, and 22, 083 transactions were falsely identified to be suspicious. While the error rate seems high, we stress that it is of utmost importance that no truly suspicious transactions gets misclassified as unsuspicious. A comparatively small error rate on unsuspicious transactions of 22% is considered a good result in our application setting.
To train the Random Forest, we used the implementation provided by scikit-learn. We combined the training and validation splits again and provided both to the Random Forest for training. After some experimentation, we found good results on the dataset using 300 estimators. All other hyperparameters are default to the provided implementation, meaning that we did not use a maximum depth of the trees, we used Gini Impurity and we used Bootstrapping. As discussed in the training of the Neural Network, we did not feel the need to further optimize the Random Forest’s hyperparameters due to a satisfying fit on the training and test data.
After fitting the Random Forest model, we observed zero transactions that were wrongly
classified to be unsuspicious. In comparison, the error rate on unsuspicious transactions was
15, 657. The Random Forest is thus able to achieve a lower error rate on the test set compared
to the Neural Network, while also not wrongly classifying any suspicious transactions.
e
c
n
a
ts
i
d
e
n
i
s
o
C
e
c
n
a
ts
i
d
e
n
i
s
o
C
e
c
n
a
ts
i
d
g
n
i
m
m
a
H
e
c
n
a
ts
i
d
g
n
i
m
m
a
H
KS-Zero KS-DataTreeSHAP-Int LIME TreeSHAP-Cond TreeSHAP-CondKS-Zero KS-DataTreeSHAP-Int LIME
4.1. Measuring explanation alignment to the expert annotations
Using the previously trained models, we extract explanations using state-of-the-art explainability
approaches. For the Random Forest, we generate explanations using TreeSHAP [
For each explainability method, we compute explanations on the fully-annotated part of the dataset and plot their distances to the respective expert annotations. As a distance measure, we use Spearman’s Footrule, Cosine distance and Hamming distance, as presented earlier. As can be seen in Fig. 5, KS-Zero performs very well for the Random Forest model in terms of median
RF LIME RF KS-Zero
RF KS-Data RF TreeSHAP-Cond
RF TreeSHAP-Int
NN LIME NN KS-Zero
NN KS-Data NN DeepSHAP
NN IG NN EG 1.00 liItrcvoaaeund liitrrsoaaeenubbNmm iftraeeeunhdd3 iittyhC ittccohnuA itaehNm ittrehhO iftraeeeunhdd1 ittryohnuC itrssehddA iittcoanhnS iftraeeeunhdd2 itskahnB ittrccoenhnuAOw ittsxTeh iiifttrccyoenhupSC iitsehunqU iiffttrseehnD itrrscoeeunubNm distance according to all metrics. The same can be said for KS-Zero for the Neural Network model, where IG also performs well. Since both IG and KS-Zero feature the same baseline of an all-zero vector, we suspect that this baseline is the main cause of their performance since most features are binary. We noticed that Hamming distance gave the clearest indication of the best performing methods. For example, consider that the most distant explanation in KS-Zero for the Random Forest is still closer than the median distance for KS-Data (see top right plot in Fig. 5).
We additionally conducted a Wilcoxon signed-rank test to test whether the diferences in computed distances between each pair of explainability methods are significant. With a significance level of = 0.01 , all diferences are statistically significant except for the diference between LIME and KS-Data for the Neural Network using Cosine distance.
Finally, we investigated how well the explanations between the annotations and each of the previous explainability methods align, and if there are patterns transcending explainability methods and model classes. The results are shown in Fig. 6. We noticed that Account hit, Other hit as well as hidden_feature_1 strongly correlate with the annotations for nearly all explainabilitity method, regardless of whether the Random Forest or Neural Network was used. This is consistent with the finding in Sec. 3 that these features are fairly imbalanced in terms of their values, meaning that unexpected values are probably highly salient. Interestingly, Bank hits shows the opposite phenomenon. In the annotations, Bank hits tended to be either uninformative or in favor of STOP (see Fig. 4), suggesting that the Neural Network and Random Forest where able to determine an efect towards GO instead. 4.2. Debugging use case: Identifying irrelevant features Next, we present a simple use case for the debugging the trained models using expert annotations. While investigating the expert annotations, we found that some features were almost always )re1.0 tt e b s i re0.8 g l)(tr a en0.6 c r e p ( 0 to0.4 e s o l c itsn0.2 o a n a lxp0.0 E KS-Data TreeSHAP-Int TreeSHAP-Cond LIME
KS-Zero
EG
KS-Data DeepSHAP IG
KS-Zero
LIME irrelevant. A particularly striking example was the feature Unique Lists, which encodes how many internal datasets contained information from the transaction. In 96% of annotations this feature was said to be irrelevant. We conducted an experiment where we investigate how important this feature was to both the Random Forest and the Neural Network, according to all previously used explainability methods. We round the generated explanations for Unique Lists to exactly zero if the absolute value was lower than 0.01 for fairer comparison. The results are shown in Fig. 7. Notice that the best performing explainability methods from the previous experiment, namely KS-Zero and IG, were among the worst methods for both models in terms of importance of feature Unique Lists. We observe that the methods that incorporate a background dataset of training data points (KS-Data, TreeSHAP-Int and EG) tend to perform better than the ones with simple all-zero baselines. However, even these methods still attribute many data points with a non-zero importance for a feature which, according to the annotations, is irrelevant most of the time. This suggests that the Random Forest and Neural Network sometimes utilized this feature in their decision making, even though the feature is highly irrelevant. With this insight gained, one could focus on specifically regularizing mostly irrelevant features such as Unique Lists during the training process, which will result in models that are more trustworthy due to them not relying on spurious correlations in irrelevant features.
We presented SancScreen, a real-world dataset containing local feature importance annotations derived from domain knowledge. By conducting two simple experiments we showed the potential of the dataset for evaluating explainability methods and debugging a trained model. Additionally, using three diferent metrics of alignment we compared diferent explainability methods, regardless of the absolute values of their explanation, to the expert knowledge. Finally, we discussed the impact of datasets like SancScreen on the future evaluations of explainability methods and outlined a procedure which we hypothesize could be used to gain more insight into up- and downsides of popular explainability methods.
In the future, we plan to incorporate some of the expert annotations during the training process of the models in order to regularize the models to be more in line with the experts. This will result in a more fair comparison, since we can take a part of the fully-annotated annotations for training to prime the model to reason along the same lines as the expert. Fully-annotated test data can then hopefully be used as ground truth annotations. Moreover, we plan to derive ways to incorporate explanations in more machine learning models besides neural networks, for which some work exists, namely by incorporating explanations in tree ensembles like Random Forests. This should lead to a wider adoption of evaluating explainability methods in application ifelds where neural networks are not commonly used.
This research has been funded by the Federal Ministry of Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence, LAMARR22A.