Regulatory requirements, such as the recently published EU AI Act, emphasize the need to explain machine learning models. Nonetheless, the limits of any given explanation have to be taken into account. From Psychology research it is known, that the human working memory capacity is limited. For this reason, any explanation must not be too complex. In this work, explanation groves are presented as a model agnostic tool to control the complexity of an explanation while simultanously maximizing the obtained degree of explanation. Explanation groves do result, if the degree of explanation is maximized over the search space of all sets of if-then rules of prespecified size. A user-friendly implementation of explanation groves is given in the R package xgrove. Its use is demonstrated for a random forest model trained on the Boston housing data. Explanation groves not only provide an easily understandable explanation but can be further used to analyze the trade-of between the obtained degree of explanation and its corresponding complexity.
not only provide an easily understandable explanation but can be further used to analyze the trade-of between the obtained degree of explanation and its corresponding complexity.
In order to find the best explanation it has to be defined what characterizes a good explanation is. In
literature, several concepts are proposed to analyze this (cf. e.g. [
(^ () − ())2 () ϒ = 1 − ()
0 where 0 is the based on () = , ∀ being the constant average prediction := (^ ()). By construction, ϒ is similar to the 2 coeficient of determination for regression problems and can be thus interpreted in a similar way: For a good explanation () will be close to 0 and thus the closer ϒ is to the value of one the better the explanation.
Based on the previous quantification of the appropriateness of an explanation, one can try to find the best explanation of by stagewise maximization of ϒ. For this purpose, an iterative approach can be used. Let
()()
be the explanation after the ℎ iteration, ∈ N. The ESD from the previous section measures the
squared loss between the model’s predictions and its explanation. As in gradient boosting theory [
() ⃒⃒ ()=(− 1)() (^ () − ())2 ⃒⃒
() ⃒⃒⃒ ()=(− 1)() = 2(^ () − (− 1)()) = : ˜ i.e. by iteratively fitting the pseudo-residuals ˜ between model and current explanation after stage ( − 1) to the data, where = 1, ..., denotes the observation index. The optimal model-agnostic rule-based explanation is then given by the sum over a set of weighted rules, which can be written as: (1) (2) (3) e s n o p s e R l e d o M
Here, 1(∈()) denotes the indicator function that returns 1 if rule () holds for and 0 otherwise.
() is a rule of the form ≤ in variable for numeric variables or ∈ with ⊂ | | for
categorical variables. is a weight that describes how the explanation changes, if rule () holds.
For this purpose, (), + and − can be computed simultaneously by fitting a gradient boosting
model using squared loss and decision trees of depth one (stumps) to the predictions ^ () of the model
of interest [
Note that the resulting optimal explanation () consists of a set of rules and corresponding
weights {((), +, − )} and thus represents a rule-based explanation as opposed to example-based
explanations. For a comparison of both approaches for model explanation cf. e.g. [
be computed in order to analyze whether there is an explanation that is both easy to understand and appropriate.
Note that an explanation grove only denotes a surrogate model [cf. 17]. This means, it is only an
approximation which mimics the model under investigation but there is no guarantee that the identified
rules correctly describe the original model. In the example, a simple and understandable explanation
could be obtained if it would be known that the underlying function is of trigononetric type. Anyway, in
practice, the type of the underlying function is usually not known and a surrogate model can neverthless
help understanding a model’s behaviour. In that sense, one may refer to the famous quote of George
Box that ‘’all models are wrong but some are useful” [
As it has been worked out before, the proposed methodology not only allows to find a set of rules of
ifxed size that maximizes the appropriateness of the resulting explanation but, furthermore, by
comparing groves of diferent size, allows to analyze the trade-of between appropriateness and complexity.
Explanation groves are implemented in the R package xgrove which is available on CRAN [
Initially, a random forest model is trained using the ranger implementation [
Note that explanation groves are model agnostic and the same code can be run for arbitrary models. As a default, it is presumed that the call predict(model, data) returns the desired predictions ^ () of the model to be analyzed (here: rf). It is possibile to define user-specific predict functions as it is done here by the function pf.
The total number splits over all 500 trees in the forest sums up to 80331 which is, of course, far too
high to be interpretable. Instead, from Psychology research it is known that humans’ working memory
# load data
library(pdp)
data(boston)
# train model
library(ranger)
set.seed(42)
rf <- ranger(cmedv ~ ., data = boston)
# define predict function, if necessary
pf <- function(model, data) {
return(predict(model, data)$predictions)
}
# include library
library(xgrove)
# specify desired grove sizes
ntrees <- c(4, 8, 16, 32, 64, 128)
# remove target variable from data
data <- boston[, colnames(boston) != "cmedv"]
# compute groves of different size
xg <- xgrove(rf, data, ntrees, pfun = pf)
# visualize achieved degree of explanation vs. complexity
plot(xg)
# print rules of the grove with at maximum eight rules
xg$rules[["8"]]
capacity is limited and restricted to a small number of items [
Finally, explanation groves are computed using the function xgrove(), which requires three arguments: the model, the data as well as the desired number of rules (ntrees). For this example, six groves of diferent size are computed where the number of rules is successively doubled from four to 128. The target variable should not be used for the explanation. For this reason it is removed from the data here.1 The additional pfun argument allows to define arbitrary predict functions and only needs to be specified if predict(model, data) does not directly return the desired predictions (cf. above).
The resulting S3 object (xg) summarizes the achieved degree of explaination ϒ as well as the corresponding number of rules for the diferent groves (cf. figure 3). In xg$groves, all groves are of diferent size are stored as specified by the ntrees argument in the call. A similar, but more convenient output is given by xg$rules where identical rules with no data points inbetween both splits are aggregated. Thus, the resulting number of rules is smaller or equal than pre-specified. An example of a resulting grove is given in table 2.
Figure 3 compares the appropriateness of the explanations given by groves of diferent size. This ifgure can be created by calling the plot() method on the output object of the xgrove() call. It can be easily seen that the degree of explanation gets better with an increasing number of rules. A value of ϒ ∼ 0.9 is already obtained for less than 20 rules in this case. On the other hand, if a degree of the 1This is done automatically if the model contains a terms component if the remove.target argument is specified as TRUE (default). Alternatively, this can be done manually as it is done here.
n o li spu .085 explanation of at least ϒ = 0.95 is required, more than 80 rules are needed. This, in turn, will be hard to interpret.
The resulting grove is given in table 2 below for a grove of six rules. It can be easily seen, that the predicted house prices decrease from 22.53 by 3.33 if the crime rate in a census tract is above 9.33 percent and the model predicts slightly higher prices for more eastern census tracts (longitude > − 71.04). A comparatively strong increase of house prices is assigned to census tracts with small percentages of persons with lower status (below 14.44 percent and even more if it is also below 4.55 percent). Finally, also a strong efect can be seen for census tracts with a high average room number above 6.84 or even above 7.44. Nonetheless, the degree of explanation given by these rules is only ϒ ∼ 0.836. Ideally, there should exist a grove of few rules and a high degree of explanation (i.e. in the top left corner of the previous graph). It is up to the user to decide whether this can be considered as suficient here, but at least, there should be awareness about the magnitude of the gap between the degree of explanation and the true model’s responses.
Explanation groves are introduced as a model-agnostic tool to extract a set of understandable rules in order to explain arbitrary machine learning models. An algorithm is proposed that allows to find the best explanation by a prespecified number of rules.
The proposed method is available in the R package xgrove on CRAN. It is demonstrated how groves of diferent size can be easily computed in order to explain arbitratry machine learning models. The results consist in an set of understandable if-then rules. By increasing the number of rules, and thus the complexity of the explanation, the appropriateness of the resulting explanation will improve but it is well-known that human’s working memory capacity is limited. In consequence, by creating groves of diferent size, explanation groves allow to analyze the trade-of between the appropriateness and the complexity of an explanation. The observed trade-of between the degree of explanation and its complexity should be taken into account whenever explainable machine learning is applied in practice.
The author has not employed any Generative AI tools.
The corresponding R package xgrove is avaliable on CRAN.