Towards machine learning (ML) for automated radio network control, we demonstrate how Explainable AI (XAI) can be used to explain and distinguish the efect of diferent factors to network performance KPIs. Additive local feature attribution methods like SHAP promise a model agnostic way to gain insight into a ML model. However, there are many variations of these methods, including marginal and conditional SHAP, as well as ways to calculate attributions that respect the causal structure of the data, such as Asymmetric SHAP. We describe two new approaches to calculating attributions that follow causal relationships more closely. One approach is based on marginal explanations of a model trained in a specific way, while the other is based on calculating the equivalent of conditional Asymmetric SHAP by training multiple models. We demonstrate the approaches on both synthetic and real-world data.
Using AI for automated radio network control is a fundamental challenge for future generation radio networks. Towards automation, a promising direction is Explainable AI (XAI), whose goal is to investigate tools and techniques aimed at opening the so-called opaque (or black-box) models (e.g., deep neural networks, DNN) or at devising intrinsically interpretable and accurate models (e.g., rule-based systems). More expressive models such as DNN or gradient boosted tree models can better fit the observation, however, they are more complex themselves. While more expressive models promise better insights, their complexity makes them more opaque, making it harder to successfully infer useful information about the system.
In this paper, we propose an XAI method to explain and distinguish the efect of diferent factors on network performance metrics. The main technical dificulty is the complex causal relationship between these factors. For example, antenna tilt configuration afects network performance indirectly via coverage, interference and cell load. These indirect metrics have a more direct efect on network performance, hence XAI will primarily find the importance of coverage, interference, and load and explain the predicted performance metric based on them, mostly ignoring the explanation of how network configuration afects this performance metric.
Local feature attribution methods based on Shapley values, such as SHAP [
In this paper, we describe our use of local feature attributions for explaining radio network performance. Radio networks used in telecommunications are complex systems with many diferent, inter-related variables, and various causal structures between these variables. Our primary goal is root cause analysis for specific behavior, and our approach is to model the overall behavior of the system and investigate the model using explanation methods. It is, however, very important that the explanations take the causal relationships between the model features into consideration, as failing to do this could easily cause misattributions in our analysis.
Reliably inferring causal relationships from observational data is generally considered to
be impossible [
In our use case, we measure the contributions of feature groups, while assuming to know the
causal structure between the groups. An important distinction between diferent
Shapley-valuebased methods is whether they use a conditional or marginal value function for evaluating the
model. A conditional value function is a way to explain the data, while a marginal value function
results in explanations that reflect the behavior of the model more closely [
The rest of this paper is structured as follows. In Section 2, we describe the mathematical background of SHAP and related terms. In Section 3, we describe two ways of calculating causal attributions for our use case; one based on marginal explanations of a causally-trained model, and one that is based on calculating the equivalent of conditional SHAP by training multiple models. In Section 4, we describe the results of our experiments on simulated and real-world data to illustrate our method. Finally in Section 5, we share our conclusions.
2.1. SHAP
The notion of Shapley values [
∑︁ ||! ( − | | − 1)! (, , ) , where
! (, , ) = ()( ∪ {}) − ()().
Here ()() is called the value function, and represents the function of evaluating the model at point while only using the coalition of features. It is, however, not evident how we can evaluate a model with coalitions of varying sizes, when the model was trained using a fixed number of features. Note that an alternative but equivalent formulation of Equation 1 is diferent coalitions of players participating and combining these results in a certain way to calculate individual contributions.
Using Shapley values to calculate additive local feature attributions for machine learning
models gained popularity in recent years. SHAP [
Given model , feature set S, and a specific point from the dataset, the Shapley value for feature can be defined as (1) (2) (3) (4) (5) ∈Π (, ) = ∑︁ 1! [︁()({ : () ≤ ()}) − ()({ : () < ()})]︁, where Π represents the set of all permutations of the ordering of model features.
We describe two ways of defining the value function ()() corresponding to the marginal and conditional variants of SHAP values. In the marginal or interventional SHAP variant ()() = (′) ︀[ ( ⊔ ′)︀] , where the expression ( ⊔ ′) represents the value of evaluating function with the incoalition feature values ∈ taken from , and the out-of-coalition feature values taken from an identically distributed random ′. In contrast, in the conditional or observational variant ()() = (′|′=) ︀[ ( ⊔ ′)︀] , where the distribution of ′ is conditioned on the known feature values from .
The diference between these two variants is whether the expectation that is taken over the
out-of-coalition features is conditioned on the values of the in-coalition features. Marginal
SHAP values can be estimated by sampling out-of-coalition values from the training data for
evaluating the value function, while conditional sampling is much harder to do and can itself
become a separate modeling task [
Marginal SHAP explanations reflect the behavior of the model, meaning that the explanations respect whatever rules the model infers from the data. If there are multiple equivalent ways in which the prediction can be calculated, then the explanations reflect whichever way the model itself uses. Another important aspect of marginal SHAP is a common criticism that the evaluation can also include invalid data points, where combining the in-coalition and the sampled out-of-coalition features results in a data point that is unrealistic in the real world.
On the other hand, conditional SHAP explanations reflect the dataset in that they are not dependent on which equivalent formulation of prediction the model uses. As a simple example, if the dataset contains two features that are identical, then the model is free to choose either of these (or any convex combination of them) for the prediction formula. Marginal SHAP attributions will mirror the model’s choice, while conditional SHAP attributions will share the contribution equally between these features regardless of the model itself. This is often cited as a drawback of marginal SHAP, as it means that the model explanation can reveal associations that are not present in the model itself.
In [
Asymmetric SHAP modifies Equation 3 to only include permutations where for each causal relationship, the causing variable is ordered before the afected variable, or more generally, allows any weighting scheme over the permutations instead of uniform weights: ( ) = ∑︁ ( )[︁()({ : () ≤ ()}) − ()({ : () < ()})]︁.
∈Π (6)
In a causal setting, the above described weighting scheme is suggested, meaning that we modify the weights such that ( ) ∝ 1 if () < () for each , where is a causal ancestor of , and ( ) = 0 otherwise. The idea behind ordering causing variables first is that these get the attribution for the change they cause in the expected value of the prediction, while the afected variables get the change that is caused by including them after the causing variables.
Our approaches rely on grouping the variables and assuming to know the causal relationships between groups. From an evaluation perspective, we treat a feature group as a single, multidimensional feature, meaning that while calculating SHAP, either all features of a feature group are included in a coalition or none of them are. This way, we get overall contributions for each group.
The causal relations between the groups can be represented in a directed acyclic graph, where the nodes are the feature groups, and the arcs point from causing group to afected group. This way we can say that feature is a causal ancestor of feature if and only if there is a path from to in the graph. We can see an example of such a graph in Figure 1. A topological ordering of this graph has the property that each variable precedes its causal descendants. In fact, the set of permutations with nonzero weight in our Asymmetric SHAP calculation, as described in Section 2.2, is exactly the set of all possible topological orderings of this graph.
One way to have the attributions respect causal relationships is by using marginal SHAP to explain a model that itself calculates the prediction in a way that respects causal relationships. Having assumed a causal order of features, as described in Section 3.1, we can attempt to force the model to prefer using features that appear earlier in the causal ordering, in hopes that the resulting explanation also attributes the prediction accordingly. If there are multiple causal orderings possible than we need to calculate the average of all of these with equal weight.
In our modeling, we primarily use gradient boosted decision trees [
One possible failure mode of this method is that there is no guarantee that the final model will be diferent from one that was trained with all features available at the same time. To see this, assume that we have the model 1(1) after training only using the first feature 1. In the next step, the modeling process can continue by adding 2(1, 2) = − 1(1) + 2′ (1, 2) such that 1 + 2 = 2′ . In other words, it is possible to cancel out the previous modeling steps in later steps. We can attempt to use regularization (eg. trying to limit capacity) to discourage the model from doing this, however, if the advantage of using 2 is great enough, it can overpower regularization.
In this section, we describe a way of using the already in place modeling process for conditioning
on the in-coalition feature values. Similar ideas are also described in [
Let us assume that we can train a separate model for each coalition, i.e. train () for each coalition , where the model is trained using only the features in . Using these, we can define =
∑︁ ||! ( − | | − 1)! (︀ ∪{}() − ())︀ , ! (7) meaning that the value function is simply equal to () instead of Equation 5. We give a heuristic argument that these two calculations approximate the same value. When modeling a regression task, we are in fact approximating an ideal function
() ≈ * () = (|) [] , with being the target variable. The SHAP value ( ) then itself can then be viewed as an approximation of ( * ), the feature contributions calculated for the ideal function * . In this sense, Equation 7 is just another way of approximating the same value that conditional SHAP estimates. This can be seen by observing the following about () and ()(): (8) (9) (10) () ≈ * () = (|) [] and ()() = (′|) ︀[ ( ⊔ ′ )︀] ≈ (′|) ︀[ * ( ⊔ ′ )︀]
= (′|) ︀[ (|′,)[]]︀ = (|) [] .
In Equations 9-10 we use Equation 8, while in Equation 10 we also use law of total expectation.
While this way we have to train a separate model for each coalition, the problem of conditioning part of the feature set on a given coalition is solved by re-using the same modeling process that we already have in place. Since we already trust this process in delivering insights about the dataset, this saves us from dealing with the conditioning as a separate sampling or modeling step. Fortunately, Asymmetric SHAP greatly reduces the number of coalitions that are of interest, as we only need to perform the evaluation for starting subsets of causal orderings of features, which makes this method computationally feasible in our case.
Given a single causal ordering, the contribution for a specific feature is given by the change in prediction when rfist including the feature in the set of features available for the model, as formalized by Equation 6. Similar to regular SHAP, we start by calculating 0 = (), which can be interpreted as the best possible model while using an empty set of features as coalition.
We run two kinds of experiments. The first one is a synthetic task where the variables and functions involved are simple enough that we know the exact result that we can expect from the explanation methods. The second is an example with real-world data.
We report global feature importance. Since simply averaging contributions over all records results in values with an expected value of zero, importance is calculated by taking average of the absolute values of the contributions. One unfortunate consequence of using this metric is that the importances don’t sum up to some fixed value. On our tables, Causal-order training refers to the approach described in Section 3.2, while Separate models refers to the approach described in Section 3.3. We denote the importance of variable by Φ .
Here we demonstrate our methods on synthetic data. We start by defining 5 variables. Let
, , ∼ [
It is clear that the target variable can be approximated from either variables , , or variable , with the latter having a mode direct relationship with the target. Let us assume that the modeling process can learn the relationship between the features and the target exactly. We can express the expected feature importance analytically for each of the cases where either , , or get full contribution for the calculating the prediction: This essentially means that in the case of marginal SHAP, on one end we get Φ = Φ = Φ = 0.5 and Φ = 0, or on the other end we get Φ = Φ = Φ = 0 and Φ ≈ 0.82. Any convex combination of these two cases is possible as an actual modeling result. Conditional SHAP is expected to distribute the contributions uniformly, meaning a feature importance of Φ = Φ = Φ = 0.25 and Φ ≈ 0.41.
We model on a training set of 104 samples, and smaller evaluation and testing sets of 103
samples. We use gradient boosted decision tree models as implemented by the Python package
LightGBM [
We demonstrate our method on a proprietary real-world dataset of mobile telecommunications, consisting of performance management (PM) data from radio access network cells with 15 minutes granularity. Model output is average downlink cell throughput for automatically determining the root cause of throughput degradations using explainers of the model. Input features of the model are described in Table 2, while Figure 1 illustrates the causal relationships between these groups. It also includes a 5th group: Cell configuration . Features of this group are not present in the dataset, but directly or indirectly afect all other features. This has the side-efect that while the groups TA distribution and Cell load can be assumed independent given
. Timing Advance distribution of mobile terminals. It is derived from radio propagation delay measurements between the mobile terminal and base stations and can be used to estimate the distance from the base station. We normalize this distance with cell range, hence these features capture cell edge versus cell center distribution of mobile terminals.
Various PM metrics describing user plane and signaling load of a given cell.
Measured uplink interference distribution in a given cell (downlink interference is unfortunately not available in our dataset).
Various downlink channel quality metrics including CQI and rank distributions.
TA distribution Cell load Cell configuration , the latter not being present in the dataset makes them no longer independent. The resulting interactions should be distributed equally between these two groups.
There are only two possible topological orderings of these feature groups. In Table 3 we report feature importance according to both separately, and also final averaged feature importance. The methods clearly have a similar efect to what we observed in our synthetic experiments, with features that appear earlier in the causal order getting increased importance. The ordering of the first two groups has a negligible efect on later groups, which suggests the possibility of not even having to recompute these values when dealing with multiple orderings.
We can observe a substantial diference between the results of the approaches described in Sections 3.2 and 3.3. While both place increased importance on the group TA distribution, the latter is still much higher. We attribute this to the possibility observed in Section 3.2 that while the model is encouraged to use features that appear earlier in the causal ordering, it is not forced to do so. There is no such problem present in the case of the other approach.
In this paper, we described two approaches to calculating causality-respecting additive local feature attributions for explaining the cause of radio network performance issues. We demonstrated their efectiveness on both synthetic and real-world datasets and concluded that these methods proved to be efective in placing more feature importance on features that appear earlier in a causal ordering.
This work has been partially funded by the Hungarian Ministry of Innovation and Technology NRDI Ofice within the framework of the Artificial Intelligence National Laboratory Program.