Counterfactual explanations highlight “actionable knowledge” which helps the end-users to understand how a machine learning outcome could be changed to a more desirable outcome. In code vulnerability detection, understanding these “actionable” corrections can be critical to proactively mitigate security attacks that are caused by known vulnerabilities. In this paper, we present the case-based explainer DisCERN for counterfactual discovery with code data. DisCERN explainer finds counterfactuals to explain the outcomes of black-box vulnerability detection models and highlight actionable corrections to guide the user. DisCERN uses feature relevance explainer knowledge as a proxy to discover potentially vulnerable code statements and then used a novel substitution algorithm based on pattern matching to find corrections from the nearest unlike neighbour. The overall aim of DisCERN is to identify vulnerabilities and correct them with minimal changes necessary. We evaluate DisCERN using the NIST Java SAR dataset to find that DisCERN finds counterfactuals for 96% of the test instances with 13 ∼ 14 statement changes in each test instance. Additionally, we present example counterfactuals found using DisCERN to qualitatively evaluate the algorithm.
Security attacks that exploit hidden software code flaws pose serious risks that compromise
system performance and services. Therefore the ability to detect these vulnerabilities in a timely
manner as well as being able to detect potential flaws is a desirable feature that can help to avoid
disastrous financial and societal consequences. Application of AI for data-driven vulnerability
detection has increased significantly in recent years [
Once vulnerabilities are detected or classified into flaw categories, the software needs to be
ifxed. Explainable AI (XAI) techniques are used for explaining AI model outcomes and to the
best of our knowledge, only very little work has been done to assist with this code revision
phase. Authors of [
Counterfactual explanations for AI (CAI) have accrued benefits from counterfactual thinking
research from psychology and GDPR guidelines for AI [
The rest of the paper is organised as follows. The introduction of the NIST Dataset and detection of code vulnerabilities using Machine Learning methods is presented in Section 2. Section 3 presents the DisCERN algorithm which discovers counterfactuals for vulnerable code segments and thereby guide the user to correct the vulnerabilities. The evaluation methodologies, performance metrics with quantitative and qualitative results are presented in Section 4, followed by conclusions and future work in Section 5.
NIST Software Assurance Reference Dataset (SARD) Project promotes the detection and correction of known security flaws in programming code. They have a publicly available repository of datasets from diferent programming languages that are labelled for flaws and possible (a) Code Segment labelled ’Vulnerable’ (b) Code Segment labelled ’Not Vulnerable’ corrections. The flaws are standardised by the community maintained Common Weakness Enumeration (CWE) list which consists of software and hardware weaknesses. In this work, we selected the Java dataset from SARD 1. Here Java files are grouped under their CWE code and each file contains a class object with one or more methods. For supervised learning we make use of two methods in the code: one with the code vulnerability and the other demonstrating the fix for the code vulnerability.
The supervised task involves a binary classification of ’Vulnerable’ or ’Not Vulnerable’ given some code segments. Forming the dataset involved the extraction of code segments and labelling them into one of the two classes and carrying out a masking step to ensure there is no target leak that can influence model training. The steps taken to organise this dataset for classification are as follows: 1. Split methods that contain a vulnerability and those that don’t into two separate files.
Java methods were labelled as ’Vulnerable’, if they contain a comment that starts with either ’FLAW’ or ’POTENTIAL FLAW’; and a ’Not Vulnerable’ label is assigned when a comment that starts with ’FIX’ is found. 2. Entity obfuscation steps are applied to each file.
a) Replace any method name with ’method’ b) Replace any Parameter type with ’Parameter’ c) Replace comments with ’/*comment*/’ d) Change all method signatures to ’public void method()’ 1https://samate.nist.gov/SARD/testsuite.php (a) Vulnerable Dataset
(b) Not Vulnerable Dataset An example of pre-processed code segments appear in Figures 1a and 1b. Here the "if" statement is used to check the size of data as a fix to the vulnerable code segment.
Using the Bag-of-Word (BoW) vectorisation method we analyse the distribution of the preprocessed dataset over the 1074 unique language tokens (see Figures 2a and 2b). The class distribution of the dataset (see Figure 3) suggests a class imbalance between the ’Vulnerable’ and ’Not Vulnerable’ data which indicates that not all files have a ’corrected’ version of the code.
The most common Machine Learning (ML) pipeline for classification with text data is to use a Tokenizer followed by a Vectoriser to transform the text data into a vector representation and then apply a classification algorithm to learn from labelled data. Code data can be seen as a text that follows grammar rules defined by the Java Compiler. In this work, the tf-idf feature vector representation is used for classification purposes where each feature value is a real-valued number that denotes the importance of that token for a code-segment instance given the full corpus of code segments. The summary of the final dataset appears in Table 1.
Results from a comparative study of classifiers on the dataset are presented in Table 2. Overall we can see that Random Forest performs extremely well on this dataset and therefore forms the AI model for our explainability evaluation studies. It is worth noting that the CAI method used here is model agnostic and we could use it with any classifier; however it makes sense to work with one that already has high accuracy so that explanations are underpinned by accurate predictions.
The use case of code vulnerability detection can benefit from diferent types of explanations. For
example, given a code segment that is labelled “Vulnerable”, a Factual Explanation will point to
the part of the code segment that led the AI model to label it as “Vulnerable”. An example factual
explanation is shown in Figure 4 where text highlights indicate “Vulnerable” code and “Not
Vulnerable” code in a Red to Green heat map scale. For an expert user this type of explanation
should be suficient given that they have the knowledge to correct the vulnerability. In contrast,
a counterfactual explanation will compare the query with a similar yet ’Not Vulnerable’ code
segment and identify how to correct the vulnerable code segment. A counterfactual example
is presented in Figure 5 where the algorithm has highlighted the substitution changes guided
by the Nearest Unlike Neighbour. This type of explanation is informative for both expert
and non-expert users. In this paper, we focus on generating counterfactual explanations
and introduce DisCERN counterfactual algorithm specifically designed for code vulnerability
correction. DisCERN was originally proposed in [
Consider a query code segment , with statements where the ℎ statement is denoted by . If the vulnerability detection pipeline used to predict the code vulnerability consists of a Tokeniser, , and a classification model, , the outcome predicted for is . = [1, 2, ..., ] = (()) (1) For a given query , there are five steps to discovering counterfactuals with DisCERN: 1. find the Nearest Unlike Neighbour (NUN), ˆ; 2. find the feature relevance weights for the query, , using the Feature Relevance Explainer
LIME [
Once the perturbed code segment achieves the desired outcome (i.e. not vulnerable), it is identified as the counterfactual of the query. Next, we will explore each of these steps in detail.
Given a query , the NUN, ˆ, is the nearest instance found in the train data with a diferent outcome. In theory, by selecting the NUN as the starting point of counterfactual discovery, we expect to minimise the actionable changes needed to flip the classifier decision. As in Equation 2, ˆ has number of statements and the predicted outcome is ˆ. Importantly, ˆ and have diferent number of statements (i.e. ̸= ) and diferent prediction outcomes (i.e. ˆ ̸= ). ˆ = [ˆ1, ˆ2, ..., ˆ] ˆ = ((ˆ)) | ˆ ̸=
Any encoder which transforms text to a vector representation can be used to model the feature
space to find the NUN. We use the Sentence Transformers library 2 to encode code segments.
Sentence Transformers Library is a publicly available collection of state-of-the-art sentence
encoder models that are based on the Sentence-BERT architectures [
Given a code segment, , the encoder generates a vector representation, , of size . From the train data set , we filter data instances for which ̸= and create the subset ′. Each data instance in ′ is encoded using the same encoder to obtain the set of vectors ′. The cosine similarity between the query , and any other instance, can be calculated as in Equation 3. (2) (3) (, ) = √︁∑︀ =1 √︁∑︀
=1 ∑︀=1
Once pair-wise similarity is computed (between and each in ′), we select the train instance from the pair with the highest similarity as the NUN of . In the rest of this paper, this function is referred to as which given, , ′ and the similarity metric returns ˆ as the output.
In DisCERN we hypothesise that if parts of the code that contributed most to the current outcome is substituted, we can find the minimal changes needed to flip the class outcome. Here Feature Relevance Explainers can provide the knowledge needed to identify important parts in the code. Accordingly, in this section, we use LIME Feature Relevance Explainer to find the feature relevance weights of the query to identify which parts of the code contributed to the current outcome.
LIME is a model-agnostic feature relevance explainer that creates an interpretable model
around a data instance to estimate how each feature contributed to the black-box model
outcome [
In the context of code segment data, the Tokenizer, , used in the vulnerability detection pipeline considers language tokens in the code as “features”. Accordingly, LIME assigns a weight for each token which indicates how much the token contributes to the classification outcome. (, , ) → {() | () ∈ R, ∈ } (4) If the vocabulary of code segments is , LIME assigns a weight for each token in as in Equation 4. A positive weight ( ≥ 0) indicates that the corresponding token contributes positively and a negative weight ( < 0) contributes negatively towards the prediction. The weights are sorted using the partial order condition in Equation 5 to obtain the list of tokens ordered by their contribution towards the current outcome as ′.
⪯ ℛ ⇐⇒ ℛ :: () ≥ ( ) (5) arg max (,^ℎ) 7: ′′ ← (′, ′, ˆ′) 8: return ′′
Given a language token in the query, the goal of the substitution algorithm is to find a matching set of statements in the query and in the NUN to adapt the query such that it contributes towards a positive outcome change (i.e. from not vulnerable to vulnerable). A Pattern Matching () algorithm is used here to find matching statement blocks as presented in Algorithm 1 for substitution. Since the Feature Relevance Explainers used with DisCERN identify important Algorithm 1 substitute Require: ′ = [′1, ′2, ..., ′]: perturbed query Require: ˆ = [ˆ1, ˆ2, ..., ˆ]: NUN as a list of statements Require: : token in the query 1: ′ ← [ ∈ ′ | ∈ ] ◁ find the list of statements in ′ that include 2: for ∈ ′ do 3: ′, ˆℎ ← ( , [′1, ′2, ..., ′], [ˆ1, ˆ2, ..., ˆ]) ◁ find the suggested change 4: = (′, ˆℎ) ◁ similarity between current and suggested code 5: end for 6: (′, ˆ′) ← ◁ select maximum similarity pair i.e. similar yet corrected ◁ replace ′ in ′ with ˆ′ ◁ return the newly perturbed query tokens, the first step involves finding the matching list of statements ′ in the query that contains the important token . We use a simple lookup function to identify all code statements in the perturbed query ′, that contain the token (Line 1). The next steps of finding the vulnerable statements and their replacements from NUN are based on the hypothesis that if a statement in ′ is vulnerable, it must be corrected in the NUN. Accordingly, for statements in ′, we use the edit changes that are proposed by a pattern matching () algorithm and select the change (based on statement-level similarity) as the most likely substitution to correct the code vulnerability.
A algorithm like the Gestalt Patten Matching or Levenshtein Edit Distance can find the minimum changes required to transform one string to another. The changes can be “replace”, “insert” and “delete” 3. We use at the statement granularity level to find what changes are proposed for a given statement, . The algorithm will return a list of matching pairs of statements to the query () and list of statements in the NUN (ˆ) where i,k,v,w are start and end indices of statements and is found within (Line 3).
Next, we calculate the similarity between the two lists of statements using Cosine similarity. Similar to Section 3.1.1 we use the all-MiniLM-L6-v2 encoder to transform the statements to vector representations and calculate Cosine similarity. Once we have all the pairs for ′, and their similarities, we select the pair, (′, ˆ′) that has the maximum similarity. We assume a vulnerable code segment and its corrected counterpart are diferent yet carry some similarities. Accordingly, by selecting the highest similarity we expect to discard any changes found by that are not vulnerability corrections. Finally in Line 7 we replace the list of statements ′ in the perturbed query ′ with the list of code statement ˆ′ to return the new perturbed query ′′.
Algorithm 2 DisCERN Algorithm Require: = [1, 2, ..., ]: query as a list of statements Require: ((.)): vulnerability detection pipeline Require: : train dataset Require: = (()): predicted outcome of the query 1: ′ ← { ∈ | ̸= } 2: ˆ ← (, ′, ) 3: {()} ← (, , ) 4: ′ ← ℛ ({()}) 5: Initialise ′ = and ′ = 6: for ∈ ′ do 7: ′ ← (′, ˆ, ) 8: ′ = ((′)) 9: if ′ ̸= then 10: Break 11: end if 12: end for 13: return ′ ◁ filter the train dataset ◁ find the NUN using function ◁ the set of weighted tokens from LIME in Equation 4 ◁ list of tokens ordered using ℛ in Equation 5 ◁ Initialise counterfactual as query
◁ from Algorithm 1 ◁ predict outcome of the perturbed query ′
◁ check if the outcome is changed ◁ stop perturbing if outcome is changed ◁ return the perturbed query as the counterfactual
DisCERN (Algorithm 2) brings together methods from Sections 3.1.1 to 3.1.3 to discover counterfactuals. Given the query , and the train dataset , in Lines 1 and 2 we find the NUN 3In this paper we used the Gestalt Pattern Matching algorithm implemented by cdiflib Python package https://github.com/mduggan/cdiflib as discussed in Section 3.1.1. Next, we find the LIME feature weights for the query and use these to sort the list of tokens that indicate which parts of the code contributed to the current outcome (Line 3 and 4). We iterate over the list of tokens where for each token we consider a substitution correction until a prediction outcome is changed. In each iteration for a given token , Algorithm 1 finds the best matching adaptation to create a new perturbed query ′. It then obtains a prediction to check the outcome class for the adapted ′ using the original classification pipeline ((.)). The iteration is terminated when a change in the outcome is observed and the algorithm returns the perturbed query ′ as the counterfactual.
This section presents the preliminary evaluation of the counterfactual DisCERN algorithm for vulnerable code correction. To the best of our knowledge, there are no other counterfactual algorithms in the literature for counterfactual discovery for this application. Accordingly, we are not able to compare any performance metrics with other algorithms in this work. Instead, we compare DisCERN algorithm with an ablated version of DisCERN as listed below. DisCERN-rand considers a random order of statements in Algorithm 2. This is instead of using LIME feature relevance as in Lines 3 and 4. This ablated version evaluates the impact of using feature relevance knowledge to guide eficient counterfactual discovery. DisCERN: is presented in the Algorithm 2.
DisCERN algorithm is evaluated using the NIST Java dataset. We only use 10019 test data instances that were classified as ’Vulnerable’ by the vulnerability detection pipeline in the DisCERN evaluation. Following metrics are used to measure the performance of the algorithm on the NIST Java dataset.
• Validity: measures the percentage of data for which DisCERN successfully finds a counterfactual. At this stage, the only requirement for a counterfactual discovered by the DisCERN algorithm is to achieve a "positive" change of outcome 4. Given the set of test instances that were predicted ’Vunerable’ are , and the subset for which DisCERN found a counterfactual is , the validity is calculated as in Equation 6. A higher percentage of validity is desirable and unlike in DisCERN for tabular data, with unstructured data (like code), we cannot yet guarantee the validity of the generated code data.
= || × 100 || (6) • Sparsity: measures the mean number of statements that are changed for a change in outcome. Given the number of statements changed in each test instance in is 4A more stringent metric would be to evaluate if the change conforms to grammar rules of the Java Compiler, which we will explore in future work.
[1, 2, ..., ], where = ||, the sparsity is calculated as in Equation 7. In Algorithm 1, number of statement changes for operations “replace”, “delete” and “insert” are calculated as ( − , ℎ − ), − and ℎ − respectively. Lower value of sparsity is desirable. =
1 ∑︁ =1 (7) In addition to Validity and Sparsity, we also report the mean number of statements in the counterfactual discovered by each algorithm. This is to show the contrast between the mean number of statements in the test instances which is 44.62. 4.1.1. Results Table 3 presents the quantitative evaluation results of DisCERN using the NIST Java dataset. DisCERN finds counterfactuals for 96.49% of the test instances while the random version only ifnds counterfactuals for 83.48% (13% less). Sparsity is comparably similar in both algorithms, this means that they both have a comparable number of statement changes 12 ∼ 14 needed to lfip the class of the query. DisCERN-rand records a slightly lower sparsity, and this could be explained by the 13% of counterfactuals additionally found by DisCERN having longer code segments (thereby increasing the number of statement changes).
Both DisCERN and DisCERN-rand record comparable values for the mean number of statements in the counterfactuals. In comparison to the mean number of statements in the query, both algorithms find counterfactuals with 4 ∼ 6 more statements. Interestingly, the mean number of statements in the NUNs was 51.81, however, this did not mean that DisCERN completely changed each test instance to its NUN in order to achieve outcome change. We explore this further in qualitative evaluation (Section 4.2).
In the qualitative evaluation we present few examples of counterfactuals found by the DisCERN algorithm. In this preliminary work, we do not check if the counterfactual based code corrections follow grammar rules defined by the Java Compiler. However, we examined a sample of the generated code to closely examine if they are indeed correcting code vulnerabilities and are sensible modifications.
Consider Figures 6a and 6b which presents two counterfactual examples. In each, the query is on the left, the NUN is on the middle and the counterfactual (CF) is on the right. The (a) Example 1: Successful Adaptation (b) Example 2: Unsuccessful Adaptation adaptations are highlighted with coloured boxes and arrows. Example in Figure 6a shows how feature relevance weights have guided the algorithm to select correct adaptations to perturb the query into a counterfactual that is similar to NUN and achieved the class outcome change. In contrast, the example in Figure 6b shows that in some cases, the adaptation does not result in grammatically correct counterfactuals. In such cases, the vulnerability detection pipeline may recognise the perturbed query as “Not Vulnerable” but the adaptations applied to the code have lost some of the original functionality. Such examples suggests, that DisCERN is a promising approach to discovering counterfactuals, however it requires further adaptation heuristics and code correction knowledge to ensure accurate code corrections from counterfactuals.
In this paper, we presented a novel approach to finding counterfactual explanations for correcting vulnerabilities in code. We used feature relevance explainer knowledge as a proxy to discovering potentially vulnerable code statements and then used a novel substitution algorithm based on pattern matching to find corrections from the nearest unlike neighbour. Overall aim of our algorithm was to identify vulnerabilities and correct them with minimal necessary changes. We evaluated our algorithm using the NIST Java SAR dataset to find that DisCERN finds counterfactuals 96% of the time with 13 ∼ 14 statement changes to the query. Additionally, we presented example counterfactuals found using DisCERN to qualitatively evaluate the algorithm. These suggest that further correction operations and heuristics are needed to ensure plausible code changes. Future work will also expand upon our evaluation to include additional SAR datasets in diferent programming languages and the use of qualitative evaluation through crowd-sourcing techniques.
This research is funded by the iSee project (https://isee4xai.com) which received funding from EPSRC under the grant number EP/V061755/1. iSee is part of the CHIST-ERA pathfinder programme for European coordinated research on future and emerging information and communication technologies. Authors would like to thank Dr. Harsha Kalutarage and Ashish Upadhyay at the Robert Gordon University for their enlightening discussions.