Finding software bugs and security vulnerabilities using static source code analysis is a viable method. Static source code analysis techniques are already suficiently matured for industrial use, and numerous tools have been developed to aid in the automatic detection of software faults. In this paper, the capabilities of three static source code analysis tools are investigated with respect to identifying null pointer dereference in Java source code. Our research uses artificial test cases as a benchmark. The study reports performance results based on five metrics. The experiments show that Facebook Infer outperforms the other tools in identifying null pointer dereference.
Static analysis technologies uncover security flaws early in the software development phase,
which saves time and cost and ensures quick feedback for the responsible programmer. The
security vulnerabilities and bugs that can be identified using static analysis techniques are in
wide ranges, starting from simple programming errors, and ending with more complex ones
like access control issues [
Null pointer dereference is a memory access error [
There are many static analysis tools – with varying capabilities – which can detect null
pointer dereference. It is known that static analysis for an “interesting” problem is undecidable,
CEUR
i.e., it is not possible to build an algorithm that produces an accurate answer in each and every
case [
With the goal of better understanding their strengths and shortcomings, this paper focuses on an empirical assessment of three static analysis tools for their performance in identifying null pointer dereference in Java source code.
For this purpose, we used the CWE476 Null Pointer Dereference test cases of the Juliet Test
Suite [
The main contributions of this work are as follows.
• We present quantitative results on the performance of three well-known, free, and opensource static analysis tools in identifying null pointer dereference in Java source code, based on the Juliet Test Suite. • We present the numbers and types of shared, unique, and missed detections of these tools. • We report on five performance metrics – recall, false alarm rate, precision, G-score, and
F-measure – for the analyzed test cases.
The main empirical observations of this work are the following.
• None of the used tools were able to detect all null pointer dereference errors in the test cases of the Juliet test suite, specifically 8% of the flawed constructs have been missed by the tools. • 14% of the vulnerabilities were detected only by Facebook Infer – these have been completely missed by the other two tools. Moreover, Infer gave the highest calculated recall, G-score, and F-measure.
• SpotBugs gave the highest calculated precision and the lowest (best) false alarm rate.
The remainder of the paper is structured as follows. Information on the static analysis tools and the Juliet benchmark, as well as some preliminaries of the research, are presented in Section 2. Then, Section 3 introduces the applied research method. The results of analyzing null pointer dereference test cases by the three tools are given in Section 4. We discuss related work in Section 5. Finally, Section 6 draws the conclusions.
Null pointer dereference is a major source of bugs and vulnerabilities in programming languages
without static typing support for nullable and non-nullable references [
The Common Weakness Enumeration (CWE) “is a community-developed list of software and
hardware weakness types. It serves as a common language, a measuring stick for security tools,
and as a baseline for weakness identification, mitigation, and prevention eforts” [
Juliet Test Suite [
When a static analysis tool correctly reports one or more positives, this is referred to as a true positive. Consequently, when a static analysis tool mistakenly reports a negative, this is referred to as a false positive. A false negative is the case when the tool misses one of the positives. Finally, a true negative is the case when a negative is, correctly, not reported.
The versions of the tools used in this study are Infer 1.1.0, SonarQube 8.8.0.42792 (Community Edition) with its SonarScanner 4.6.0.2311, and Spotbugs 4.2.3. Moreover, we have used the most up-to-date version of Juliet Java, which is version 1.3.
Let us first explain the design of the experiment, including the used metrics. Then we present the details of the implementation of the experiment.
Our study follows a two-factorial experiment design aimed to explore the abilities of three selected static analysis tools in the detection of null pointer dereference in the Juliet Java test suite. The two factors (i.e., independent variables) are the following: (1) static analysis tool, and (2) type of vulnerability.
The levels of the first factor are the specific tools used for evaluation. We started the tool selection process by compiling a survey that consisted of twelve commercial tools and eight opensource tools. The main characteristics of each tool were extracted from their documentation. The criteria for making the selection were as follows: (1) it has to be free and widely used; (2) it should specifically identify null pointer dereference; (3) it should support Java. Therefore, we excluded all the commercial tools, and from the list of the eight open-source tools, we ifrst excluded the tools that did not support Java and tools that did not include specific rules for identification of null pointer dereference. Hence we ended up choosing Facebook Infer, SonarQube, and SpotBugs.
For the second factor, there is a single level: we investigate null pointer dereference only, which is CWE476.
As response variables (i.e. dependent variables) we used five metrics: recall, false alarm rate, precision, G-score, and F-measure; these reflect several dimensions of the performance of the tools. To compute these response variables, we start by calculating true positives (TP) and false positives (FP) from the reports of each tool. These, along with the number of positives and negatives are then used to compute the above metrics. False alarm rate =
Precision =
G score = F measure = + 2 ∗ ∗
+ 2 ∗ ∗ + (1) (2) (3) (4) (5)
Recall describes the capability of accurately detecting vulnerabilities; it is defined as the ratio of true positives to the total number of positives (Eq. (1)). Hence, Recall is restricted to faulty constructs and shows the percentage of flawed constructs successfully recognized by a tool.
False Alarm Rate is exclusively concerned with unflawed test cases, and it reflects the percentage of unflawed test cases that have been misidentified as flawed ones. It is the ratio of false positives to the total number of negatives, or the ratio of mistakenly reporting unflawed constructions as flawed ones. It is given by Eq. ( 2).
Specificity (used in the computation of G-score) is directly related to false alarm rate. It is the ratio of true negatives to all negatives (hence it is equal to 1 − False Alarm Rate).
Precision, given with Eq. (3), provides the ratio of true positives to the sum of true positives and false positives. Hence Precision is concerned with all the reported test cases; it shows the percentage of successfully detected flawed constructs to the number of (flawed and unflawed) constructs reported by a tool.
Eq. (4) defines G-score, which is the harmonic mean of recall and specificity. It enables us to merge two key metrics into a single one. Similarly, the F-measure is the harmonic mean of recall and precision (Eq. (5)). They both describe the accuracy of the analysis.
High recall, precision, G-score, and F-measure, as well as low false alarm rate values, indicate
higher performance, and they are all in the interval [
The execution of our experiment can be structured into six major steps, as described in the lfowchart shown in Figure 1.
Step 1: Evaluate CWE476 test cases. According to previous research [
ence. Based on the documentation of Infer, we determined the analyzers which target null pointer dereference. This step is important because there may be some checkers that are not turned on by default. The analyzers are Pulse and Biabduction. The latter works by default when running Infer, while Pulse should be explicitly activated. Note that SonarQube and SpotBugs related analyzers are all activated by default.
analyzers of Infer mentioned in Step 2 have been run on all the test cases that were compiled in
Step 1. This resulted in an output report which contains all detections. Find more details about
the used command and other technicalities in the author’s GitHub account [
the shared, unique, and missed detections, we used the results obtained for Infer from Step 4,
and we also brought here the analysis results for SonarQube and SpotBugs from our previous
study [
We computed true positives and false positives in Step 3 as listed in Table 1. Those values together with positives and negatives from Step 1 are used in this step to compute the metrics mentioned in Section 3.1.
The quantitative results of the execution of the experiment described in Section 3 and their qualitative analysis are presented now.
In Step 1 of the experiment, the test cases in Juliet Java 1.3 were thoroughly investigated. It turned out that although documentation of Juliet mentions that the number of positives is 198 for CWE476, 17 of them are incorrectly labeled as positives (they do not result in a null pointer dereference). Consequently, we have excluded them from the positive test cases. Moreover, since the number of negatives is not mentioned in the documentation, we could simply count the number of “good” methods (which is 496) – however, after the manual review 30 of them turned out to be incorrectly labeled as good, and hence they must be excluded from negatives. The final number of positives and negatives used in this study are 181 and 466, respectively.
More details about the excluded test cases and the script used for counting negatives are
provided in the author’s GitHub account [
The results of running the three tools on Juliet’s null pointer dereference test cases are shown
in Table 1. The figures presented here for SonarQube and SpotBugs originate from our previous
research [
In SpotBugs, detected bugs have priorities and they can be shown at three levels: (i) high, (ii) high & normal, and (iii) all priorities. These priorities present confidence levels. They have been named as SpotBugs(h), SpotBugs(h&n), and SpotBugs(all) in Table 1. In our analysis below we rely on the SpotBugs(h&n) configuration because it gives the best results among the three options. The best result in this context means the one with a good balance of high true positives and low false positives.
We can see from Table 1 that currently, the best performing tool for this particular problem is Facebook Infer. This tool gave the highest number of true positives with a reasonably low number of false positives. The low rate of false positives in Infer is due to its ability to avoid infeasible control paths.
For a better understanding of how well the tools are performing in the detection of CWE476, it
is important to further analyze the results of Section 4.1 with the help of a manual review, and
to investigate the number and characteristics of detected test cases shared among some, or all,
of the tools. It is also instructive to see those true and false positives which are detected by a
single tool, and those cases which have been missed by all the tools (viz. false negatives). For
more details see the author’s GitHub account [
Shared and Unique Detections. Figure 2 presents the number of shared and unique detections by the tools. The most important observations are the following: 1. There are 25 positives (15% of the 166 true positive detections, and 14% of all 181 positives) which were detected by Infer only, and not by the other tools. These test cases require data flow analysis among methods located in diferent classes and files (as an example, see Listing 1). 2. The three tools detected 106 positives in common (64% of the 166 true positive detections, and 59% of the 181 positives). These test cases of Juliet Test Suite are straightforward cases of null pointer dereference, and the dereference occurs in clearly feasible execution paths. According to the terminology of the Juliet Test Suite, these test cases belong to the baseline and control flow category, and they require control flow and data flow analysis between diferent methods of the same class definition. 3. There is not a single tool that is able to detect all of the positives. 4. The tools have detected 92% of the positives. In particular, Infer was able to detect this 92% of positives itself, i.e. it could detect everything that other tools could detect (disregarding Spotbugs(all) here, as that configuration yields an unacceptable amount of false positives). 5. The tools have reported 92 false positives. Infer and SonarQube have 8 in common. These negatives rely on the value of a modifiable instance field; the tools behave conservatively by assuming that such a field might be changed somewhere. One can also observe that Spotbugs(h&n) performed extremely well in this respect by not giving any false positives.
Listing 1: An example for unique detections by Infer in Juliet Java 1.3 public class CWE476_NULL_Pointer_Dereference__int_array_22a extends AbstractTestCase { /∗ The public static variable below is used to drive control flow in the sink function . ∗ The public static variable mimics a global variable in the C/C++ language family. ∗/ public static boolean badPublicStatic = false ; public void bad () throws Throwable { int [] data = null ; data = null ; /∗ POTENTIAL FLAW: data is null ∗/ badPublicStatic = true ; (new CWE476_NULL_Pointer_Dereference__int_array_22b()).badSink(data ); } public class CWE476_NULL_Pointer_Dereference__int_array_22b { public void badSink( int [] data ) throws Throwable { if (CWE476_NULL_Pointer_Dereference__int_array_22a.badPublicStatic) {
IO. writeLine ( ”” + data . length ) ; /∗ POTENTIAL FLAW: null dereference will occur if data is null ∗/ } else { /∗ INCIDENTAL: CWE 561 Dead Code, the code below will never run ∗ but ensure data is initialized before the Sink to avoid compiler errors ∗/ data = null ; }
} 1 2 3 4 5 6 7 8 9 10 11 } 12 13 14 15 16
Missed Test Cases (False Negatives) We mentioned earlier that the tools of this study detected 92% of the positives. This means that only 8% of the positives was missed by Infer, and these have been missed by all the other tools as well. The missed test cases all rely on passing a data structure (an array, a Vector, or a LinkedList) storing a null value, or a serialized object having a field with a null value as a parameter to a method. The detection of vulnerabilities of this type may require built-in (lexical) knowledge about the standard library (the logic of data structures and the behavior of serialization), which – considering the volume of the Java standard library – is indeed a really demanding requirement.
Listing 2 presents a test case missed by Infer and the other two tools. The test case illustrates a vulnerability that involves more than one method. A static analyzer is expected to detect the bug either at line 10, or line 19, or both. Identifying this bug requires analysis that crosses method boundaries, which may be achieved by global analysis approaches. Infer can successfully detect such positive test cases in general. However, this test case requires maintaining (and passing around) information about an element of an array, which proved to be too sophisticated for Infer. Similar missed positives occur in Infer when an element of some other data structure is set to null.
Listing 2: An example of missed test cases in Juliet Java 1.3
The last step of our experiment described in Section 3 is to calculate the specified metrics. Figure 3 presents these for the three tools, characterizing their performance in the classification of the null pointer dereference test cases of the Juliet Test Suite. As mentioned in Section 4.1, in the case of SpotBugs, results of high & normal priority have been considered in this research (i.e., SpotBugs(h&n)).
The Recall values are shown in the first column. It represents how good the identification of the positives is, without taking into account the faulty reports of negatives. Higher recall indicates better performance. The findings revealed that SonarQube and SpotBugs had comparable recall values, whereas Infer had the best recall.
Column 2 displays the values for the false alarm rate. Smaller values here indicate better performance since the false alarm rate involves unflawed structures being mistakenly classified as faulty. It can be observed from Figure 3 that SpotBugs had the lowest false alarm rate among the three tools, but Infer performed also quite well in this respect.
The third column in Figure 3 presents the precision of the three tools in identifying null pointer dereference. The precision measures how good the identification of faulty methods is compared to all the reports. Higher precision indicates better performance. SpotBugs has the highest performance for precision, followed closely by Infer.
The G-score and the F-measure enable us to combine two metrics into one, as discussed in Section 3. Recall & false alarm rate are combined for G-score and recall & precision are combined for F-measure. For both metrics, values close to 1 indicate that the tool can detect a specific weakness well, with no or very few false positives. This can be observed in the case of Infer (0.95 and 0.92, respectively). SpotBugs and SonarQube performed significantly worse: both G-score and F-measure for SpotBugs(h&n) are 0.83, while SonarQube yields 0.73 and 0.62, respectively.
A variety of methodologies for evaluating static analysis tools are in use today. For benchmarking purposes, some researchers work with real code bases, while others rely on synthetic ones, or even both. Our study falls in the second category.
In a prior study [
Stephan Lipp and others [
Goseva-Popstojanova and Perhinschi [
Wouter Stikkelorum [
The results of this research revealed that Facebook Infer is quite reliable when used for identifying null pointer dereference in the Juliet Java test suite. We can also conclude that SpotBugs is fairly good in this vulnerability with the “high & normal” configuration, while SonarQube performed worst mostly due to the high number of false positives.
• 14% of the flawed constructs were detected only by Infer, and not the other tools. • All the flawed constructs that have been detected by SonarQube and SpotBugs have also been detected by Infer, with a negligible number of false positives. • SpotBugs is another very good static analysis tool, which, with well-chosen settings, gives highly accurate results.
• There is still room for improvement in static analysis tools.
Midya Alqaradaghi has been supported by the Stipendium Hungaricum program. The research of Tamás Kozsik has been supported by project no. TKP2021-NVA-29, which is implemented with the support provided by the Ministry of Innovation and Technology of Hungary from the National Research, Development, and Innovation Fund, financed under the TKP2021-NVA funding scheme.