Finding bugs in software using automated tools has enjoyed generous attention for as long as humans wrote software. With the increase in computing capacity and advancement in compiler theory, static analyzer tools like the Clang Static Analyzer have been an unexpandable part of several software development projects. The Clang Static Analyzer, like many other tools of its kind, employs heuristics, which may lead to reports on correct code - so-called false positives. The evaluation and the potential ifxing of the reports cannot be automated, requiring valuable time from the most expensive and least available resource during development, a human expert. While the finding of bugs stands in the focus of academic research, the intelligible presentation of those findings to the user was less frequently discussed. In our paper, we survey several reports emitted by the Clang Static Analyzer to understand what makes a bug report hard to understand. Our study aims to pave the way for further research to fix the problems discussed here.
Maintenance costs take a considerable chunk out of the budget of any development team. Most
of these expenses are spent fixing bugs. The earlier a bug is detected, the lower the cost of the
ifx [
Static analyzers provide early feedback on the quality of software by design. It is a popular
method for finding bugs and code smells, and various tools implement it [
Most static analysis methods apply heuristics, meaning they may often underestimate or
overestimate program behaviour [
In this paper, we survey bug reports generated by the Clang Static Analyzer, inspect various selected examples, and discuss where communication with the user is lacking. Alongside an evaluation of how many reports are intelligible from the complete report set, we highlight two specific faults observed in several reports.
In this section, we discuss the way we approached surveying the Clang Static Analyzer’s reports.
A decent simplified explanation of how symbolic execution [
The Clang Static Analyzer [
Following the above analogy, the interpreter is the analyzer’s engine, and checkers define pre- and postconditions. For instance, core.DivByZero defines that the denominator must not be 0. If the engine can prove that the denominator is 0, a warning is emitted; otherwise, it is assumed that it is non-zero. Some checkers implement more sophisticated semantics – unix.MismatchedDeallocator keeps track of whether malloc() or operator new was allocated some memory and whether the corresponding deallocator was used to release it.
The former example is a checker that relies strongly on the correct operation of the engine. While the latter example relies on the engine as well, its own bookkeeping is a large component of its logic. For this reason, we chose to limit our investigations to core.DivByZero, because we are more interested in the general mechanics and faults of bug report generation and less so in reports on specific domains. Acid fmpeg LLVM + Clang OpenSSL postgres QTBase Vim Xerces Total
Total reports 1 6 11 1 2 7 2 1 31
We tested the latest version of the analyzer1 on the following open-source C and C++ projects:
Acid [
Table 1. partially summarizes our findings. We sorted the results we surveyed into the 3 categories: • Acceptable: It is possible to understand the report, and whether it stands, even if it could be improved. • Not enough info: It is not possible understand the report, but it is possible to say which function calls / value changes the analyzer neglected to explain, and a domain expert may possess the missing information and judge whether the report stands. • Incomprehensible: The entire bug report is incomprehensible, and its doubtful that even a domain expert can judge the report.
Overall, we found that out of 31 reports, 11 division by zero reports were not acceptable. Of that, 6 reports were incomprehensible; 4 was reported in QTBase, and concerned code where numerous integers and floating point numbers were used in a hard-to-follow manner. An example is shown on Figure 1. For the other reports, it was more apparent that the analyzer was at fault for failing to generate a proper report, rather than the source code being messy. We feel that the analyzer was also the tool to blame for the reports that lacked more information. An example for this case can be seen in Figure 2.
Out of 31 reports, only 20 reports were at all acceptable. That is not to say that extracting the necessary information was easy but at the very least possible. We identified a pattern of indirect reasoning, shown in Figure 3. The example shows that i’s value is 0 when we assume it to be equal to NumElts, which is why the division-by-zero warning occurs at the remainder operator. 1The latest version of Clang at of time of writing is 16.0.6. While the example is only a few lines, we found that this indirect reasoning in other reports is often spread over several function calls, several hundred lines of code, and several logical statements that wear on the memory of the user. It would be easier to understand this report if the warning stated “NumElts is known to be 0 because it is equal to i, which has a value of 0.”.
Supporting findings of a previous research [
Symbolic execution, a static analysis technique, is a powerful tool to find deeply rooted programming errors. Despite its strengths, it often emits bug reports that leave much to be desired, demanding even more of the most expensive resource and least available in a software development project: human experts.
We surveyed a a popular static analyzer tool, the Clang Static Analyzer, that checks code of the C family of languages with a complex technique called symbolic execution. We discussed that it consists of an engine that interprets the code and several checkers defining programming semantics. We took measurements of a checker that does relatively little modelling and relies mainly on the engine and generalized bug reporting techniques.
We found that out of 31 reports, 6 are incomprehensible, 5 do not contain enough information to understand the finding, and 20 are acceptable, even if poor. We identified two specific classes of problems: one where chains of logical statements need to be deciphered by the user to understand the report, and one where only the last few notes of the report were actually important.