-One of the challenges in the area of software engineering research has been the low rate of adoption by industry of the tools and methods produced by university researchers. We present a model to improve the situation by providing tangible evidence that demonstrates the real-world effectiveness of such tools and methods. A survey of practising software engineers indicates that the approach in the model is valid and applicable. We apply and test the model for providing such evidence and demonstrate its effectiveness in the context of static analysis using FindBugs. This model can be used to analyse the effectiveness of academic research contributions to industry and contribute towards improving their adoption.
The success of software engineering research in
universities can be measured in terms of the industrial adoption of
methods and tools developed by researchers. Current adoption
rates are low [
One approach to providing such evidence is to conduct experiments that demonstrate the effectiveness of research tools in a real-world context. We apply this approach to analyse the effectiveness of a static analysis tool. In doing so, we demonstrate that such experimentation can contribute to closing the gap between research and practice.
The structure of this paper is as follows: Section II provides
the background of our work leading to the proposed model;
Section III presents a survey which shows that real world
evidence can positively influence the decision of software
engineers to use research tools; Section IV explains our
experimental method which uses FindBugs [
Since the 1970’s there have been ongoing efforts to increase
the adoption of research outcomes outside of universities [
Researchers in universities are working towards addressing
significant problems. The outcome of their work can be a
tool or method which may or may not achieve wide-spread
industry adoption. A key factor limiting the readiness of these
research outcomes for adoption in industry is a lack of tangible
evidence that they would be effective in practice [
Figure 1 depicts our model for demonstrating the
realworld effectiveness of research tools and methods. The model
involves 4 steps with intermediate activities. First step is
to identify a problem to address. Step 2 is to develop a
tool or a method to address the problem. The intermediate
iterative activity involved between these 2 steps is the process
of solution formulation, which involves adding new ideas to
the available state of the art. These steps are followed by
iterative testing for validation in Step 3. Step 3 confirms
the readiness of the research outcome for adoption. An idea
should be validated in a practical setting [
University Research 1
In order to understand the impact of evidence from a userscenario on real-world decisions to use a research tool, we conducted an on-line survey of software developers.
The survey uses static analysis as an example and was
prepared and delivered using our university’s online polling
system [
III.B.1 III.B.2
III.B.3
We invited 20 software industry practitioners and around 10 computer science researchers with industry experience in software development.
The survey data consisted of responses to the sequence of questions depicted in Figure 2 and described below.
1) Software engineering experience: We gathered information about the level of software development expertise of each participant so that we can understand any relationship between experience and use of static analysis tools.
2) Static analysis knowledge: We asked the following questions to determine each participant’s level of understanding and use of static analysis tools.
to find potential bugs without actually executing the software. Have you heard of static analysis before?’
Answers to these questions were used to determine the final question we asked, as indicated in Figure 2.
3) The impact of the tangible evidence: At the end of the survey each participant who has not used static analysis tools was asked a question to determine the impact that tangible evidence (that the tool can identify real bugs early in the software development life-cycle) might have on their approach to static analysis. The exact question asked depended on their answers to questions described in Section III-B2. Specifically: 1) Participants who had no knowledge of static analysis were asked ‘Would our research results interest you in gaining knowledge of static analysis and adoption of automated static analysis tools?’. 2) Participants who knew about static analysis but had not used any static analysis tools (our primary group of interest) were asked to rate the impact that the following factors would have on their decision to adopt static analysis tools. A Likert scale was used with 5 options (No influence, May be, Likely, Highly Likely, and Definitely). a) Effectiveness of tool in finding bugs b) Ease of use c) Integration of tool to development environment. d) License type. e) The availability of tangible evidence that the tool can identify real bugs early in the software development life-cycle - before they are reported by users.
The response rate for our survey was high with 27 responses out of 30 invitations. Responses to the survey indicate that tangible evidence of real-world effectiveness of a tool has positive impact on decisions to adopt static analysis tools.
Analysis of the survey results provides the following specific findings: 1) Software Engineering Experience - As expected, participants had varied level of experience in software development. However, we do not find any direct relation between the experience and usage of tools. 2) Static Analysis Knowledge - Survey results show that 9 participants (33%) had no prior knowledge of static analysis. It is noteworthy that while the remaining 18 participants knew about static analysis, only 4 of them had used static analysis tools. 3) Impact of the tangible evidence - Our survey results show that tangible evidence has a positive impact on decisions to adopt static analysis tools. Out of the 9 respondents who had no prior knowledge of static analysis, 8 said that tangible evidence would interest them in gaining knowledge of static analysis and adopting automated static analysis tools. This is a valuable information indicating that providing evidence of effectiveness could contribute to improved adoption of research tools in industry. Of the 14 participants who had knowledge of static analysis but who had not used any tools, 7 participants (50%) indicated that tangible evidence would be Highly Likely or would Definitely influence their decision to adopt static analysis tools (Figure 3). Another four participants indicated that such evidence would be Likely to influence their decision. Considering the response of three participants as May be, it is possible that they respond positively, which will add to the percentage of participants agreeing that tangible evidence will influence the decision. 4.5 4 3.5 s tn 3 a p i itc 2.5 r a fP 2 o rbe 1.5 m uN 1 0.5 0
None
May be Likely Highly likely Positive influence on decisions to adopt static analysis tool Definitely
As shown in Figure 4, our results also indicate that other factors such as Ease of use, IDE integration and License type have a positive impact on decisions to adopt static analysis tools. It is interesting to note that under the May be and Definitely category, the top two influencing factors are License and Tangible evidence.
Our results clearly show that tangible evidence is an important factor in influencing decisions to adopt research tools in industry. 7 6 ts 5 n a iitcp 4 r a fop 3 r e bm 2 u N 1 0
None
May be Likely Highly likely Definitely
Influence of the factor
In order to test the applicability of our proposed model,
we first had to identify an appropriate tool developed by
researchers and a scenario to test its effectiveness. FindBugs
version 3.0 was chosen for our research as according to the
tool’s website, there are few organizations using FindBugs
[
FindBugs analyses Java class files to report warnings and
potential errors in the associated source code. The tool
performs analysis on Java class files without needing access to
the source code. It is stable, easy to use, and as mentioned
in [
We identified Eclipse as the object of analysis as it is a large widely used open source project with good records of user reported bugs over many years and versions of the software. For our experimentation we focused on the analysis of Eclipse version 4.4 (Luna) because it was the current version at the time of our experimentation.
Java Jars associated with Eclipse versions 4.4 were analysed using FindBugs version 3.0. Findbugs generated a list of warnings pertaining to code considered as faulty.
The Eclipse project uses Bugzilla to track bugs reported by users. In order to identify bugs that could be associated with FindBugs warnings, we needed to identify bug reports that included a documented stack-trace. This was achieved by performing an advanced Bugzilla search for bugs that satisfied the following criteria:
Status: NEW, ASSIGNED, VERIFIED 1.
We then inspected the query results to identify those bug reports that included a documented stack-trace.
Our last step was to match warnings generated by FindBugs (Section IV-C) with user-reported bugs (Section IV-D). This was achieved using the following steps: 1) For each of the bugs identified using the procedure described in Section IV-D, we identified the Java class that was the likely source of the reported bug. This class was usually the one appearing at the top of the stacktrace. In some cases, we had to traverse through lower levels in the stack-trace to find matching classes. 2) We then searched for the above classes in the warnings generated by FindBugs (Section IV-C) and analysed the code associated with the warning. We did this by using the FindBugs class name filter feature to show warnings related to the class of interest. 3) Finding a matching line of code in the FindBugs warnings establishes a connection between the warnings generated by FindBugs and the bugs reported by users.
1Because our experiment looks at the ability of FindBugs to find bugs that have not been fixed, we ignore the bugs with CLOSED status. In addition, we do not consider the possibility of bugs that have been closed incorrectly.
Our analysis of FindBugs warnings for Eclipse version 4.4 showed that static analysis of Eclipse version 4.4 generated warnings in the categories of Correctness (652), Bad Practice (547), Experimental (1), Multithreaded correctness (390), Security (3), and Dodgy code (55) under the rank range of Troubling (Rank 1-14), as depicted in Figure 5. We focused on the Scariest warnings (Rank 1-4), considering them as real problems requiring attention. There were 82 Scariest warnings in total. These comprised 81 warnings in the Correctness category and one in the Multi-threaded correctness category.
Additional investigation found that warnings in the Correctness category included a range of coding errors such as comparisons of incompatible types, null pointer dereferences and reading of uninitialized variables.
Dodgy code
Security s g irnn Multithreaded correctness a fryoow Experimental tgeaC Bad Practice
Correctness 0 100 200 300 400 500 600 700
Number of warnings
ported bugs: Execution of the query described in Section IV-D resulted in a dataset of 2575 bugs, which included 347 enhancements. We excluded the enhancements from our analysis. Out of the remaining bugs, we have analysed 1185 bug reports so far, 90 of which included a documented stacktrace.
We used the method described in Section IV-E to compare the stack-trace in these 90 bug reports with the warnings generated by FindBugs (Section IV-F1). We found that six of the user reported bugs could be associated with FindBugs warnings as presented in Table I. The data presented in the table includes the Bugzilla Bug ID, a description of the bug, and a description of the warning generated by FindBugs.
The model proposed in this paper considers that in order to improve the adoption of university research outcomes, researchers need to demonstrate its effectiveness in a real-world scenario and think about the value of their research outcomes beyond the lab boundaries. Our main purpose in conducting the experiment described was to test the applicability of our proposed model by analysing the value of a research tool in industrial practice. Specifically, we evaluated the performance of the FindBugs static analysis tool by analysing its capability FindBugs Warning Method call passes null for nonnull parameterThis method call passes a null value for a nonnull method parameter. Either the parameter is annotated as a parameter that should always be nonnull, or analysis has shown that it will always be dereferenced. Bug kind and pattern: NP - NP NULL PARAM DEREF Load of known null value. The variable referenced at this point is known to be null due to an earlier check against null. Although this is valid, it might be a mistake (perhaps you intended to refer to a different variable, or perhaps the earlier check to see if the variable is null should have been a check to see if it was nonnull). Bug kind and pattern: NP - NP LOAD OF KNOWN NULL VALUE Possible null pointer dereference There is a branch of statement that, if executed, guarantees that a null value will be dereferenced, which would generate a NullPointerException when the code is executed. Of course, the problem might be that the branch or statement is infeasible and that the null pointer exception can’t ever be executed; deciding that is beyond the ability of FindBugs.
Bug kind and pattern: NP - NP NULL ON SOME PATH Non-virtual method call passes null for nonnull parameter A possibly-null value is passed to a nonnull method parameter. Either the parameter is annotated as a parameter that should always be nonnull, or analysis has shown that it will always be dereferenced. Bug kind and pattern: NP - NP NULL PARAM DEREF NONVIRTUAL Boxing/unboxing to parse a primitive A boxed primitive is created from a String, just to extract the unboxed primitive value. It is more efficient to just call the static parseXXX method.
Bug kind and pattern: Bx - DM BOXED PRIMITIVE FOR PARSING Possible null pointer dereference There is a branch of statement that, if executed, guarantees that a null value will be dereferenced, which would generate a NullPointerException when the code is executed. Of course, the problem might be that the branch or statement is infeasible and that the null pointer exception can’t ever be executed; deciding that is beyond the ability of FindBugs.
Bug kind and pattern: NP - NP NULL ON SOME PATH Possible null pointer dereference There is a branch of statement that, if executed, guarantees that a null value will be dereferenced, which would generate a NullPointerException when the code is executed. Of course, the problem might be that the branch or statement is infeasible and that the null pointer exception can’t ever be executed; deciding that is beyond the ability of FindBugs.
Bug kind and pattern: NP - NP NULL ON SOME PATH in generating warnings relating to real-world bugs reported by users of the Eclipse IDE.
Our results indicate that FindBugs is capable of identifying bugs that will manifest themselves as bugs reported by users. Since real-world evidence would influence the decision to adopt a tool as indicated in the survey results, FindBugs needs to improve the percentage of such warnings to make the evidence convincing. Currently, FindBugs does not have the intelligence to track the bug among the list of false positives that can manifest. Our research provides improvement directions to FindBugs in identifying the important bugs from the warning base by analysing the historical data and user requirement. FindBugs results can be improved by introducing a new ‘user-impact’ category to classify the warnings which will potentially have an impact in the client environment and hence, need immediate attention. For this, sufficient information from industrial practice needs to be applied into testing the research tool.
The model appears simple but it is challenging for researchers to identify a scenario and approach users and/or data to demonstrate the effectiveness of their tool or methods. It might be difficult for them to find the user-filed data always. Also, once they have the data to demonstrate the effectiveness, they need a mechanism to propagate it to industry. Also, the concern about the relevance of research suggests that researchers need to think about the relevance of the problem they are trying to address. These factors indicate that universities and industry need to start collaborating at an early stage and consider co-developing whenever possible and feasible. This can pave a good start towards improving the relevance and adoption of research outcomes in general, and bridging the gap between industry and academia.
Our experiments were limited by the small number of Eclipse bugs reported with a documented stack-trace. It is important to note that the ability to analyse only 90 of the 1185 bugs considered reflects a limitation of our approach and does not reflect a limitation of FindBugs. There are some limitations to the validity of our experiments: 1) FindBugs does not always point to the exact line number referred to in the stack trace. It might be possible that the source of error could be different from the warnings provided by FindBugs. 2) While it is likely that the FindBugs warnings listed in
Table I are the actual cause of the listed real-world bugs
reported by Eclipse users, we cannot be certain of this.
3) As there is lack of literature detailing the use of static
analysis tools like FindBugs by the Eclipse development
team, a comparison study was not feasible.
4) This test highlights the relevance of static analysis tools, low-adoption rates were due to a lack of awareness of static
though the effectiveness in real-world projects, particu- analysis tools among developers.
larly large scale projects, cannot be confirmed as the sam- None of the work described above analyses the effectiveness
ple size of our survey is small. However, by considering of FindBugs in identifying problems that manifest themselves
the sample sizes of 20 and 18 used in related studies [
Adoption of software engineering research outcomes in
industry practice has been a concern [
Various experiments have been conducted to demonstrate
the effectiveness of the FindBugs static analysis tool by
showing that it is able to detect the specific problems it has been
designed to detect [
Al Bessey et al. [
Johnson et al. [
We have proposed a model to contribute towards improving the adoption of research tools by industry by demonstrating the effectiveness of the tool in real world scenario. We have presented a mechanism which involves a research tool as a medium of building tangible evidence. A survey of software developers supports our hypothesis that such tangible evidence of effectiveness of a tool can have a positive influence on realworld decisions to adopt static analysis tools.
Further experiment for testing the applicability of the model in the static analysis context was conducted. In this experiment, by establishing a connection between user-reported bugs and warnings generated by the FindBugs static analysis tool, we have demonstrated the ability of static analysis tools to eliminate some defects before software is deployed. However, the evidence needs to be stronger regarding the number of such connections in order to be more convincing and improving the industrial adoption of the tool.
Future research would present more detailed analysis of the complete list of the bugs found in Section IV-F2, which will provide us precise data about the effectiveness of the tool according to our approach. Our approach also presents a scenario where industry and university researchers can work together to create more useful tools. We plan to discuss these results with the FindBugs development team to explore the possibility of strengthening the evidence and devising a new classification user-impact to indicate the warnings that would manifest in client-environment.
Finally, we would like to adapt this approach to explore the effectiveness of research tools involved in other phases of the software development life-cycle.