In this paper, we describe the formatting guidelines for ACM SIG Proceedings. Software testing is an essential process during software development and maintenance for improving software quality. Test code, the artefact during software testing, has been widely used in many software quality assurance techniques. Traditionally, software quality assurance techniques, e.g., automatic bug repair, fault localization, test case prioritization, and mining API usage from test code are based on the hypothesis of a sound quality of the test code. However, via empirical study on four open source projects, we found that the quality of test code is quite low comparing with corresponding source code, and this might hurt the above software quality assurance techniques. In this paper, we studied more than 140,000 LOC(lines of code) test code from four large scale and widely used open source projects and found that it is common for test code to be unregulated and of low-quality in open source projects. First, the comment clone ratio, unreleased resource ratio and clone code ratio of test code is much higher than that of corresponding source code; second, bug-fixed coverage is down to 0. We have learned the following lessons: the quality of test code is quite low comparing with corresponding source code, and the low quality test code may misguide existing software quality assurance techniques.
Software testing is an essential process during software development and maintenance. Test code is widely used as the artefact during software testing for ensuring software quality. Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes
Therefore, it’s critical to maintain high quality test code. Existing
work [
Traditionally, software quality assurance techniques, e.g.,
automatic bug repair [
In this paper, via empirical study on four large scale and widely used open source projects, we found that the quality of test code is quite low comparing with corresponding source code, and this might have negative impact on the above software quality assurance techniques. We studied more than 140,000 LOC test code from four large scale and widely used open source projects and found that it is common for test code to be unregulated and low-quality in open source projects. Specifically, first the comment clone ratio, unreleased resource ratio and clone code ratio of test code is much higher than that of corresponding source code; second, bug-fixed coverage is down to 0.
We have learned the following lessons: the quality of test code is quite low comparing with corresponding source code, and the low quality test code may misguide existing software quality assurance techniques. The main contributions of this work include: 1. We proposed five criteria for the measurement of test code quality. 2. Based on proposed criteria, we measured test code quality of four large scale, widely used open source projects. Results show that the quality of test code is quite low comparing with corresponding source code. We further discuss the potential impact of low quality test code on existing software quality assurance techniques. To the best of our knowledge, this is the first work to report that the test code is low-quality and untrustworthy, which should be taken seriously.
In the remainder of this paper, section II presents points on which the author would like to get the most advice on; Section III presents essential background and related work of our study; Section IV shows our motivation; Section V present the three categories for the measurement of the test code quality. Section VI explains how we conduct our empirical study; Section VII discusses the threats to this work; Section VIII concludes this paper and discuss our future work.
As for the points on which we would like to get the most advice on, we are thinking about the possibilities of proceeding our research further. Specifically, we plan to conduct quantitative and qualitative studies to explore how exactly the low-quality test code could impact software quality assurance techniques, i.e., automatic bug repair, fault localization, test case prioritization, and mining API usage. We will appreciate it if mentors could give insightful suggestions on whether the work is valuable and how it could be effectively done. Also, any feedback on the structure and content of the paper would be welcome.
Automatic Bug Repair: is the process of automatically
generating patches for repairing bugs. A lot of studies have been
carried out to address this issue. Weimer et al. [
Fault Localization: is the indispensable process to identify
exactly where the bugs are before fixing them. Xuan et al. [
execution order of test cases to maximize specific objectives.
Elbaum et al. [
and learning the correct usage of APIs of libraries are significant
but complex activities for developers. Ghafari et al. [
Test Code Quality: It is critical to detect the quality of test code, however there are not enough work done in this field.
Athanasiou et al. [
As is mentioned above, test code is closely related with software quality assurance techniques, e.g., automatic bug repair, fault localization, test case prioritization, and mining API usage from test code. These studies are based on the hypothesis of a sound quality of the test code. Thus, the quality of test code is critical for the performance of these techniques. In this study, we try to conduct a pilot study to explore the quality of test code according to the five criteria.
In this section, we present the three categories for the measurement of test code quality.
This category consists of test criteria that focus on measuring the error detection ability of the code. Unreleased resource and code clone are the main criteria for this category.
Unreleased resource is a kind of incorrect use of APIs. Unreleased resource occurs when developers fail to release resource such as File, ResultScanner and so on. When developers finished the input and output operations on a file object, developers should close file and release resources, or there will be a potential memory leak vulnerability. Moreover, developers should close the resource with finally clause to ensure the resource is closed no matter what happens in the try block.
The following real test case from Ant 1.9.4, is an example where the method did not close the file object it opened.
Ratio# 0.20 # Clone Com(Comment) are the number of test classes which clone comment, and Clone Comment Ratio represents the number of clone comment classes out of the total number of test classes. Unreleased Resources are the number of test classes which do not release resources, and Unreleased Resource Ratio represents the number of test classes which do not release resource out of the total number of test classes. Code Clones represent the lines of clone code, and Clone Code Ratio represents the lines of clone code out of the total lines of code. 1 public void testPassFile () throws Exception { buildRule . executeTarget (”test3 ”); File f = new File ( buildRule . getProject (). getBaseDir () , “testpassfile .tmp ”); assertTrue ( . . . ) ; assertEquals ( . . . ) ;
In this study, the unreleased resource ratio is defined as the following: Unreleased Resource Ratio = (#unreleased resource classes)/TC where #unreleased resource classes shows the number of the classes which did not release resources, and TC shows the total number of the classes.
We have developed a tool to automatically examine the code, and checked whether every resource was closed after being opened and used. If the resource was closed, we checked whether the close statement is enclosed in finally blocks. We counted both the resources which were not closed and those that were not closed in finally block for the unreleased resources.
Code clones are separate fragments of code that are very similar.
They are a common phenomenon in the open source systems
which have been under development for some time. Clone codes
are often referred to due to the difficulty it makes in changing and
maintaining the open source systems since developers have to
locate and update many fragments frequently. For example,
Fowler [
Clone Code Ratio = (#clone code lines)/LOC where #clone code line shows the lines of clone code, and LOC shows the lines of code.
To estimate the clone code, we use PMD’s Copy/Paste Detector
(CPD) [
Criteria that fall inside this category focus on measuring the loss detection ability of the code. To identify the loss of code, we choose code coverage and bug-fixed coverage as the main criteria.
Code coverage is the most frequently used metric for test code
quality assessment [
In general, after fixing a bug, new unit test should be created and added to the regression test suite to ensure this bug will not be reintroduced in the following versions of the projects. The bugfixed coverage is a measure used to describe the degree to which extent the fixed bug is covered by test code. If a fixed bug has not been covered by test cases, this fixed bug will be under high risk of reopening. In this study, the bug-fixed coverage is defined as the following:
Bug-fixed Coverage = (#tested bugs)/ (#fixed bugs) where #tested bugs is the number of fixed bugs which is tested in the current version of the project, and #fixed bugs is the number of fixed bugs in the current version of the project.
To calculate the bug-fixed coverage, we first collected the fixed bugs for the responding version of the four open source projects from their release notes; second, we found the fixed bugs which are not tested in the current version, manually. Specifically, the first author is responsible for the collection of un-tested fixed bugs, after which the second author recollected the data. Then results are merged, and conflicts were get resolved by a joint pairinspection of all three authors.
This category consists of test criteria that focus on measuring the detection of the unreadable code. Clone comment ratio is the main criterion for this category.Comments make the code developerreadable. They generate code documentation in predefined format. Clone comments are the same prologue comment of different methods, which makes the code inconsistent with the code documentation and hard to understand. Often this occurs when developers miss to change the clone comment.
Consider the following real test code from Ant 1.9.4: 4 5 } 9
In this study, the clone comment ratio is defined as the following:
Clone Comment Ratio = (#clone comment classes)/TC where #clone comment classes shows the number of the classes which clone comment, and TC shows the total number of the classes.
The logic of the two test cases are different, yet the comments are the same, which might cause the developers’ confusion in terms of the understanding of the test logic of these two test cases. We developed a simple tool to detect cloned comments, which could look into the code and locate the functions that share comment, and collect the clone comments automatically.
In this section, we present the quality of test code of the four open source projects according to the five criterion respectively.
In this paper, we try to explore the quality of test code of four large-scale and widely used open source projects, i.e., Ant, Maven, Log4j, and Commons Math, using the five criteria. For each project, we extracted the test code and source code separately from the latest version. Details of these projects are shown in Table 1.
As is shown in Table 1, in the four projects, the unreleased resource ratio for test code is up to 32% and on average it is 20%, while the average ratio for source code is less than 7%. Overall, all the unreleased resource ratios in test code are much higher than that in the corresponding source code.
The results indicate that the quality of source code is much higher
than test code. As the test code is widely used for mining API
usage examples [
As is shown in Table 1, in the open source projects, the clone code ratio for test code is up to 40% and on average the ratio is about 23%, while the average ratio for source code is about 17%. Overall, all the clone code ratios in test code are much higher than that in corresponding source code.
Clone test code is harmful for software quality, which increases test maintenance overhead and propagates any pre-existing errors. Clone test code always cover similar source code, which reduces the discrimination among test code and might hurt fault localization and test case prioritization.
As is shown in Table 2, in the four projects, the code coverage for test code varies from 75% to 85%, while the average ratio for code coverage is around 80 %. Overall, the code coverage are high enough to basically cover the test of the source code.
Results are shown in Table 2, Bug-fixed Coverage in our studied versions of Ant and Log4j is 0, which means none of the fixed 20 bugs in Ant 1.9.4 and Log4j 1.2.17 has been tested after fixing. In Maven, and Commons Math, values of Bug-fixed Coverage are also quite low, more than 70% fixed bugs are not addressed in test code. Overall, the bug-fixed coverage is so low that the test for the fixed bugs are not sufficient.
The low Bug-fixed Coverage values in software projects make it hard to practice test case prioritization, and also hurt the effectiveness of automatic bug repair. Since test case prioritization aims to arrange test cases based on the code coverage for accelerating bug detection, the low Bug-fixed Coverage means that when un-tested fixed bugs are re-introduced, prioritized test cases might not reveal these bugs; for automatic bug repair, the low Bug-fixed Coverage makes the evaluation inefficient. Automatic bug repair evaluates its repair patch by running all test cases. If all test cases are passed, the generated patch will be treated as an effective repair patch for a bug. However the low Bug-fixed Coverage means many fixed bugs are not tested by the existing test cases. So when evaluating generated repair patches, even all test cases are passed, these un-covered fixed bugs might be re-introduced.
As is shown in Table 1, the average clone comment ratio for test code is about 20% and the average clone comment ratio for source code is less than 10%. Overall, the clone comment ratio in test code is much higher than that in source code of the four projects. The results indicate that the quality of source code is much higher than the test code in terms of comment clone.
Program comments are important for developers to understand
code. Moreover, comments that are inconsistent with code can
easily confuse and misguide developers to introduce bugs in
subsequent versions [
In this paper, we study the quality of test code. we present the five criteria for the measurement of test code quality. However, other criteria that we have overlooked may also can measure the quality of test code.
In this work we investigate the quality of test code in terms of five proposed criteria on open source projects. However, it is possible that our approach may not work well on some closed-source software, or small scale open source software projects. The purpose of this work is to study the impact of low quality test code on several software assurance techniques, however, not all projects maintain valid test code. Our approach is not suitable for these projects without test code.
This paper found that it is common for test code to be unregulated and of low quality in the open source projects. We studied 1164 test classes and more than 140,000 LOC test code from the current version of four open source projects. Results indicate that the quality of test code is much lower than that for the corresponding source code in terms of the proposed criteria, e.g., unreleased resource, code clone and comment clone, and the the coverage of test code for fixed bug is not sufficient.
We further discussed the potential impact of low quality on the existing software quality assurance techniques. To the best of our knowledge, this is the first work to report that the test code is lowquality and untrustworthy, which should be taken seriously. Future work. Our research is in a final stage. We explored the quality of test code in terms of five proposed criteria and the impact of test code on software quality assurance techniques. In the future, we plan to conduct quantitative and qualitative studies to explore how exactly the low-quality test code could impact software quality assurance techniques, i.e., automatic bug repair, fault localization, test case prioritization, and mining API usage.