Test smells can be found in test code using a
variety of tools. In this paper, we present our
integration of a test smell detection tool into
Better Code Hub (BCH), an online
environment for monitoring code quality and
identifying problems in it. We extended BCH with
test smell detection and observe how
developers react to various instances of test smells. By
integrating this detection into BCH, we gain
access to a wide range of developers working
on both open-source and commercial projects
using their own code. We study whether
developers consider these test smells important
and what they do with them in future code
changes. From our preliminary results, we
found out a high test smell detection
accuracy for most test smells; however, developers
are only willing to remove a small portion of
them.
The majority of the developer focus is spent on
writing and improving production code quality, while test
code quality is often not prioritized [
SIG, a consultancy company with the headquarter in Amsterdam (NL), developed a tool to analyze GitHub repositories code quality called Better Code Hub (BCH)1. BCH checks the GitHub codebase against 10 easy to follow software engineering guidelines. Currently the guideline for test quality in BCH only include assertions density and test code LOC. We extended the existing test quality metrics with test smell detection and integrated it into BCH to do our research on test smells.
Existing research on test smells focuses on detection
of test smells [
We aim to answer the following research questions: RQ1: What is the perception of developers on test smells in their codebase?
RQ2: Which test smells developers consider to be important?
Our research on test smells is di erent in the following aspects: (1) we investigate all instances of test smells on code the developers have previously interacted with; (2) by using Better Code Hub, we can 1https://bettercodehub.com/ cover more projects and users to gain a more diverse data set; (3) by giving developers concrete examples from their own codebase, we expect the developers to be aware of the context in which both the production and test code was developed and how the test smell was created. Furthermore, for each detected test smell in the code base, we ask the developers (through a survey) their opinion on the importance of the detected smell. Because the survey is integrated directly in BCH, we are able to invite users outside of SIG to take part in the survey and also analyze their own code.
After conducting our experiment, we found that developers recognize test smell instances once they are presented to them with high accuracy. However, depending on the test smell, they are less willing to refactor the test suite to remove this test smell even for test smells which are rated as having an high impact on software maintainability.
The paper is structured as follows: section 2 contains the related works on test smell research, section 3 presents the research questions and experiment design, section 4 provides the results followed by section 5 with an analysis of the results, followed by our future plans in section 6 and concludes with section 7. 2
The concept of test smells originates from the work of
van Deursen et al. [3] and test smells have been
expanded with several other instances [
Currently there is ongoing research into the
discovery and classi cation of test code smells and their
impact on software quality and reliability [
We rst modi ed Better Code Hub2 (BCH) using a Chrome extension to display di erent metrics for test code quality than the current production build. The \Automate Tests" metric results were modi ed to include the detected test smells in the analyzed source code along with the test smell type. The modi cation is shown in Figure 1. The user can then click on the individual smelly method and see the source code. There is a possibility to submit their feedback on whether the test code segment is, according to their opinion, a valid instance of the test smell. This feedback can be given for each found test smell. A sample view of the code view and feedback form is in Figure 2. We also modi ed the guideline explanation in the sidebar to have a brief explanation of all of the detected test smells to ensure that the users are aware of how the test smell classi cation was performed.
We selected the open-source tool tsDetect [
The tool tsDetect can detect multiple test smells;
however, for the purpose of this study we restricted
our analysis to the following test smells: conditional
test logic; mystery guest; redundant assertion;
sensitive equality; verbose test; sleepy test; eager test
and resource optimism. The following test smells are
not part of the original test smells proposed by van
Deursen et al. [3], but are part of the tsDetect tool:
conditional test logic, redundant assertion, verbose
test; sleepy test. Based on testing done on a
manually selected dataset, the tsDetect tool is highly
reliable at detecting instances of these test smells [
We created the survey to be integrated into BCH for each test smell based on the following criteria: (1) ability to evaluate if given test smell instance is valid in the project context, (2) option to classify test smell instances as something to x (refactoring candidate) or as something which will take too much time and e ort to x (technical debt), and (3) at the same time allow developers to rate the subjective importance of the found test smell on the project's maintainability. The rst part of the survey helps us answer RQ1, while the second scale rating helps us answer RQ2. 3.4
Our participants were selected from a pool of Java developers with at least two years of development experience within the host company. Each of them was asked to evaluate parts of the codebase written in Java they were actively working on, while using BCH for the evaluation. 4
This preliminary study was done to determine the viability of using BCH as a platform for test smell perception in developers outside of the host organization to gain a larger sample size. We asked 4 developers to use our modi ed tool and then interviewed them afterwards. The results are summarized in Figures 3 and 4. After a separate interview session with the developers, we were able to obtain additional results not covered by the BCH survey. From the results we can see that the majority of the observed test smells are considered as refactoring candidates. Sensitive Equality shows that the majority of the instances found were false positives, and those which were not considered false positives were ignored. The conditional test logic is evenly split between marking as a refactoring candidate and taking no action. 5
The test smell rated as the one which developers decided to ignore the most is conditional testing logic, where the threshold was more than 1 branch statement per test case. During the interviews, developers have repeatedly identi ed that rewriting the test cases to not use conditions would be either impossible or take too much e ort with no additional gain for test maintainability. A similar view was held for eager test, where for certain test cases it was deemed desirable to call multiple production methods in a row before doing an assert. The alternative would be to move the rst production call to another method which sets up the test; however, this method might only be used by the one speci c test case, negating any bene ts of moving to a separate function. Sensitive equality was deemed the most as a false positive or no action item. This was primarily due to limitations of tsDetect and some parts of the code working with parsers, where toString() was required to evaluate the output. Other test smells were primarily rated as refactoring candidates. These test smells could then be used as a metric for evaluating the quality of the unit tests, as these are the smells developers are willing to remove. From the rankings of test smell presence on maintainability, we can see that the highest impact is sleepy test and redundant assertion. Due to the high false positive rate for sensitive equality, the developers ranked the issue as not severe or hard to avoid. 5.1
RQ1 Based on the results, we can see that developers consider most test smells to have a negative e ect on the codebase and should be removed by refactoring. For test smells which are detected reliably, the developers consider the best course of action to take is refactoring the test method to remove the test smell instance. For eager test, the developers expressed that refactoring the test would be di cult and not improve the overall test suite quality. Conditional test logic was the test smell most considered as one to keep (take no action). This can be because the developers consider the test smell hard to remove in the presented cases. Gaining more data about sensitive equality would require using a di erent tool with a lower false positive rate. 5.2
RQ2 Verbose test was rated always as a refactoring candidate, despite on average being rated as having a low impact. On inspection of the found test smell instances, we found that the tests often contained several blocks with comments and were repeatedly testing the same methods with di erent values, indicating a co-occurence with lazy test. Sleepy test was rated as having both the highest impact and was always rated as a refactoring candidate, indicating that the removal of dependency on threading in unit tests is perceived as high priority. Resource optimism is considered medium severity and always a refactoring candidate, indicating that this test smell can be easily refactored by adding the extra le existence checks. Nonetheless, if the test fails due to a le not being found, it is easy to detect. Eager test was rated on average as below medium severity and primarily a refactoring candidate, indicating that it is not perceived as a problem. Constructor initialization was rated as not severe problem and could be refactored into setup methods. 6
The internal preliminary study shows promising results to deploy this modi ed version of BCH to the general public. This will enable us to analyze multiple projects, both open and closed source from their developers. We plan to determine if there is a relationship between the test smell perception and project experience (in terms of time and commits) and the familiarity with the le where the test smell resides by the commit history. 7
We extended Better Code Hub with test smell detection and ran a preliminary study. In this study, we investigated the developer perception of test smells using a sample of developers from the host company on a codebase they were working on. The results show that the tool can be deployed for the general public and used to gather a larger sample size from the users.
We would like to thank the developers at Software Improvement Group for taking part in this study. [1] Gabriele Bavota et al. \Are test smells really harmful? An empirical study". In: Empirical Software Engineering 20 (2014), pp. 1052{1094. [3] Arie van Deursen et al. \Refactoring Test Code".
In: Proceedings of the 2nd International Conference on Extreme Programming and Flexible Processes in Software Engineering (XP). 2001, pp. 92{95.