-Refactoring is a set of techniques used to enhance the quality of code by restructuring existing code/design without changing its behavior. Refactoring tools can be used to detect specific code smells, propose relevant refactorings, and in some cases automate the refactoring process. However, usage of refactoring tools in industry is still relatively low. One of the major reasons being the veracity of the detected code smells, especially smells that aren't purely syntactic in nature. We conduct an empirical study on some refactoring tools and evaluate the correctness of the code smells they identify. We analyze the level of confidence users have on the code smells detected by the tools and discuss some issues with such tools.
Refactoring improves various qualities of code/design like
maintainability (understandability and readability),
extensibility, etc. by changing the structure of the code/design without
changing the overall behaviour of the system. It was first
introduced by Opdyke and Johnson [
Many code smell detection tools support (semi) automatic
refactoring process [
Despite the known benefits of refactoring tools, their usage
is not widespread due to users’ lack of trust on the tools’
code smell detection capability, learning curve involved, and
the inability of users to understand the code smell detection
results [
Correctness of the identified code smells among the chosen tools Deviation in confidence levels of developers in open source code smell detection tools that detect semantic code smells
The rest of the paper is structured as follows: Section II provides a brief overview of the code smells discussed in this paper and section III presents some related work. In section IV, we detail the study design. Section V and VI present the results and analysis of the results. Based on the study, we provide some guidelines for increasing the correctness of detecting some code smells in section VII. Finally, in section VIII we discuss some limitations to our work.
II. CODE SMELLS
Complexity related metrics like coupling are commonly used in tools to detect certain semantic code smells. Threshold values are established for various metrics and the code smells are identified based on the threshold values. Code smells like Feature envy, long methods, god class, etc. are widely studied in the literature. In our study, we primarily target the following code smells:
Feature envy: A method is more interested in some other class than the one in which it is defined.
Long methods: Method that is too long (measured in terms of lines of code or other metrics), possibly leading to low cohesion and high coupling God class: A God class performs too much work on its own delegating only minor details to a set of trivial classes.
Large Class: A class with too many instance variables or methods. It may become a God class.
Shotgun Surgery: A change in one classes necessitates a change in many other classes Refused Bequest: A sub-class not using its inherited functionality
Fontana et al. [
Pinto et al. [
Ferme et al. [
Ouni et al. [
The objective of our study is to analyze the correctness of
tools relevant to semantic code smells by performing a study
with human subjects with prior refactoring experience. Tools
like JDeodorant, PMD, InCode, iPlasma, and Stench Blossom,
and two large systems like JHotDraw and GanttProject that
have been widely studied in refactoring research were
considered. Choosing these tools and systems helped us in cross
validating our work with prior work done by us [
Initially, 35 human subjects volunteered to be part of the study. The exact hypothesis of the study was not informed to the subject to avoid biasing the study results. The subjects were only informed of the specific tasks that needed to be done, i.e. to assess certain refactoring tools with respect to their ability to detect code smells based on a given criteria and fill in a template.
All the human subjects had varying levels of prior
knowledge about code smells and tool based refactorings. However,
they had not worked on the specific tools used for this study.
The subjects had varied experience (13% subjects had 1-5
years of experience, 31% had 5-10 years of experience and
rest had less than a year of experience). The detailed statistics
of subjects is available at [
JHotDraw - a Java GUI framework for technical and structured Graphics. Its design relies heavily on some well-known design patterns.
The subject systems are available for use under open source
license and are fairly well documented. In addition to being
a part of Qualitas Corpus [
There are several commercial and open source code smell detection tools. Some are research prototypes meant for detection of specific code smells while others identify a wide range of code smells. In addition to detection, some tools refactor the code smells automatically, while others are semi-automated. Semi-automated refactoring tools propose refactorings that require human intervention and can be automated to a certain extent by changing rules of detection. Some of these are more or less like recommender systems that recommend certain type of refactorings but leave it to the developers to refactor or ignore the recommendations.
Some tools are integrated with the IDE itself, while others are written as plugins that can be installed on a need basis. For example, checkstyle, PMD, etc. are some eclipse based plugins that are well known. For our study, we focused on tools widely studied in the literature, their semantic code smell detection ability and usage in the industry. The list of tools and the detected code smells relevant to our study are shown in table II.
Table III provides a cumulative summary of code smells
detected and disagreements (weighted average of results from
32 results) by our human subjects. To show the disparity
in results, we compare our results for the GanttProject with
Fontana et al.’s results [
Feature Envy: The number of Feature Envy methods detected by different tools varies significantly. Some tools consider any three or more calls to method of the other class as a code smell and hence give rise to large number of false positive code smells. In such cases, when it’s just counting numbers, it becomes tedious to filter out the actual smells.
The degree of disagreement was found to be 9/44 for JDeodorant and 3/4 for inCode for the JHotDraw project.
Disagreements by the human subjects were prevalent across
all tools for detected feature envy smells. We also observed
a significant difference between the results of Fontana’s [
God Class: The number of detected God classes detected
by JDeodorant for GanttProject in Fontana et al.’s [
This inconsistency between tools reduces the confidence level of the results.
The degree of disagreement to jDeodrant code smells for jHotDraw project was 10 out of 56. Our human subjects observed that tools were considering the data model classes (getters and setters) and parser classes as smells. Usually these classes are needed and are necessary evils. As a result, there is a need of building some sort of intelligence/metrics to detect these kinds of classes which can safely be ignored to reduce the false positives.
Large Class: This code smell is detected by the code size and is subjective to the threshold limit of LOC set by the user.
Classes that contain project utility methods can grow in size with a lot of small utility methods and usually developers make these classes as singleton objects. Refactoring such smells requires one to be able understand the intent of the sentences before an extract method or extract class refactoring is applied.
Refused Bequest: Some tools considered interface methods and abstract methods that are normally overridden by the respective concrete classes as a code smell resulting in a lot of false positives.
Compared to Fontana et al.’s [
Shotgun Surgery: The results of our study for shotgun surgery on v2.7 were similar to the one’s for the GanttProject (v1.11.1). However, from our study we can conclude that the way shotgun surgery is detected in the different code smells differs from tool to tool rather that different versions of the same tool. The degree of disagreement to the detected code smells was 2 for GanttProject using inCode and 7 using iPlasma.
Long Method: This code smell is related to number of lines of code present in the method and subject to the threshold limit set by the user for detecting the code smell. As per the disagreements documented by our subjects, methods containing large switch statements should not be counted as a long method. iPlasma had a complex detection mechanism that considers such kind of things while detecting the Long Method code smells. On the contrary jDeodrant listed fairly small methods as a long method code smell.
The degree of disagreement to the code smells detected for GanttProject was 12 in jDeodrant, 27 in Stench Blossom and 8 in PMD. For JHotDraw project, it was 10 in jDeodrant, 12 in Stench Blossom, 11 in iPlasma and 8 in PMD.
The detections from jDeodrant tool were significantly high
as compared to 57 (v1.111.1) of GanttProject from Fontana et
al. [
C, C++, Java
The major challenge in assessing correctness of detected smells is the knowledge possessed by human subjects in regards to the code smells and the behavior of the code itself. Since the subjects (users) were not familiar with the tools chosen for the study, they complained about the time consumed in understanding and using the tool. At times, the focus seemed to be more on the user-interface of the tool rather than detected code smells. The authors had to revert back to the subjects to get additional clarification about the comments written in the templates regarding specific code smells.
Some tools require explicit specification of rules (for example, PMD) for detecting a specific code smell. So, selecting rules from the entire list of rules was tedious and timeconsuming. Most of the users seemed to struggle with the setup and configuration issues of the tools. Few tools like jDeodorant had high memory utilization and performance issues. So re-compilation after every change and re-running the detection process was laborious.
Contrary to our belief that the same code smell must be
identified in the same way by different tools, the disagreement
in the correctness of the detected code smells between the
various tools for the same type of smell was pretty high
(as shown in table III). Additionally, the results were not
complimenting the results provided in prior research [
To validate the results, we had to further cross-check the tools with some dummy examples. For instance, the dummy code (shown in Listing 1) was detected as a Feature Envy smell in some of the tools (for example, jDeodorant). If ’doCommit’ method is moved to any of the classes (A, B and C in this example) we must pass the other two classes as a reference parameter, that in turn increases the coupling.
Moreover, semantically it makes sense to call all commit methods in ’doCommit’ method itself.
An interesting observation can be made from the tools that detected Request Bequest code smell instances. For example, code resembling the dummy code (shown in Listing 2) reveals that there is no behavior written in the base method and just because it was being overridden without any invocation of base methods, it was detected as a smell. In other words, intuitively the authors could conclude even such code smells are primarily being thought about as syntactic where in the tool is just looking for redundant names in the superclass and subclass. Ideally, the tool should check if there is any meaningful behavior attached to the base method and only then should it be detected as a smell.
The dummy example (shown in Listing 3) was not detected as feature envy except by stench blossom tool. The probable reason for non detection is the declaration of the phone object inside the getMobilePhoneNumber method. Logically may not be detected as a code smell but from a semantic perspective getMobilePhoneNumber should be part of ”phone” class. This issue poses a question of correct semantic code smell detection by tools.
The dummy example (shown in Listing 4) has shotgun surgery code smell. The example shows a common mistake that users commit while creating database connection and querying tables. Users tend to create individual connections and command objects in each of the methods as shown in the example. If we have a connection timeout occurs semantically is make sense to create a utility method that takes SQL as an argument and returns result set ”DBUtility.getList” 4 conn = DriverManager.getConnection(DB_URL, in this method we will open connection and create SQL 5 USER, PASS); statements. InCode and iPlasma tools did not detect this code 6 stmt = conn.createStatement(); 7 String sql; smell. 8 sql = "SELECT * FROM Employees";
Although, several tools detect code smells, they do not 9 ResultSet rs = stmt.executeQuery(sql); consider the semantic intent of the code and hence end up 10 return rs.toList<Employee>(); with lot of false positives. Reducing false positives is the 11 } first step towards increasing the confidence levels of users and 12 public List<Customer> getCustomerList() { 13 Connection conn = null; proportionately increasing the usage of refactoring tools. 14 Statement stmt = null; 15 Class.forName("com.mysql.jdbc.Driver"); Listing 1: Feature Envy 16 conn = DriverManager.getConnection(DB_URL, 17 USER, PASS); 18 stmt1 = conn1.createStatement(); 19 String sql; 20 sql = "SELECT * FROM Customers"; 21 ResultSet rs = stmt1.executeQuery(sql); 22 return rs.toList<Customer>(); } 1 public class Main { 2 public void doCommit(){ 3 a.commit(); 4 b.commit(); 5 c.commit();}} 6 public class A { 7 public void commit() { 8 //do something 9 } } 10 public class B { 11 public void commit() { 12 //do something 13 } } 14 public class C { 15 public void commit() { 16 //do something 17 } } Listing 2: Refused Bequest 1 public class Base{ 2 protected void m1() { } 3 } 4 public class Inherited extends Base { 5 protected void m1() { //do something } 6 } Listing 3: Feature Envy 1 public class Phone { 2 private final String unformattedNumber; 3 public Phone(String unformattedNumber) { 4 this.unformattedNumber = unformattedNumber; 5 } 6 public String getAreaCode() { 7 return unformattedNumber.substring(0,3); 8 } 9 public String getPrefix() { 10 return unformattedNumber.substring(3,6); 11 } 12 public String getNumber() { 13 return unformattedNumber.substring(6,10); 14 } } 15 public class Customer { 16 public String getMobilePhoneNumber() { 17 Phone m_Phone = new Phone("111-123-2345"); 18 return "(" + m_Phone.getAreaCode() + ") " 19 + m_Phone.getPrefix() + "-" 20 + m_Phone.getNumber(); 21 } } Listing 4: Shotgun Surgery 1 public List<Employee> getEmployeeList() { 2 Connection conn = null; 3 Statement stmt = null;
Are referencing methods from multiple classes. Check if moving a method increases coupling and references to the target class.
Check if mutual work like object dispose, commit of multiple transactions is accomplished in the method. Is it semantically performing a cumulative task.
Take domain knowledge and system size into account before detecting smell, use semantic/information retrieval techniques to identify domain concepts
Ignore utility classes, parsers, compiles, interpreters exception handlers Ignore Java beans, Data Models, Log file classes Transaction manager classes
Normalized results with system size
Check if large switch blocks are written in the methods if multiple code blocks can be extracted with different functionality
Ignore Interface methods, they are meant to be overridden.
Check if base method has any meaning full behaviour attached to it.
Our initial study has provided evidence of disagreement towards the detected code smell from tools by our human subjects. We presented sample code for incorrectly detected code smells and semantic smells that were not detected by the tools. The authors strongly felt the need of taking semantic intent of the code into consideration while detecting smells and proposing (semi) automatic refactoring.
The disagreement in detected code smells correlates to the confidence levels of users. We saw that the results from all the users were not the same for the same code smell detection tools. So, the accuracy of these results can always be questioned. The results of Fontana et al. were used for cross-validation of our work. For the degree of disagreement to the detected code smells, we took an average of the overall disagreement report by the users. But, code smell disagreement is subjective until a standardized method for detection and measurement is proposed.
As a future work we would to extend our experiment with more industry level users. We would like to share the disagreement detected by our human subjects with the actual developers of the system to validate our findings. Towards understanding the correctness of the semantic code smell we would like to compare detection logic by tools.