-Managing technical debt effectively to prevent it from accumulating too quickly is of great concern to software stakeholders. To pay off technical debt regularly, software developers must be conscious of the existence of technical debt items. The first step is to make technical debt explicit; that is the identification of technical debt. Although there exist many kinds of static source code analysis tools to identify code-level technical debt, identifying non-code-level technical debt is very challenging and needs deep exploration. This paper proposed an approach to identifying non-code-level technical debt through issue tracking data sets using natural language processing and machine learning techniques and validated the feasibility and performance of this approach using an issue tracking data set recorded in Chinese from a commercial software project. We found that there are actually some common words that can be used as indicators of technical debt. Based on these key words, we achieved the precision of 0.72 and the recall of 0.81 for identifying technical debt items using machine learning techniques respectively.
Technical debt refers to delayed tasks and immature artifacts that constitute a “debt” because they incur extra costs in the future in the form of increased cost of change during evolution and maintenance [1]. An appropriate amount of technical debt would accelerate the process of software development; however, too much of it would impede the progress and even abort the project [2]. Typically, some startup software companies tend to incur technical debt strategically to speed up the development at the early stage of development process for the purpose of capturing the market. But with the growth of the size and complexity of the software, it may become increasingly more difficult to maintain and evolve the product due to intertwined dependencies between modules or components without paying off technical debt regularly. As a result, software stakeholders need to pay off technical debt regularly to prevent it from accumulating too quickly. Different from bugs or defects existing in a software system, technical debt is invisible as the software often works well from users’ perspective and even developers are often unconscious of the existence of technical debt. The invisibility of technical debt increases the risks of rigid software design and huge maintenance cost in the future significantly. Therefore, it is essentially critical for development teams to be able to identify technical debt items existing in the current software system at any point in time as it is the prerequisite to conduct other management activities of technical debt including measurement of technical debt, estimation of
Partially sponsored by a Canada NSERC grant, a MITACS grant and the Institute for Computing, Information and Cognitive System (ICICS) at UBC effort to be expended, payment of technical debt, risk evaluation, etc. Once technical debt can be identified systematically, software development teams would be able to estimate future budget, prioritize future tasks, allocate limited resources and evaluate potential risks. They could also make informed decisions about when technical debt should be paid off to maximize their profits.
Due to the importance of identification of technical debt, a number of studies empirically explored various approaches to detecting technical debt. Some of these researches focused on employing source code analysis techniques to detect technical debt. Code smells and automatic static analysis (ASA) are two most-used source code analysis techniques for the identification of technical debt. Code smells was first introduced by Fowler et al. to describe the violation of object-oriented design principles (e.g., abstract, encapsulation and inheritance) [3], whereas ASA techniques aim at identifying violations of recommended programming practices that might degrade some of software quality attributes (e.g., maintainability, efficiency).
Other studies aimed to identify technical debt of large
granularity that’s undetectable by source code analysis
techniques, such as architecture and requirement technical debt
[
As a complement to existing approaches, we try to identify non-code-level technical debt through issue trackers. We hope to acquire developers’ points of view on technical debt and understand how they communicate technical debt in issue trackers since they use issue trackers to record, track, prioritize various kinds of issues in software projects. Further, developers’ standpoints of technical debt will in turn help refine our understanding of technical debt and should be taken into consideration for an improved definition of technical debt.
However, it is difficult and impractical to identify technical debt manually through issue trackers due to substantial effort involved, especially when a large project comprises a large number of issues. In this context, we exploited natural language processing (NLP) and machine learning techniques to automate the process. NLP techniques were applied to extract features from unstructured text data and machine learning techniques were used to decide whether a certain issue is an instance of technical debt or not. We performed an exploratory study on a commercial software project to validate the efficacy of our approach to the identification of technical debt through issue trackers. Experimental results demonstrate that our approach is effective in identifying non-code-level technical debt, especially requirement debt, design debt, and UI debt, which cannot be detected by source code analysis techniques.
We address the following questions through this research: • RQ1: How do software practitioners communicate technical debt issues in issue trackers?
• RQ2: Are there text patterns that are an indication that technical debt exist which can be used to identify potential technical debt using NLP and machine learning techniques automatically?
The rest of this paper is organized as follows: Section II discusses related work. Section III describes our approach. Section IV reports and analyzes experimental results of our exploratory study. Section V presents the threats to validity. Finally, we conclude our research and envision future work in Section VI.
Many researches have been done to identify code-level technical debt. This kind of technical debt can be detected using static program analysis tools based on the measurement of various source code metrics. Marinescu proposed metric-based detection strategies to help engineers directly localize classes or methods affected by the violation of object-oriented design principles and validated the approach on multiple large industrial case studies [4]. Munro et al. refined Marinescu’s detection strategies by introducing some new metrics and justification for choosing the metrics and evaluated the performance of the approach in identifying two kinds of code smells (lazy class and temporary field) in two case studies [5]. Olbrich et al. investigated the relationship between two kinds of code smells (god class and shotgun surgery) and maintenance cost by analyzing the historical data of two major open source projects, Apache Lucene and Apache Xerces 2 J [6]. Wong et al. proposed a strategy to detect modularity violations and evaluated the approach using Hadoop and Eclipse [7]. Besides, some researchers explored identifying technical debt through comments in source code [8] [9].
Other researches aimed at exploring approaches to
identifying other types of technical debt such as architecture
technical debt. Brondum et al. proposed a modelling approach
to visualizing architecture technical debt based on analysis of the
structural code [
Issue tracking systems are widely used in open source projects as well as in software industry to record, triage and track different kinds of issues occurred during the lifecycle of software: bugs finding, defects fixing, adding new features, future tasks, requirements updating, etc. They play an important role in facilitating software development teams to manage development and maintenance activities and thus promoting the success of software projects. Some researches have focused on mining issue tracking databases to retrieve valuable information for improved definition, development management, quality evaluation, predictive models, etc.
Antoniol et al. applied NLP and machine learning techniques
(alternating decision trees, naïve Bayes classifier, and logistic
regression) to automate the process of distinguishing bugs from
other kinds of issues, compared the performance of this
approach with that of using regular expression matching and
concluded machine learning techniques outperforms regular
expression matching in terms of predictive accuracy [
Runeson et al. developed a prototype tool which detects
duplicate defect reports in issue tracking systems using NLP
techniques, evaluated the identification capabilities of this
approach in a case study and concluded that about 2/3 of the
duplicates can possibly be found using this approach [
Other work focused on concerned aspects of software quality
attributes, say security. Cois et al. proposed an approach to
detecting security-related text snippets in issue tracking systems
using NLP and machine learning techniques [
For this research, we cooperated with a local software company to access the issue data set of a commercial software product which have been in development for more than two years rather than just using issue data sets from open source software projects in order to make the classifier we developed more adaptable to the style of issue data from commercial software products. The issues are recorded mainly in Mandarin with a few English words as the developers of this product are Chinese.
We first exported the issue data set and saved it in a spreadsheet which makes it easier for researchers to read the issues and to process the data. The fields or variables of the data set we used are id, type, priority, state, summary, description and label. We also tuned the character coding format so that Chinese characters can be displayed normally and removed issues with messy code to render the data set clean and tidy. Finally, we got 8,149 issues in total. Figure 1 shows an overview of our approach.
We tagged each issue or task in the issue data set as technical debt or not technical debt manually by reading the summary and description based on the following classification criteria: 1. Is it a request for requirement change from the client? If yes, we definitely tag this issue as not technical debt. 2. Is it a task to add new functions or introduce new features to the product?
If yes, we also definitely tag this issue as not technical debt. 3. The description of the issue is too short or insufficient to decide whether the issue is a technical debt item.
4. Is it a defect that important and critical functions or features are not implemented correctly?
5. Is it a defect that is not critical from the client’s perspective but weakens the performance and capabilities of the system and will be fixed in the future?
6. Is it a task to redesign some function or feature as current design does not meet or meet the requirement partially?
8. Is it an issue related to bad coding practices such as dead code and no proper comments?
9. Is it a UI related issue such as inconsistent UI style or ugly UI elements that degrades user experience?
10. Is it a limitation of design in architecture level that may exert a negative impact on the performance of the system or on the evolution and maintenance of the system such as the violation of modularity?
types of issues existing in the issue tracker. Actually, some issues can be tagged as either technical debt or not technical debt, which, to a large extent, depends on your personal understanding of technical debt. Typically, there exist wide discrepancies regarding whether defects should be viewed as a type of technical debt among researchers and developers. In this study, we divided defects into two categories: 1. critical defects that may cause fatal errors occurring when using the software; 2. tolerable defects that may exert a marginal negative impact on the use of the software and are not fixed immediately after being detected. We tagged the first type of defects as not technical debt and the second type of defects as technical debt.
After we finished tagging all the issues, we asked a known expert in software engineering and technical debt external to our research team to validate the results of our manual classification. The expert classified a random subset of the issues independently. With respect to discrepancies in the classification of some issues, we exchanged our respective points of view about why we classified a certain issue as the category to solve our discrepancies. If we did not achieve agreements in the classification of a certain issue, we discussed the issue with developers to gain insight into the issue itself and their opinions on the classification.
Finally, we found 331 technical debt issues in total whose distribution is shown in Table II. Requirement debt and design debt are the main technical debt types, including 105 and 141 instances respectively.
Different from English, Chinese is written without spaces
between words. So before extracting key words from the
Chinese texts, we have to convert each text to a word sequence
using a Chinese text segmentation tool. For this research, we
used Jieba (https://github.com/fxsjy/jieba/) [
After conducting Chinese text segmentation, we extracted key words using Jieba. Jieba integrated two key word extraction algorithms: TF-IDF and TextRank. We used both of them to extract key words for detecting technical debt. We took the union of two sets of key words extracted using these two different algorithms, removed the key words referring to domain knowledge from the union set, and finally added some key words based on our intuition. To make this paper more readable, we only list the meaning of key words instead of original Chinese characters in the below:
“at present”, “now”, “current”, “previously”, “in the past”, “in the future”, “time”, “actually”, “in reality”, “users”, “clients”, “strengthen”, “change”, “modify”, “replace”, “update”, “delete”, “cancel”, “suggest”, “optimize”, “simplify”, “perfect”, “improve”, “refactor”, “decouple”, “again”, “re-”, “replant”, “tidy”, “integrate”, “merge”, “adjust”, “extend”, “expect”, “plan”, “management”, “maintenance”, “function”, “requirement”, “design”, “rule”, “theory”, “strategy”, “mechanism”, “algorithm”, “data structure”, “logic”, “code”, “structure”, “architecture”, “style”, “format”, “performance”, “efficiency”, “sufficiency”, “security”, “compatibility”, “scalability”, “maintainability”, “stability”, “generality”, “usability”, “readability”, “real-time”, “limitation”, “more friendly”, “more specialized”, “more accurate”, “problem”, “configuration”, “priority”, “inconsistent”, “unreasonable”, “inconvenient”, “convenient”, “not clear”, “inaccurate”, 'not intuitive', “not pretty”, “incongruous”, “not smooth”, “inconformity”, “incomplete”, “abnormity”, “defect”, “impact”, “experience”, “habit”, “operation”, “difficulty”, “delay”, “UI”, “risk”, “optimize”, “refactor”, “SonarQube”
There are 114 key words in total, among which 104 words are Chinese words and 10 words are English words. As some words express the similar or same meaning, we merged these words. All these words to some extent indicate or imply the concept of technical debt from different perspectives. To be specific, “at present”, “now”, “current”, “previously”, “in the past”, “in the future”, “time”
These words indicating time concept accumulation. may imply “strengthen”, “change”, “modify”, “replace”, “update”, “delete”, “cancel”, “optimize”, “simplify”, “perfect”, “improve”, “refactor”, “decouple”, “again”, “re-”, “replant”, “tidy”, “integrate”, “merge”, “adjust”, “extend”
These words indicate the modification of code, design or architecture, or the enhancement of functionality, capability, performance, efficiency, etc.
“security”, “compatibility”, “scalability”, “maintainability”, “stability”, “generality”, “usability”, “readability”, “real-time”, “limitation”
These words indicate concerned aspects of software quality attributes.
“inconsistent”, “unreasonable”, “inconvenient”, “convenient”, “unclear”, “inaccurate”, 'not intuitive', “not pretty”, “incongruous”, “not smooth”, “inconformity”, “incomplete”, “abnormity”, “defect”, “limit”, “impact”, “experience”, “habit”, “operation”, “difficulty”, “delay”
These words indicate defects or design limitation such as inconsistent UI style, unreasonable design, etc.
• • • •
Once key words were extracted from the issue data set, features for text classification can be derived by checking the presence or absence of each key word in each issue text. Given the set of key words is [“users”, “change”, “modify”, “improve”, “refactor”, “decouple”, “priority”, “button”, “architecture”, “deploy”, “rules”], consider this issue description: “design change: to keep a consistent design with different pages, we are moving the clear-all-rules button to the front of the deploy rules into a sequence of words and removed stop words (words that are too common to indicate any semantic meaning for our classification). Thus, the text is converted to a string list: [“design”, “change”, “keep”, “consistent”, “design”, “different”, “pages”, “moving”, “clear-all-rules”, “button”, “front”, “deploy”, “rules”, “table”]. Then we could check whether this string list contains each of key words, i.e. [contain(“users”), contain(“change”), contain(“modify”), contain(“improve”), contain(“refactor”), contain(“decouple”), contain(“priority”), contain(“button”), contain(“architecture”), contain(“deploy”), contain(“rules”)]. This vector checking the presence or absence of each key word is called feature space. The dimension of feature space depends on the size of the set of key words. Finally, we got the feature vector of the issue sample based on the feature space: [false, true, false, false, false, false, false, true, false, true, true].
The feature space actually not only includes unigram features that are a single word like “design”, “decouple”, but also has bigram and trigram features which comprised adjacent word pair and triplet respectively, such as “design change” and “improve unit test”; that is to say, the feature space in the previous example can be
extended to [contain(“users”), contain(“change”), contain(“modify”), contain(“improve”), contain(“refactor”), contain(“decouple”), contain(“priority”), contain(“button”), contain(“architecture”), contain(“deploy”), contain(“rules”), contain(“design change”), contain(“improve unit test”)]. Then the feature vector is turned into [false, true, false, false, false, false, false, true, false, true, true, true, false]. Figure 2 shows the process of feature extraction.
Naïve Bayes is a simple classification algorithm that is based on an assumption that the features are conditionally independent of each other given the category. It determines the category of a given sample
with n-dimensional features ( 1, … , ) by calculating the probability that the sample belongs to each category and then assigning the most probable category c to it, which can be described as: = arg
max ∈{1,…,K} ( | 1, … , ), ( | 1, … , ) can be decomposed as: where is the kth category, and K is the size of the set of categories. Using Bayes’ theorem, the conditional probability ( | 1, … , ) = ( 1,…, | ) ∑ℎ=1 ( 1,…, | ℎ) ( ). probability ( | 1, … , ) can be transformed into: With the conditional independence assumptions, the conditional
To perform our experiments, we used a popular natural
language toolkit for building Python programs to process human
language data (NLTK http://www.nltk.org) [
Feature Extraction from Text Data text = “design change: to keep a consistent design with different pages, we are moving the clear-all-rules button to the front of the deploy rules table. (Consistent with event page)” t = tokenize(text) = [“design”, “change”, “keep”, “consistent”, “design”, “different”, “pages”, “moving”, “clear-all-rules”, “button”, “front”, “deploy”, “rules”, “table”] RQ2: Are there text patterns that are an indication that technical debt exist which can be used to identify potential technical debt using NLP and machine learning techniques automatically?
The experimental results demonstrate that text patterns indicating technical debt indeed exist and can be used to identify technical debt. In general, technical debt issues are characterized by two aspects of properties including rework whether it is code refactoring or feature redesign and accumulation which is implied by some words indicating time such as previously, at present, and in the future. 20 most informative features that are strongly correlated to technical debt are shown in Table III. Each of these features may contribute differently to the identification of different types of technical debt. Intuitively, the presence of “style” and “experience” may indicate UI debt while “simplify” and “efficiency” are more likely to be indicators of design debt.
To evaluate the performance of our classifier, the average precision and recall were calculated for 10 repeated random subsampling validations. Precision measures the fraction of technical debt instances identified by our classifier that were proved to be correct classification. Recall measures the fraction of correctly classified technical debt items out of the total number of technical debt issues. In our experiments, the average precision and recall were 72% and 81% respectively for 10 repeated random sub-sampling validations shown in Table IV.
There are two main threats to the validity of our study: threats to internal validity and threats to external validity. Threats to internal validity can be caused by the level of subjectivity in manual analysis and classification of issues as we definitely have personal bias in the understanding of issue description. To counter the threats, we had an expert external to our research team classify random samplings of the issues and solved our discrepancies by discussion. We also had discussions with the developers of the product to gain insight into the issues that we were not sure we classified correctly. Threats to external validity concern the generalization of our findings. We performed a case study on an issue data set from a commercial software project. The data set of issues we used may not be representative; that is to say, we cannot guarantee the same results will be obtained when our approach is applied to other commercial software projects. In particular, our approach may not be applicable to those projects for which issue trackers are not used to record issues.
This paper presents an exploratory study of applying NLP and machine learning techniques to identify technical debt issues through issue trackers. We have demonstrated that we can automate the process of detecting technical debt issues through issue trackers and achieve an acceptable performance using NLP and machine learning techniques. We found that some common words in software engineering are directly or indirectly related to technical debt and these words can be used as features to decide whether a certain issue is technical debt or not. We believe the performance of our classifier will improve further when more sophisticated feature extraction and classification techniques are applied.
This exploratory study was based on a rather limited data set of 8,149 issues. Our approach needs to be validated with issue data sets from a wider range of software projects. Furthermore, we will improve the performance of our classifier by exploring more sophisticated feature extraction techniques such as mapping phrases with regular expressions and extracting semantically meaningful information based on the context and applying other classification techniques such as random forest, SVM, and deep learning. In addition, we will also develop a multi-classifier to identify technical debt of a specific type.
We would like to acknowledge the support of Jean Su, Steven Zhu, Billy Liu for this research. We also thank Robert Nord, Rob Fuller, Randy Hsu for their valuable review comments. This research was partially supported by a Canada NSERC grant, a MITACS grant and the Institute for Computing, Information and Cognitive System (ICICS) at UBC.