Toward Measuring Defect Debt and Developing a Recommender system for their prioritization∗ Shirin Akbarinasaji Data Science Lab Ryerson University Toronto, Canada shirin.akbarinasaji@ryerson.ca ABSTRACT Every project requires to be completed and delivered under Software development managers make a release decision with- certain constraints. Project management body of knowl- out fully resolving the defects from current and previous re- edge modeled these constraints as the iron triangle of time, leases due to tight deadlines. Deferring the defects would cost and quality (scope). Figure 1 illustrates the project accumulate a tremendous amount of technical debt in the management triangle. Typically, changing one side of the system. Typically, the defect debts are defined as the type triangle has an effect on the other side of the triangle. For of defect that should be fixed. However, due to compet- instance, increasing the quality (scope) leads to the growth ing priorities and the limited amount of time and resources, in time and cost. Similarly, tight time (budget) constraints they would be postponed to the next release. In order to may cause increasing of the budget (time) and reducing the aid practitioners ,who make release decisions, to observe the quality. Complex software projects certainly need to tackle amount of debt, there is a need for quantifying the defect the triple project constraints as well. The quality of soft- debt. Software bug repositories roughly provide us with in- ware often diminished over time since the software mainte- formation about the amount of time the defect debt exist in nance projects are performed under tight time and resource the system, the time the defects are resolved and the sever- constraints. The project managers need to make balance ity of the defect. We suggest categorizing the defect into between time and cost properties and the required quality the regular defect and debt prone defect by analyzing this level. To achieve the system equilibrium, they might delay information. Afterwards, we compare the regular defect and some maintenance activities such as documentation, testing debt-prone defect to determine the principal, interest and or even fixing bugs. The consequence of delaying these tech- interest probability of defect debt. We also propose the re- nical development activities can be interpreted as a type of inforcement learning for scheduling which defect debt needs debt and it will affect the long term maintenance and devel- to be paid and when they need to be paid. opment activities. The trade-off between short term benefits of delaying these activities and long-term effect of postpon- Categories and Subject Descriptors ing them is articulated as “technical debt”. H.4 [Information Systems Applications]: Miscellaneous; D.2.8 [Software Engineering]: Metrics—complexity mea- sures, performance measures General Terms Theory Keywords Technical Debt Measurement, Defect Debt, Reinforcement Learning, Software Maintainability Figure 1: Project Triangle 1. INTRODUCTION ∗Copyright ©2015 for this paper by its authors. Copying Generally, the technical debt describes the delayed devel- permitted for private and academic purposes. opment activity due to the time and resource constraints. Likewise the financial debt, principal is an amount of effort in terms of time or cost require paying off the debt (i.e com- plete the task). Interests are the potential penalty in terms of extra amount of effort required to pay as a compensation for what is borrowed [24]. Accumulation of technical debt in the system has tremendous effect on the quality of the sys- tem [24]. There is always a choice between paying down the whole principal or continuing to pay off the interest. And, in small software projects, the managers may implicitly de- cide on the amount of the debt to be paid off and payment 15 schedule. However, in the large projects, there is a need for quantifying the defect debt and we seek feedback on our a comprehensive system that is able to track and manage research approach and our proposed solution. the technical debt [19]. Additionally, technical debts may occur intentionally or un- 3. RELATED WORK intentionally [14]. Intentional debts are the kind of debts The technical debt was first introduced two decades ago by that the developers and managers are aware of their exis- Ward Cunningham [3]. He described that “Shipping first tence and they occur deliberately due to the strategic and time code is like going into debt. A little debt speeds devel- tactical mission. Unintentional debts occur due to the de- opment so long as it is paid back promptly with a rewrite”. velopers'lack of attention or understanding and the team is Although it is a fairly recent metaphor in software engineer- not aware of their existence and their location. Further- ing, it is highly related to well-researched issue like software more, the debts may also be classified according to its type. decay by Lehman and Belady [11] and software aging by According to Li et al. technical debts are classified into 10 Parnas [17] . Software decay is a complexity of software types of debts including: requirement, architectural, design, due to continuous changes and software aging is a disability code, test, build, documentation, infrastructure, versioning of software to meet required changes. The introduction of and defect debt[13]. In this study we particularly focus on technical debt facilitates managing these concepts. defect debts which refer to the defects, bugs or failures found but not fixed in the current release[19]. Martin Fowler [7] suggested the quadrilateral categorization of technical debt considering intention (deliberate or inad- Since, quantifying the technical debt is a key factor in mak- vertent) versus awareness (reckless or prudent). Brown et al. ing decision about incurring, paying off and deferring tech- [1] extended the concept of technical debt from the code level nical debt, we will initially propose a new approach for mea- metaphor to architectural and detailed design. Since then, suring the principals and interests of defect debt. Most of the many practitioners and researchers relied on the definition quantifying approaches in the literature are based on the dis- of the technical debt in order to explain various cost-drive tance between the violation of the code from the ideal code. issues in software engineering. Tom et al. [24] explored However, our approach differentiates from the existing ap- the technical debt concept focusing on dimension of tech- proaches since it concentrates on mining of bug repositories nical debt, the advantages and drawbacks of allowing them and collecting data from issue tracking system. in a system and its origins. In another study, Tom et al. [23] did a systematic literature review focusing on the state On the other hand, software development team always faces of academic research for technical debt. Li et al. [13] did a the high volume of defect reports and change requests in systematic mapping study on technical debt and its manage- every cycle of release. The challenge is to figure out which ment. Seaman and Guo [19] reviewed some issues associated instance of defects should be addressed in this release and with technical debt and proposed the framework for mea- in which order they are required to be fixed. Therefore, an suring and monitoring the debt based on risk management important next step is recommending a system to the de- approaches. Their framework for managing technical debt velopers and managers to prioritize instances of defect debt was identifying debt, measuring debt and monitoring debt. in the current release. Our proposed solution for scheduling For identifying debt, especially code-based debt, there are of the defect debt is based on reinforcement learning. Our various tools. They are designed to detect code smell [18], ultimate goal is finding an optimal action-selection policy modularity violation [25], grime buildup [9] and potential for paying off the defect debt in limited time in such a way defect. Zazworka et al. [27] found that different tools did that minimizing the amount of interest. We can summarize not reveal overlapping results, rather they each pointed out our contribution as follows: a different problem. Letouzney [12] proposed the SQALE (Software Quality As- • Proposing a new approach for quantifying the defect sessment based on Lifecycle of Expectations) for evaluat- debt, ing the technical debt. Sonar tool1 is a popular tool for • Developing an automated framework for prioritization evaluating the technical debt but it is not perfect as it is of defect debt widely discussed in technical debt literature [5]. Sonar ap- plied quality heuristic for identifying code duplication, vio- lation of code standard, lack of testing, and potential latent The remainder of the paper is as follows: in section 2, we bugs. Nord et al. [15] presented a metrics based on architec- will briefly explain the current research issues and what we ture and measurement approach for managing the technical would like to get advice on. Section 3 will review the related debt. Nugroho et al. [16] redefined the technical debt as work. In section 4, we will specifically discuss our research fixing cost of the technical issue. They performed static objectives. Section 5 is a brief explanation of research ap- analysis to identify debt and estimated the debt principle proach. Finally, section 6 will review next step and section based on the percentage of changed line code, code duplica- 7 is conclusion and summary. tion, dependency and parameter count and also complexity metric. They calculated the interest estimation by assigning 2. CURRENT RESEARCH ISSUES these metrics to risk categories. Guo et al. [8] investigated Although the technical debt attracts interests of many re- the effect of technical debt in a real software project by es- searchers in recent years, most of the work only capture the timating the principal debt according to effort estimation. theoretical aspect of the technical debt and there is still a Zazworka [28] applied cost-benefit analysis for prioritizing lack of empirically based studies in the literate. In this re- 1 search, we would like to propose an empirical approach for http://www.sonarsource.org/) 16 the God class debt. He ranked the cost of paying debt and Jifeng et al. referred to the defect debt by another terminol- the impact of debt on quality to determine which refactoring ogy as “debt prone bugs” and described it as any software activities need to be performed initially. Singh et al. [20] bugs which remain in the system because of any immature, monitored developer activities to estimate the debt interest. incomplete process of fixing the bugs[26]. Several studies proposed considering interest probability for interest estimation [2], [6]. This is the probability of debt, if From the perspective of defect debt definition, the bugs are not paid, how it would affect other tasks in a negative way divided into the bugs which constitute the technical debt [19]. There are also several studies that are presented and and the bugs which do not form any debt. Hereafter, we classified the technical debt by mapping study by [13]. refer to the latter as regular bugs. Therefore, two type of bugs exist in any bug repositories: Xuan et al. proposed the concept of debt-prone bugs and identified three types of debt :tag bugs, reopen bugs and du- plicate bugs. Tag bugs are the bugs which “fixme, todo, xxx” • Regular bugs which refer to the bugs that are submit- tags annotated to them. Reopened bugs are the bugs which ted and resolved in the same release are solved by developers but reopen later, and duplicate bugs • Debt prone bugs which refer to the bugs that are sub- are the bugs with the same root as existing ones.They con- mitted in one release but are not resolved in that re- ducted a case study on Mozilla to investigate effect of debt lease prone bugs on software quality. [26].Snipes et al. identified and categorized cost related to fixing the defects or deferring the defects. They stated that decision factor for managing 5.2 Measuring defect debt principal and in- defects from the technical debt perspective are: severity, an terest existence of a workaround, an urgency of fix required by In management level of any software organization, they are customer, the effort to implement the fix, the risk of the interested in achieving a quantitative understanding of time proposed fix and the scope of testing required[21]. constraints for fixing the bugs. However, the time con- straints that debt prone bugs may impose to the system In this study,we particularly concentrate on defect debts and is totally different from the regular bug fixing time. The we propose how to measure the principal and interest and fixing time for debt prone bugs is the summation of the interest probability of defect debts. However, the main con- standard amount of time required for fixing the bugs and cern of development team is how to prioritize defect debts in an extra amount of time required to fix them as a penalty order to minimize the total amount of interest. We propose of deferring the process. In other word, this fixing process reinforcement learning to schedule debt prone bugs pay off. for debt prone bugs may exceed the standard fixing time for For future step, we plan to collect data from bug reposi- regular bugs because the accumulation of unresolved bugs tory of a real project to show the feasibility of our proposed in the system would make it more complicated to repair the model. defects. Will Snipes et al. identified the following type of time (cost) for fixing defects[21]: 4. RESEARCH OBJECTIVE The general objective of this study is to propose an approach that provides the developers and practitioners to make a de- • Investigation time or the time of diagnosing, verifying cision under uncertainty during project management regard- and finding the alternative solution for defects. ing the defect debt. In particular, the goals of this proposal • Modification time which refers to the time of applying can be summarized as following: the solution to fix defects • Work around time that deals with providing the bypass • To propose a simple and straightforward approach for for defects which are not resolved immediately measuring the principal and interest of any defect prone bug instance: In most of the previous studies, the tech- • Customer support time is the time of providing sup- nical debt has been measured by comparing the current port for the customer because of the defect that exists state of the code and the ideal target for the code. In in the current release this study, we would like to propose a new approach to measure the technical debt by mining the bug reposi- • Patch time or the time of finding temporary solution tory. for fixing the bugs • To develop a framework for automating the prioritiza- • Validation time that refers to the time of testing the tion of defect debt: Once technical debt is measured, systems. the next step is to provide a guide to practitioners a decision technique and approach to determine whether or not to pay off the debt at a particular point of time. Investigation, modification and validation time are incurred for fixing the regular bugs. However, postponing the fix- ing process of the bugs may impose additional time such as 5. RESEARCH APPROACH workaround, customer support time and patch time to the 5.1 Definition of Defect Debt system as well. Besides, the investigation cost may increase Defect debts refer to the defects, bugs or failures found but because of the accumulation of more defects make the diag- not fixed in the current release, because there are higher pri- nosis process more complicated[21]. As mentioned earlier, ority bugs to fix or there are limited resources to fix them. technical debt principal is the amount of effort in terms of 17 time that are required for fixing bugs. And the interest is are dependent to current bugs need to be divided by the an extra amount of time as a type of penalties which are number of all existing bugs in the current release. required for fixing the defect. Consequently, principal value for debt prone bugs may include three components: investi- 5.3 Applying Reinforcement Learning for Pri- gation, modification and validation time. The interest value may compromise additional workaround, customer support oritization of Defect Debt Reinforcement learning is a machine learning task of find- time and patch time. ing an optimal action-selection behavior of an agent in or- der to maximize the total amount of reward[22]. Reinforce- In order to help software development team to better esti- ment learning is inspired by a concept in psychology “re- mate if the defect debt can be absorbed in current release or inforcement”. In behavioral psychology, reinforcement is a not, a fair approximation for principal and interest are re- consequence that will motivate the agent behavior by of- quired. Suppose that two similar bugs with the same charac- fering specific stimulus followed by the behavior. Basically, teristic such as severity, same assignment, and same product reinforcement learning is a problem of an agent who inter- exist in bug repositories. One bug is treated as a regular bug acts with the environment and tries to achieve an optimal and one is treated as debt prone bug. For any regular bug, goal. Reinforcement learning has many applications in dif- we can easily estimate the regular fixing time by retriev- ferent domains: robotic control, scheduling, chess playing, ing the historical data from bug tracking system. In any backgammon, etc. [10]. Reinforcement learning is based bug tracking system, the bugs are submitted by develop- on Markov Decision Process (MDP) and the components of ers and testers with an ID, description of bugs, its version, basic reinforcement learning model are as following:[22]: the reported date and its severity. The developers who are interested in working on those bugs may assign different la- bels to bugs such as new, unconfirmed, reopened and etc. • a set of actions a ∈ A They also contain an adequate information regarding the last time the bugs are modified and the history of modifica- • a set of states in the environment s ∈ S tion. The fixing time for regular bugs is equivalent to the • a set of transactions between the states difference between reporting time of the bugs and the re- solving time of the bugs. Therefore, we are able to build a • The rules that determine an immediate reward for any prediction model based on the data from regular bug fixing transition (r) time to predict the principal for debt prone bugs. The po- • The rules that determine what the agent are able to tential input features for feeding the prediction model would observe from an environment be severity, submitter, owner, priority, the indicator if the bug internally discovered or externally discovered, etc. The agent interacts with an environment over potentially In order to evaluate the interest for debt prone bugs, ini- infinitive discrete time steps of t = 1, 2, 3, . The time step tially the variance between the fixing time and the estimated is not necessarily fixed interval time, it can be determined principal needs to be calculated. The interest amount is based on point of time that the state change or new action proportional to this variance. The weighting factor for ad- has been performed[4]. At each time, the agent based on justing that is called severity. Severity is the potential effect state of an environment chooses an action at . Then, the of defect in the functionality of the system and customer agent receives a reward based on chosen action and will be requirement[21]. For instance, one reason for deferring the transferred to new state st+1 . The agent can evaluate its bugs fixation to the next release is that the bug is very triv- performance based on the received reward. The reward de- ial. Suppose the customer is requesting changes related to fines the goal in the learning problem. The Policy is the background color of the system. In this case, the impact rule agent uses to select actions and for each state it assigns of the defect is a minor irritation. Therefore, the severity a probability to each possible action. The agent tries to of the bug is very low and postponing the defect for many adjust policy in order to maximize the accumulated reward years may not affect the functionality of the system. There- [4]. Figure (2) depicts a standard framework of reinforce- fore, the interest amount for this particular bug is the vari- ment learning. ance between the regular fixing time and estimated principal times the severity of the bugs. From this point of view, we can conclude that the interest amount is the time difference between principal and real fixing time multiplying the co- efficient based on severity. Fundamentally, the more severe bugs pose more interests to the system if they are not fixed in the same release as they reported. Therefore, an interest amount can be calculated as below: InterestAmount = (RealF ixingT ime−EstimatedP rincipal) ∗ Severity Figure 2: Reinforcement learning Framework Another component of the interest is the interest probability which refers to the probability that defect debt, if not fixed, will negatively affect the fixing time of other defects[19].For Prioritization of defect debt can be modeled as a reinforce- the sake of estimating interest probability, all the bugs that ment learning. The main reason is that in scheduling of de- 18 fect debt, we are facing a changing environment and, there- Where B is the long-term reward in the system, π is the fore, a traditional fixed policy schedule would not work in action (policy) the developers decide to take. r is a reward such a fluctuating environment. Additionally, we are not at each state and 0 < γ < 1 is the discount factor which able to anticipate the long-term consequence of debt prone needs to be determined initially. The optimum decision is bugs in the system. In software defect detection, practition- to optimize the long-term reward: ers are looking for recommendations beyond classification. Identifying and recommending a course of action for defect π(s) = argmaxBπ (s) debt is a complex task that traditional classification algo- The optimal π would return the sequence of the defect debt rithms may not suffice. A classification algorithm takes a that developers should fix to save the maximized amount of given data to learn and build a model for making a single interest. prediction or decision. However, human learning takes place in three phases, classification, memorization and procedural by interacting with the environment in making decisions. 6. PLANNED NEXT STEPS When building recommender systems in complex environ- In order to check the feasibility of our approach, we need ments such as software defect debt prediction, we need to to apply it in real world projects and analyze the results. use learning techniques such as reinforcement learning where Therefore, we plan our next steps of this research as below: an algorithm considers the effect of actions by trying things to learn. It is different than dynamic programming as we do The initial next step is collecting data from both open source not know the effects of actions in the case of prioritization projects and also commercial software companies and ap- of defect debts. Since technical debt is hidden in the code plying our proposed techniques for quantifying their defect and we do not know its future effect, we need a learning al- debt. We also need to consult with developers in each do- gorithm that considers this uncertainty and learns from its main to assess our approach. The second step is determin- environment by interacting with it. Such learning is more ing which reinforcement learning method is best situated for realistic and closer to how humans learn, but it is difficult our model such as temporal difference learning, Q-learning, to implement as a learning algorithm. SARSA , etc. The last step is to extend our approach to other types of technical debt. The potential framework for prioritization of debt is as fol- lows: Suppose that the developer is the agent in the model 7. CONCLUSION that should adjust policy in order to maximize the amount In this study, we described a new procedure for measur- of interest we could save. A set of states in environment ing the principal, interest and interest probability for defect might be the amount of time remaining before upcoming debt. Despite most of the existing approaches which are release. A set of actions is whether to pay debt i in the based on the violation of the program from the right code upcoming release or not. Reward is equal to amount of in- practice, our proposed method is based on mining the bug terest we save if we pay the debt i at current time instead repository and standardizing the measurement based on the of postponing it to the next release. Note that after fixing actual status of the system. Our proposed approach presents a defect debt bugs, the target is achieving zero (minimum) a simple heuristic for practitioners with limited knowledge amount of debt before the release or maximizing amount of about the architecture of the code in quantifying the defect interest that we saved. We assume that the developers are debt. Typically, we concentrate on the bugs that are opened fully aware of the current state of the model. They have and resolved in the same release and use them as an original all the required information about how much debt is in the required fixing time. Then, by comparing the defect debt system and how much time they have to work on debt prone prone to the regular defect, we calculate the interest. bugs. Furthermore, we would like to develop a novel learning frame- Assume that in each release the developer team decides to work for scheduling of the defect debt. Our proposed method assign a specific amount of time T to fix some defect debts. is reinforcement learning because of the dynamic nature of At t = 1 the developer face totally N defect debts to be the defect debt environment. We believe that determining fixed. He needs to decide which debt to pay at time t = 1. which defect debt to work on is a very complicated problem Suppose that the developer decides to fix defect debt i and and developers need an automated framework that suggest it would take x1 minutes to fix it. He would save an interest them a sequence of defect debt they can fix in limited time of r(s1 ). At t = 2, the remaining time is equal to T − x1 . that can return the maximum amount of reward to them. As, defect debt i has been fixed in the system, it would effect the number of defect debt we may face in the system since it will facilitate fixing of some other bugs which are 8. ACKNOWLEDGMENT dependent in defect debt i. For instance, the number of I would like to express my sincere thanks to Dr. Ayse Bener defects may change to N + m − 1. Suppose, at time t = 2 and Dr. Bora Caglayan for providing me with their valuable the developer decides to fix defect debt j and it would take advice and guidance. I am very grateful for their support x2 . The remaining time would be T − x1 − x2 and he would and feedback that help me to improve my proposal. save r(s2 ). The developer will continue till he reaches time T . He needs to make the decision based on the long-term reward the system would return to him. The expected long- 9. REFERENCES [1] N. Brown, Y. Cai, Y. Guo, R. Kazman, M. Kim, term rewards at each state s for policy ,π, would be: P. Kruchten, E. Lim, A. MacCormack, R. Nord, I. Ozkaya, et al. Managing technical debt in Bπ (s) = Exp[r(st ) + γ ∗ r(st+1 ) + γ 2 ∗ r(st+2 ) + ...|st = s] software-reliant systems. In Proceedings of the 19 FSE/SDP workshop on Future of software engineering 1–8. ACM, 2011. research, pages 47–52. ACM, 2010. [17] D. L. Parnas. Software aging. In Proceedings of the [2] Y. Cai, R. Kazman, C. Silva, L. Xiao, and H.-M. 16th international conference on Software engineering, Chen. A decision-support system approach to pages 279–287. IEEE Computer Society Press, 1994. economics-driven modularity evaluation. [18] J. Schumacher, N. Zazworka, F. Shull, C. Seaman, and Economics-Driven Software Architecture, 2013. M. Shaw. Building empirical support for automated [3] W. Cunningham. The wycash portfolio management code smell detection. In Proceedings of the 2010 system. ACM SIGPLAN OOPS Messenger, ACM-IEEE International Symposium on Empirical 4(2):29–30, 1993. Software Engineering and Measurement, page 8. ACM, [4] M. K. P. DARSINI. Application of Reinforcement 2010. learning algorithms to software verification. PhD [19] C. Seaman and Y. Guo. Measuring and monitoring thesis, M. Sc. Thesis, Universite Laval, 2006. technical debt. Advances in Computers, 82:25–46, [5] R. J. Eisenberg. A threshold based approach to 2011. technical debt. ACM SIGSOFT Software Engineering [20] V. Singh, W. Snipes, N. Kraft, et al. A framework for Notes, 37(2):1–6, 2012. estimating interest on technical debt by monitoring [6] C. Fernández-Sánchez, J. Dı́az, J. Pérez, and developer activity related to code comprehension. In J. Garbajosa. Guiding flexibility investment in agile Managing Technical Debt (MTD), 2014 Sixth architecting. In System Sciences (HICSS), 2014 47th International Workshop on, pages 27–30. IEEE, 2014. Hawaii International Conference on, pages 4807–4816. [21] W. Snipes, B. Robinson, Y. Guo, and C. Seaman. IEEE, 2014. Defining the decision factors for managing defects: A [7] M. Fowler. Technical debt quadrant. Bliki [Blog]. technical debt perspective. In Managing Technical Available from: http://www. martinfowler. Debt (MTD), 2012 Third International Workshop on, com/bliki/TechnicalDebtQuadrant. html, 2009. pages 54–60. IEEE, 2012. [8] Y. Guo, C. Seaman, R. Gomes, A. Cavalcanti, [22] R. S. Sutton and A. G. Barto. Reinforcement learning: G. Tonin, F. Q. Da Silva, A. L. Santos, and C. Siebra. An introduction, volume 1. MIT press Cambridge, Tracking technical debtâĂŤan exploratory case study. 1998. In Software Maintenance (ICSM), 2011 27th IEEE [23] E. Tom, A. Aurum, and R. Vidgen. A consolidated International Conference on, pages 528–531. IEEE, understanding of technical debt. In ECIS, page 16, 2011. 2012. [9] C. Izurieta and J. M. Bieman. How software designs [24] E. Tom, A. Aurum, and R. Vidgen. An exploration of decay: A pilot study of pattern evolution. In technical debt. Journal of Systems and Software, Empirical Software Engineering and Measurement, 86(6):1498–1516, 2013. 2007. ESEM 2007. First International Symposium on, [25] S. Wong, Y. Cai, M. Kim, and M. Dalton. Detecting pages 449–451. IEEE, 2007. software modularity violations. In Proceedings of the [10] L. P. Kaelbling, M. L. Littman, and A. W. Moore. 33rd International Conference on Software Reinforcement learning: A survey. Journal of artificial Engineering, pages 411–420. ACM, 2011. intelligence research, pages 237–285, 1996. [26] J. Xuan, Y. Hu, and H. Jiang. Debt-prone bugs: [11] M. M. Lehman and L. A. Belady. Program evolution: technical debt in software maintenance. processes of software change. Academic Press [27] N. Zazworka, C. Izurieta, S. Wong, Y. Cai, Professional, Inc., 1985. C. Seaman, F. Shull, et al. Comparing four approaches [12] J.-L. Letouzey. The sqale method for evaluating for technical debt identification. Software Quality technical debt. In Proceedings of the Third Journal, 22(3):403–426, 2014. International Workshop on Managing Technical Debt, [28] N. Zazworka, C. Seaman, and F. Shull. Prioritizing pages 31–36. IEEE Press, 2012. design debt investment opportunities. In Proceedings [13] Z. Li, P. Avgeriou, and P. Liang. A systematic of the 2nd Workshop on Managing Technical Debt, mapping study on technical debt and its management. pages 39–42. ACM, 2011. Journal of Systems and Software, 101:193–220, 2015. [14] S. McConnell. Technical debt. 10x software development. Blog]. Available at: http://blogs. construx. com/blogs/stevemcc/archive/2007/11/01/technicaldebt- 2. aspx, 2007. [15] R. L. Nord, I. Ozkaya, P. Kruchten, and M. Gonzalez-Rojas. In search of a metric for managing architectural technical debt. In Software Architecture (WICSA) and European Conference on Software Architecture (ECSA), 2012 Joint Working IEEE/IFIP Conference on, pages 91–100. IEEE, 2012. [16] A. Nugroho, J. Visser, and T. Kuipers. An empirical model of technical debt and interest. In Proceedings of the 2nd Workshop on Managing Technical Debt, pages 20