Measuring Technical Debt is important in guiding software development teams to make informed decisions and prioritize refactoring initiatives. This study presents an exploratory literature review of studies published between 2010 and 2023 to investigate the current state of Technical Debt measurement research. Through a set of four research questions, this study identifies the prevalent methodologies, metrics, and obstacles entailed in quantifying Technical Debt. Specifically, this study focuses on what is proposed to be measured through Technical Debt, the measurement solutions proposed for measuring Technical Debt, and how these approaches categorize and evaluate various aspects of Technical Debt. By scrutinizing the diverse approaches and challenges, this exploratory literature review identifies gaps (and related issues) in Technical Debt measurement research and contributes to a nuanced understanding of Technical Debt measurement practices, offering insights into enhancing software sustainability and maintainability.
The Technical Debt (TD) metaphor was first introduced by Ward Cunningham in 1992 [
Since then, the definition and understanding of TD has evolved [
Measuring TD is important for software organizations to understand the extent of the problem and
prioritize areas that require immediate attention. By quantifying TD, organizations can make informed
decisions about when and where to invest resources to address the accumulated debt [
Researchers have explored various approaches to quantify TD, and the widely adopted approach
involves leveraging code analysis tools that evaluate the quality of the codebase by employing
predefined metrics such as code complexity, code duplication, and coding style violations [
The most common approach for measuring TD involves various approaches that focus on the
different aspects of TD quantification. These approaches include identifying smells, quantifying the
Return on Investment (ROI) of refactoring, comparing the ideal state with the current state of software
quality, and evaluating alternative development paths to reduce technical debt [
Except for the literature review of TD in requirements [
The remainder of this paper is organized as follows. Section 2 presents the systematic literature review method used in this study. Section 3 presents and discusses the results. Section 4 concludes the paper with a summary of key findings and directions for future work.
The following section describes the review method used for conducting this exploratory literature
review through a Systematic Literature Review (SLR), a rigorous and transparent method for
comprehensively analyzing existing research on a specific topic [
To delve deeper into technical debt measurement, this exploratory literature review addresses the research questions (RQs) presented in Table 1, along with their rationales.
To identify relevant literature to this exploratory review, we conducted a comprehensive search across five reputable digital libraries: Scopus, ScienceDirect, ACM Digital Library, IEEE Xplore, and SpringerLink. The search encompassed peer-reviewed publications published between January 2010 and December 2023. The choice of 2010 as the starting point for the exploratory literature review on technical debt measurement, instead of 1992 when the technical debt metaphor was introduced, is primarily due to the significant increase in research activity around technical debt in the last decade. This study focuses on this period to capture the most recent and relevant developments in the field. Additionally, the selection of this time frame helps ensure that the review includes comprehensive and up-to-date techniques and approaches, reflecting the current state of technical debt measurement practices. These specific databases were chosen because of their extensive coverage of computer science and engineering research, ensuring a high likelihood of capturing pertinent studies on technical debt measurements within this domain.
The search string was based on the key terms identified in the RQs as well as the commonly used terminology associated with TD and measurement. Initially, the main search terms were combined using the Boolean operator “OR” to their corresponding related keywords. Subsequently, these main terms were linked with one another using “AND” to ensure a focused and relevant set of results. Table 2 lists the complete search strings used in this study.
Owing to potential variations in search engine syntax across different databases, we carefully adapted our search strings to optimize retrieval across each database (Scopus, ScienceDirect, ACM Digital Library, IEEE Xplore, and SpringerLink). The search focused on titles, abstracts, and keywords to ensure relevant studies were captured. To manage the retrieval process, we conducted separate searches on each database, followed by consolidation of the identified papers. Subsequently, we employed EndNote reference management software to identify and remove duplicate studies, ensuring streamlined and non-duplicate studies for further analysis. 2.3.
The selection aimed to identify relevant primary studies that addressed the measurement of technical debt using the following inclusion criteria (IC) and exclusion criteria (EC):
Inclusion criteria: • Studies written in English and published between January 2010 and December 2023. • Studies published in peer-reviewed journals, conference proceedings, workshop proceedings, and book chapters. • Studies where full texts are available. • Studies that propose, evaluate, or discuss techniques, metrics, or approaches for measuring technical debt.
Exclusion criteria: • Duplicate publications of the same study. • Studies where full texts are not available. • Studies titles and abstracts with a focus on non-technical debt aspects of software development. • Studies on TD but not on TD measurement. • Studies on TD without a clear focus on measurement aspects • Studies have focused solely on identifying TD (without measurement) or lacking details on the measurement approach.
A three-stage process was followed to select the studies for this review.
In the first stage, 808 primary studies were identified from the five digital libraries. Seventy (70) studies were discarded as duplicate publications of the same study.
In the second stage, an initial screening of the search results (738 studies) was performed based on titles and abstracts. Studies that full texts are not available were excluded (228 studies). Additionally, 480 studies that did not explicitly mention technical debt measurement or quantification techniques, metrics, or approaches were excluded. After this stage, we obtained 30 full-text studies reviewed.
In the third stage, a full-text review of the remaining studies (30 remaining studies) was conducted. During this phase, we applied the defined inclusion and exclusion criteria to ensure the relevance and quality of the selected studies. The exclusion criteria were as follows: • Studies on TD but not on TD measurement. • Studies on TD without a clear focus on measurement aspects. • Studies have focused solely on identifying TD (without measurement) or lacking details on the measurement approach.
To ensure the reliability of the study selection process, two researchers independently performed screening and full-text reviews. Any disagreements or conflicts were resolved by discussion and consensus. After this phase, the 21 remaining studies were included in the review. The study selection process, with the total number of studies retrieved and included in each phase, is shown in Figure 1.
Identification of studies via databases n o i t a c i f i t n e d I
Records identified (n=808):
ACM (n = 292) IEEE Xplore (n = 161) ScienceDirect (n = 72) Scopus (n = 182)
SpringerLink (n = 101)
Records removed before screening: Duplicate records removed
(n = 70) • • • • •
Records excluded (n = 228):
Reports excluded (n = 480):
technical debt aspects of software development Reports excluded (n = 9):
on measurement aspects
Quality assessment was used to assess the relevance and credibility of the selected studies. The
papers were selected from five well-known databases of papers reviewed by experts before their
publication. The quality of the included primary studies was assessed using a customized quality
checklist adapted from Kitchenham and Charters [
The checklist consists of the following four quality criteria: • QA1: Are the research aims clearly stated? • QA2: Is the technical debt measurement approach clearly described? • QA3: Are the findings clearly stated? • QA4: Are the limitations of the study discussed? Only studies that answered at least three of the above questions were selected. After this phase, all the 21 studies met three of the above questions. 2.5.
A data extraction form was developed to retrieve relevant information from the included primary studies, addressing the RQs. It should be noted that not all the selected studies addressed all four RQs. Data synthesis aimed to collate and summarize the results of the included primary studies. We identified and grouped all relevant data to answer the RQs using a descriptive synthesis.
This section provides answers to the SLR research questions based on a synthesis of the selected studies. 3.1. (TD)?
The scope of the RQ1 covers a triple question “why”, “how” and “what”. Why measuring TD enables to understanding the reasons behind measuring technical debt helps in recognizing its importance and the benefits it brings to software projects. How to Measure TD focuses on the methodologies and approaches used to quantify and analyze TD. What Should be Measured about TD identifies the specific aspects and metrics that need to be quantified to understand and manage technical debt effectively.
The primary measurement goals identified in the selected studies are: 1. Sizing TD in terms of the effort required to reduce it to zero (S1, S3, S4, S5, S8, S19, S20, S21). 2. Assessing the impact of TD on software quality attributes such as functionality, performance, maintainability, reliability, and security (S1, S2, S6, S9, S10, S17, S18). 3. Estimating the rework/refactoring efforts required to enhance evolvability and mitigate accumulated TD (S3). 4. Evaluating the accuracy and usefulness of TD measurement tools (S11). 5. Comparing different TD identification techniques (S13).
Several approaches have been proposed for measuring (quantifying) TD. Existing approaches measure TD in terms of what is proposed to be measured through TD in the following ways: • Eight studies (S1, S3, S4, S5, S8, S19, S20, S21) base their quantification on the identification of code, design, architectural smells, or defects. • Seven studies (S3, S4, S5, S8, S19, S20, S21) have attempted to quantify the return on investment (ROI) of refactoring activities to remove technical debt. • Seven studies (S1, S2, S6, S9, S10, S17, S18) compare an ideal state with the current state of the software in terms of quality attributes, such as maintainability and modularity. • Seven studies (S3, S4, S5, S8, S19, S20, S21) compare alternative development paths with the aim of reducing rework and quantifying the impacts of taking on technical debt versus not taking it on.
In summary, the key measurements relate to sizing TD in terms of the required remediation effort, assessing the impact on software quality attributes, estimating rework efforts for mitigating TD, and evaluating the effectiveness of TD measurement approaches and tools. 3.2. RQ2: What are the existing measurement solutions for measuring technical debt (TD)?
The selected studies proposed various measurement solutions, including: • Code metrics (complexity, coupling, cohesion, size, duplication, etc.) (S1, S3, S6, S9, S10, S11,
S14, S16, S18). • Modularity and design quality metrics (S3, S14). • Defect proneness, change proneness, and maintenance effort metrics (S7, S9, S10, S15, S17). • Static code analysis issues/violations (S2, S9, S19, and S20). • Test coverage and quality metrics (S2, S9). • Technical debt principal and interest calculations (S5, S7, S12, S15, S19, S20, and S21). • Machine Learning models for TD identification and forecasting (S6, S9, and S10).
Table 3 presents the measurement solutions/metrics proposed for measuring Technical Debt (TD) across the selected studies.
As shown in Table 3, the measurement solutions/metrics proposed across the selected studies covered a wide range of aspects, including code metrics, design/architecture metrics, static analysis metrics, quality metrics, effort/cost metrics, machine learning features, and other metrics related to defect proneness, change proneness, and maintainability. 3.3. of TD?
The selected studies categorize and evaluate various aspects of TD in the following ways: • Based on the type of TD: code debt, design debt, architectural debt, documentation debt, test debt, etc. (S3, S4, S10, S12, S18, S19, and S20). • Based on architectural levels: path, activity, and application levels (S3). • Based on quality characteristics/attributes affected: reliability, security, maintainability, portability, etc. (S4, S13, S19, and S20). • Based on severity/priority of issues (S1, S19, S20). • Based on financial cost/effort estimations (S5, S19, S20, and S21).
In summary, the approaches categorize and evaluate TD based on numerous factors, including the types of TD (code, design, architectural, etc.), architectural levels affected, quality characteristics impacted, severity or priority of issues, and financial implications or effort estimations. By considering these different categories and evaluation perspectives, this study aims to provide an understanding of TD, its manifestations, its impact on various aspects of software quality, and the potential costs and efforts required for its remediation. 3.4.
From the analysis of selected studies, gaps (and related issues) in TD measurement research were identified by the researchers themselves: • Lack of validation on real-world projects (S3). • Measuring other types of debt beyond code debt (documentation debt, test debt, architectural debt, etc.) (S4, S10). • Integrating developers’ opinions with code characteristics to improve TD severity identification (S1). • Holistic approaches combining financial and technical factors (S14). • Interpretable thresholds in metric-based approaches (S14). • Quantifying interest costs, risks/liabilities, and opportunity costs of TD (S19 and S20). • Generalizability and application to complex systems (S21). • Need for effective tooling and methodologies for managing TD across the software development lifecycle (S10).
These gaps and issues highlight the need for further research and improvements in TD measurement, including validation on real-world projects, comprehensive coverage of different TD types, integration of developer perspectives, holistic approaches, interpretable thresholds, consideration of TD costs and risks, generalizability to complex systems, effective tooling and methodologies, direct TD quantification, and accounting for non-functional requirements.
Addressing these gaps and issues can contribute to more accurate, practical, and comprehensive TD measurement approaches, thereby enabling better management and decision-making processes related to TD in software development projects. 3.5.
This study presents several key findings. Firstly, the study identified a variety of measurement goals such as sizing TD, assessing its impact on software quality attributes, estimating rework efforts for TD mitigation, and evaluating the effectiveness of TD measurement tools. The research highlights the prevalent methodologies, including code metrics, design/architecture metrics, static analysis metrics, quality metrics, effort/cost metrics, and machine learning features for identifying and forecasting TD.
The findings indicate significant research gaps and areas for further exploration. Researchers are encouraged to focus on: 1. Validation in Real-World Projects: There is a need for more empirical validation of TD measurement tools and methodologies in real-world software development projects to enhance their practical applicability.
2. Comprehensive Coverage of TD Types: Future research should expand beyond code debt to include other types such as documentation debt, test debt, and architectural debt. This holistic approach will provide a more accurate picture of TD and its implications.
3. Integration of Developer Perspectives: Incorporating insights from developers regarding the severity and impact of TD can improve the accuracy and relevance of measurement tools.
4. Holistic Approaches Combining Financial and Technical Factors: Developing measurement approaches that consider both financial and technical aspects of TD can offer more comprehensive management strategies.
5. Interpretable Thresholds in Metric-Based Approaches: Establishing clear and interpretable thresholds for various TD metrics will facilitate better decision-making processes for software teams.
Practitioners can benefit from the study’s insights by: 1. Adopting Diverse Measurement Solutions: Utilizing a combination of code metrics, design/architecture metrics, static analysis metrics, and machine learning models can provide a multifaceted understanding of TD, aiding in more effective management and reduction strategies.
2. Focusing on High-Impact Areas: By identifying and prioritizing areas with high TD, practitioners can allocate resources more efficiently, addressing the most critical issues that affect software quality and maintainability.
3. Continuous Monitoring and Refinement: Implementing continuous TD measurement and monitoring processes will help in early detection and mitigation of TD, thereby reducing long-term costs and improving software sustainability.
From the synthesis of over a decade of research, several high-level concepts and lessons emerge: 1. The Complexity of TD Measurement: The measurement of TD is inherently complex, involving multiple dimensions such as code quality, design, architecture, and financial implications. This complexity necessitates sophisticated and integrated measurement approaches.
2. The Importance of Contextual Factors: The impact of TD varies significantly depending on the context of the software project, including factors like project size, complexity, and team expertise. Tailoring TD measurement approaches to specific project contexts can enhance their effectiveness.
3. Need for Standardization and Tool Integration: There is a pressing need for standardization in TD measurement approaches and better integration of tools to facilitate more consistent and reliable measurements across different projects and organizations.
4. Continuous Evolution of Measurement Techniques: As software development practices evolve, so too must the techniques and tools for measuring TD. Keeping abreast of emerging trends and incorporating new methodologies will be crucial for maintaining effective TD management practices.
In summary, addressing these insights and integrating them into both research and practice can significantly enhance the management of TD, contributing to the long-term sustainability and quality of software systems.
This exploratory literature review aimed to provide a comprehensive overview of state-of-the-art in technical debt measurement research. However, there are potential threats to validity that should be acknowledged: • While the search string was carefully constructed to capture relevant studies, it is possible that some relevant publications may have been missed because of the use of different terminologies or the presence of relevant studies in sources not included in the selected digital libraries. This review focused exclusively on studies published between January 2010 and December 2023. Consequently, relevant earlier works or very recent publications may have been unintentionally excluded. • Although the study selection process followed well-defined inclusion and exclusion criteria and was conducted independently by two researchers, there is an inherent risk of bias in the selection and interpretation of studies. • The quality assessment of the included studies was based on a customized checklist adapted from established guidelines. However, the assessment process may have introduced bias because of the subjective interpretation of quality criteria. • The included studies exhibited substantial heterogeneity in terms of research methodologies, measurement approaches, and evaluation contexts. This diversity may introduce challenges in synthesizing and comparing findings across studies.
Despite these potential threats, we used rigorous and systematic methods to conduct the literature review, including following established guidelines involving multiple researchers in the study selection and data extraction processes. Additionally, we have transparently acknowledged the limitations and potential threats to validity, which can inform the interpretation and applicability of the findings.
The quantification and measurement of TD are important for software development teams and
organizations. By understanding the extent and impact of technical debt, organizations can make
informed decisions about when and where to invest resources to address it. A number of technical debt
issues have been investigated by researchers over the years, but relatively few have focused on technical
debt measurements/metrics. This 2010-2023 SLR in studies proposing TD measurements followed the
guidelines proposed by Kitchenham and Charters [
The key findings are as follows: • RQ1: Why, how, and what should be measured about technical debt (TD)? The primary measurement goals identified were: 1) sizing TD in terms of required remediation effort, 2) assessing the impact of TD on software quality attributes, 3) estimating rework efforts for TD mitigation, 4) evaluating the effectiveness of TD measurement approaches and tools. • RQ2: What are the existing measurement solutions for measuring TD?
A wide range of measurement solutions has been proposed, including code metrics (complexity, coupling, cohesion, etc.), design and architecture metrics, static analysis metrics, quality metrics, effort/cost metrics, machine learning features, and metrics related to defect proneness, change proneness, and maintainability.
• RQ3: How do these approaches categorize and evaluate various aspects of TD? The approaches categorized and evaluated TD based on factors such as the types of TD (code, design, architectural, etc.), architectural levels affected, quality characteristics impacted, severity or priority of issues, and financial implications or effort estimations.
• RQ4: What are the gaps (issues) identified in TD measurement research?
The key gaps identified included the lack of real-world validation, limited coverage of non-code debt types, need for holistic approaches integrating technical and financial factors, lack of interpretable thresholds, quantification of TD costs and risks, generalizability to complex systems, and the need for effective tooling and methodologies.
In summary, this review identified a diversity of measurement solutions and categorization approaches for technical debt while also highlighting significant gaps and areas for further research and improvement in this field. The findings provide a nuanced understanding of the current state of technical debt measurement research and offer insights into enhancing software sustainability and maintainability through effective technical debt management practices. 5.2.
The findings from this exploratory literature review highlight promising directions for future research on technical debt measurements. There is a need for more comprehensive measurement approaches that can effectively quantify and consolidate distinct types of technical debt such as design debt, architectural debt, documentation debt, and test debt. Further research could also explore the identification and measurement of technical debt, which can be derived from software functional requirements not yet implemented, as well as from system non-functional requirements not implemented and that can be implemented in software functions distributed across a software environment.
We are grateful to anonymous reviewers.
This appendix contains the supporting documentation for our article. The list of the 21 selected primary studies to perform the SLR is presented in Table 1.
Quantification of Technical
Debt
Maintainability Index, SIG Method, and SQALE for Technical Debt Identification
Technical Debt Principal Assessment through Structural
Metrics Towards an Open-Source Tool for Measuring and Visualizing the Interest of Technical Debt
Applying Metrics to
Identify and Monitor Technical Debt Items during Software Evolution
Comparing four approaches for technical debt identification
Investigating the Impact of Code Smells Debt on Quality
Code Evaluation
Estimating the Size, Cost, and Types of Technical Debt
Estimating the Principal of an Application’s Technical
Debt
Technical Debt and Interest
Engineering
IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
Applied Computing (SAC '20)
Managing Technical Debt (MTD)
on Managing Technical Debt (MTD '11)