Test smells are suboptimal patterns in test code that can negatively impact the effectiveness [1] of software testing by reducing test maintainability, reliability, and eventually hampering software quality [2,3,4,5]. Developers usually introduce test smells due to less awareness in design and program comprehensibility [6]. Test smells can occur in both categories of software development environments-industrial and open-source projects [7,8]. Due to time, budget constraints, and software release pressure in the industry, systems are developed in a different setting compared to open-source projects. Additionally, open-source software grows more quickly than industrial projects which could have an impact on the occurrences of test smells and test writing practices between these two contexts [9].
Test smells were initially introduced to highlight issues in test code within software systems [10], but later they have been used as a measure of quality of software projects. Researchers found that test smells are associated with software quality [2,11], fault-proneness [12,13], and maintainability issues in software systems [2,14,15]. Although there has been extensive research on test smells in open-source projects, to the best of our knowledge, no studies have focused on test smell detection across both open-source and industrial projects. Identifying and addressing frequent test smells in both contexts can significantly improve software maintainability and reliability. This leads to the following research questions: To address these research questions, an empirical analysis was conducted on 75 Python projects, including 64 popular open-source projects and 11 industrial projects. Opensource projects have been selected based on criteria such as consistent development activity, unique project purpose, and substantial contributions. On the other hand, industrial projects were selected using convenience sampling [4,16] from software companies based on data accessibility and industry willingness to share information. The selected industrial projects have been developed following the standard Software Development Life Cycle (SDLC) model and are currently used by clients. A set of 18 test smells has been chosen for the investigation. To detect these test smells, PyNose [17] frequently in open-source projects compared to industrial projects, even though the volume of test code is not significantly higher in open-source projects. For the remaining 6 test smells, the analysis showed no significant difference in their frequency between open-source and industrial projects. Additionally, two-tailed hypothesis testing supports this finding, indicating minimal differences in test code volume across environments, with 6 out of 7 null hypotheses accepted.
The rest of the paper is organized as follows. Section 2 discusses existing work in the field of test smell analysis and detection. Section 3 describes the methodology used to design this empirical study. Section 4 presents the results and provides a detailed analysis based on the research questions. In Section 5, we address the threats to the validity of our study, and we conclude in Section 6.
In recent years, test smells have become a crucial area of research in software engineering. Studies have focused on the automated detection, mitigation, and impact evaluation of test smells on software quality. Early work aimed to define various types of test smells [10], while more recent studies have explored detection techniques [15,18] and the development of specialized tools for detecting test smells across different programming languages, such as Java, C#, Python, and Scala [19,20,17,21]. Furthermore, several studies have investigated the evolution of test smells and their influence on software maintenance and quality [3,22,2,23,14].
The concept of test smells was introduced by Van Deursen et al. [10], who defined them as signs of poor practices in test code. They identified 11 test smells, which laid the groundwork for understanding how suboptimal testing can impact software quality and maintenance. Expanding on this, Hauptmann et al. [24] introduced Natural Language Test Smells (NLTS) to address issues related to the maintainability, readability, and efficiency of manual test cases. Soares et al. [25] further refined this idea by cataloging eight NLTS, validated through empirical research and natural language processing (NLP) analysis. Additionally, the Open Catalog of Test Smells offers a comprehensive resource by consolidating information from 127 sources to support the identification and understanding of test smells in various contexts [26].
In recent years, specialized tools have emerged to aid in the identification and mitigation of test smells. Initially, research concentrated on statically typed languages like Java and Scala. Virginio et al. [20] developed JNose, a tool for detecting 21 types of test smells in Java, followed by tsDetect [19], which detects 19 types. Later, tools for Python, including pytest-smell [21] and PyNose [17], were introduced, with PyNose also introducing the "suboptimal assert" test smell. Pontillo et al. [18] proposed machine learning-based detection methods, while Palomba et al. [15] expanded test smell detection to automatic techniques using information retrieval methods. More recently, Lucas et al. [27] explored the use of large language models (LLMs) like ChatGPT-4, Mistral Large, and Gemini Advanced for test smell detection, demonstrating the potential of LLMs in this domain.
Test smells are often linked to flaky tests, which produce inconsistent results and undermine test reliability. Garousi et al. [7] and Martins et al. [11] found that certain test smells, such as external dependencies and non-deterministic behavior, contribute to flaky tests, masking genuine defects. Palomba and Zaidman [28] identified key smells that frequently co-occur with flaky tests, while Camara et al. [29] showed that incorporating test smells into flaky test prediction models outperforms traditional vocabulary-based approaches. These findings underscore the importance of mitigating test smells to enhance software reliability.
Further research has explored the impact of test smells on software quality, maintainability, and reliability. Spadini et al. [2,6] investigated the relationship between test smells and software quality, while Kim et al. [23] examined how test smells evolve and affect software maintenance. Qusef et al. [12] established a connection between test smells and fault-proneness, indicating their potential negative impact on software reliability.
In the context of open-source versus industrial projects, Fushihara et al. [30] showed that test smells persist and evolve over time, particularly in open-source projects. The differences between these two environments are often attributed to varying levels of emphasis on test quality. Recent studies have also examined how test smells and refactoring are discussed within online communities like Stack Exchange, revealing insights from both industry and academia [11].
While previous research has explored test smells in various software environments, there is a notable lack of comparative studies focusing on how these smells manifest in open-source versus industrial projects. Given the differences in development practices and priorities, understanding these variations is crucial for tailoring detection techniques and improving test quality. To address this gap, we conduct an empirical analysis. This study investigates the types of test smells prevalent in both open-source and industrial projects. It also examines how the prevalence of these smells differs between the two environments, as well as how the volume of test code compares across them.
This study aims to understand the occurrences, types and the distribution of test smells in open-source and industrial projects. It also seeks to assess the extent to which the distribution of test smells differs between open-source and industrial projects. To achieve this, the study analyzes 64 open-source projects and 11 industrial projects, all written in Python. Test smells within these projects are identified using a tool named PyNose1 . Then, the collected data are analyzed using the metrics Spreadity [31] and Inverse Smell Frequency (ISFi) [32]. Finally, statistical hypothesis tests are conducted to evaluate differences in test smell frequency and test code volume between open-source and industrial projects. The overall methodology is described in detail in the following subsections.
A dataset comprising 75 extensive Python projects-64 opensource projects and 11 industrial projects-serves as the foundation for our analysis. The open-source projects were chosen based on specific criteria: they must have more than 50 stars on GitHub, not be forked, have over 1,000 commits, include at least 10 contributors, possess a development history spanning more than two years, be actively maintained within the last two years, and primarily use Python as their programming language.
In selecting industrial project, convenience sampling [4] strategy was employed. This non-random technique was used because data collection on industrial projects largely depends on the accessibility, willingness, and readiness of software companies to share their data. In this study, 11 industrial projects were collected from different software companies of Bangladesh. Selection criteria included a minimum project size of 6,000 lines of code (LOC), a minimum of 500 commits to ensure development activity, and comprehensive test code (with at least 1,000 lines of test code).
The dataset was carefully curated to represent a diverse range of testing practices, providing a robust foundation for comparing test smells in both open-source and industrial projects. Key attributes, such as lines of code (KLOC), number of methods (NOM), and number of classes (NOC), are detailed in Table 1 and Table 2. These tables offer a clear overview of the projects' core characteristics, presented without any initial analysis. The complete dataset used in this study is available in the online appendix.2
Detecting test smells in Python typically involves a static code analysis process that inspects the structure of test code to identify patterns known to impact test maintainability, readability, and reliability. This approach systematically scans Python test classes-particularly those derived from the unittest framework-focusing on elements such as assertion statements, control structures, and setup and teardown methods. By examining these components, static analysis tools can identify common test smells like Assertion Roulette, Eager Test, and Test Duplication, which complicate debugging and future modifications. Consider the code snippet in Listing 1, which provides an example of a common test smell.
Listing 1: Test Smell Example import u n i t t e s t c l a s s E x a m p l e T e s t ( u n i t t e s t . T e s t C a s e ) :
def t e s t _ v a l u e s ( s e l f ) : r e s u l t = g e t _ v a l u e s ( ) s e l f . a s s e r t E q u a l ( r e s u l t [ 0 ] , 1 0 ) s e l f . a s s e r t T r u e ( r e s u l t [ 1 ] > 5 ) s e l f . a s s e r t E q u a l ( r e s u l t [ 2 ] , " e x p e c t e d _ v a l u e " ) i f __name__ == ' __main__ ' : u n i t t e s t . main ( )
In this example, the test contains multiple assertions without descriptive messages, which is a classic case of the "Assertion Roulette" test smell. This occurs when multiple assertions are made without context, making it difficult to diagnose failures. Such smells are detected using rule-based methods, as described by Aljedaani et al. [33], which identify tests with multiple assertions lacking clarity. Metrics like cosine similarity can also highlight a "Lack of Cohesion" in the test case.
The methodology uses predefined thresholds for various metrics, such as local variables and hard-coded values, based on best practices in software testing. This structured approach identifies issues like redundant setups, poorly named tests, and excessive complexity. For our study, we utilize PyNose [17], which is effective in detecting a broad range of test smells, as demonstrated in previous research [34,35,36].
To systematically analyze the distribution of test smells within our dataset, we use two key metrics: Spreadity and Inverse Smell Frequency (ISFi) [32], along with statistical hypothesis testing techniques to examine the differences in test smell occurrences between open-source and industrial projects. These metrics are commonly used to evaluate the distribution and frequency of smells in software research [31].
Spreadity: To measure the presence of test smells across the analyzed projects, we utilize the metric called Spreadity as shown in Equation 1. Spreadity calculates the proportion of projects that contain a specific test smell, providing insight into how common a particular smell is within the dataset.
where 𝑥 denotes the test smell, 𝑛(𝑥) is the number of projects containing the smell 𝑥, and 𝑁 represents the total number of projects analyzed. A higher Spreadity value indicates that a larger proportion of projects are affected by the test smell, with a maximum value of 1 if all projects contain the smell.
Inverse Smell Frequency (ISFi): To gain a deeper understanding of the occurrence of test smells, we calculate the Inverse Smell Frequency (ISFi) metric using the Equation 2. This metric is used to analyze code smell frequency and occurrences [31]. It normalizes the frequency of a test smell relative to specific software metrics, such as Test Lines of Code (TLOC), Test Number of Methods (TNOM), and Test Number of Classes (TNOC).
where the metric values could be TLOC, TNOM, or TNOC, and Smell Count𝑖 represents the occurrence of a particular test smell 𝑖. Lower ISFi values indicate a higher frequency of the test smell, suggesting that it is more prevalent in the code.
Statistical Tests of Hypothesis: The differences in test smell occurrences between open-source and industrial projects are examined using t-tests, a widely accepted method for comparing the means of two independent groups.
A one-tailed t-test is performed to determine if there are significant differences in the prevalence of specific test smells between open-source and industrial projects. The one-tailed approach is used because we hypothesize that the occurrence of test smells will be higher in opensource projects compared to industrial projects, based on previous observations of faster development cycles and less formal testing practices in open-source environments. The alternative hypothesis suggests that the occurrence of test smells is higher in industrial projects, while the null hypothesis assumes no significant difference. The test is conducted at a 5% significance level.
Additionally, a two-tailed t-test was used to assess whether there were any significant differences in the volume of test code between open-source and industrial projects. In this analysis, the alternative hypothesis proposes a difference in test code volume, while the null hypothesis assumes no difference. The test was conducted at a 5% significance level, with specific metrics analyzed to determine any significant variations between the two environments.
This section presents the findings from our study, addressing each research question by analyzing the occurrences of test smells in industrial and open-source projects. We highlight the types of test smells, their frequency, differences in occurrence between environments, and insights into testwriting practices in the software industry.
To address RQ1, we analyze the presence of test smells in both open-source and industrial projects. Our study identifies 14 distinct test smells across these environments, with notable differences in their presence and frequency, as shown in Table 4.
The The analysis reveals that test smells such as Assertion Roulette (AR) and Conditional Test Logic (CTL) are notably more prevalent in open-source projects compared to industrial projects. This conclusion is based on both the spread and average frequency metrics, as shown in Figure 2. Additionally, Figures 3, 4, and 5 illustrate the ISFi scores for TLOC, TNOM, and TNOC across all the test smells under study. The four common smells that did not appear in any system are excluded, as discussed in response to RQ1. 2.
In order to corroborate these observations, a one-tailed t-test with a 5% significance level is used to conduct a statistical hypothesis test. The p-value is displayed in the final column of Table 4, and the value that rejects the null hypothesis is bolded. null hypothesis states that there no difference the occurrence of a particular test smell between open-source industrial projects, while the alternative hypothesis refers that the open-source system has higher occurrences. The p-values in Table 4 indicate that the null hypothesis is rejected for 8 out of the 14 test smells, suggesting a statistically significant higher occurrence of these smells in open-source projects. These findings emphasize that open-source projects are more susceptible to frequent and diverse test smells compared to industrial projects, highlighting the need for targeted improvements in test-writing practices within the opensource community. The null hypothesis (𝐻0) posits that there is no significant difference in the volume of test code between open-source and industrial Python projects (𝜇 Industry = 𝜇Open Source). In contrast, the alternative hypothesis (𝐻1) suggests that the test code volume differs between the two environments (𝜇 Industry ̸ = 𝜇Open Source). Table 5 provides the t-test results comparing these metrics across open-source and industrial projects. While most metrics, such as Total Test Files, Test LOC, Test NOC, and Test NOM, did not show statistically significant differences, the Test NOC per File metric, with a pvalue of 0.009, indicates a notable distinction in the number of test classes per file between the two environments. This finding suggests a potential structural difference in test code organization across open-source and industrial projects.
The t-test was conducted for all test code metrics extracted from both industrial and open-source projects. The null hypothesis was accepted for 6 metrics, indicating no statistically significant difference in test code volume or structure between the industrial and open-source projects for most metrics. However, the null hypothesis was rejected for 1 metric, Test NOC per File (marked in bold in Table 5), supporting the claim that the number of test classes per file in open-source projects is significantly different from that in industrial projects, suggesting a distinct approach to organizing test classes. These findings are summarized in Table 5, showing that for the majority of metrics (6 out of 7), there is no significant difference in test code volume between open-source and industrial projects. This suggests that, with respect to basic testing characteristics like the number of test files, lines of code, and methods, both environments follow relatively similar practices.
However, the significant difference in the Test NOC per File metric highlights that open-source projects may organize test classes differently, potentially due to variations in project structure, coding standards, or collaborative development practices typical in open-source environments.
This section addresses construct, external, and reliability threats that may affect the validity of our empirical investigation [37].
The accuracy of PyNose, our test smell detection tool, may affect construct validity. Any inaccuracies in detecting test smells could lead to incorrect conclusions regarding their presence and significance. To minimize this risk, we used PyNose, a widely recognized tool in test smell detection with reliability confirmed in prior studies.
Our findings' generalizability is limited by several external validity threats. First, this study focuses on Python-based projects, so results may not fully apply to projects in other languages like Java or C#. Additionally, the dataset comprises 11 industrial projects from Bangladesh and 64 open-source projects, which may not represent the broader global industry practices. The industrial projects come from a specific sector that may have unique testing practices. To address these threats, we included well-known open-source projects with diverse testing practices.
Reliability concerns arise from the need to ensure replicability, particularly for industrial data. Due to confidentiality constraints, we cannot disclose details about the industrial projects, limiting direct replication. However, open-source project findings are fully replicable, as these projects and the PyNose tool are publicly accessible. For further validation, future studies could replicate the analysis with similar industrial datasets.
This research investigates the occurrence, distribution, and frequency of test smells in both industrial and opensource projects, highlighting the differences between these two environments. The inclusion of industrial projects alongside open-source ones adds depth to the study, as it reveals distinct patterns in the prevalence of test smells. Although Assertion Roulette (AR) is the most common test smell, appearing in 92% of open-source projects and 81% of industrial projects, a one-tailed t-test shows significant differences in the overall prevalence of test smells between these two types of projects.
Furthermore, 10 out of 18 test smells were observed in industrial projects, while 14 out of 18 test smells were detected in open-source projects. Notably, 8 out of the 14 test smells showed a significantly higher occurrence in opensource projects. Analyzing test writing practices, we found no significant differences in 6 out of 7 metrics between industrial and open-source projects, except for the number of test classes per file (TNOC). These findings will assist developers in reducing test smells during software development and provide insights for researchers to refine smell refactoring techniques, particularly in terms of frequency and distribution. Future work should explore how specific development practices contribute to these differences between project types, offering a deeper understanding of how coding conventions and workflows influence test smell occurrence. Further research could also investigate automated refactoring techniques that account for these structural and volumetric differences to better support quality improvements in both open-source and industrial projects.