Understanding the Prevalence of Test Smells in Open-source and Industrial Software: An Empirical Study on Python Projects Md Arif Hasan, Toukir Ahammed Institute of Information Technology, University of Dhaka, Dhaka, Bangladesh Abstract Test smells, a type of anti-pattern, refer to poorly designed tests that can undermine the effectiveness, maintainability, and overall quality of test suites, as well as the reliability of production code. Test smells have been extensively investigated in the literature for open-source projects from different viewpoints such as detection, analysis, impact on maintainability and quality issues. However, whether the occurrence of test smells differs between open-source and industrial projects has not yet been investigated. This study aims to bridge that gap by investigating the types, occurrence and distribution of test smells in both open-source and industrial projects. Moreover, this study explores whether the volume of test code differs between open-source and industrial projects to gain insights into the test-writing practices in both environments. For this purpose, 64 open-source and 11 industrial Python projects were analyzed to detect and examine the prevalence of 18 distinct test smells. 14 types of test smells were identified in open-source projects, averaging 601.58 smells per project, while 10 types were identified in industrial projects, with an average of 48.72 smells per project. Assertion Roulette and Conditional Logic Test are the two most frequently occurring test smells observed in both environments. The results of the study indicate that test smells occur more frequently in open-source projects compared to industrial projects, as confirmed by one-tailed t-test, while a two-tailed t-test revealed no significant difference in the volume of test code between the two environments. These findings suggest that open-source projects and industrial projects have differences in maintaining clean and high-quality test code. Future work should explore how specific development practices contribute to this difference across both project types. Keywords Test Smells, Python Testing, Industrial Projects, Open Source Projects, Test Code Quality 1. Introduction • RQ3: How does the volume of test code differ between open-source and industrial Python Test smells are suboptimal patterns in test code that can projects? negatively impact the effectiveness [1] of software testing by reducing test maintainability, reliability, and eventually To address these research questions, an empirical analysis hampering software quality [2, 3, 4, 5]. Developers was conducted on 75 Python projects, including 64 popular usually introduce test smells due to less awareness in open-source projects and 11 industrial projects. Open- design and program comprehensibility [6]. Test smells source projects have been selected based on criteria such can occur in both categories of software development as consistent development activity, unique project purpose, environments—industrial and open-source projects [7, 8]. and substantial contributions. On the other hand, industrial Due to time, budget constraints, and software release projects were selected using convenience sampling [4, 16] pressure in the industry, systems are developed in a different from software companies based on data accessibility and setting compared to open-source projects. Additionally, industry willingness to share information. The selected open-source software grows more quickly than industrial industrial projects have been developed following the projects which could have an impact on the occurrences standard Software Development Life Cycle (SDLC) model of test smells and test writing practices between these two and are currently used by clients. A set of 18 test smells contexts [9]. has been chosen for the investigation. To detect these test Test smells were initially introduced to highlight issues smells, PyNose [17] - a popular test smell detection tool for in test code within software systems [10], but later they Python test code, is used. have been used as a measure of quality of software projects. After detecting test smells, frequent test smells are Researchers found that test smells are associated with identified and assessed whether the occurrence of test software quality [2, 11], fault-proneness [12, 13], and smells differs between open source and industrial projects. maintainability issues in software systems [2, 14, 15]. The results of the study show that 14 types of test smells Although there has been extensive research on test smells were found in open-source projects and 10 types of test in open-source projects, to the best of our knowledge, smells in industrial projects. Assertion Roulette (AR) is no studies have focused on test smell detection across the most common test smell found in both open-source and both open-source and industrial projects. Identifying industrial projects, indicating widespread issues in assertion and addressing frequent test smells in both contexts practices across environments. In industrial projects, the can significantly improve software maintainability and top 3 occurring test smells are Assertion Roulette (AR), reliability. This leads to the following research questions: Magic Number Test (MNT), and Conditional Test Logic (CTL). Meanwhile, in open-source projects, the top three • RQ1: What types of test smells are prevalent in are Assertion Roulette (AR), Conditional Test Logic (CTL), open-source and industrial Python projects? and Duplicate Assertion (DA). This finding highlights the • RQ2: How does the prevalence of test smells distinct patterns of test smell occurrence in open-source differ between open-source and industrial Python and industrial projects. projects? Our analysis shows that test smells are more prevalent QuASoQ 2024: 12th International Workshop on Quantitative Approaches and concentrated in open-source projects, with an average to Software Quality, December 03, 2024, Chongqing, China of 601.58 test smells per project compared to 48.72 per $ bsse1112@iit.du.ac.bd (M. A. Hasan); toukir@iit.du.ac.bd (T. Ahammed) project in industrial projects. Assertion Roulette (AR) is the © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License most widespread test smell in both environments, though it Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 19 appears in 92% of open-source projects and 81% of industrial ChatGPT-4, Mistral Large, and Gemini Advanced for test projects. On average, Assertion Roulette occurs for every smell detection, demonstrating the potential of LLMs in this 262 lines of test code in industrial projects and every 212 domain. lines in open-source projects. Statistical analysis conducted Test smells are often linked to flaky tests, which produce using a one-tailed t-test shows that 8 out of 14 test smells inconsistent results and undermine test reliability. Garousi occur more frequently in open-source projects compared et al. [7] and Martins et al. [11] found that certain test to industrial projects, even though the volume of test code smells, such as external dependencies and non-deterministic is not significantly higher in open-source projects. For the behavior, contribute to flaky tests, masking genuine defects. remaining 6 test smells, the analysis showed no significant Palomba and Zaidman [28] identified key smells that difference in their frequency between open-source and frequently co-occur with flaky tests, while Camara et al. industrial projects. Additionally, two-tailed hypothesis [29] showed that incorporating test smells into flaky test testing supports this finding, indicating minimal differences prediction models outperforms traditional vocabulary-based in test code volume across environments, with 6 out of 7 approaches. These findings underscore the importance of null hypotheses accepted. mitigating test smells to enhance software reliability. The rest of the paper is organized as follows. Section 2 Further research has explored the impact of test smells discusses existing work in the field of test smell analysis on software quality, maintainability, and reliability. Spadini and detection. Section 3 describes the methodology used to et al. [2, 6] investigated the relationship between test smells design this empirical study. Section 4 presents the results and software quality, while Kim et al. [23] examined how and provides a detailed analysis based on the research test smells evolve and affect software maintenance. Qusef questions. In Section 5, we address the threats to the validity et al. [12] established a connection between test smells and of our study, and we conclude in Section 6. fault-proneness, indicating their potential negative impact on software reliability. In the context of open-source versus industrial projects, 2. Related Work Fushihara et al. [30] showed that test smells persist and evolve over time, particularly in open-source projects. The In recent years, test smells have become a crucial area of differences between these two environments are often research in software engineering. Studies have focused on attributed to varying levels of emphasis on test quality. the automated detection, mitigation, and impact evaluation Recent studies have also examined how test smells and of test smells on software quality. Early work aimed to refactoring are discussed within online communities like define various types of test smells [10], while more recent Stack Exchange, revealing insights from both industry and studies have explored detection techniques [15, 18] and academia [11]. the development of specialized tools for detecting test While previous research has explored test smells in smells across different programming languages, such as various software environments, there is a notable lack Java, C#, Python, and Scala [19, 20, 17, 21]. Furthermore, of comparative studies focusing on how these smells several studies have investigated the evolution of test smells manifest in open-source versus industrial projects. Given and their influence on software maintenance and quality the differences in development practices and priorities, [3, 22, 2, 23, 14]. understanding these variations is crucial for tailoring The concept of test smells was introduced by Van detection techniques and improving test quality. To address Deursen et al. [10], who defined them as signs of poor this gap, we conduct an empirical analysis. This study practices in test code. They identified 11 test smells, which investigates the types of test smells prevalent in both laid the groundwork for understanding how suboptimal open-source and industrial projects. It also examines how testing can impact software quality and maintenance. the prevalence of these smells differs between the two Expanding on this, Hauptmann et al. [24] introduced environments, as well as how the volume of test code Natural Language Test Smells (NLTS) to address issues compares across them. related to the maintainability, readability, and efficiency of manual test cases. Soares et al. [25] further refined this idea by cataloging eight NLTS, validated through empirical 3. Methodology research and natural language processing (NLP) analysis. Additionally, the Open Catalog of Test Smells offers a This study aims to understand the occurrences, types and comprehensive resource by consolidating information from the distribution of test smells in open-source and industrial 127 sources to support the identification and understanding projects. It also seeks to assess the extent to which the of test smells in various contexts [26]. distribution of test smells differs between open-source and In recent years, specialized tools have emerged to aid industrial projects. To achieve this, the study analyzes 64 in the identification and mitigation of test smells. Initially, open-source projects and 11 industrial projects, all written research concentrated on statically typed languages like in Python. Test smells within these projects are identified Java and Scala. Virginio et al. [20] developed JNose, a using a tool named PyNose1 . Then, the collected data are tool for detecting 21 types of test smells in Java, followed analyzed using the metrics Spreadity [31] and Inverse Smell by tsDetect [19], which detects 19 types. Later, tools for Frequency (ISFi) [32]. Finally, statistical hypothesis tests are Python, including pytest-smell [21] and PyNose [17], were conducted to evaluate differences in test smell frequency introduced, with PyNose also introducing the "suboptimal and test code volume between open-source and industrial assert" test smell. Pontillo et al. [18] proposed machine projects. The overall methodology is described in detail in learning-based detection methods, while Palomba et al. [15] the following subsections. expanded test smell detection to automatic techniques using information retrieval methods. More recently, Lucas et al. [27] explored the use of large language models (LLMs) like 1 https://github.com/JetBrains-Research/PyNose.git 20 Table 1 Production Code Metrics KLOC NOM NOC Projects Category Number of Projects Total Average Total Average Total Average Open-Source Project 64 18012 281 603118 9428 145659 2276 Industrial Project 11 2592 236 86045 7822 23355 2123 Table 2 Test Code Metrics Test KLOC Test NOM NOC Projects Category Number of Projects Total Average Total Average Total Average Open Source Project 64 2882 45 317810 4966 23730 371 Industrial Project 11 62 6 7422 675 650 59 3.1. Dataset Collection Listing 1: Test Smell Example import u n i t t e s t A dataset comprising 75 extensive Python projects—64 open- source projects and 11 industrial projects—serves as the clas s ExampleTest ( u n i t t e s t . TestCase ) : def t e s t _ v a l u e s ( s e l f ) : foundation for our analysis. The open-source projects result = get_values ( ) were chosen based on specific criteria: they must have s e l f . assertEqual ( result [0] , 10) s e l f . assertTrue ( result [1] > 5) more than 50 stars on GitHub, not be forked, have over s e l f . assertEqual ( result [2] , " expected_value " ) 1,000 commits, include at least 10 contributors, possess a i f __name__ == ’ __main__ ’ : development history spanning more than two years, be u n i t t e s t . main ( ) actively maintained within the last two years, and primarily use Python as their programming language. In this example, the test contains multiple assertions In selecting industrial project, convenience sampling [4] without descriptive messages, which is a classic case of the strategy was employed. This non-random technique was "Assertion Roulette" test smell. This occurs when multiple used because data collection on industrial projects largely assertions are made without context, making it difficult to depends on the accessibility, willingness, and readiness of diagnose failures. Such smells are detected using rule-based software companies to share their data. In this study, 11 methods, as described by Aljedaani et al. [33], which identify industrial projects were collected from different software tests with multiple assertions lacking clarity. Metrics like companies of Bangladesh. Selection criteria included a cosine similarity can also highlight a "Lack of Cohesion" in minimum project size of 6,000 lines of code (LOC), a the test case. minimum of 500 commits to ensure development activity, The methodology uses predefined thresholds for various and comprehensive test code (with at least 1,000 lines of test metrics, such as local variables and hard-coded values, code). based on best practices in software testing. This structured The dataset was carefully curated to represent a diverse approach identifies issues like redundant setups, poorly range of testing practices, providing a robust foundation for named tests, and excessive complexity. For our study, we comparing test smells in both open-source and industrial utilize PyNose [17], which is effective in detecting a broad projects. Key attributes, such as lines of code (KLOC), range of test smells, as demonstrated in previous research number of methods (NOM), and number of classes (NOC), [34, 35, 36]. are detailed in Table 1 and Table 2. These tables offer a clear overview of the projects’ core characteristics, presented 3.3. Test Smell Analysis without any initial analysis. The complete dataset used in this study is available in the online appendix.2 To systematically analyze the distribution of test smells within our dataset, we use two key metrics: Spreadity and Inverse Smell Frequency (ISFi) [32], along with statistical 3.2. Test Smell Detection hypothesis testing techniques to examine the differences in Detecting test smells in Python typically involves a static test smell occurrences between open-source and industrial code analysis process that inspects the structure of test code projects. These metrics are commonly used to evaluate the to identify patterns known to impact test maintainability, distribution and frequency of smells in software research readability, and reliability. This approach systematically [31]. scans Python test classes—particularly those derived from Spreadity: To measure the presence of test smells across the unittest framework—focusing on elements such as the analyzed projects, we utilize the metric called Spreadity assertion statements, control structures, and setup and as shown in Equation 1. Spreadity calculates the proportion teardown methods. By examining these components, static of projects that contain a specific test smell, providing analysis tools can identify common test smells like Assertion insight into how common a particular smell is within the Roulette, Eager Test, and Test Duplication, which complicate dataset. debugging and future modifications. Consider the code 𝑛(𝑥) Spreadity(𝑥) = (1) snippet in Listing 1, which provides an example of a common 𝑁 test smell. where 𝑥 denotes the test smell, 𝑛(𝑥) is the number of projects containing the smell 𝑥, and 𝑁 represents the total 2 https://figshare.com/s/4789bd212185042cdad0 number of projects analyzed. A higher Spreadity value 21 indicates that a larger proportion of projects are affected Table 3 by the test smell, with a maximum value of 1 if all projects Test Smell Occurrences in Industrial and Open-Source Projects contain the smell. Number of affected projects Inverse Smell Frequency (ISFi): To gain a deeper Test Smell Industrial Open source understanding of the occurrence of test smells, we calculate Assertion Roulette 9 59 the Inverse Smell Frequency (ISFi) metric using the Equation Conditional Test Logic 7 59 2. This metric is used to analyze code smell frequency and Duplicate Assertion 6 55 occurrences [31]. It normalizes the frequency of a test smell Exception Handling 4 51 relative to specific software metrics, such as Test Lines of Magic Number Test 8 49 Code (TLOC), Test Number of Methods (TNOM), and Test Obscure In-Line Setup 4 40 Number of Classes (TNOC). Redundant Assertion 4 31 Redundant Print 2 29 Metric Value Test Maverick - 19 ISFi(metric) = (2) Smell Count𝑖 Sleepy Test - 17 where the metric values could be TLOC, TNOM, or TNOC, Empty Test 1 17 Lack of Cohesion - 7 and Smell Count𝑖 represents the occurrence of a particular Suboptimal Assert 2 7 test smell 𝑖. Lower ISFi values indicate a higher frequency Constructor Initialization - 2 of the test smell, suggesting that it is more prevalent in the code. Statistical Tests of Hypothesis: The differences in Assertion Roulette (AR) emerges as the most frequently test smell occurrences between open-source and industrial occurring test smell in both open-source and industrial projects are examined using t-tests, a widely accepted projects based on the Table 3. It is occurred in 9 out of method for comparing the means of two independent 11 industrial and 59 out of 64 open-source projects. This groups. indicates a widespread issue with AR in both environments, A one-tailed t-test is performed to determine if there suggesting a need for better assertion practices in test are significant differences in the prevalence of specific code. In open-source projects, the top 3 test smells based test smells between open-source and industrial projects. on project count are Assertion Roulette (AR), Conditional The one-tailed approach is used because we hypothesize Test Logic (CTL), and Duplicate Assertion (DA), which are that the occurrence of test smells will be higher in open- consistently present and indicative of complex assertion source projects compared to industrial projects, based on patterns in open-source projects. previous observations of faster development cycles and However, industrial projects display a slightly different less formal testing practices in open-source environments. ranking of test smells. When sorted by the number of The alternative hypothesis suggests that the occurrence of projects in which they are found, the order is Assertion test smells is higher in industrial projects, while the null Roulette (AR) > Magic Number Test (MNT) > Conditional hypothesis assumes no significant difference. The test is Test Logic (CTL). In contrast, when ranked by their total conducted at a 5% significance level. occurrences (frequency) in the open-source projects, as Additionally, a two-tailed t-test was used to assess shown in Table 4, the order shifts to Assertion Roulette (AR) whether there were any significant differences in the volume > Conditional Test Logic (CTL) > Magic Number Test (MNT). of test code between open-source and industrial projects. This variation highlights that while some test smells like AR In this analysis, the alternative hypothesis proposes a are universally problematic, other smells vary in significance difference in test code volume, while the null hypothesis depending on the environment, indicating different testing assumes no difference. The test was conducted at a practices or priorities between industrial and open-source 5% significance level, with specific metrics analyzed to projects. determine any significant variations between the two The dominance of test smells like AR and CTL highlights environments. common weaknesses in both open-source and industrial projects, pointing to the need for improved testing strategies 4. Result Analysis that prioritize clarity and simplicity in test code. Identifying and addressing these frequent test smells can significantly This section presents the findings from our study, addressing enhance the maintainability and reliability of software each research question by analyzing the occurrences of test projects in both environments. smells in industrial and open-source projects. We highlight Moreover, we identify a few test smells that are entirely the types of test smells, their frequency, differences in absent in both open-source and industrial projects. These occurrence between environments, and insights into test- test smells include Default Test (DT), General Fixture (GF), writing practices in the software industry. Ignored Test (IT), and Unknown Test (UT). Additionally, four specific test smells—Constructor Initialization (CI), Lack of 4.1. RQ1: What types of test smells are Cohesion of Test Cases (LCTC), Sleepy Test (ST), and Test Maverick (TM)—are present exclusively in open-source prevalent in open-source and industrial projects and are completely absent from industrial projects. Python projects? This absence indicates a divergence in testing practices To address RQ1, we analyze the presence of test smells between these environments, with certain test smells being in both open-source and industrial projects. Our study specific to open-source development contexts. identifies 14 distinct test smells across these environments, with notable differences in their presence and frequency, as shown in Table 4. 22 Table 4 Comparison of Test Smell Occurrences between Industrial and Open-source Projects Industrial Projects Open-source Projects Avg Freq./ Avg Freq./ Test Smell Total Freq. Spreadity ISFi(TLOC) ISFi(TNOM) ISFi(TNOC) Total Freq. Spreadity ISFi(TLOC) ISFi(TNOM) ISFi(TNOC) p-value Proj. Proj. Assertion Roulette(AR) 235 21.36 0.82 262 32 3 13607 212.61 0.92 212 23 2 0.068 Conditional Test Logic(CTL) 115 10.45 0.64 536 65 6 10136 158.38 0.92 284 31 2 0.002 Constructor Initialization(CI) 0 0 0.00 NA NA NA 130 2.03 0.03 22167 2445 183 0.119 Duplicate Assertion(DA) 51 4.64 0.55 1209 146 13 6185 96.64 0.86 466 51 4 0.0002 Empty Test(ET) 3 0.27 0.09 20561 2474 217 106 1.66 0.27 27186 2998 224 0.082 Exception Handling(EH) 10 0.91 0.36 6168 742 65 1494 23.34 0.80 1929 213 16 0.0001 Lack of Cohesion of Test Cases(LCTC) 0 0 0.00 NA NA NA 544 8.50 0.11 5297 584 44 0.010 Magic Number Test(MNT) 72 6.55 0.73 857 103 9 3058 47.78 0.77 942 104 8 0.156 Obscure In-Line Setup(OIS) 5 0.45 0.36 12337 1484 130 669 10.45 0.62 4308 475 35 0.049 Redundant Assertion(RA) 16 1.45 0.36 3855 464 41 114 1.78 0.48 25278 2788 208 0.178 Redundant Print(RP) 4 0.36 0.18 15421 1856 163 542 8.47 0.45 5317 586 44 0.036 Sleepy Test(ST) 0 0 0.00 NA NA NA 240 3.75 0.27 12007 1324 99 0.012 Suboptimal Assert(SA) 25 2.27 0.18 2467 297 26 744 11.63 0.11 3873 427 32 0.129 Test Maverick(TM) 0 0 0.00 NA NA NA 898 14.03 0.30 3209 354 26 0.001 4.2. RQ2: How does the prevalence of test smells differ between open-source and industrial Python projects? The aim of RQ2 is to identify the most frequent and widespread test smells that significantly affect both industrial and open-source projects. Our analysis reveals that test smells are generally more frequent in open-source projects, with an average frequency of 601.57 test smells per project compared to 48.72 in industrial projects. Figure 2: Average frequency of each test smell Assertion Roulette (AR) is observed as the most widespread test smell in both open-source and industrial projects, with spreadity values of 0.92 and 0.81, depicted in Figure 1. Similarly, Conditional Test Logic (CTL) has a spreadity of 0.92 in open-source projects but only 0.63 in industrial projects. While Magic Number Test (MNT) ranks as the second most frequent smell in industrial projects, its position drops to fifth in open-source projects, despite having a slightly higher spreadity in open-source (0.76) than in industrial environments (0.72). Certain test smells, such as Test Maverick (TM), Sleepy Test (ST), Lack of Cohesion of Test Cases (LCTC), and Constructor Initialization (CI), are found exclusively in open-source Figure 3: ISFi(TLOC) Score Comparison for each test smell projects and are completely absent from industrial projects. This indicates a clear divergence in test practices, with these smells being specific to the open-source context. Figure 4: ISFi(TNOM) Score Comparison for each test smell The results suggest that Assertion Roulette occurs in Figure 1: Spreadity of each test smell industrial projects for every 262 test lines of code, and in open-source projects for every 212 test lines of code. This pattern is shown in Figure 3 and Table 4. The second most The analysis reveals that test smells such as Assertion frequent test smell is Conditional Test Logic, as for every Roulette (AR) and Conditional Test Logic (CTL) are notably 536 lines of test code in the industrial projects and 284 lines more prevalent in open-source projects compared to of test code in the open-source projects, the smell prevails. industrial projects. This conclusion is based on both the Conversely, Empty Tests are rare in both type of projects, spread and average frequency metrics, as shown in Figure 2. occurring only for every 20,561 and 27,186 lines of test code, Additionally, Figures 3, 4, and 5 illustrate the ISFi scores for respectively. Considering the test methods, one Assertion TLOC, TNOM, and TNOC across all the test smells under Roulette test smell is found for every 32 test methods in the study. The four common smells that did not appear in any industrial projects and 23 test methods in the open-source system are excluded, as discussed in response to RQ1. projects, depicted in Figure 4. In terms of the number of 23 Table 5 t-test Results for Normalized Test Code Metrics (Mean) Between Industrial and Open-source Projects Metric Industrial Projects (Mean) Open-source Projects (Mean) t-test (2-tail) p-value Total Test Files 0.086297 0.166895 0.0848 Test LOC 0.119178 0.181811 0.2889 Test NOC 0.175402 0.161391 0.8047 Test NOM 0.524443 0.576357 0.8027 Test LOC per File 1.02626 1.020543 0.9772 Test NOC per File 2.070511 1.01271 0.009 Test NOM per File 5.157113 3.488941 0.0592 4.3. RQ3: How does the volume of test code differ between open-source and industrial Python projects? To explore differences in test code volume between open- source and industrial Python projects, we analyzed key metrics including the number of test files, Test Lines of Code (TLOC), number of test classes (TNOC), and number of test methods (TNOM). These metrics, normalized for project size variations, were assessed to enable a fair comparison between the two environments. Analyzing Figure 5: ISFi(TNOC) Score Comparison for each test smell test code volume provides a foundational understanding that complements the examination of test smells across these projects. Differences in test smell prevalence could classes, for every 3 test classes in the industrial projects be influenced by the ratio of test code to production code, and 2 test classes in the open-source projects, one Assertion making test code volume an essential aspect to assess. This Roulette test smell is seen. approach sheds light not only on the presence of test smells The second most frequent test smell is Conditional Test but also on potential structural distinctions in test code Logic based on ISFi(TNOM) and ISFi(TNOC) as well. In organization and density across open-source and industrial fact, the smell is so frequent that for every 6 classes and projects. 65 test methods in the industrial projects and 2 classes The null hypothesis (𝐻0 ) posits that there is no significant and 31 test methods in the open-source projects, one CTL difference in the volume of test code between open-source test smell is found. In contrast, the third most frequent and industrial Python projects (𝜇Industry = 𝜇Open Source ). In test smell in industrial projects is the Magic Number Test. contrast, the alternative hypothesis (𝐻1 ) suggests that the Specifically, this smell is found once for every 857 test lines test code volume differs between the two environments of code, 103 test methods, and 9 test classes. However, with (𝜇Industry ̸= 𝜇Open Source ). Table 5 provides the t-test results 466 TLOC, 51 TNOM, and 4 TNOC, Duplicate Assertion comparing these metrics across open-source and industrial is the third most occurring test smell in the open-source projects. While most metrics, such as Total Test Files, Test projects. Additionally, one of every 2,474 test methods and LOC, Test NOC, and Test NOM, did not show statistically 217 test classes in the industrial projects and 2,998 test significant differences, the Test NOC per File metric, with a p- methods and 224 test classes in the open-source projects value of 0.009, indicates a notable distinction in the number has been identified as an Empty Test. Other smells are not of test classes per file between the two environments. This as prevalent, as highlighted in the observations of the study finding suggests a potential structural difference in test code in Table 2. organization across open-source and industrial projects. In order to corroborate these observations, a one-tailed The t-test was conducted for all test code metrics t-test with a 5% significance level is used to conduct a extracted from both industrial and open-source projects. statistical hypothesis test. The p-value is displayed in the The null hypothesis was accepted for 6 metrics, indicating final column of Table 4, and the value that rejects the no statistically significant difference in test code volume or null hypothesis is bolded. The null hypothesis states that structure between the industrial and open-source projects there is no difference in the occurrence of a particular for most metrics. However, the null hypothesis was rejected test smell between open-source and industrial projects, for 1 metric, Test NOC per File (marked in bold in Table 5), while the alternative hypothesis refers that the open-source supporting the claim that the number of test classes per system has higher occurrences. The p-values in Table 4 file in open-source projects is significantly different from indicate that the null hypothesis is rejected for 8 out of the that in industrial projects, suggesting a distinct approach to 14 test smells, suggesting a statistically significant higher organizing test classes. These findings are summarized in occurrence of these smells in open-source projects. These Table 5, showing that for the majority of metrics (6 out of 7), findings emphasize that open-source projects are more there is no significant difference in test code volume between susceptible to frequent and diverse test smells compared open-source and industrial projects. This suggests that, with to industrial projects, highlighting the need for targeted respect to basic testing characteristics like the number of improvements in test-writing practices within the open- test files, lines of code, and methods, both environments source community. follow relatively similar practices. However, the significant difference in the Test NOC per File metric highlights that open-source projects may 24 organize test classes differently, potentially due to variations These findings will assist developers in reducing test in project structure, coding standards, or collaborative smells during software development and provide insights development practices typical in open-source environments. for researchers to refine smell refactoring techniques, par- ticularly in terms of frequency and distribution. Future work should explore how specific development practices 5. Threats to Validity contribute to these differences between project types, offer- ing a deeper understanding of how coding conventions and This section addresses construct, external, and reliability workflows influence test smell occurrence. Further research threats that may affect the validity of our empirical could also investigate automated refactoring techniques that investigation [37]. account for these structural and volumetric differences to better support quality improvements in both open-source 5.1. Construct Validity and industrial projects. The accuracy of PyNose, our test smell detection tool, may affect construct validity. Any inaccuracies in detecting test References smells could lead to incorrect conclusions regarding their presence and significance. To minimize this risk, we used [1] T. Virgínio, R. Santana, L. A. Martins, L. R. Soares, PyNose, a widely recognized tool in test smell detection H. Costa, I. Machado, On the influence of test smells on with reliability confirmed in prior studies. test coverage, in: Proceedings of the XXXIII Brazilian Symposium on Software Engineering, 2019, pp. 467– 5.2. External Validity 471. [2] D. Spadini, F. Palomba, A. Zaidman, M. Bruntink, Our findings’ generalizability is limited by several external A. Bacchelli, On the relation of test smells to software validity threats. First, this study focuses on Python-based code quality, in: 2018 IEEE international conference projects, so results may not fully apply to projects in on software maintenance and evolution (ICSME), IEEE, other languages like Java or C#. Additionally, the dataset 2018, pp. 1–12. comprises 11 industrial projects from Bangladesh and 64 [3] G. Meszaros, xUnit test patterns: Refactoring test code, open-source projects, which may not represent the broader Pearson Education, 2007. global industry practices. The industrial projects come from [4] S. Baltes, P. Ralph, Sampling in software engineering a specific sector that may have unique testing practices. To research: A critical review and guidelines, Empirical address these threats, we included well-known open-source Software Engineering 27 (2022) 94. projects with diverse testing practices. [5] A. Vahabzadeh, A. M. Fard, A. Mesbah, An empirical study of bugs in test code, in: 2015 IEEE international 5.3. Reliability Validity conference on software maintenance and evolution (ICSME), IEEE, 2015, pp. 101–110. Reliability concerns arise from the need to ensure [6] G. Bavota, A. Qusef, R. Oliveto, A. De Lucia, D. Binkley, replicability, particularly for industrial data. Due to An empirical analysis of the distribution of unit test confidentiality constraints, we cannot disclose details about smells and their impact on software maintenance, in: the industrial projects, limiting direct replication. However, 2012 28th IEEE international conference on software open-source project findings are fully replicable, as these maintenance (ICSM), IEEE, 2012, pp. 56–65. projects and the PyNose tool are publicly accessible. For [7] V. Garousi, B. Küçük, Smells in software test code: further validation, future studies could replicate the analysis A survey of knowledge in industry and academia, with similar industrial datasets. Journal of systems and software 138 (2018) 52–81. [8] A. Peruma, K. S. Almalki, C. D. Newman, M. W. 6. Conclusion Mkaouer, A. Ouni, F. Palomba, On the distribution of test smells in open source android applications: An This research investigates the occurrence, distribution, exploratory study (2019). and frequency of test smells in both industrial and open- [9] J. W. Paulson, G. Succi, A. Eberlein, An empirical study source projects, highlighting the differences between these of open-source and closed-source software products, two environments. The inclusion of industrial projects IEEE transactions on software engineering 30 (2004) alongside open-source ones adds depth to the study, as it 246–256. reveals distinct patterns in the prevalence of test smells. [10] A. Van Deursen, L. Moonen, A. Van Den Bergh, Although Assertion Roulette (AR) is the most common test G. Kok, Refactoring test code, in: Proceedings smell, appearing in 92% of open-source projects and 81% of the 2nd international conference on extreme of industrial projects, a one-tailed t-test shows significant programming and flexible processes in software differences in the overall prevalence of test smells between engineering (XP2001), Citeseer, 2001, pp. 92–95. these two types of projects. [11] L. Martins, D. Campos, R. Santana, J. M. Junior, Furthermore, 10 out of 18 test smells were observed in H. Costa, I. Machado, Hearing the voice of experts: industrial projects, while 14 out of 18 test smells were Unveiling stack exchange communities’ knowledge detected in open-source projects. Notably, 8 out of the 14 of test smells, in: 2023 IEEE/ACM 16th International test smells showed a significantly higher occurrence in open- Conference on Cooperative and Human Aspects of source projects. Analyzing test writing practices, we found Software Engineering (CHASE), IEEE, 2023, pp. 80–91. no significant differences in 6 out of 7 metrics between [12] A. Qusef, M. O. Elish, D. Binkley, An exploratory study industrial and open-source projects, except for the number of the relationship between software test smells and of test classes per file (TNOC). fault-proneness, IEEE Access 7 (2019) 139526–139536. 25 [13] Y. Fushihara, H. Aman, S. Amasaki, T. Yokogawa, [28] F. Palomba, A. Zaidman, The smell of fear: On M. Kawahara, Fault-proneness of python programs the relation between test smells and flaky tests, tested by smelled test code (2024). Empirical Software Engineering 24 (2019) 2907– [14] D. J. Kim, T.-H. Chen, J. Yang, The secret life of 2946. URL: https://link.springer.com/article/10.1007/ test smells-an empirical study on test smell evolution s10664-018-9662-1, retracted on 26 March 2020. and maintenance, Empirical Software Engineering 26 [29] B. Camara, M. Silva, A. Endo, S. Vergilio, On (2021) 1–47. the use of test smells for prediction of flaky tests, [15] F. Palomba, A. Zaidman, A. De Lucia, Automatic test in: Proceedings of the 6th Brazilian Symposium on smell detection using information retrieval techniques, Systematic and Automated Software Testing, 2021, pp. in: 2018 IEEE International Conference on Software 46–54. URL: https://doi.org/10.1145/3482909.3482916. Maintenance and Evolution (ICSME), IEEE, 2018, pp. doi:10.1145/3482909.3482916. 311–322. [30] Y. Fushihara, H. Aman, S. Amasaki, T. Yokogawa, [16] D. Falessi, N. Juristo, C. Wohlin, B. Turhan, J. Münch, M. Kawahara, A trend analysis of test smells in python A. Jedlitschka, M. Oivo, Empirical software test code over commit history, in: 2023 49th Euromicro engineering experts on the use of students and Conference on Software Engineering and Advanced professionals in experiments, Empirical Software Applications (SEAA), IEEE, 2023, pp. 310–314. Engineering 23 (2018) 452–489. [31] M. M. Rahman, A. Satter, M. M. A. Joarder, K. Sakib, [17] T. Wang, Y. Golubev, O. Smirnov, J. Li, T. Bryksin, An empirical study on the occurrences of code smells I. Ahmed, Pynose: a test smell detector for python, in open source and industrial projects, in: Proceedings in: 2021 36th IEEE/ACM international conference on of the 16th ACM/IEEE International Symposium on automated software engineering (ASE), IEEE, 2021, pp. Empirical Software Engineering and Measurement, 593–605. 2022, pp. 289–294. [18] V. Pontillo, D. Amoroso d’Aragona, F. Pecorelli, [32] H. Schütze, C. D. Manning, P. Raghavan, Introduction D. Di Nucci, F. Ferrucci, F. Palomba, Machine learning- to information retrieval, volume 39, Cambridge based test smell detection, Empirical Software University Press Cambridge, 2008. Engineering 29 (2024) 55. [33] W. Aljedaani, A. Peruma, A. Aljohani, M. Alotaibi, [19] A. Peruma, K. Almalki, C. D. Newman, M. W. Mkaouer, M. W. Mkaouer, A. Ouni, C. D. A. Ouni, F. Palomba, Tsdetect: An open source test Newman, A. Ghallab, S. Ludi, Test smell detection smells detection tool, in: Proceedings of the 28th tools: A systematic mapping study, in: Proceedings ACM joint meeting on european software engineering of the 25th International Conference on Evaluation conference and symposium on the foundations of and Assessment in Software Engineering, 2021, pp. software engineering, 2020, pp. 1650–1654. 170–180. [20] T. Virgínio, L. Martins, L. Rocha, R. Santana, A. Cruz, [34] A. Eghbali, M. Pradel, Dynapyt: a dynamic analysis H. Costa, I. Machado, Jnose: Java test smell detector, framework for python, in: Proceedings of the in: Proceedings of the XXXIV Brazilian Symposium 30th ACM Joint European Software Engineering on Software Engineering, 2020, pp. 564–569. Conference and Symposium on the Foundations of [21] A. Bodea, Pytest-smell: a smell detection tool for Software Engineering, 2022, pp. 760–771. python unit tests, in: Proceedings of the 31st [35] R. Sandouka, H. Aljamaan, Python code smells ACM SIGSOFT International Symposium on Software detection using conventional machine learning Testing and Analysis, 2022, pp. 793–796. models, PeerJ Computer Science 9 (2023) e1370. [22] G. Bavota, A. Qusef, R. Oliveto, A. De Lucia, D. Binkley, [36] D. Fernandes, I. Machado, R. Maciel, Tempy: Test smell Are test smells really harmful? an empirical study, detector for python, in: Proceedings of the XXXVI Empirical Software Engineering 20 (2015) 1052–1094. Brazilian Symposium on Software Engineering, 2022, [23] D. J. Kim, An empirical study on the evolution of pp. 214–219. test smell, in: Proceedings of the ACM/IEEE 42nd [37] P. Runeson, M. Host, A. Rainer, B. Regnell, Case International Conference on Software Engineering: study research in software engineering: Guidelines Companion Proceedings, 2020, pp. 149–151. and examples, John Wiley & Sons, 2012. [24] B. Hauptmann, M. Junker, S. Eder, L. Heinemann, R. Vaas, P. Braun, Hunting for smells in natural language tests, in: 2013 35th International Conference on Software Engineering (ICSE), IEEE, 2013, pp. 1217– 1220. [25] E. Soares, M. Aranda, N. Oliveira, M. Ribeiro, R. Gheyi, E. Souza, I. Machado, A. Santos, B. Fonseca, R. Bonifácio, Manual tests do smell! cataloging and identifying natural language test smells, in: 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, 2023, pp. 1–11. [26] The open catalog of test smells, 2024. URL: https:// test-smell-catalog.readthedocs.io/en/latest/, accessed: [2024-11-09]. [27] K. Lucas, R. Gheyi, E. Soares, M. Ribeiro, I. Machado, Evaluating large language models in detecting test smells, arXiv preprint arXiv:2407.19261 (2024). 26